Multipart Upload
Definition
Multipart upload is a specific protocol, and it is worth separating from chunked upload, which is the general idea of splitting a transfer. Here the sequence is fixed and the vocabulary is fixed. A create call returns an UploadId. Each part is sent with that ID and a part number between 1 and 10000, and each response returns an ETag for that part. A complete call submits the full list of part numbers and ETags, and only then does the object exist at its key. Parts must be at least 5 MB apart from the last one, and a single object can reach 5 TB this way against a 5 GB ceiling for an ordinary single PUT. The name collides confusingly with multipart form data, a browser request encoding that has nothing to do with any of this.
Why It Matters
Past 5 GB there is no choice: an S3-compatible store will simply refuse a single PUT, so the protocol is the only route. Below that it is still usually faster, because parts upload in parallel and six concurrent streams on a link with 200 ms of latency will beat one stream comfortably. The failure mode is financial rather than dramatic. An upload that is started and never completed leaves its parts in the bucket, invisible in a normal object listing, and they are billed as stored bytes until something aborts them. Teams have found hundreds of gigabytes of orphaned parts years after the client that created them was decommissioned.
How It Works
CreateMultipartUpload returns the UploadId, which the client keeps for the whole transfer. UploadPart is a PUT carrying the upload ID, the part number and that slice of bytes; the response header ETag identifies the stored part. The client collects those pairs and sends CompleteMultipartUpload with them in ascending part order, and the store assembles the object server-side without the bytes travelling again. AbortMultipartUpload discards everything if the client gives up. One detail catches people out: the ETag of a completed multipart object is not the MD5 of the file. It is a hash over the concatenated part hashes, followed by a hyphen and the number of parts, so a value ending in a hyphen and 12 means twelve parts and nothing more.
Real-World Example
A photographer publishes a 4.2 GB ZIP archive of raw wedding files for a client to collect. Sent as a single request it would be rejected outright; sent as 84 parts of 50 MB it takes about eleven minutes with four streams running at once, and a part that failed on the first attempt was retried on its own. 99helpers completes the upload, stores it under one key, and hands over a link at hart-wedding-raws.99helpers.site.
Common Mistakes
- ✕Sending a middle part smaller than 5 MB — the completion call rejects the whole upload with EntityTooSmall and the work is wasted
- ✕Discarding the UploadId after the client restarts — the parts still exist and still cost money, but nothing can complete or abort them without it
- ✕Comparing the object's ETag against a locally computed MD5 — for a multipart object those values are not meant to match
- ✕Leaving no lifecycle rule to abort incomplete uploads, so abandoned parts sit in the bucket indefinitely
Related Terms
Chunked Upload
Chunked upload is the technique of splitting one large file into smaller pieces and sending each piece as its own request. If the connection drops, only the piece in flight is lost rather than the whole transfer.
Object Storage
Object storage keeps each file as a self-contained object — its bytes, its metadata and a key that names it — inside a flat container reached over HTTP. There is no directory tree and no file handle to open, seek and write.
Storage Bucket
A storage bucket is the top-level container in an object storage system: a named, region-bound namespace that holds objects and carries the settings applied to all of them.
Checksum
A checksum is a short fixed-length value calculated from a file's bytes, used to confirm that the copy you received is identical to the copy that was sent. Same bytes in, same value out — and one changed bit produces a completely different value.
ETag
A short identifier a server attaches to a file so a cache can later ask whether that exact version is still current. If it is, the answer comes back as a 304 with no body, and nothing is downloaded.
Put a file online in seconds
Drop in a document, an image, a page or a whole static website and share the link — free, with no build step and no server to set up.
Host a file free →