Large file uploads are common in modern applications – whether users are sharing design assets, video content, machine learning datasets, or large project archives. But many development teams encounter a critical limitation when building upload workflows on cloud storage: handling files reliably at scale.
This challenge often surfaces when working with Amazon S3 pre-signed URLs, a widely used method that allows clients to upload files directly to storage without exposing backend credentials. It’s secure, efficient, and simple to implement.
However, there’s a catch.
When using a single PUT request with a pre-signed URL, Amazon S3 imposes a 5GB limit. That works perfectly during testing, but once real users start uploading larger assets like 8GB videos, 10GB project bundles, or even bigger datasets, the system can suddenly fail.
That’s exactly the situation I recently encountered while building a file upload feature. The solution required a shift in approach: moving from simple uploads to AWS Multipart Uploads.
What initially seemed like a workaround turned out to be a much more scalable and resilient architecture.
The “What” and the “Why” of Multipart Upload
A standard upload approach is like trying to ship a grand piano in one piece. If something goes wrong during transport, the entire shipment fails.
Multipart upload works differently. Think of it as shipping the piano in carefully packed components.
With AWS Multipart Upload, the file is divided into smaller parts before uploading. Each part can then be transferred independently.
The process typically looks like this:
- Split the large file into manageable chunks (for example, 10MB–100MB each).
- Generate individual pre-signed URLs for each chunk.
- Upload multiple chunks simultaneously in parallel.
- Capture the ETag identifier returned by S3 for every uploaded part.
- Once all chunks are uploaded, instruct S3 to assemble them into the final file.
This approach introduces a bit more orchestration, but the advantages are significant.
Why Multipart Uploads Are a Game Changer
Resumability
If a network interruption happens during a traditional upload, the entire transfer fails and must restart.
Multipart uploads eliminate this frustration. If a connection drops at 90%, you only retry the failed chunks—not the whole file. This is the same principle behind the pause-and-resume functionality seen in services like cloud storage platforms.
Resilience on Unstable Networks
Smaller uploads are naturally more reliable than one massive transfer. Each chunk has a much higher probability of completing successfully, especially on slower or unstable connections.
Faster Transfers Through Parallelism
Instead of uploading a file sequentially, a multipart upload allows multiple parts to be uploaded simultaneously.
By parallelizing the transfer process, applications can fully utilize available bandwidth and dramatically reduce total upload time.
True Enterprise Scalability
Most importantly, multipart uploads allow systems to handle extremely large files, far beyond the 5GB single-upload limit. In fact, S3 multipart uploads support files up to 5 terabytes.
This makes the architecture suitable for data-heavy platforms such as media systems, analytics platforms, and engineering workflows.
The Mindset Shift: From “Works” to “Works Better”
The biggest takeaway from this experience wasn’t just technical, but it was architectural.
Initially, the workflow was straightforward:
Get pre-signed URL → Upload file → Done
With multipart upload, the process becomes more state-driven:
Initialize upload → Generate part URLs → Upload parts → Track ETags → Finalize upload
This requires maintaining additional metadata, such as:
- Upload ID
- Part numbers
- ETags returned from S3
In many implementations, this information is stored temporarily in a database so the upload can resume safely if needed.
At first glance, it appears more complex. But in practice, each component becomes smaller, more predictable, and more resilient. The result is a system that behaves reliably even under real-world network conditions.
What once felt like a fragile, all-or-nothing operation becomes a background process that users can trust.
A Small Architectural Choice That Improves User Experience
In engineering, the smallest architectural decisions often have the biggest impact on user experience.
For large-file workflows, such as media platforms, data pipelines, or enterprise collaboration tools, upload reliability directly affects how users perceive the product. A failed upload after several minutes can quickly erode trust.
By adopting multipart upload strategies early, teams can build systems that are more scalable, fault-tolerant, and performant from day one.
Final Thought: Building Systems That Scale
One reminder that should be mindful during still stands:
Growth happens when we replace “it works” with “it works better.”
That mindset pushes engineers to revisit assumptions, refine architectures, and design systems that perform well not just in development, but in real-world usage.
Multipart uploads may introduce a bit more orchestration, but they transform large file transfers into something far more reliable and scalable.
Instead of relying on a single fragile request, the system becomes a coordinated process of smaller, resilient operations. And in distributed systems, that shift, from one large step to many smaller ones, is often what makes the difference between something that merely works and something that works well.
Learn how AWS Multipart Upload helps you bypass the 5GB limit, improve upload speed, reliability, and efficiently handle large files in scalable applications.
