Multi-Part Upload in AWS S3

Overview: Multi-part upload is a feature in AWS S3 that allows users to upload large files in smaller parts, which can be done in any order. This method is particularly useful for large files, enhancing the upload speed and providing a safer way to handle transfer failures.

When to Use Multi-Part Upload:

• Recommended for files larger than 100 MB.
• Mandatory for files over 5 GB.

Benefits of Multi-Part Upload:

• Parallel Uploads: Speeds up transfers by uploading parts in parallel.
• Network Utilization: Maximizes network bandwidth.
• Error Recovery: Only failed parts need to be re-uploaded.
• Efficiency: Saves time on retries.

Key Points:

• Maximum of 10,000 parts per upload.
• Parts are uploaded to an Amazon S3 bucket and combined at the end.
• A 'Complete' request is made to concatenate all parts into the original large file.
• Lifecycle policies can automate the deletion of unfinished uploads.

Lifecycle Rule Configuration:

• Rule Name: For example, 'demo multi-part'.
• Application Scope: Can apply to all objects in the bucket.
• Actions: Delete expired delete markers or incomplete multi-part uploads.
• Cleanup: Set to delete incomplete multi-part uploads after a specified number of days (e.g., 3 days) to clear any residual parts.