I recently had a need to upload a few large (11GB) files to S3 for offsite archive purposes. Amazon announced support for objects >5GB back in November 2010, so I had assumed that most S3 clients and libraries had support for it already. I was wrong. It turns out that in order to support large files, you need to use the new “multipart upload” feature. What I found was that many S3 clients do not yet support multipart uploads.
In case this helps save anyone else a few hours of time, here is a summary of what I found regarding multipart upload support in the various S3 clients:
As of Feb 18, 2011:
- Multipart uploads supported:
- Multipart uploads NOT supported:
To solve my immediate need I used S3 Browser on Windows 7. It was the only free client I found that supported multipart uploads. I mostly use a Mac, however, so if someone knows of a free S3 client for Mac please let me know.
I did not look at which S3 libraries support multipart uploads yet. I have heard that my favorite python library - boto - supports it already. And I have seen some reports that RightScale’s right_aws ruby library may support it on some platforms. I’m not sure about the others. Post in the comments if you want to confirm support for multipart uploads in your favorite S3 client or library.
Update (3/10/2010): Cyberduck 4.x was released this week and includes support for multipart uploads and 5TB uploads. A Windows version is also available now. Pretty cool, good job Cyberduck.
Update (4/18/2012): s3cmd 1.1.0-beta2 is available and supports multipart uploads along with many other new S3 and Cloudfront features. Grab the beta or wait for 1.1.0 final to be released