Big improvements in Globus performance on Amazon S3 endpoints
November 11, 2014 | Vas Vasiliadis
Our first implementation for transferring files to/from Amazon S3 endpoints (buckets) was a good proof of concept but performance was, at best, average. In our latest release, we’ve upped the stakes significantly and used multiple techniques to improve performance: (1) using parallel TCP file streams, (2) quickly retrying on transient faults, and (3) using multipart-upload only for files greater than 100MB.
Initial tests show throughput peaks of 5.2Gbps; in one test case we transferred 1.2 terabytes in 48 minutes. These transfers were performed with using Globus Connect Personal on Amazon EC2 instances with the following configurations: Instance type: m3.xlarge (network performance: High/1000 Mbps); Instance type: cr1.8xlarge, (network performance: 10Gbps). Sustained throughput on the m3 instance was approximately 880Mbps; on the cr1 instance it was approximately 3.5Gbps.
We're extremely pleased with these results and hope you will be too. If you'd like to try the Globus S3 uploader please fill this form to request access to the beta program. Note that S3 endpoint support requires a Globus Provider plan subscription.