Dr. Dobb's | S3 Meets R3 (Reliability, Robustness, and Resilience)

S3 Meets R3 (Reliability, Robustness, and Resilience)

Assessing the Performance of Amazon Simple Storage Service (S3) Web Services

August 18, 2006
URL:http://www.drdobbs.com/web-development/s3-meets-r3-reliability-robustness-and-r/192202171

Mamoon Yunus is CTO of Forum Systems and a pioneer of SOA Gateways & Firewalls. Prior to Forum, Mr. Yunus was at webMethods where he developed XML-based techology. Mamoon holds two Graduate Degrees in Engineering from MIT.

Rizwan Mallal is Technology Director at Crosscheck Networks and Chief Security Architect of Forum Systems. Previously, Rizwan held various positions at Sonicwall and Raptor (now Symantec). Rizwan holds a Masters in Computer Science from University of Vermont.

Amazon's entry into the online storage business has created a lot of buzz in the industry. The well-touted S3 (Simple Scalable Storage) web service holds the promise of providing reliable and robust virtual storage for users over the web. It’s provided as a storage service exposed via standard interfaces such as SOAP Web Services and REST that developers can leverage to build custom storage applications for end users.

The success of Amazon's S3 depends on two factors: first, consumers building successful business models from such services; and second, S3 providing a reliable and robust infrastructure to deliver such services. The robustness, reliability and resilience can create a positive user experience. Robustness and reliability in turn depend on the performance characteristics of this service, specifically the latency that determines the delay in storing or retrieving files.

To determine the latency that a user might experience with S3, we embarked on a project to measure typical file storage, file retrieval and other miscellaneous file operations. The latency of these file operations was based on the time it would take to store and retrieve files of multiple sizes ranging from a few kilobytes up to a few megabytes. Keeping in mind that performance is relative, we introduced a comparable latency metric by measuring how long it would take us to store/retrieve files of similar sizes to a remote server over the WAN (Wide Area Network) using Secure Copy (SCP) protocol. Granted, measuring the transfer time between the two different standards on different endpoints is not exactly an apples to apples comparison but it would still give us a fair estimation of where Amazon S3 service stands in comparison to a well-known industry standard such as SCP.

Based on our goals we decided to run multiple test results and produce metrics that would highlight the latency involved when accessing Amazon S3.

Test Runs

Multiple file operations are available to a user of an Amazon S3 service. In our tests, we focused on two key file operations that dominate a user’s access pattern. The two key operations of interest to us are read (PUT) and write (GET). Realizing that users can store or retrieve files of varying sizes, we broke the latency test into four test cases:

Test Case I. Store (PUT) files ranging from 10-to-100 KB.

Test Case II. Store (PUT) files ranging from 1-to-10 MB.

Test Case III. Retrieve (GET) files ranging from 10-to-100 KB.

Test Case IV. Retrieve (GET) files ranging from 1-to-10 MB.

To test the latency of Amazon S3 Web Services, we relied on SOAPSonar Enterprise Edition, a Web Services client tool produced by our company, Crosscheck Networks. SOAPSonar performs functional and load testing as well as vulnerability assessment to ensure that Web Services are reliable and robust before they are deployed. To measure the latency of Amazon S3 Web Services, we used SOAPSonar Enterprise Edition for a full cycle of testing, which included tasks such as generating SOAP traffic and measuring performance metrics.

Figure 1 illustrates the setup used to test Amazon S3 Web Service. In all of our tests, files were written to Amazon S3 using SOAP with attachments based primarily on the MIME standard. We also ran tests using the DIME standard as a comparison and encountered different characteristics than the MIME standard.

Figure 1: Amazon S3 vs. SCP Test Setup

The following components & steps are required to run the performance metrics:

Download and Install SOAPSonar Enterprise Edition as the S3 Client.
Register with Amazon S3 to obtain the AWS Key and the Secret Key.
Load the Amazon WSDL and Schema from the Amazon S3 in SOAPSonar.

Test Case I: Writing Small Files

The first test case involves writing a set of small files to Amazon S3 Web Services and to a remote SCP enabled server. The set of files (10k to 100k) are relatively small in size given that one can write up to 5GB in a single operation to Amazon S3. The objective of this test case to observe how Amazon S3 behaves with files of small size. Figure 2 illustrates the average transfer time it takes to write multiple small files to Amazon S3 and an SCP server.

Figure 2: Amazon S3 vs. SCP Small Size File Writes (PUT).

Observations: For Small file size writes (PUT), we make the following observations:

ü Latency increases almost linearly with an increase in file sizes from 10-100k. This linear increase in latency holds true for both Amazon S3 and the SCP.

ü Throughput ranges from ~ 160 Kbps -- 270 Kbps.

Test Case II: Writing Large Files

The second test case involves writing a set of large files to Amazon S3 Web Services and to a remote SCP enabled server. Figure 3 illustrates the time it takes to write large multiple files to Amazon S3 and an SCP server.

Figure 3: Amazon S3 vs. SCP Large Size File Writes (PUT).

Observations: For Large File size writes (PUT), we make the following observations:

ü Latency increases almost linearly with an increase in file sizes from 1-10 MB. This linear increase in latency holds true for both Amazon S3 and the SCP.

ü Throughput ranges from ~ 256 Kbps -- 347 Kbps.

Test Case III: Reading Small Files

The third test case involves reading a set of small files from Amazon S3 Web Services and from a remote SCP enabled server.

Figure 3 illustrates the time it takes to read multiple small files from Amazon S3 and an SCP server.

Figure 4: Amazon S3 vs. SCP Small Size File Reads (GET).

Observations: For Small file size reads (GET), we make the following observations:

ü The latency for Amazon S3 service is remarkably lower than SCP reads.

ü Throughput ranges from ~ 0.8 Mbps -- 1.3 Mbps, well within the advertised download bandwidth of 6-8Mbps.

Test Case IV: Reading Large Files

The fourth test case involves reading a set of large files from Amazon S3 Web Services and from a remote SCP enabled server. Figure 5 illustrates the time it takes to read multiple large files from Amazon S3 and an SCP server.

Figure 5: Amazon S3 vs. SCP Large Size File Reads (GET).

Observations: For Large file size reads (GET), we make the following observations:

ü The latency for Amazon S3 service is significantly lower than SCP reads.

ü Throughput ranges from ~ 4.75 Mbps -- 7.5 Mbps. The S3 reads consume all available bandwidth advertised at 6-8 Mbps.

Overall Observations and Assessment

S3 reads (GET) are faster than S3 writes (PUT). There are three overall factors that may contribute to this difference:

Cable Modem Download is over a magnitude faster than Upload.
Most likely, a conservative write scheme is deployed by Amazon to ensure that the files being stored are backed up for high reliability.
For S3 reads, potential in-memory file caching by S3 removes latency of reading from disks.

Regardless of message size, the Amazon S3 Service reads (GET) are noticeably faster than SCP reads. The following factors may contribute to this difference:

Most likely, Amazon deploys SSL concentrators or accelerators. The SCP protocol has no such cryptographic acceleration.
The SCP reads do not support caching whereas, most likely, S3 has sophisticated caching enables for reads.

S3 Service writes (PUT) are somewhat slower than SCP writes. Most likely, S3 has a data mirroring schemes that requires multiple writes, where SCP directly writes to a single file system.

S3 supports both MIME and DIME standards for SOAP with Attachments. For writes (PUT), there was little difference in performance between DIME and MIME for small files, with DIME performing slightly better. However, for large files DIME performed much better than MIME. For reads (GET), only DIME is returned and the user has no control over switching to MIME.

Enterprise-Class Service

Amazon’s S3 Services provide a standards-based and high performance mechanism for managing content. S3’s performance characteristics make it ideal for end-user as well as enterprise applications where cost-effective data back up is desired. During our regression we never had a problem with reading or writing files. The S3 service had enterprise-class availability and reliability.

We anticipate that a number of both end-user application as well as complex enterprise-class applications will proliferate the market that are built on storage services such as S3. Such storage services and their derived applications will fulfill regulatory requirements for industries that have third party auditing and record keeping mandates. We also anticipate a number of privacy and integrity related services to be built on top of such storage services. The best part -- after a couple of weeks of testing with numerous uploads, downloads and large file storage, the Amazon S3 service cost us a mere 52 cents.