AMAZON S3 AND EC2 PERFORMANCE REPORT

Transcription

1 AMAZON S3 AND EC2 PERFORMANCE REPORT OVERVIEW A frequently asked question regarding the Amazon Web Services (AWS) cloud computing platform is how well their storage system (S3) performs with their computing platform (EC2). As a file sharing solution that runs entirely within the AWS cloud, HostedFTP.com has, based on our internal performance data, created this report to discuss the performance you can expect when storing and retrieving files between an EC2 instance and S3. We will also be reporting on how the AWS infrastructure performs over time. Each month we will publish updates to the data to give you an insider's view on how well AWS scales as they continue to add capacity and customers. THE PERFORMANCE MODEL When storing or retrieving a file with S3 we expect the performance to be comprised of two parts: a fixed transaction cost that is not related to file size and a variable bandwidth cost that is related to file size. In other words, we expect a linear performance model for storing and retrieving files between S3 and EC2. Determining the fixed transaction and variable bandwidth costs is the goal of our data analysis.

2 THE DATA Here is the data, grouped by file size, for storing and retrieving files: Data set 1: Storing a file File size range Sample Size Average file size Average time Performance (ms) (MB/s) From To 0 KB 100 KB KB KB 200 KB KB KB 300 KB KB KB 400 KB KB KB 500 KB KB KB 600 KB KB KB 700 KB KB KB 800 KB KB KB 900 KB KB KB 1.0 MB KB MB 1.1 MB MB MB 1.2 MB MB MB 1.3 MB MB MB 1.4 MB MB MB 1.5 MB MB MB 1.6 MB MB MB 3.2 MB MB MB 6.4 MB MB MB 12.8 MB MB MB 25.6 MB MB MB 51.2 MB MB MB MB MB MB 5 GB MB

3 ANALYSIS The variable bandwidth cost when storing a file is between 10 and 12 MB/s. To determine the fixed transaction cost we can perform a linear regression. The following chart illustrates: This chart shows that the fixed transaction cost when storing a file is around 140 ms. Data set 2: Retrieving a file File size range Sample Size Average file size Average time Performance (ms) (MB/s) From To 0 KB 100 KB KB KB 200 KB KB KB 300 KB KB KB 400 KB KB KB 500 KB KB KB 600 KB KB KB 700 KB KB KB 800 KB KB KB 900 KB KB KB 1.0 MB KB MB 1.1 MB MB MB 1.2 MB MB MB 1.3 MB MB MB 1.4 MB MB MB 1.5 MB MB MB 1.6 MB MB MB 3.2 MB MB MB 6.4 MB MB MB 12.8 MB MB MB 25.6 MB MB MB 51.2 MB MB MB MB MB MB 5 GB MB

4 ANALYSIS The variable bandwidth cost when retrieving a file is again between 10 and 12 MB/s. Unlike with storing files there appears to be no discernible fixed transaction cost CONCLUSION We can conclude the following from the above analysis: 1. The variable bandwidth cost when storing and retrieving files is between 10 and 12 MB/s 2. The fixed transaction cost when storing a file is roughly 140ms and negligible when retrieving a file METHODOLOGY We tracked the number of milliseconds (ms) it takes to store and retrieve files from S3 using large EC2 instances in the us-east-1a availability zone. We used the JetS3t Java Library to handle the actual storing and retrieving of files which in turn uses the Commons HttpClient Library. We started the timer at the point when the file is being stored or retrieved to S3. We included 50,000 total data points (stores and retrieves) in our analysis, drawn randomly from the month of February. These data points cover all days of the week and hours of the day. LIMITATIONS AND OTHER CONSIDERATIONS The maximum throughput (around 50 MB/s) you can expect when using S3 from a large EC2 instance is discussed here. Since we actively load balance our instances we don't anticipate that this limit had any discernible impact on our results. To track the time it takes to store and retrieve a file we use the Java System.currentTimeMillis() function. From the documentation on this function: Note that while the unit of time of the return value is a millisecond, the granularity of the value depends on the underlying operating system and may be larger. For example, many operating systems measure time in units of tens of milliseconds. Since we have a large number of data points at small file sizes this should have a very limited impact on our results. Our times do not include failed attempts to store or retrieve a file from S3.

5 ABOUT HOSTEDFTP.COM HostedFTP.com is a cloud file sharing solution that's secure, reliable and easy to use. Designed for use with both web browsers and FTP clients, HostedFTP.com improves network security and saves your business money. For more information please visit our website