205 Industry Report A Benchmark Comparison of Speed, Availability and Scalability
Executive Summary Both 203 and 204 were record-setting years for adoption of cloud services in the enterprise. More than percent of large enterprises reported using cloud services as an integral part of their infrastructure, and it is safe to say that cloud services are no longer just for early adopters. Cloud service provider (CSP) offerings have matured and are quickly becoming an essential component of any leading enterprise s critical infrastructure. One of the most widely adopted and fastest growing cloud services is public cloud storage. Some analysts project a 33 percent compound annual growth rate for public and private cloud storage over the next five years, as enterprises find new ways to leverage the unlimited capacity and enterprise-grade reliability offered by most leading CSPs. The challenge for enterprises, however, has been how to take full advantage of what these providers have to offer without increasing internal costs for support, maintenance and custom development. At Nasuni, we have created a unique service that enables enterprises to scale their storage capacity without scaling their internal operations, to provide unlimited backups of their data, and to provide global access from any device. To do this, we leverage integrations with the major CSPs for unlimited capacity and global footprint. Our goal is to always provide the best infrastructure for our customers, and to that end, we regularly test these CSPs services. This year, we have restricted our tests to the three leading CSPs: Microsoft Azure s Blob Storage, Amazon s Simple Storage Service (S3), and Google s Cloud Storage. We found that the results are similar to last year: Microsoft has come out as the highest performer, particularly in the benchmark tests. Amazon, as expected, is a close second, and edged out Microsoft in some aspects of scalability and availability, but ultimately fell short in some of the critical benchmark evaluations. Google continues to show promise albeit without a significant difference in its status from 203. Cohen, Reuven, The Cloud Hits the Mainstream: More than Half of U.S. Businesses Now Use Cloud Computing, Forbes, April 6, 203. 2 Rebello, Jagdish, Enterprise cloud computing: future market size, growth and competitive landscape, IHS Quarterly. 2 United States:.800.208.348 International:.8.433.6200
Why Does Nasuni Publish This Report? We regularly benchmark the major cloud service providers to ensure that we offer our customers the most reliable and best performing cloud storage available. A few years ago, we decided to publish these results to ) demonstrate the reasoning behind our choice to work with specific technology partners, and 2) provide some useful benchmarking and analysis in a market that is becoming increasingly commoditized. How We Evaluate CSPs One of the things that we take pride in doing well at Nasuni is making the most out of what all the cloud service providers have to offer. The Nasuni Service employs many different cloud services including NoSQL databases, elastic computing, and messaging queues from a variety of providers. But storage is central to what we offer, so, accordingly, we are invested in evaluating the public cloud storage capabilities of the major providers. Our thinking around how to test public cloud storage comes from our own experience providing missioncritical storage services to hundreds of enterprises. Our benchmark tests reflect the patterns of use that we see from our customers. The scalability tests are designed to stress the critical aspects of the storage environment, such as the durability of containers and the behavior of the service under stress, as well as to simulate some of our toughest use cases, such as disaster recovery and initial migrations. Where we set up our tests and how frequently we test are also considerations that are designed to replicate the global footprint of our customers and help us to identify weak points not just in the storage itself, but also in the network infrastructure that is delivering it. provider is more a decision about the price and performance trade-off than about features. We hope that, by sharing our findings on the performance of these providers, we are informing the market with critical information to enable readers to distinguish between these so-called commodity services. Although this report represents a detailed review of Nasuni s latest findings, it is important to note that we conduct these tests primarily to identify CSPs that meet the needs of Nasuni and its customers. They do not necessarily represent the optimal metrics for any provider tested. However, these independent test results should be of interest to any organization that is considering the use of cloud storage as a component within a larger IT infrastructure. Comparison Metrics As with most technology evaluations, three main criteria govern the decision to purchase: Functionality: what a service offers Price: the cost of the service Performance: how well that service is operates For each of the cloud service providers, the functionality is essentially the same: we are able to create containers for objects, write objects into that container, read objects from that container, and then delete objects. For most of the major CSPs, price is based on some combination of data stored per month, and the bandwidth consumed with both writes and reads. Price competition has been happening for years in this industry, and the real distinction in pricing is around the capacity tiers of service. But, for the most part, the price differences are minimal and only matter on the margin. This leaves us with performance as the main set of comparison metrics. Sharing Critical Performance Information Public cloud storage is often referred to as a commodity that a CSP offers: a basic service that differs little from its competitive alternatives. Since most of the major cloud storage services are at feature parity with each other, choosing the right 3 United States:.800.208.348 International:.8.433.6200
Performance Performance is the primary yardstick by which Nasuni measures any publicly available CSP, testing the operation and stability of CSPs over long periods of time. In fact, Nasuni has been testing and comparing CSPs since 2009. Before considering any CSP for use in a production environment, it must meet minimum performance benchmarks across three areas: Speed Availability Scalability Speed This simple test measures the raw ability of each CSP to handle large numbers of writes, reads, and deletes (W/R/D). We test each CSP with files of varying sizes: KB 0 KB 00 KB MB 0 MB 00 MB GB Using different levels of concurrency: Thread 0 This test runs for 2 hours, using multiple testing machine instances and several non-serial test runs to reduce the likelihood that external network issues could bias the results. Availability: This test takes place over a 30-day period and measures each CSP s response time to a single W/R/D process at 60-second intervals: Write a randomly-generated KB file. Read a randomly-selected previously-written file. Delete a selected file. Reading and deleting a random file forces each CSP to prove their ability to be responsive to all of the data, all of the time, and not merely to the last piece of cached data. This test calculates the entire time required to complete the three requests, including any required retries. This ensures examination of not only responsiveness, but also of CSP reliability and latency. Scalability: Similar to the availability test, this is also an extended test that measures each CSP s ability to perform consistently as the number of objects under management increases. Performance under increasing object counts is often the Achilles heel of a cloud storage system, and this test measures each CSP s ability to maintain performance levels as the total number of objects stored in a single container increases to hundreds of millions. This is particularly important to Nasuni, because unlimited scalability is a key feature of our service. We have customers with large data sets that have been keeping version snapshots for years. Methodology Due to dynamics in the marketplace, the list of platforms evaluated continues to change from year to year. The CSPs tested this year are Amazon S3, Microsoft Azure, and Google Cloud Storage. HP s Cloud Object Storage and IBM s SoftLayer were considered, and even tested in a limited capacity; however, we ultimately decided not to include them. In HP s case, it was partly because there was a change in their strategic direction and we were unsure what their service would look like over the coming year. With IBM, our experience with their service was checkered with scheduled outages on their part. Working around those proved to be difficult and a concern for us. We hope to be able to re-visit both services in the future. We also looked to outside analysts to help validate our decisions to focus our efforts this year. Gartner has listed our three target CSPs in their most recent Magic Quadrant report as the only Leaders (Amazon and Microsoft) and Challengers (Google) in the public cloud storage space. That additional perspective helped us justify our decision to restrict the scope of this year s report. Unlike last year, we did not invite the CSPs to participate in the evaluation process. 4 United States:.800.208.348 International:.8.433.6200
Test Setup Nasuni engineers conducted all tests between October 204 and February 205, using virtual machines across most of the major cloud-compute platforms. The virtual machines had the following specifications: RAM: 3.5-4 GB vcpus: Operating system: Ubuntu 4.04, 64 bit - Ubuntu 4.04 LTS File Size & Tested Number of Concurrent 0 KB 0 KB 00 KB MB 0 MB 00 MB Table 2: File size and thread count combinations tested GB Each CSP s benchmark tests were run using four outside machines. For example, Amazon EC2 was not used to test Amazon S3. Instead, S3 was tested using Microsoft, HP, Google, and IBM. The test centers were also spread geographically throughout the eastern and central regions of the United States. We ran all tests using a variety of times, locations, virtual machines, and dates to minimize the risk of external network bias. Benchmark Tests: The benchmark tests are designed to evaluate the performance of CSPs under file-server data load, and the file data set represents the same distribution of file sizes used by actual Nasuni enterprise customers across thousands of installations over several years. Here is our typical breakdown of files by size: KB 0KB 00KB MB 0MB 00MB GB 6.8% 24.6% 26.2% 9.7% 22.2% 0.4% 0.% Table : File size distribution In addition, this distribution of file sizes closely matches a well-documented breakdown that Microsoft did over a number of years. For each speed test, the test evaluated 23 combinations of file sizes and thread counts as shown in Table 2. The results are averaged based on the weighting of customer file-server data (Table ) and are then indexed to the performance of the top performer. The results compare all the CSPs to the performance of the top performer across all file sizes and thread counts. This allows hundreds of individual tests to be evaluated using a single benchmark metric. Detailed raw results by CSP are included in the appendix. As the results show, the raw speed performance varies significantly as object sizes and thread counts vary. Specifically, small object sizes and smaller thread counts highlight the transactional overhead of any platform. The effect of transactional overhead becomes most noticeable during writes, which contains three steps: Preparation Transmission Acknowledgment For small files, Transmission is only a small portion of the total transaction, so any inefficiency in the performance of Preparation and Acknowledgment has greater impact. Those object stores that are built with efficient Preparation and Acknowledgment steps perform best when handling small files. However, as file sizes or thread counts increase, the time associated with Transmission increasingly dominates the overall time associated with the transaction. Inefficiencies in Preparation or Acknowledgement become less and less critical. Many CSPs overly focus their efforts to improve the efficiency of the Transmission stage of the 5 United States:.800.208.348 International:.8.433.6200
transaction, and thus perform better under the load of larger object sizes or thread counts. This may be fine for use-cases such as media archives, but for file-server data, which is often dominated by small files, performance on small files is critical. Availability Test: Availability tests were run from a single VM running in Rackspace. The metric used is response time, which measures each CSP s response time to a single W/R/D process at 60-second intervals. Because response time also includes any time associated with retries or delays, it is a more effective metric for availability than a simple ping test. Scalability Test: Scalability tests were conducted using internal machines to reach the highest scalability numbers in the shortest amount of time, for example, Amazon EC2 writing to Amazon S3. As object counts increase, the performance of some CSPs degrades or becomes variable. Depending on CSP architecture, some systems are designed to scale across containers, not within them. This type of architectural limitation can become a significant bottleneck after months or years of usage. An ideal scenario for anyone seeking to leverage cloud storage is to partner with a CSP whose performance and responsiveness are unchanging, regardless of the number of objects under management. Just as with traditional in-house storage, customers expect a consistent level of performance. Results Write Benchmark Similar to last year, Microsoft was the top write performer, excelling in 3 of the 23 individual combinations tested, thereby making it the optimal write target for file-based data. Amazon is a strong second and Google performed at less than half of the average response time of Microsoft. Figure : Indexed write speed with all file sizes However, for files larger than MB, Amazon had the overall best write performance. In Figure 2, Google and Microsoft are on par with one another, essentially tying for second place to Amazon. Under this test, all of the CSPs were loaded with new objects as quickly as possible: up to 00 million objects or 30 days, whichever came first. The variance represents how much the speed of loading objects changed over time, causing inconsistency and variability as objects were loaded. This year we added a new dimension write speed vs. write speed variance so others can evaluate the tradeoff of time-to-complete their large tasks vs. the variability in the performance. Figure 2: Indexed write speed with file sizes >MB 6 United States:.800.208.348 International:.8.433.6200
Read Benchmark In terms of read performance, Microsoft still consistently outperforms the other CSPs. However, Amazon trails Microsoft by less than it did in the write performance tests. Google, however, consistently performs at about half of the level of Microsoft in both read and write. Delete Benchmark Microsoft s real performance superiority is seen in the delete benchmark: it is more than twice as fast at deleting files as Amazon and nearly 5x as fast as Google. Different CSPs implement delete in different ways. Some CSPs acknowledge the delete and then do the work in the background. Others do the actual delete operation before responding to the request. This could explain some of the results in this benchmark. Figure 3: Indexed read speed with all file sizes Similar to the write benchmarks, Amazon significantly outperforms both Google and Microsoft on the large file sizes. Figure 5: Indexed DELETE speed with all file sizes Figure 4: Indexed read speed with file sizes >MB 7 United States:.800.208.348 International:.8.433.6200
Availability Amazon and Microsoft nearly tied on our availability metrics, averaging a response time of. and.4 seconds, respectively, over a 30-day period. Google, again, trailed the top two with an average response time of.5 seconds, nearly 5x slower. In addition to system and data availability, the test also measures overall uptime or percent of the time that the CSP is reachable. Unlike in previous reports, each CSP had perfect scores of 00 percent uptime. Scalability In previous reports, we measured scalability resilience using three metrics: ) the variance of write speed in writing 00M objects, 2) the number of write misses and 3) the number of read misses. This year we decided to add a fourth dimension to this analysis, the tradeoff between variance (measured as (obj/s) 2 ) and write speed. Google had by far the lowest variance of the three, an order of magnitude smaller than both Microsoft and Amazon. Microsoft had the worst of the three with exceptionally large variance. Figure 6: Average response time (shorter bars are better) Examining the results over the month of testing also gives some insight into the variability of the numbers. Amazon shows the most consistency and the smallest daily uptime values. Microsoft is a close second. Google, however, is both more variable and slower. Figure 8: Variance in write speed (shorter bars are better) The read/write error analysis showed less of a difference between CSPs. All CSPs had zero read errors for 00M objects. Google and Microsoft showed zero write errors, while Amazon had five write errors for an error rate of.000005 percent. Figure 7: Average daily response time 8 United States:.800.208.348 International:.8.433.6200
The average write speed/variance trade-off tells a different story. Microsoft performs worst with the lowest average write speed and the highest variance. Amazon appears to be the top performer with the highest write speed; however, that comes at a price of a substantially higher variance over Google, which has the lowest variance but also a substantially lower average write speed. For most applications, however, the ability to write quickly is more important than the ability to write consistently, which makes Amazon the preferred choice for simply scaling quickly. Amazon performed slightly better than Microsoft in our availability test, both in overall average response time and in daily average response time. However, Google was a distant third, with almost five times the response time. Scalability results were mixed. Amazon had a relatively low variance and the highest average write speed, but also was the only CSP to show any write errors. Although Microsoft had the highest variance, it also posted the second highest speed and a perfect record of zero read and zero write errors. Overall, Microsoft and Amazon are the two clear leaders in public cloud storage. This year, Microsoft out-performed Amazon, specifically, in our critical benchmark tests. However, Amazon continues to demonstrate the robustness of its platform, particularly in its ability to scale quickly and reliably. Figure 9: Variance (/V) vs. average write speed Conclusion Similar to our 203 report, we find that there are only two significant competitors in the public cloud storage market: Microsoft and Amazon. And, for the second year in a row, Microsoft is the top CSP for public cloud storage. Disclaimer The tests reported upon in this document are conducted by us using our own test tools under test conditions chosen by us. The test conditions were chosen by us to reasonably represent what our customers would experience using our Service with their representative environments and workloads. The tests have been designed by us to only look at the performance aspects of the CSPs that we believe are relevant to our customers it is intentionally narrow in scope. Nasuni is not in the business of benchmarking CSPs, certifying test results or selling performance metrics. We have attempted to make sure the tests are fair and consistent within our selected parameters and have worked with several of the vendors to confirm our results. Our tests are not meant to indicate performance from each CSP under ideal conditions to the CSP, and, in any event, performance should only be one factor of many in a CSP selection process. In our most important suite of tests the benchmark tests Microsoft consistently performed better than Amazon. It delivered the best speeds across small and medium-sized files and, in some cases, beat Amazon by nearly 2x. For large files, Amazon showed better write and read performance, but Microsoft s delete speeds topped Amazon s. 9 United States:.800.208.348 International:.8.433.6200
Appendix Raw Results Microsoft Azure Write Benchmark Results (KB/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 62 53 3,485 9,654 8,075 35 3,242 7,2 28,947 27,82 472 4,36 9,745 29,838 28,896 59 4,606 9,973 30,04 30,4 Read Benchmark Results (KB/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 3 863 4,67 0,27 22,886 63 6,04 22,748 33,854 27,462,229 9,600 27,06 72,474,773,285 9,778 26,28 79,6,493 Delete Benchmark Results (Obj/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 07 05 05 29 755 799 806 795 695,267,0,294,032 659,272,294,276,244 435 Note: Microsoft Azure Blog Storage does not support objects larger than 64MB in a single upload. 0 United States:.800.208.348 International:.8.433.6200
Amazon S3 Write Benchmark Results (KB/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 9 9 560 3,003 9,24,737,85 95 96 5,55 27,23 48,88 43,77 23 2,24 4,4 65,409 02,036 48 4,707 27,849 2,97 9,267 Read Benchmark Results (KB/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 9 75,480 8,729 20,067 32,65 36,042 90,88 3,783 62,96 89,80 86,477 474 4,560 3,833 83,684 82,460 996 9,354 56,899 86,346 75,570 Delete Benchmark Results (Obj/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 7 8 8 7 8 6 3 74 66 82 75 80 57 470 448 473 46 44 957 955 968 945 8 United States:.800.208.348 International:.8.433.6200
Google Cloud Storage Write Benchmark Results (KB/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 5 53 343 2,496 3,89 23,72 28,24 54 5 3,533 23,094 32,0 32,040,267 8,769 30,958 32,327 26 2,533 5,736 30,89 32,086 Read Benchmark Results (KB/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 23 88,436 5,095 9,658 28,966 3,598 20,90 2,27 27,97 32,930 33,994 58 4,740 9,056 8,520 29,843 975 7,488 22,440 4,73 24,288 Delete Benchmark Results (Obj/s) 0 KB 0 KB 00 KB MB 0 MB 00 MB GB 9 8 6 9 0 8 8 87 88 6 86 77 60 22 209 49 28 92 44 49 296 30 264 2 United States:.800.208.348 International:.8.433.6200
About Nasuni Files are everywhere and they are a pain. Nasuni eliminates this pain forever by delivering file storage for distributed enterprises using a combination of cloud capacity, Nasuni software, and NAS appliances. Nasuni gives customers unlimited storage with built-in data protection and DR, secure global file sharing and mobile access, all managed from a single web console. Nasuni is cloud-based NAS for the distributed enterprise. Our team is made up of enterprise storage, security and networking industry veterans with a shared vision of transforming the way enterprise organizations view data storage. We believe that storage should be as easy to purchase, consume and manage as the electricity that keeps the lights on. Nasuni Corporation 205, All Rights Reserved 33 Speen Street, Natick, MA 0760-538.8.433.6200 WP_3.5