Choosing Between Commodity and Enterprise Cloud With Performance Comparison between Cloud Provider USA, Amazon EC2, and Rackspace Cloud By Cloud Spectator, LLC and Neovise, LLC.
1 Background Businesses and enterprises have quickly realized the power and efficiency of cloud computing. The ability to deploy servers in minutes rather than days, scale them elastically and nearly eliminate upfront cost has made infrastructure as a service (IaaS) attractive to both new and established businesses. Unfortunately, with the vast number of IaaS providers in the market, it has become hard to truly distinguish between them. Some providers emphasize the low cost of their cloud, while others focus on delivering strong customer service and security. Yet, with so many providers offering the same features, a distinction between Provider A and Provider B is no longer easily made. Even the line between basic cloud types commodity clouds and enterprise clouds has become blurry. Virtual server performance, a little explored but extremely vital consideration for IT decision-makers interested in migrating or beginning projects in the cloud, is becoming a factor when choosing between commodity and enterprise cloud services. In 2012, Neovise partnered with Cloud Spectator to conduct a series of performance tests comparing Amazon EC2, Rackspace Cloud, and Cloud Provider USA. Tests ran from December 8, 2012 until January 8, 2013, and measured CPU power, disk I/O, and internal network speeds. The two analyst groups jointly published this white paper to clarify the performance differences between two commodity clouds Amazon EC2 and Rackspace Cloud and the enterprise cloud offered by Cloud Provider USA. Commodity Clouds Versus Enterprise Clouds Two basic types of public infrastructure-as-a-service (IaaS) clouds have emerged: commodity clouds and enterprise clouds. As their names suggest, commodity clouds are built with commodity-class infrastructure and enterprise clouds are built with enterprise-class infrastructure. However, that is just the beginning. These broad classifications have other important attributes associated with them as well. Before describing the additional attributes of these clouds, it is important to note that they are often discussed as extremes. As with many technologies and services in the IT industry, strong opinions have formed regarding these different approaches. There are some camps that claim commodity clouds are the single best approach to IaaS and there are opposing camps that claim enterprise clouds are the single best approach to IaaS. In reality there is room for both approaches as well as some in-between approaches. Reliability One attribute debated by the commodity and enterprise camps is reliability. The commodity cloud extremists say that reliability in the cloud comes from the application, not the infrastructure. They suggest that two or three nines of availability from the cloud infrastructure is sufficient since applications can be distributed across multiple availability zones in order to reach application availability of three or four nines. While it is true that many IaaS clouds, both commodity and enterprise, provide facilities for redundancy such as multiple availability zones, the majority of existing applications have not been designed with that in mind. Enterprise clouds not only use enterprise-class hardware that tends to be more reliable, they often have a greater focus on overall reliability backed by more stringent service-level agreements (SLA). Performance SLAs themselves serve as another point of debate between commodity and enterprise clouds. Commodity cloud extremists have two primary arguments as to why cloud SLAs don't matter. First, they say that cloud providers cannot deliver on key elements of SLAs such as performance. Since multitenant cloud environments consist of unpredictable users competing for shared resources, they claim that cloud service providers have no way of guaranteeing performance levels. Yet, in direct conflict with this claim, there are some enterprise cloud providers with infrastructure architectures that enable performance commitments. The second argument from commodity cloud extremists against the need for performance guarantees in SLAs builds off the first one. They build in their own application performance, even though commodity cloud service providers cannot guarantee the performance of
2 the underlying infrastructure. As with increasing reliability through application design, application developers can use scale-out architectures to increase application performance. Yet, once again, most existing applications have not been written to scale out and scale back in as demand fluctuates. Additionally, scale-out architectures do not solve every performance-related problem. For example, it may be possible for a transaction-oriented application to process multiple transactions in parallel, dramatically increasing the number of transactions that can be handled per second. However, assuming individual transactions cannot be broken down further into even more parallel tasks, a scale-out architecture does not help complete individual transactions any faster. That would instead require some combination of a faster processor, lower latency network and faster storage, attributes more typically found in an enterprise cloud. Real-World Clouds Rather than simply focus on the general differences often found between commodity and enterprise clouds, it is important to evaluate specific real-world clouds to determine how well they meet your needs. Organizations of all kinds from small businesses to large enterprises, as well as government organizations are reminded that determining your cloud requirements is not a battle between commodity and enterprise clouds. Choosing one extreme view or another is not likely to produce the best results. Selecting a cloud service provider is much more about understanding the requirements of your individual applications. Additionally, keep in mind that a given large organization is likely to have a mix of applications, including some that will run well on a commodity cloud and others that require an enterprise cloud. Of course there is also significant variation between individual clouds of a given type, commodity or enterprise. Amazon and Rackspace are perhaps the most commonly cited examples of commodity clouds, yet even they have major differences in the variety of services and levels of support offered. Evaluating Public Cloud Performance Organizations seeking public clouds that deliver high levels of performance should expect to see major differences between commodity cloud offerings such as Amazon and Rackspace versus enterprise clouds such as Cloud Provider USA. The performance results achieved by Cloud Provider USA as part of a comprehensive performance analysis completed by Cloud Spectator demonstrate this clearly. CloudSpecs Score The CloudSpecs score takes the pricing of each provider s server and sets it in relation to server performance to create a score. The higher the score, the better the value to the customer (highest score is 100). Cloud Provider USA s server is $0.73/hour. Rackspace Cloud s server is $0.96/hour. Amazon EC2 s server is $0.64/hour. For more details on how to calculate the CloudSpecs score, please see the methodology in Appendix 1. Summary of Findings Cloud Provider USA, the enterprise cloud, performs on a superior level compared to Amazon EC2 and Rackspace Cloud in areas of CPU performance, disk I/O, and internal network speeds. All three areas are important to understand when selecting a high-performance cloud. All three tests can easily be reproduced. CPU performance is measured with the open source UnixBench test. Disk I/O is measured with the open source Dbench test. Internal network speed is gauged by the Linux terminal ping command. For disk I/O, Cloud Provider performed over 7x better than Amazon EC2 and over 9.5x better than Rackspace Cloud over the same 30-day period. Cloud Provider USA also outperforms Amazon EC2 by 8.3x and Rackspace by 2.3x in internal network speeds. For CPU performance on average, over 30 days, Cloud Provider USA performed almost 2.5x better than Amazon EC2 and almost 3x better than Rackspace Cloud. For a detailed methodology, please see Appendix 1.
3 Disk Relative to hard disk drives (HDD), solid state drives (SSDs) represent a major advance in performance, in both latency and throughput. Latency measures how long it takes to begin a data transfer, while throughput measures how fast data transfers once the transfer begins. With all SSD storage, Cloud Provider USA delivers the ultimate in raw storage performance. Additionally, beyond maximum performance levels, SSDs deliver higher quality of service (QoS) by reducing I/O contention that is problematic in some multitenant public cloud environments. This enables guaranteed IOPs, allowing I/O sensitive applications and databases to achieve extreme levels of performance. Dbench is a benchmark designed by the Samba project as a free alternative to netbench. It contains file-system calls for testing disk performance. Dbench can be used to stress a filesystem or a server, and can also be used for prediction analysis to determine how many concurrent clients/applications performing this workload the server can handle before response starts to lag. Provider 30-day Average MiB/s CloudSpecs Score Cloud Provider USA 907 100 Amazon 233 29 Rackspace 105 9
4 Network The data center network in a public cloud environment serves in many ways as the foundation of performance for all users and all applications. If the network becomes a bottleneck, all tenants in the cloud are impacted even those using the network minimally. When the network slows, even briefly, the performance of multi-tier and other forms of distributed applications can deteriorate significantly. To avoid this, Cloud Provider USA has deployed a 40Gbps network throughout its cloud environment. Keep in mind that consistent network performance can be just as important as average network performance. Note the large fluctuations shown in the performance of the Amazon network. By default, the Linux Terminal Interface includes the ping command. The command measures the delay between two servers. Because virtual servers may be physically separate from the SAN or other virtual servers, in order to maintain fast application and processing functionality, it is important the network is able to transfer data quickly enough. In cloud environments, the network becomes even more vital, as many virtual servers may share a limited network. Provider 30-day Average µs CloudSpecs Score Cloud Provider USA 170 100 Amazon 1,412 11 Rackspace 403 92
5 Servers As discussed earlier, many commodity clouds are built on the notion that performance and scale can be achieved solely through scale-out architectures. Yet, relational databases are not generally designed to scale out. They rely on high-end servers to achieve performance and scale. Low latency, transaction-oriented applications have a similar dependency on high-end servers. At the same time, raw server performance is not the only attribute that matters. The total cost required to achieve the desired performance is also key. The CPU performance tests clearly show that Cloud Provider USA outperformed both Amazon and Rackspace on a price/performance basis. The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system. The UnixBench is a suite of approximately 10 tests that can measure how many instructions per second a CPU can perform. If CPU A performs 2x better than CPU B at only 1.5x the cost, then CPU A provides more performance per unit of cost. Provider 30-day Average Score CloudSpecs Score Cloud Provider USA 2,864 100 Amazon 1,152 46 Rackspace 966 26 Closing Thoughts Readers should keep in mind that simply throwing together high-performance infrastructure components is insufficient. The value of a cloud is not determined by the quality of its components alone, or even its end-to-end performance. Cloud Provider USA goes beyond performance to deliver deep transparency and control, robust service and support and meaningful SLAs. The result is an extremely high-value, flexible, elastic, pay-as-you-go cloud offering.
6 Appendix 1 Cloud Spectator sets up two servers on each cloud provider that meets the following minimum requirement: Primary Server Secondary Server CPU Cores 4 vcpus CPU Cores 1 vcpu RAM in GB 16 RAM in GB 4 Disk in GB 50 Disk in GB 50 Because Amazon and Rackspace offer packaged cloud servers with preset allocations of CPU, RAM, and disk, the closest package that meets the minimum requirements is selected. AMAZON The servers are set up on Amazon EC2 availability zone US-East-1a. Virtualized server OS is Amazon Linux. Primary Server Secondary Server CPU Cores 4 vcpus CPU Cores RAM in GB 15 Disk in GB 1,690 1 vcpus RAM in GB 3.75 Disk in GB 410 RACKSPACE The servers are set up in Rackspace s Dallas (DFW) data center. Virtualized server OS is Debian 6. Primary Server Secondary Server CPU Cores 4 vcpus RAM in GB 15.872 Disk in GB 620 CPU Cores 2 vcpus RAM in GB 4 Disk in GB 160 CLOUD PROVIDER USA The servers are set up in Cloud Provider USA s Virtualized server OS is Debian 6. Primary Server CPU Cores 4 vcpus RAM in GB 16 Disk in GB 50 Secondary Server CPU Cores 1 vcpus RAM in GB 4 Disk in GB 50 Cloud Spectator maintains root access to all servers. Using SSH, the CloudSpecs application is installed on to the servers. Both Primary and Secondary servers are located within the same virtual private network. Once the CloudSpecs performance application is installed, it automatically runs 4 times a day, 365 days a year. PERFORMANCE TESTS Cloud Spectator runs basic, open source benchmark tests that are precise and accurate for measuring the performance of a cloud server. Tests included come from the Unixbench Suite (https://code.google.com/p/byte-unixbench) and Phoronix Test Suite (http://www.phoronix-test-suite.com). The benchmarks for network tests and the disk benchmark for local file copy may be reproduced using the Linux command line interface. The network tests assume $SERVER is the IP of the secondary server and $CLIENT is the IP of the primary server. Command line instructions for the internal network tests are included below in each of the tests within the network description. DATA COLLECTION The CloudSpecs test results for this publication run from December 8, 2012 to January 8, 2013 (a total of 30 days). Cloud Spectator obtains cloud servers by either purchasing the server space directly from the client as an anonymous user, or the client reimburses Cloud Spectator for data collected relevant to that active project. THE CLOUDSPECS SCORE CALCULATION The CloudSpecs Score represents the true value a cloud user receives when using a server. A higher score means better value. The highest value performer receives a score of 100 and other providers are pegged in relativity to value. Value is calculated based on performance over time and the cost of the Primary Server hourly (730 hours per month). Provider Amazon EC2 East $0.64 Rackspace $0.96 Cloud Provider USA $0.32 Cost per Hour
7 Below is the formula process for calculating the CloudSpecs Score (Provider s CloudSpecs Score is the calculation for the final CloudSpecs score of each provider). The CloudSpecs score is rounded off to the nearest whole number. The CloudSpecs score in this report were calculated over a period of 30 days: provider_value(p) = [Provider test score over a period of time] / [Provider price] best_provider_value = max(provider_values) Provider s CloudSpecs Score = 100 * provider_value(p) / best_provider_value About Cloud Spectator Cloud Spectator is the premier cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud Spectator has monitored the cloud Infrastructure industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 4 times a day, 365 days a year with use of open source benchmark tests. The data is compared relatively with pricing from each provider to allow business decision-makers to smart decisions. Cloud Spectator 800 Boylston Street 16 TH Floor Boston, MA 02199 Website: www.cloudspectator.com Phone: (USA) 1-617-300-0711 About Neovise Based on independent research and analysis, Neovise delivers essential knowledge and guidance to cloud-related technology vendors, service providers and systems integrators, as well as business and IT organizations that purchase and use cloud-related services and technology. Our offerings include research, advisory and collateral development services that help our customers and their customers make optimal decisions and formulate winning strategies. Research. Analyze. Neovise. For more information, visit www.neovise.com.