Cloud Computing Performance. Benchmark Testing Report. Comparing ProfitBricks vs. Amazon EC2

Transcription

1 Cloud Computing Performance Benchmark Testing Report Comparing vs. Amazon EC2 April 2014

2 Contents The Cloud Computing Performance Benchmark report is divided into several sections: Topics.Page Introduction... 1 Executive Summary... 2 Benchmark Testing Methodology... 2 UnixBench Results... 4 DBENCH Results... 7 Iperf Results Next Steps We invite you to contact one of our Cloud Computing consultants by ing inbound- us@profitbricks.com or calling We also have a 14- day, no obligation trial account option that does not require a credit card. Please visit to get started. Comparing Cloud Computing Performance Amazon EC2 and

3 Introduction At, Cloud Computing performance is a primary focus for our teams of engineers. We take a comprehensive view on IaaS performance benchmarking. We contract with 3 rd parties, explore, research and develop new testing methodologies in-house, and discover the bugs and limitations of traditional and new benchmarking tools. As a Cloud Computing service provider we realize that we are an edge case our responsibility for high performance extends from hardware and network architecture to the virtualization layer, to the software we develop to manage the environment. Cloud Computing s multi-tenant architecture requires that we keep and maintain performance levels that exceed our customers expectations, and we strive to remain the highest performance cloud available in the market. The performance engineering team continually tests the performance of other cloud platforms and services. Performance is our passion and the team is dedicated to publishing accurate and repeatable results. Benchmark testing Cloud Computing environments that are not similar requires a thorough understanding of the respective environments, the effects that each component may have on performance, in addition to ensuring that the configurations are similar. It s also essential that each benchmark test is run on top of a similar stack and software configuration. We welcome questions about our methodologies. strives to share our expertise with the community and to engage in an open dialogue about cloud performance. Comparing Cloud Computing Performance Amazon EC2 and - Page 1

4 Executive Summary In this report, the performance engineering team presents our latest series of standardized benchmark testing results (UnixBench, DBENCH, Iperf), comparing virtual data centers and instances with Amazon s EC2 instances. We strive to create an apples to apples comparison of virtual servers/instances at both and Amazon EC2, and we believe our methodologies are fully transparent (see description in Section 3). In January 2014 Amazon introduced a new line of instance sizes, based on new CPUs and new architecture. engineers spent eight weeks configuring and testing these new EC2 m3 instance types and compared them to similar instances alongside Amazon s legacy m1 instances. The results show that continues to be the performance leader in the cloud on all tested configurations, using all three standard benchmarks, showing results that are at least twice the performance of Amazon. In some cases the performance difference between and Amazon EC2 is an astounding 17x (Iperf). Benchmark Testing Methodology All benchmarking tools were compiled on Ubuntu (64-bit) servers running a Linux kernel, and each test was performed three times on three separate days with the results averaged. By taking advantage of granular scaling features, we were able to compare Amazon s latest EC2 offerings to servers with similar resource specifications. For example, we contrasted the performance of an EC2 m3.large with 2 vcpus and 7.5GB of RAM with a server containing 2 dedicated cores and 7.5GB of dedicated RAM (and equivalent InfiniBand-powered double redundant storage devices). In all cases, these are standard configurations at both and Amazon. No special or optional services were added to the configurations. For each report, three test runs were completed on each instance for three days. Note: we continually see performance variations (sometimes large) on Amazon EC2 services that vary by instance size, by hour, and by data center. This is the main reason for multiple runs we want to ensure that we ve determined a representative composite result. Comparing Cloud Computing Performance Amazon EC2 and - Page 2

5 Server Configuration: Amazon virtual instances are sold and packaged with a pre-packaged quantity of vcpus, RAM and temporary storage. virtual instances are not pre-packaged, thus every instance can be uniquely configured. We ve done our best to match equivalent instances to Amazon instances on a hardware basis not a cost basis. OS Configuration: Operating System: Ubuntu (64-bit) with Linux kernel Each instance had the following packages installed if the provider did not include them with their images: make, GCC, automake, libpopt-dev, zlib1g-dev, g++, git All benchmark software used in these tests are open source and we have provided the information you would need (including configuration details) to run these tests. You can download the tools from these URLs: UnixBench: DBENCH: Use git to create a local branch of the official git repository and download the tree: `git clone git://git.samba.org/sahlberg/dbench.git dbench` Iperf: For DBENCH and UnixBench, both of which test storage performance, a separate block device (EBS on EC2 vs. block storage) was formatted with ext4 and mounted: mkfs.ext4 /dev/second_device mount /dev/second_device /bench Comparing Cloud Computing Performance Amazon EC2 and - Page 3

6 UnixBench Results UnixBench is one of the most popular open source benchmark performance testing tools, which combines multiple tests to assess various aspects of a system s performance in a Unix- like system. performance engineers run these tests multiple times over numerous days. The entire set of index values is then combined to calculate an overall performance index for the system. In this UnixBench benchmark test, exceeds the performance of Amazon s new m3 instances by 2 to 4.4x, and Amazon s m1 instances by 2.2 to 3.1x depending on the instance size. These tests were run over three separate days and the results and test configuration can be found in the UnixBench test results on the next two pages. UnixBench test configuration: Get HZ environment variable to what is configured as "CONFIG_HZ_?" in the kernel configuration: "grep 'CONFIG_HZ_' /boot/config virtual" In case of Ubuntu LTS: 250 HZ Edit Makefile and change the following line: - from "OPTON = -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall" - to "OPTON = -march=native -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall" Compile UnixBench: - -HZ="250" make Run UnixBench: - HZ="250"./Run -c NR_OF_CORES - NR_OF_CORES equals the number of virtual cores available to each VM Comparing Cloud Computing Performance Amazon EC2 and - Page 4

7 UnixBench Results: Cloud Server Instances vs. Amazon AWS EC2 Instances type m UnixBench Average Score UnixBench score AWS EC2 UnixBench score 0 m3.medium m3.large m3.xlarge Instance Size EC2: m3.medium, equivalent (1 CPU Core/3.75GB RAM) EC2: m3.large, equivalent (2 CPU cores/7.5gb RAM) EC2: m3.xlarge, equivalent (4 CPU cores/15gb RAM) UnixBench score AWS EC2 UnixBench score Performance Advantage x x x Comparing Cloud Computing Performance Amazon EC2 and - Page 5

8 UnixBench Results: Cloud Server Instances vs. Amazon AWS EC2 Instances type m UnixBench Average Score UnixBench score AWS EC2 UnixBench score 0 m1.medium m1.large m1.xlarge Instance Size EC2: m1.medium, equivalent (1 CPU Core/3.75GB RAM) EC2: m1.large, equivalent (2 CPU cores/7.5gb RAM) EC2: m1.xlarge, equivalent (4 CPU cores/15gb RAM) UnixBench score AWS EC2 UnixBench score Performance Advantage x x x Comparing Cloud Computing Performance Amazon EC2 and - Page 6

9 DBENCH Results DBENCH is a popular open source performance testing tool that generates I/O workloads for a file system or networked CIFS of NFS server. It is used to stress a file system to see which workload becomes saturated. In this DBENCH benchmark test, exceeds the performance of standard EBS volumes on Amazon s new m3 instances by 10.5 to 16x and Amazon s m1 instances by 9.5 to 12.7x depending on instance size. These tests were run over three separate days and the results and test configuration can be found on the DBENCH test results on the next two pages. DBENCH test configuration: Compile DBENCH: - CFLAGS= - march=native./autogen.sh - CFLAGS= - march=native./configure - make Run DBENCH: -./dbench backend=fileio t 60 D /benchmark/bench loadfile=loadfiles/client.txt 48 Comparing Cloud Computing Performance Amazon EC2 and - Page 7

10 DBENCH Results: Cloud Server Instances vs. Amazon AWS EC2 Instances type m3 700 DBENCH Average Score (MB/s) DBENCH (MB/s) AWS EC2 DBENCH (MB/s) 0 m3.medium m3.large m3.xlarge Instance Size EC2: m3.medium, equivalent (1 CPU Core/3.75GB RAM) EC2: m3.large, equivalent (2 CPU cores/7.5gb RAM) EC2: m3.xlarge, equivalent (4 CPU cores/15gb RAM) DBENCH (MB/s) AWS EC2 DBENCH (MB/s) Performance Advantage x x x Comparing Cloud Computing Performance Amazon EC2 and - Page 8

11 DBENCH Results: Cloud Server Instances vs. Amazon AWS EC2 Instances type m1 700 DBENCH Average Score (MB/s) DBENCH (MB/s) AWS EC2 DBENCH (MB/s) 0 m1.medium m1.large m1.xlarge Instance Size EC2: m1.medium, equivalent (1 CPU Core/3.75GB RAM) EC2: m1.large, equivalent (2 CPU cores/7.5gb RAM) EC2: m1.xlarge, equivalent (4 CPU cores/15gb RAM) DBENCH (MB/s) AWS EC2 DBENCH (MB/s) Performance Advantage x x x Comparing Cloud Computing Performance Amazon EC2 and - Page 9

12 Iperf Results Iperf is a popular open-source network testing tool that creates TCP and UDP data streams and measures the throughput of the network that it is running on. In this Iperf benchmark test, exceeds the performance of Amazon s new m3 instances by 7.3 to 17x and Amazon s m1 instances by 4.9 to 7.2x depending on the instance size. These tests were run over three separate days and the results and test configuration can be found on the Iperf test results on the next two pages. Iperf test configuration: VMs were configured on both providers for both the client and the server instances appeared to run on separate hardware to ensure that a local bridge was not used in these tests. Compile iperf:./configure make Run iperf: Server:./src/iperf s Client:./src/iperf c IP_OF- SERVER f m t 60 P 4 Comparing Cloud Computing Performance Amazon EC2 and - Page 10

13 Iperf Results: Cloud Server Instances vs. Amazon AWS EC2 Instances type m Iperf (Mbit/s) Iperf (Mbit/s) AWS EC2 Iperf (Mbit/s) m3.medium m3.large m3.xlarge Instance Size EC2: m3.medium, equivalent (1 CPU Core/3.75GB RAM) EC2: m3.large, equivalent (2 CPU cores/7.5gb RAM) EC2: m3.xlarge, equivalent (4 CPU cores/15gb RAM) Iperf (Mbit/s) AWS EC2 Iperf (Mbit/s) Performance Advantage x x x Comparing Cloud Computing Performance Amazon EC2 and - Page 11

14 Iperf Results: Cloud Server Instances vs. Amazon AWS EC2 Instances type m Iperf (Mbit/s) Iperf (Mbit/s) AWS EC2 Iperf (Mbit/s) m1.medium m1.large m1.xlarge Instance Size EC2: m1.medium, equivalent (1 CPU Core/3.75GB RAM) EC2: m1.large, equivalent (2 CPU cores/7.5gb RAM) EC2: m1.xlarge, equivalent (4 CPU cores/15gb RAM) Iperf (Mbit/s) AWS EC2 Iperf (Mbit/s) Performance Advantage x x x Comparing Cloud Computing Performance Amazon EC2 and - Page 12

15 Next Steps is here to help you evaluate us as your cloud provider, from benchmark testing to pricing questions our team of knowledgeable Cloud Computing Consultants and Sales Engineers are available by ing or calling offers a 14-day, no obligation trial account that does not require a credit card for activation. Visit today! Finally, for up-to-the-minute information about our products and services, be sure to check out our Resource Center at Inc. 15 Tudor Street Cambridge, MA Phone: Fax: info-us@profitbricks.com twitter.com/profitbricksusa blog.profitbricks.com 2014, Inc. All rights reserved., the logo and Data Center Designer are trademarks of Inc. All other trademarks are the property of their respective owners. reserves the right to make changes without further notice. Comparing Cloud Computing Performance Amazon EC2 and - Page 13