Performance Analysis: Benchmarks of Bare-Metal & Virtual Clouds Benchmarking offerings from Internap, Amazon Web Services and By Cloud Spectator October 213
Table of Contents Abstract and Introduction 2 Executive Summary 4 System Performance 5 CPU Performance 6 RAM Performance 9 Internal Network Performance 1 Disk Performance 12 Database Performance 17 Price-Performance 18 Conclusion 2 Appendices 21 About Cloud Spectator 3 1
Abstract Virtualized infrastructure-as-a-service (IaaS) has drawn increasing attention and usage due to its granular billing, ease-ofuse, and broad network access. Bare-metal cloud services, which are essentially physical servers that can be deployed on demand and billed hourly, can offer significant improvements over virtualized IaaS in performance, consistency and cost efficiency for many applications. Benchmark tests comparing similarly sized virtual and bare-metal cloud configurations reveal that bare-metal cloud yields superior CPU, RAM, storage, and internal network performance. Predictably, solid-state drive () configurations showed substantially higher disk performance measures when compared with spinning serial ATA () drives, although processor, memory and network results between the two configurations were virtually identical. Cost efficiency, measured by performance-weighting the prices, was also measurably higher with the tested bare-metal cloud deployment inferring superior value from bare-metal cloud, all-elseequal. Introduction Bare-metal Cloud Bare-metal cloud offerings refer to a type of high-performance hosted-server offering that combines the advantages of traditional dedicated servers with those of IaaS virtual servers. With bare-metal cloud, the entire physical server s resources are dedicated to a single user, purportedly offering better performance than a comparable virtualized server. Since resources are not being shared, no hypervisor layer is needed, thereby allowing more of the server s processing power to be devoted to the application. Unlike typical dedicated servers, which can take days or weeks to deploy, baremetal cloud offerings can be provisioned in a matter of minutes. Bare-metal cloud servers are also able to be implemented using Application Programming Interface (API) methods or management portals and billed on a usagebased model. Similar to cloud IaaS virtual offerings, bare-metal cloud is billed only when the server is in use. This flexibility allows bare metal to accommodate for peak demand quickly similar to virtual cloud servers. Afterwards, additional resources can be deleted, with the customer only being charged for the hours of additional resources, compared to monthly fees typical with traditional dedicated servers. This study tests the degree of performance improvement and value enhancement that can be obtained using bare-metal cloud relative to two well-known virtualized IaaS offers. Bare metal vs. Virtual The difference between bare-metal cloud servers and virtual servers comes down to two key elements: the hypervisor and multi-tenancy. The hypervisor is used to virtualize the resources of physical machines, creating multiple virtual machines on each physical server for a multi-tenant environment. Bare-metal servers do not have a hypervisor; instead, the entire physical server s resources are reserved for a single tenant. However, bare-metal cloud is similar to virtual cloud in that the physical servers are linked together with an orchestration layer, so that the resources can be provisioned and scaled quickly using an API or other remote management tool. The level of flexibility for both offerings is similar as both operate in a usage-based billing model and offer similar features and services. It is commonly known that adding a hypervisor layer will result in some performance degradation for the virtual servers. Also, multi-tenancy and oversubscription of some services can cause an individual user to fight for shared resources on the server. In contrast, bare-metal cloud resources are fully dedicated to a single user. It follows that bare-metal cloud users routinely report greater processing power with more consistent disk and network I/O performance. 2
The Study Performance Testing Cloud Spectator monitors the CPU, RAM, storage, and internal network performance of over 2 of the world s most wellknown IaaS services to understand important aspects of virtual server performance. Tests are run at least three times per day, 365 days per year to capture variability in addition to performance level. Tests are chosen based on reliability and practicality. The goal is to provide an indication of where certain providers perform well relative to others. This can give consumers an indication of which services would be best for their application(s) by understanding the performance of provider resources most critical to that application. Singular benchmarks alone should not be the only deciding factor in the provider selection process. A feature set, configuration matches, pricing and ancillary services such as security, compliance, and disaster recovery should always factor into any vendor selection process. However, performance is a very important piece to the puzzle. The Comparison For the purpose of generating this document, Cloud Spectator measured the performance of Internap s bare-metal cloud offering against virtual offerings from Amazon and. The goal was to quantify how much of a performance penalty users are taking by choosing virtual cloud servers. Over a period of ten days, Cloud Spectator ran benchmark tests across Internap bare-metal cloud, AWS Elastic Compute Cloud (EC2) with Amazon Elastic Block Storage (EBS), and OpenCloud with Block Storage. Each test was run to understand the unique performance capabilities of each offering s CPU, internal network, RAM and disk. Cloud Spectator accounted for performance capability and stability for each provider to understand the value each one delivers to its users. Tests were run on 8GB servers for Internap and, and 7.5GB servers for Amazon (for more details, see Methodology section in Appendix B). While other factors can sway the perception of provider comparisons, including value-added services, features, support, security, etc., Cloud Spectator chose to focus objectively on performance to derive numerical relationships between the levels of service delivered from each provider. Performance testing and benchmarking of cloud computing platforms is a complex task, compounded by the differences between providers and the use cases of cloud infrastructure users. IaaS services are utilized by a large variety of industries, and performance metrics cannot be completely understood by simply representing cloud performance with a single value. When selecting a cloud computing provider, IT professionals consider many factors: feature-sets, cost, security, location and more. However, performance is a key issue that drives many others, including cost. In many cases, four primary resources affect overall server performance: central processing unit (CPU), random access memory (RAM), disk, and internal network. These four resources are what this comparison focuses on. vs. A growing number of service providers are now offering their customers the option to choose between solid-state drives () or traditional hard drives. While the focus of this study centers on comparing similarly priced virtual instances to Internap bare-metal cloud servers outfitted with a, it also assesses the relative performance of a lower-priced, bare-metal configuration. The results display that while CPU, RAM, and internal network performance are comparable between the bare-metal and instances, the disk tests show significant variation. Certain CPU or RAM-intense scenarios that require less disk IO could benefit from the lower-cost option relative to and/or virtual cloud instances (see Appendix A). 3
Executive Summary Overall Performance Internap bare-metal cloud has both higher performance and stability than Amazon and in nearly 7% of the benchmark tests across the 1-day testing period. System Performance Internap bare-metal cloud has both the highest performance and stability in the general system test. Internap bare-metal cloud scored 7x higher than the second best performing provider. Internap bare-metal cloud and have very stable system performance: Coefficient of Variation (CV) less than 1%. Internap bare-metal cloud outperformed by 7x and Amazon by 1.1x in the system test. CPU Performance Internap bare-metal cloud has both the highest performance and stability across all the CPU tests. Internap bare-metal cloud outperformed Amazon and in 3 out of 3 CPU tests. Internap bare-metal cloud was more stable than Amazon and in 3 out of 3 CPU tests. Internap bare-metal cloud outperformed by an average 2.5x and Amazon by 4.4x in the CPU tests. RAM Performance Internap bare-metal cloud has the highest performance across the RAM test and is also the most stable. Internap bare-metal cloud outperformed both Amazon and in the RAM test. Internap bare-metal cloud has very stable RAM performance: CV of 1% or less. Internap bare-metal cloud outperformed by 1.7x and Amazon by 2.1x in the RAM test. Internal Network Performance Internap bare-metal cloud has the highest performance and stability across the internal network tests, showing high throughput and low latency. Internap bare-metal cloud outperformed Amazon and in 2 out of 2 internal network tests. Internap bare-metal cloud was more stable than Amazon and in 2 out of 2 internal network tests. Internap bare-metal cloud outperformed by an average 2.5x and Amazon by 2.5x in the internal network tests. Disk Performance Internap bare-metal cloud has both the highest performance and stability in 6% of the disk performance tests. Internap bare-metal cloud outperformed Amazon and in 4 out of 5 disk tests. Internap bare-metal cloud was more stable than Amazon and in 4 out of 5 disk tests. Internap bare-metal cloud outperformed by an average 8.4x and Amazon by 4.9x in the disk tests. Database Performance Internap bare-metal cloud has the highest performance across the database test and has the second highest stability. Internap bare-metal cloud outperformed Amazon and in the database test. Internap bare-metal cloud was more stable than Amazon in the database test. Internap bare-metal cloud outperformed by 67% and Amazon by 65% in the database test. 4
System Performance UnixBench Test Description Unixbench runs a set of individual benchmark tests, aggregates the scores, and creates a final score to gauge the performance of Linux systems. According to the Unixbench homepage (https://code.google.com/p/byte-unixbench/): The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value. The entire set of index values is then combined to make an overall index for the system. Some very simple graphics tests are included to measure the 2D and 3D graphics performance of the system. Multi-CPU systems are handled. If your system has multiple CPUs, the default behavior is to run the selected tests twice - once with one copy of each test program running at a time, and once with N copies, where N is the number of CPUs. This is designed to allow you to assess: - the performance of your system when running a single task - the performance of your system when running multiple tasks - the gain from your system's implementation of parallel processing Do be aware that this is a system benchmark, not a CPU, RAM or disk benchmark. The results will depend not only on your hardware, but on your operating system, libraries, and even compiler. System Performance Results The results of the Unixbench benchmark test show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap s bare-metal cloud has an average score of 6415, outperforming s average of 917 by 7x, and Amazon s average of 637 by 1.1x. For performance stability, Internap bare-metal cloud has the least variable performance, with the lowest Coefficient of Variation (CV) of.4%, followed by with.8%, and Amazon with 9.9%. 7 6 5 SYSTEM TEST - Unixbench: Internap bare-metal vs. Amazon vs. Score 4 3 2 1 Amazon Internap Bare Metal Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares system performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the UnixBench benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 6415 24.4% 6474 6358 Amazon 637 63 9.9% 784 554 917 7.8% 931 92 5
CPU Performance File Compression Test Description With p7zip s integrated benchmark feature, we test the performance of the virtual CPU by measuring the millions of instructions per second (MIPS) that it can handle when compressing a file. This benchmark test is a multi-threaded CPU test. File Compression Performance Results The results of the File Compression benchmark show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap bare-metal cloud has an average speed of 17626 MIPS, outperforming s average of 5983 MIPS by 3x and Amazon s average of 2831 MIPS by 6.2x. For performance stability, Internap bare-metal cloud has the lowest Coefficient of Variation (CV) of 1.8%, followed by with 2.9%, and Amazon with 14.3%. CPU TEST - File Compression: Internap bare-metal cloud vs. Amazon vs. 2 18 16 14 MIPS 12 1 8 6 4 2 Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Amazon Internap The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the File Compression benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 17626 MIPS 322 MIPS 1.8% 1822 MIPS 16976 MIPS Amazon 2831 MIPS 45 MIPS 14.3% 355 MIPS 2136 MIPS 5983 MIPS 173 MIPS 2.9% 6254 MIPS 561 MIPS 6
Audio Encoding Test Description From the Phoronix Benchmark Test Suite, this test times how long it takes to encode a WAV file to MP3 format using LAME, an MP3 encoder. This encoding test is single-threaded, indicating the CPU performance of a single core on each provider. To compare the performance difference between single-core and multi-core encoding of each provider, see results for CPU Test: Video Encoding. Audio Encoding Performance Results The results of the Audio Encoding benchmark show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap bare-metal cloud has an average time of 15 seconds, outperforming s average of 34 seconds, and Amazon s average of 37 seconds. For performance stability, Internap bare-metal cloud has the lowest CV of.1%, followed by with 1.6% and Amazon with 9.9%. CPU TEST - Audio Encoding: Internap bare-metal cloud vs. Amazon vs. Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 5 Seconds 1 15 2 25 3 Amazon Internap 35 4 45 The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Audio Encoding benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 15 Seconds Seconds.1% 15 Seconds 15 Seconds Amazon 37 Seconds 4 Seconds 9.9% 4 Seconds 3 Seconds 34 Seconds 1 Seconds 1.6% 36 Seconds 34 Seconds 7
Video Encoding Test Description By using the x264 Video Encoding Test within the Phoronix Benchmark Suite, we can test the performance of the CPU through the compute-intensive task of converting video, and score the provider s virtual CPU performance based on how many frames are converted per second, i.e., Frames Per Second (FPS). To compare the performance difference between single-core and multi-core encoding of each provider, see results for CPU Test: Audio Encoding. Video input format: YUV4MPEG2 Video output format: H.264/MPEG-4 AVC format Video Encoding Performance Results The results of the Video Encoding benchmark show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap bare-metal cloud has an average of 132 FPS, outperforming s average of 44 FPS by 3x and Amazon s average of 24 FPS by 5.5x. For performance stability, Internap bare-metal cloud has the lowest CV of.5%, followed by with 3.8%, and Amazon with 9.%. 16 14 CPU TEST - Video Encoding: Internap bare-metal cloud vs. Amazon vs. Frames Per Second 12 1 8 6 4 Amazon Internap 2 Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Video Encoding benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 132 FPS 1 FPS.5% 134 FPS 131 FPS Amazon 24 FPS 2 FPS 9.% 29 FPS 21 FPS 44 FPS 2 FPS 3.8% 46 FPS 41 FPS 8
RAM Performance RAMSpeed SMP Test Description The RAMSpeed test is an aggregate of several tests that measure COPY, SCALE, ADD and TRIAD functions for both integer and floating point values. More information on COPY, SCALE, ADD, and TRIAD: COPY transfers data from one memory location to another (A = B) SCALE multiplies the data with a constant value before writing it (A = Bn) ADD reads results from two different locations, adds those results and writes then to the new location (A = B + C) TRIAD merges ADD and SCALE. It reads data from the first memory location, scales it (multiplies it), then adds data from the second one and writes to the new location (A = Bn + C) Each test results in a score in MB/s. All scores are averaged to come up with the final score. RAMspeed SMP Performance Results The results of the RAMspeed Test show Internap bare-metal cloud with the highest performance and stability throughout the testing period, followed by, then Amazon. Internap bare-metal cloud has an average of 12544 Mb/s, outperforming s average of 7299 Mb/s by 72% and Amazon s average of 67 Mb/s by 2.1x. Internap baremetal cloud has nearly zero instability with a CV of.1%, followed by with a CV of 2.4% and Amazon proving its instability with a CV of 45.2%. RAM TEST - RAMspeed SMP Test: Internap bare-metal cloud vs. Amazon vs. 18 16 14 MiBytes/sec 12 1 8 6 4 Amazon Internap 2 Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares RAM performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the RAMspeed SMP benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the majority of the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 12544 Mb/sec 16 Mb/sec.1% 12557 Mb/sec 1257 Mb/sec Amazon 67 Mb/sec 2716 Mb/sec 45.2% 15476 Mb/sec 332 Mb/sec 7299 Mb/sec 175 Mb/sec 2.4% 7526 Mb/sec 6894 Mb/sec 9
Internal Network Performance Iperf Test Description Running Iperf tests the network throughput between two virtual machines (VMs) within the same private network inside the data center. The results are important in understanding the possible network bottlenecks. Applications that require databases, which may pull from storage located off of the local server, need a large pipe to transfer data as efficiently as possible to load quickly from the server side. This is especially true for big data applications like Hadoop, which push massive amounts of data. Throughput and bandwidth are often confused; while both measure the size of the pipe, bandwidth is the theoretical capacity, while throughput is the actual bandwidth the user actually receives. This is an important distinction for a cloud environment, where a 1GBit pipe may be split among many users, thereby resulting in lower actual throughput per user. Internal Network Throughput Results The results of the Iperf benchmark show Internap bare-metal cloud with the highest throughput speed and stability. Internap bare-metal cloud has an average network throughput of 913 Mbits/sec, outperforming Amazon s average of 289 Mbits/sec by 3.2x and s average of 288 Mbits/sec by 3.2x. Internap bare-metal cloud has zero instability with a CV of.%. Amazon and are both stable, with CVs of 1.5% and.6% respectively. NETWORK TEST - Iperf Test: Internap bare-metal cloud vs. Amazon vs. Mibits/sec 1 9 8 7 6 5 4 3 2 1 Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Amazon Internap The graph above compares internal network performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Iperf benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. Amazon and s performance are nearly identical, resulting in s trendline to overlap Amazon s in the graph above. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 913 Mbits/sec Mbits/sec.% 913 Mbits/sec 913 Mbits/sec Amazon 289 Mbits/sec 4 Mbits/sec 1.5% 294 Mbits/sec 276 Mbits/sec 288 Mbits/sec 2 Mbits/sec.6% 291 Mbits/sec 283 Mbits/sec 1
Ping Test Description In contrast to throughput testing, ping commands can be used to measure the latency between nodes on a private network. Higher latency severely degrades application performance in a distributed computing environment, and can adversely impact user experience, especially with regard to time-sensitive applications. 1 microseconds is 1 millisecond. Cloud Spectator runs the ping test from the Linux Terminal with the following parameters: ping c 1 i.25 s w 3 W 5 Latency Results The results of the Ping benchmark show the relative stability of Internap Bare Metal, while Amazon and s performance was erratic and unstable for the duration. Internap bare-metal cloud has the lowest average latency of 271 microseconds, performing 8% better than s average latency of 1381 microseconds, and 87% better than Amazon s average latency of 2153 microseconds. Internap bare-metal cloud has the most stable latency with a CV of.8%, followed Amazon with 22.2% and with 36.4%. NETWORK TEST - Ping: Internap bare-metal cloud vs. Amazon vs. Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 5 Microseconds 1 15 2 25 3 Amazon Internap 35 4 45 The graph above compares internal network performance on a Linux server between Internap bare-metal, Amazon and. The graph shows the scores from the Ping benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap baremetal has the least amount of latency compared to Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 271 µs 2 µs.8% 274 µs 265 µs Amazon 2153 µs 478 µs 22.2% 3823 µs 1825 µs 1381 µs 52 µs 36.4% 276 µs 699 µs 11
Disk Performance IOzone Test Description IOzone measures HDD Performance read and write speed. This test produces measurements from a variety of file operations and is used in the general file system analysis of a system. The average between HDD read and write speeds make up the final score. IOzone performs the following operations: read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread, nmap, aio_read, and aio_write. IOzone Performance Results The results of the IOzone benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, outperforming Amazon and by about 6x. Internap bare-metal cloud leads with an average of 4453 Mb/s, outperforming Amazon s average of 73 Mb/s by 6.3x, and s average of 7 Mb/s by 6.4x. Internap bare-metal cloud has the most stable performance for the testing period with a CV of 1.7%, followed by with 6.9% and Amazon with 13.%. DISK TEST - IOzone Test: Internap bare-metal cloud vs. Amazon vs. 5 45 4 MiBytes/sec 35 3 25 2 15 1 5 Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Amazon Internap The graph above compares disk performance on a Linux server between Internap bare-metal, Amazon and. The graph shows the scores from the IOzone benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 4453 Mb/sec 74 Mb/sec 1.7% 4536 Mb/sec 4279 Mb/sec Amazon 73 Mb/sec 91 Mb/sec 13.% 938 Mb/sec 619 Mb/sec 7 Mb/sec 48 Mb/sec 6.9% 726 Mb/sec 531 Mb/sec 12
MongoPerf Read w/out Cache Test Description Cloud Spectator tests disk read operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. By eliminating the cache functionality, data must always be read directly from the disk, eliminating the added benefit of RAM, for a true measure of disk IO. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:false, r:true, w:false, recsize:4, syncdelay:1 MongoPerf Read w/out Cache Performance Results The results of the Read without Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by Amazon then. Internap bare-metal cloud has the highest average of 245 Ops/sec, outperforming Amazon s average of 5399 Ops/sec by 3.7x, and s average of 693 Ops/sec by 28.9x. While having the lowest performance, has the most stable performance throughout the testing period with a CV of 17.6%, followed by Internap bare-metal cloud with a CV of 22.3%, and Amazon proving to be very unstable with a CV of 75.6%. 3 25 DISK TEST - Read w/out Cache Test: Internap bare-metal cloud vs. Amazon vs. Operations/sec 2 15 1 5 Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal, Amazon and. The graph shows the scores from the Read w/out Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 245 Ops/sec 4469 Ops/sec 22.3% 273 Ops/sec 146 Ops/sec Amazon 5399 Ops/sec 484 Ops/sec 75.6% 16663 Ops/sec 57 Ops/sec 693 Ops/sec 122 Ops/sec 17.6% 1136 Ops/sec 563 Ops/sec 13
MongoPerf Write w/out Cache Test Description Cloud Spectator tests disk write operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. By eliminating the cache functionality, data must always be written directly to the disk, eliminating the added benefit of RAM, for a true measure of disk IO. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:false, r:false, w:true, recsize:4, syncdelay:1 MongoPerf Write w/out Cache Performance Results The results of the Write without Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by, and then Amazon. Internap bare-metal cloud has an average of 2292 Ops/sec, outperforming s average of 15 Ops/sec by 53%, and Amazon s average of 1157 Ops/sec by 98%. Internap bare-metal cloud also has the most stable performance throughout the testing period with a CV of.5%, followed by with 7.1%, and Amazon with 9.5%. 25 DISK TEST - Write w/out Cache Test: Internap bare-metal cloud vs. Amazon vs. 2 Operations/sec 15 1 5 Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Write without Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 2292 Ops/sec 11 Ops/sec.5% 23 Ops/sec 2267 Ops/sec Amazon 1157 Ops/sec 19 Ops/sec 9.5% 136 Ops/sec 78 Ops/sec 15 Ops/sec 16 Ops/sec 7.1% 1583 Ops/sec 981 Ops/sec 14
MongoPerf Read w/ Cache Test Description Cloud Spectator tests disk read operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:true, r:true, w:false, recsize:4, syncdelay:6 MongoPerf Read w/ Cache Performance Results The results of the Read with Cache benchmark show Amazon with the highest performance throughout the testing period, followed by Internap Bare Metal, and then. Amazon has an average of 1629198 Ops/sec, outperforming Internap Bare Metal s average of 195523 Ops/sec by 49%, and s average of 274943 Ops/sec by 5.9x. While Amazon is the highest performing, it is also the least stable with a CV of 31.9%. Internap bare-metal cloud is the most stable with a CV of 1.3%, followed by with 8.8%. 25 DISK TEST - Read w/ Cache Test: Internap bare-metal cloud vs. Amazon vs. 2 Operations/sec 15 1 5 Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Read w/ Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Amazon outperforms Internap bare-metal cloud and over the majority of the test period. Amazon is also the most variable over the test period compared to the other providers. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 195523 14418 Ops/sec 1.3% 112652 Ops/sec 17247 Ops/sec Ops/sec Amazon 1629198 52328 Ops/sec 31.9% 237536 Ops/sec 593233 Ops/sec Ops/sec 274943 Ops/sec 24261 Ops/sec 8.8% 333211 Ops/sec 222528 Ops/sec 15
MongoPerf Write w/ Cache Test Description Cloud Spectator tests disk write operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:true, r:false, w:true, recsize:4, syncdelay:6 MongoPerf Write w/ Cache Performance Results The results of the Write with Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by, then Amazon. Internap bare-metal cloud has an average of 25238 Ops/sec, outperforming s average of 163262 Ops/sec by 26%, and Amazon s average of 17239 Ops/sec by 11.9x. Internap bare-metal cloud has the most stable performance throughout the test period with a CV of 1.1%, followed by Amazon with 24.5% and proving to be very unstable with a CV of 6.%. 35 3 DISK TEST - Write w/ Cache Test: Internap bare-metal cloud vs. Amazon vs. Operations/sec 25 2 15 1 5 Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Write w/ Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap baremetal cloud outperforms Amazon and over the majority of the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 25238 Ops/sec 2222 Ops/sec 1.1% 29886 Ops/sec 2152 Ops/sec Amazon 17239 Ops/sec 4231 Ops/sec 24.5% 27519 Ops/sec 11747 Ops/sec 163262 Ops/sec 97884 Ops/sec 6.% 315132 Ops/sec 67373 Ops/sec 16
Database Performance MySQL Test Description The MySQL Performance test is MySQL s native benchmarking tool. This tests time to completion of MySQL s database queries/commands (alter, insert, update, and delete). Lower scores are better. MySQL Performance Results The results of the MySQL Performance benchmark show Internap bare-metal cloud outperforming Amazon and throughout the testing period. Internap bare-metal cloud outperformed the other offerings by approximately 3x with an average of 6 seconds, with Amazon and following with average scores of 17 seconds, 18 seconds respectively. has the most stable performance with a CV of 6.3%, followed by Internap bare-metal cloud with 9.5%, and Amazon with 12.%. DATABASE TEST - MySQL Performance Test: Internap bare-metal cloud vs. Amazon vs. Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Seconds 5 1 15 Amazon Internap 2 25 The graph above compares database performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the MySQL Performance benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 6 Seconds 1 Seconds 9.5% 7 Seconds 4 Seconds Amazon 17 Seconds 2 Seconds 12.% 2 Seconds 12 Seconds 18 Seconds 1 Seconds 6.3% 21 Seconds 16 Seconds 17
Price-Performance Measuring Value Raw performance numbers do not always tell the full story. While some users may have performance as their first priority and be willing to pay extra for a quality solution, others have value in mind. These users try to understand which services give the best resource performance for their dollar. Users focused solely on price may not get the best value. If one virtual server has twice the CPU performance of another with an identical configuration, but is priced only 5% more, then this offering provides better value to the user. Instead of paying for two or more servers to get sufficient performance, the user only pays for one and saves money. Extrapolating this idea to a much bigger distributed environment like a Hadoop cluster enhances the importance of accounting for price-performance value. The Calculation After obtaining the correct performance metrics, the calculation for each test is quite simple: Performance/price = price-performance value To further simplify, we take the highest price-performance metric and divide that into all other price-performance metrics for the given test. Then we multiply by 1. This gives the offering providing the most value for that test a score of 1, and pegs all other offerings in relativity to that top score of 1. We call this value out of 1 the CloudSpecs Score. This represents our final price-performance score. For example, this is how we calculate price-performance for the UnixBench test: Offering Score Divided by Price Divided by Highest Score Multiply by 1 UnixBench CloudSpecs Score Internap bare-metal cloud ($.36/hour) 6,415/$.36 = 17,818 17,818/17,818 = 1. 1.*1 = 1. 1. Amazon ($.25/hour) 637/$.25 = 2,548 2,548/17,818 =.143.143*1 = 14.3 14.3 ($.49/hour) 917/$.49 = 1,871 1,871/17,818 =.15.15*1 = 1.5 1.5 These scores indicate that for the UnixBench test, Internap s bare-metal cloud provides: 7x more value than Amazon 9.5x more value than The Results The following tables provide price-performance results for all tests involving Internap bare-metal cloud servers outfitted with versus nominal virtual instances of Amazon and. System CPU RAM UnixBench Compression Audio Encoding Video Encoding RAMSpeed INTERNAP 1. 1. 1. 1. 1. 14.3 23.1 56.6 26.2 69. 1.5 24.9 31.4 24.6 42.8 18
Storage IOzone Read(w/o Cache) Write(w/o Cache) Read(w/ Cache) Write(w/ Cache) INTERNAP 1. 1. 1. 46.7 1. 22.7 38.8 72.7 1. 12.1 11.6 2.5 48.8 8.6 58.4 Internal Network Database Iperf Ping MySQL INTERNAP 1. 1. 1. 45.6 18.1 51. 23.1 14.4 24.3 Results Summary Internap s bare-metal cloud offers the best price-performance ratio in 12 out of 13 total tests. 1/1 system test 3/3 CPU tests 1/1 RAM test 4/5 storage tests 2/2 internal network tests 1/1 database test 19
Conclusion Study Results In terms of raw performance, UnixBench, the lone system test in this study, shows Internap bare-metal cloud surpassing the next best virtualized option, Cloud, by 7 times. Although bare-metal did not perform the best in every test, it outperformed virtual servers by a significant margin in most. It can therefore be concluded from the data that users will realize much higher performance in a bare-metal cloud offering in each resource. Along with the raw performance numbers, price-performance value was also examined. Although the bare-metal cloud server was priced higher than one of the virtual IaaS providers tested (Amazon), the results show that its relative performance more than compensates for its higher price. Bare metal offers over 6 times more price-performance value than any other offering and provides more value in 12 of 13 total tests. As indicated by the results in Appendix A, lower priced bare metal instances leveraging drives can also yield significantly more value than virtualized IaaS in scenarios where storage access is less intensive. Implications for IaaS Users As the data show, bare-metal cloud could provide users the opportunity to derive more value for certain IaaS workloads. With the option to use a dedicated server as they would a cloud server, users can potentially better manage workloads that typically require direct access to physical hardware, such as databases and calculation-intensive applications. In addition, organizations that have limited their IT deployments to long-term hosted or owned environments due to performance concerns can look at bare-metal cloud as a way to maintain quality while improving agility and asset efficiency. Areas for Additional Research Possible areas for future study include testing performance differentials across specific use cases to better understand where bare metal and virtualized IaaS provide the best price/value fit, expanding the value equation to include additional services such as on-demand load balancing, back-up and storage, database services etc. 2
Appendix A: Traditional Versus & Virtual Instances Bare-metal Performance The bare-metal cloud instances with drives perform similarly to the bare-metal cloud instances with drives in comparison to the virtual instances offered by Amazon and. Because both of the bare-metal instances have the same CPU, RAM and internal network, performance was comparable for the two instances. In the disk benchmarks, the bare-metal instance performed comparably to significantly lower than the bare-metal instances. The stability of performance was comparable between the bare-metal instances for the majority of tests as well. System Performance Results The results of the Unixbench benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and in general system performance. The Internap instance performed higher than the Internap instance by 3%, Amazon by 1.3x and by 7.2x. Internap, Internap and had CVs under 1%, while Amazon reached 9.9%. Average Score SYSTEM TEST - UnixBench: Internap vs Internap vs Amazon vs 8 6 4 2 Coefficent of Variation SYSTEM TEST - UnixBench Performance Stability of vs vs Amazon vs.% 5.% 1.% 15.% File Compression Performance Results The results of the File Compression benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and. The Internap instance performance differed from the Internap instance by less than 1%, but higher than Amazon by 6.2x and by 3x. Internap, Internap and had CVs under 3%, while Amazon reached 14.3%. Average MIPS CPU TEST - File Compression: Internap vs Internap vs Amazon vs 2 15 1 5 Coefficent of Variation CPU TEST - File Compression Performance Stability of vs vs Amazon vs.% 5.% 1.% 15.% 2.% 21
Video Encoding Performance Results The results of the Video Encoding benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and. The Internap instance performance differed from the Internap instance by less than 2%, but higher than Amazon by 5.4x and by 2.9x. Internap, Internap had CVs less than 1%, while was 3.8% and Amazon was 9%. Average FPS 15 1 5 CPU TEST - Video Encoding: Internap vs Internap vs Amazon vs Coefficent of Variation CPU TEST - Video Encoding Performance Stability of vs vs Amazon vs.% 5.% 1.% Audio Encoding Performance Results The results of the Audio Encoding benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and. The Internap instance performance differed from the Internap instance by less than 1%, but higher than Amazon by 6% and by 57%. Internap, Internap and had CVs under 2%, while Amazon reached 9.9%. Average Seconds 1 2 3 4 CPU TEST - Audio Encoding: Internap vs Internap vs Amazon vs Coefficent of Variation CPU TEST - Audio Encoding Performance Stability of vs vs Amazon vs.% 5.% 1.% 15.% 22
RAMSpeed SMP Performance Results The results of the RAMSpeed SMP benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and. The Internap instance performance differed from the Internap instance by less than 1%, but higher than Amazon by 2.1x and by 72%. Internap, Internap and had CVs under 3%, while Amazon reached 45.2%. Average MB/sec RAM TEST - RAMSpeed SMP: Internap vs Internap vs Amazon vs 15 1 5 Coefficent of Variation RAM TEST - RAMSpeed SMP Performance Stability of vs vs Amazon vs.% 2.% 4.% 6.% IOzone Performance Results The results of the IOzone benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and. The Internap instance performance differed from the Internap instance by less than 1%, but higher than Amazon by 6.4x and by 6.4x. Internap, Internap had CVs under 2%, followed by with 6.9% and Amazon with 13%. Average MB/sec 6 4 2 DISK TEST - IOzone: Internap vs Internap vs Amazon vs Coefficent of Variation DISK TEST - IOzone Performance Stability of vs vs Amazon vs.% 5.% 1.% 15.% 23
MongoPerf Write w/ Cache Performance Results The results of the Write with Cache benchmark test show the Internap instance matching Amazon, but performing lower than the Internap instance and. The Internap instance performed lower than the Amazon by 35%, the Internap instance by 95% and by 93%. Internap had the lowest CV with 1.1%, followed by Amazon with 24.5%, with 6% and Internap with 92.5%. Average Ops/sec DISK TEST - Write Ops (w/ Cache): Internap vs Internap vs Amazon vs 3 2 1 Coefficent of Variation DISK TEST - Write Ops (w/ Cache) Performance Stability of vs vs Amazon vs.% 5.% 1.% MongoPerf Read w/ Cache Performance Results The results of the Read with Cache benchmark test show the Internap instance matching the Internap instance, while performing lower than Amazon and higher than. The Internap instance performance differed from the Internap instance by less than 1%, but lower than Amazon by 33% and higher than by 4x. Internap and Internap had CVs under 2%, followed by with 8.8% and Amazon with 31.9%. Average Ops/sec DISK TEST - Read Ops (w/ Cache): Internap vs Internap vs Amazon vs 2 15 1 5 Coefficent of Variation DISK TEST - Read Ops (w/ Cache) Performance Stability of vs vs Amazon vs.% 1.% 2.% 3.% 4.% 24
MongoPerf Write w/out Cache Performance Results The results of the Write without Cache benchmark test show the Internap instance performing lower than the Internap instance, Amazon and. The Internap instance performed lower than the Internap instance by 94%, Amazon by 88% and by 91%. Internap had the lowest CV with.5%, followed by Internap with 4.5%, with 7.1% and Amazon with 9.5%. Average Ops/sec 3 2 1 DISK TEST - Write Ops (w/out Cache): Internap vs Internap vs Amazon vs Coefficent of Variation DISK TEST - Write Ops (w/out Cache) Performance Stability of vs vs Amazon vs.% 5.% 1.% MongoPerf Read w/out Performance Results The results of the Read without Cache benchmark test show the Internap instance performing lower than the Internap instance, Amazon and. The Internap instance performed lower than the Internap instance by 99%, Amazon by 95% and by 6%. Internap had the lowest CV with 5.4%, followed by 17.6%, Internap with 22.3% and Amazon with 75.6%. Average Ops/sec DISK TEST - Read Ops (w/out Cache): Internap vs Internap vs Amazon vs 3 2 1 Coefficent of Variation DISK TEST - Read Ops (w/out Cache) Performance Stability of vs vs Amazon vs.% 2.% 4.% 6.% 8.% 25
Iperf Performance Results The results of Iperf benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and. The Internap instance performance differed from the Internap instance by less than 1%, but higher than Amazon by 3.2x and by 3.2x. Internap, Internap and had CVs less than 1%, while Amazon was slightly higher at 1.5%. Average Mibits/sec 1 NETWORK TEST - Iperf: Internap vs Internap vs Amazon vs 5 Coefficent of Variation NETWORK TEST - Iperf Performance Stability of vs vs Amazon vs.%.5% 1.% 1.5% 2.% Ping Performance Results The results of the Ping benchmark test show the Internap instance matching the Internap instance, while outperforming Amazon and. The Internap instance performance differed from the Internap instance by less than 1%, but higher than Amazon by 87% and by 8%. Internap, Internap had CVs less than 1%, while was 36.4% and Amazon was 22.2%. Average Microseconds 1 2 3 NETWORK TEST - Ping: Internap vs Internap vs Amazon vs Coefficent of Variation NETWORK TEST - Ping Performance Stability of vs vs Amazon vs.% 1.% 2.% 3.% 4.% 26
MySQL Performance Results The results of the MySQL Performance benchmark test show the Internap instance performing lower than the Internap instance, while slightly outperforming Amazon and. The Internap instance performance was lower than the Internap instance by 2.6x, but higher than Amazon by 8% and by 14%. Internap had the lowest CV with 6.2%, closely followed by with 6.3%, and then Internap with 9.5% and Amazon with 12%. Average Seconds 5 1 15 2 DATABASE TEST- MySQL Performance: Internap vs Internap vs Amazon vs Coefficent of Variation DATABASE TEST - MySQL Performance Performance Stability of vs vs Amazon vs.% 5.% 1.% 15.% 27
Appendix B Methodology Server Setup: Two servers are set up on each provider that match the following configurations as closely as possible. The Primary Server runs all tests specific to CPU and storage. The Secondary Server is used in conjunction with the Primary Server to test the internal network. Primary Server 4 vcpu 8 GB RAM 5 GB Storage Secondary Server 1 vcpu 2 GB RAM 5 GB Disk The following server configurations were set up on each provider. The pre-configured VMs that characterize Amazon and offerings make it impossible to match target configurations exactly. Amazon Web Services EC2 http://aws.amazon.com/ec2/instance-types/ Primary: Data Center: Availability Zone East-1a OS: Ubuntu 12.4 vcpus: 2 (with 4 ECU) RAM: 7.5GB Disk: 5GB (EBS) Secondary: Data Center: Availability Zone East-1a OS: Ubuntu 12.4 vcpus: 1 (with 1 ECU) RAM: 1.7GB Disk: 16GB Internap AgileSERVER bare-metal cloud http://www.internap.com/agile/flexiblecloud-hosting-solutions/enterprise-publiccloud-solutions/bare-metal-cloud/ Primary: Data Center: Santa Clara OS: Ubuntu 12.4 vcpus: 4 RAM: 8GB Disk: 12GB (and 2TB ) Secondary: Data Center: Santa Clara OS: Ubuntu 12.4 vcpus: 1 RAM: 2GB Disk: 5GB OpenCloud http://www..com/cloud/server s/pricing/ Primary: Data Center: Dallas Fort Worth (DFW) OS: Ubuntu 12.4 vcpus: 4 RAM: 8GB Disk: 5GB (Block) Secondary: Data center Dallas Fort Worth (DFW) OS: Ubuntu 12.4 vcpus: 2 RAM: 2GB Disk: 8GB Tests Used The benchmarks used to test the providers listed in this report are derived from a number of sources. CPU Internal network RAM Storage Database File Compression Iperf General RAM Test Iozone MySQL Audio Compression Ping Read w/ Cache Video Encoding Write w/cache Read w/out Cache Write w/out Cache Timeframe The test period for the systems ranged from 9/26/213 to 1/5/213. Data Collection Performance data for each test was collected three times per day, every day throughout the above testing periods. Data shown in the graphs and tables only take the highest and lowest point of each day into account. Rather than showing all the data points collected, only showing the highest and lowest points of the day let customers determine the range of best to worst performance they could expect from a virtual server from the providers. Cloud Spectator obtains cloud servers by purchasing the server space directly from the providers as any user would. For certain providers, the client may reimburse Cloud Spectator for the sever space needed for data collection relevant to that 28
Cloud Spectator Performance Report: Internap, and September 213 active project. Cloud Spectator collects and compiles the data into the CloudSpecs database and translates it into a visual display. Terms and Definitions For the purpose of understanding the relational values of the data, all numbers within the tables below each graph are expressed as whole numbers except the percentages, which are expressed to the tenth decimal point. Percentages are expressed in that manner to account for instances when the coefficient of variation (CV), which is expressed as a percentage, falls below 1%, indicating a high degree of performance predictability. Average When describing averages, Cloud Spectator refers to the average numerical value over a period of 1 days from 9/26/13 until 1/5/13. Average provider scores can be found inside the tables underneath each graph within this document. The average is used to summarize the data from the charts in a simplified overview. Standard Deviation The standard deviation is calculated over a period of 1 days from 9/26/13 until 1/5/13. The standard deviation can be found inside the tables underneath each graph within this document. The standard deviation is used to understand the amount of variation from the average benchmark score of a provider; i.e., how predictable a provider s server performance is for that test. The standard deviation can only be used to understand the amount of variation within a certain provider, and cannot be used to compare among providers. Coefficient of Variation (CV) The coefficient of variation is expressed as a percentage. The CV can be found inside the tables underneath each graph within this document. The CV is a measure of precision. It normalizes the standard deviation as a percentage of the average, which can be compared across providers. A lower CV means more stable performance. [(Standard Deviation) / (Average)] * 1 1-Day Highs and Lows From the tested period between 9/26/13 until 1/5/13, Cloud Spectator extracts and presents the highest and lowest achieved scores by each provider in the tables underneath the graphs within this document. 29
Cloud Spectator Performance Report: Internap, and September 213 About Cloud Spectator Cloud Spectator is the premier, international cloud analyst group focused on infrastructure pricing and server performance. Since 211, Cloud Spectator has monitored the cloud Infrastructure industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 3 times a day, 365 days a year with use of open source benchmark tests. Currently, the CloudSpecs system actively tracks 2 of the top IaaS providers around the world. Cloud Spectator 8 Boylston Street, 16 th Floor Boston, MA 2199 Website: www.cloudspectator.com Phone: (USA) 1-617-3-711 3