Performance Analysis: Benchmarks of Bare-Metal & Virtual Clouds

Transcription

1 Performance Analysis: Benchmarks of Bare-Metal & Virtual Clouds Benchmarking offerings from Internap, Amazon Web Services and By Cloud Spectator October 213

2 Table of Contents Abstract and Introduction 2 Executive Summary 4 System Performance 5 CPU Performance 6 RAM Performance 9 Internal Network Performance 1 Disk Performance 12 Database Performance 17 Price-Performance 18 Conclusion 2 Appendices 21 About Cloud Spectator 3 1

3 Abstract Virtualized infrastructure-as-a-service (IaaS) has drawn increasing attention and usage due to its granular billing, ease-ofuse, and broad network access. Bare-metal cloud services, which are essentially physical servers that can be deployed on demand and billed hourly, can offer significant improvements over virtualized IaaS in performance, consistency and cost efficiency for many applications. Benchmark tests comparing similarly sized virtual and bare-metal cloud configurations reveal that bare-metal cloud yields superior CPU, RAM, storage, and internal network performance. Predictably, solid-state drive () configurations showed substantially higher disk performance measures when compared with spinning serial ATA () drives, although processor, memory and network results between the two configurations were virtually identical. Cost efficiency, measured by performance-weighting the prices, was also measurably higher with the tested bare-metal cloud deployment inferring superior value from bare-metal cloud, all-elseequal. Introduction Bare-metal Cloud Bare-metal cloud offerings refer to a type of high-performance hosted-server offering that combines the advantages of traditional dedicated servers with those of IaaS virtual servers. With bare-metal cloud, the entire physical server s resources are dedicated to a single user, purportedly offering better performance than a comparable virtualized server. Since resources are not being shared, no hypervisor layer is needed, thereby allowing more of the server s processing power to be devoted to the application. Unlike typical dedicated servers, which can take days or weeks to deploy, baremetal cloud offerings can be provisioned in a matter of minutes. Bare-metal cloud servers are also able to be implemented using Application Programming Interface (API) methods or management portals and billed on a usagebased model. Similar to cloud IaaS virtual offerings, bare-metal cloud is billed only when the server is in use. This flexibility allows bare metal to accommodate for peak demand quickly similar to virtual cloud servers. Afterwards, additional resources can be deleted, with the customer only being charged for the hours of additional resources, compared to monthly fees typical with traditional dedicated servers. This study tests the degree of performance improvement and value enhancement that can be obtained using bare-metal cloud relative to two well-known virtualized IaaS offers. Bare metal vs. Virtual The difference between bare-metal cloud servers and virtual servers comes down to two key elements: the hypervisor and multi-tenancy. The hypervisor is used to virtualize the resources of physical machines, creating multiple virtual machines on each physical server for a multi-tenant environment. Bare-metal servers do not have a hypervisor; instead, the entire physical server s resources are reserved for a single tenant. However, bare-metal cloud is similar to virtual cloud in that the physical servers are linked together with an orchestration layer, so that the resources can be provisioned and scaled quickly using an API or other remote management tool. The level of flexibility for both offerings is similar as both operate in a usage-based billing model and offer similar features and services. It is commonly known that adding a hypervisor layer will result in some performance degradation for the virtual servers. Also, multi-tenancy and oversubscription of some services can cause an individual user to fight for shared resources on the server. In contrast, bare-metal cloud resources are fully dedicated to a single user. It follows that bare-metal cloud users routinely report greater processing power with more consistent disk and network I/O performance. 2

4 The Study Performance Testing Cloud Spectator monitors the CPU, RAM, storage, and internal network performance of over 2 of the world s most wellknown IaaS services to understand important aspects of virtual server performance. Tests are run at least three times per day, 365 days per year to capture variability in addition to performance level. Tests are chosen based on reliability and practicality. The goal is to provide an indication of where certain providers perform well relative to others. This can give consumers an indication of which services would be best for their application(s) by understanding the performance of provider resources most critical to that application. Singular benchmarks alone should not be the only deciding factor in the provider selection process. A feature set, configuration matches, pricing and ancillary services such as security, compliance, and disaster recovery should always factor into any vendor selection process. However, performance is a very important piece to the puzzle. The Comparison For the purpose of generating this document, Cloud Spectator measured the performance of Internap s bare-metal cloud offering against virtual offerings from Amazon and. The goal was to quantify how much of a performance penalty users are taking by choosing virtual cloud servers. Over a period of ten days, Cloud Spectator ran benchmark tests across Internap bare-metal cloud, AWS Elastic Compute Cloud (EC2) with Amazon Elastic Block Storage (EBS), and OpenCloud with Block Storage. Each test was run to understand the unique performance capabilities of each offering s CPU, internal network, RAM and disk. Cloud Spectator accounted for performance capability and stability for each provider to understand the value each one delivers to its users. Tests were run on 8GB servers for Internap and, and 7.5GB servers for Amazon (for more details, see Methodology section in Appendix B). While other factors can sway the perception of provider comparisons, including value-added services, features, support, security, etc., Cloud Spectator chose to focus objectively on performance to derive numerical relationships between the levels of service delivered from each provider. Performance testing and benchmarking of cloud computing platforms is a complex task, compounded by the differences between providers and the use cases of cloud infrastructure users. IaaS services are utilized by a large variety of industries, and performance metrics cannot be completely understood by simply representing cloud performance with a single value. When selecting a cloud computing provider, IT professionals consider many factors: feature-sets, cost, security, location and more. However, performance is a key issue that drives many others, including cost. In many cases, four primary resources affect overall server performance: central processing unit (CPU), random access memory (RAM), disk, and internal network. These four resources are what this comparison focuses on. vs. A growing number of service providers are now offering their customers the option to choose between solid-state drives () or traditional hard drives. While the focus of this study centers on comparing similarly priced virtual instances to Internap bare-metal cloud servers outfitted with a, it also assesses the relative performance of a lower-priced, bare-metal configuration. The results display that while CPU, RAM, and internal network performance are comparable between the bare-metal and instances, the disk tests show significant variation. Certain CPU or RAM-intense scenarios that require less disk IO could benefit from the lower-cost option relative to and/or virtual cloud instances (see Appendix A). 3

5 Executive Summary Overall Performance Internap bare-metal cloud has both higher performance and stability than Amazon and in nearly 7% of the benchmark tests across the 1-day testing period. System Performance Internap bare-metal cloud has both the highest performance and stability in the general system test. Internap bare-metal cloud scored 7x higher than the second best performing provider. Internap bare-metal cloud and have very stable system performance: Coefficient of Variation (CV) less than 1%. Internap bare-metal cloud outperformed by 7x and Amazon by 1.1x in the system test. CPU Performance Internap bare-metal cloud has both the highest performance and stability across all the CPU tests. Internap bare-metal cloud outperformed Amazon and in 3 out of 3 CPU tests. Internap bare-metal cloud was more stable than Amazon and in 3 out of 3 CPU tests. Internap bare-metal cloud outperformed by an average 2.5x and Amazon by 4.4x in the CPU tests. RAM Performance Internap bare-metal cloud has the highest performance across the RAM test and is also the most stable. Internap bare-metal cloud outperformed both Amazon and in the RAM test. Internap bare-metal cloud has very stable RAM performance: CV of 1% or less. Internap bare-metal cloud outperformed by 1.7x and Amazon by 2.1x in the RAM test. Internal Network Performance Internap bare-metal cloud has the highest performance and stability across the internal network tests, showing high throughput and low latency. Internap bare-metal cloud outperformed Amazon and in 2 out of 2 internal network tests. Internap bare-metal cloud was more stable than Amazon and in 2 out of 2 internal network tests. Internap bare-metal cloud outperformed by an average 2.5x and Amazon by 2.5x in the internal network tests. Disk Performance Internap bare-metal cloud has both the highest performance and stability in 6% of the disk performance tests. Internap bare-metal cloud outperformed Amazon and in 4 out of 5 disk tests. Internap bare-metal cloud was more stable than Amazon and in 4 out of 5 disk tests. Internap bare-metal cloud outperformed by an average 8.4x and Amazon by 4.9x in the disk tests. Database Performance Internap bare-metal cloud has the highest performance across the database test and has the second highest stability. Internap bare-metal cloud outperformed Amazon and in the database test. Internap bare-metal cloud was more stable than Amazon in the database test. Internap bare-metal cloud outperformed by 67% and Amazon by 65% in the database test. 4

6 System Performance UnixBench Test Description Unixbench runs a set of individual benchmark tests, aggregates the scores, and creates a final score to gauge the performance of Linux systems. According to the Unixbench homepage ( The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value. The entire set of index values is then combined to make an overall index for the system. Some very simple graphics tests are included to measure the 2D and 3D graphics performance of the system. Multi-CPU systems are handled. If your system has multiple CPUs, the default behavior is to run the selected tests twice - once with one copy of each test program running at a time, and once with N copies, where N is the number of CPUs. This is designed to allow you to assess: - the performance of your system when running a single task - the performance of your system when running multiple tasks - the gain from your system's implementation of parallel processing Do be aware that this is a system benchmark, not a CPU, RAM or disk benchmark. The results will depend not only on your hardware, but on your operating system, libraries, and even compiler. System Performance Results The results of the Unixbench benchmark test show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap s bare-metal cloud has an average score of 6415, outperforming s average of 917 by 7x, and Amazon s average of 637 by 1.1x. For performance stability, Internap bare-metal cloud has the least variable performance, with the lowest Coefficient of Variation (CV) of.4%, followed by with.8%, and Amazon with 9.9% SYSTEM TEST - Unixbench: Internap bare-metal vs. Amazon vs. Score Amazon Internap Bare Metal Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares system performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the UnixBench benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap % Amazon % %

7 CPU Performance File Compression Test Description With p7zip s integrated benchmark feature, we test the performance of the virtual CPU by measuring the millions of instructions per second (MIPS) that it can handle when compressing a file. This benchmark test is a multi-threaded CPU test. File Compression Performance Results The results of the File Compression benchmark show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap bare-metal cloud has an average speed of MIPS, outperforming s average of 5983 MIPS by 3x and Amazon s average of 2831 MIPS by 6.2x. For performance stability, Internap bare-metal cloud has the lowest Coefficient of Variation (CV) of 1.8%, followed by with 2.9%, and Amazon with 14.3%. CPU TEST - File Compression: Internap bare-metal cloud vs. Amazon vs MIPS Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Amazon Internap The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the File Compression benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap MIPS 322 MIPS 1.8% 1822 MIPS MIPS Amazon 2831 MIPS 45 MIPS 14.3% 355 MIPS 2136 MIPS 5983 MIPS 173 MIPS 2.9% 6254 MIPS 561 MIPS 6

8 Audio Encoding Test Description From the Phoronix Benchmark Test Suite, this test times how long it takes to encode a WAV file to MP3 format using LAME, an MP3 encoder. This encoding test is single-threaded, indicating the CPU performance of a single core on each provider. To compare the performance difference between single-core and multi-core encoding of each provider, see results for CPU Test: Video Encoding. Audio Encoding Performance Results The results of the Audio Encoding benchmark show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap bare-metal cloud has an average time of 15 seconds, outperforming s average of 34 seconds, and Amazon s average of 37 seconds. For performance stability, Internap bare-metal cloud has the lowest CV of.1%, followed by with 1.6% and Amazon with 9.9%. CPU TEST - Audio Encoding: Internap bare-metal cloud vs. Amazon vs. Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 5 Seconds Amazon Internap The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Audio Encoding benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 15 Seconds Seconds.1% 15 Seconds 15 Seconds Amazon 37 Seconds 4 Seconds 9.9% 4 Seconds 3 Seconds 34 Seconds 1 Seconds 1.6% 36 Seconds 34 Seconds 7

9 Video Encoding Test Description By using the x264 Video Encoding Test within the Phoronix Benchmark Suite, we can test the performance of the CPU through the compute-intensive task of converting video, and score the provider s virtual CPU performance based on how many frames are converted per second, i.e., Frames Per Second (FPS). To compare the performance difference between single-core and multi-core encoding of each provider, see results for CPU Test: Audio Encoding. Video input format: YUV4MPEG2 Video output format: H.264/MPEG-4 AVC format Video Encoding Performance Results The results of the Video Encoding benchmark show Internap bare-metal cloud outperforming Amazon and in speed and stability. Internap bare-metal cloud has an average of 132 FPS, outperforming s average of 44 FPS by 3x and Amazon s average of 24 FPS by 5.5x. For performance stability, Internap bare-metal cloud has the lowest CV of.5%, followed by with 3.8%, and Amazon with 9.% CPU TEST - Video Encoding: Internap bare-metal cloud vs. Amazon vs. Frames Per Second Amazon Internap 2 Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Video Encoding benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 132 FPS 1 FPS.5% 134 FPS 131 FPS Amazon 24 FPS 2 FPS 9.% 29 FPS 21 FPS 44 FPS 2 FPS 3.8% 46 FPS 41 FPS 8

10 RAM Performance RAMSpeed SMP Test Description The RAMSpeed test is an aggregate of several tests that measure COPY, SCALE, ADD and TRIAD functions for both integer and floating point values. More information on COPY, SCALE, ADD, and TRIAD: COPY transfers data from one memory location to another (A = B) SCALE multiplies the data with a constant value before writing it (A = Bn) ADD reads results from two different locations, adds those results and writes then to the new location (A = B + C) TRIAD merges ADD and SCALE. It reads data from the first memory location, scales it (multiplies it), then adds data from the second one and writes to the new location (A = Bn + C) Each test results in a score in MB/s. All scores are averaged to come up with the final score. RAMspeed SMP Performance Results The results of the RAMspeed Test show Internap bare-metal cloud with the highest performance and stability throughout the testing period, followed by, then Amazon. Internap bare-metal cloud has an average of Mb/s, outperforming s average of 7299 Mb/s by 72% and Amazon s average of 67 Mb/s by 2.1x. Internap baremetal cloud has nearly zero instability with a CV of.1%, followed by with a CV of 2.4% and Amazon proving its instability with a CV of 45.2%. RAM TEST - RAMspeed SMP Test: Internap bare-metal cloud vs. Amazon vs MiBytes/sec Amazon Internap 2 Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares RAM performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the RAMspeed SMP benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the majority of the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap Mb/sec 16 Mb/sec.1% Mb/sec 1257 Mb/sec Amazon 67 Mb/sec 2716 Mb/sec 45.2% Mb/sec 332 Mb/sec 7299 Mb/sec 175 Mb/sec 2.4% 7526 Mb/sec 6894 Mb/sec 9

11 Internal Network Performance Iperf Test Description Running Iperf tests the network throughput between two virtual machines (VMs) within the same private network inside the data center. The results are important in understanding the possible network bottlenecks. Applications that require databases, which may pull from storage located off of the local server, need a large pipe to transfer data as efficiently as possible to load quickly from the server side. This is especially true for big data applications like Hadoop, which push massive amounts of data. Throughput and bandwidth are often confused; while both measure the size of the pipe, bandwidth is the theoretical capacity, while throughput is the actual bandwidth the user actually receives. This is an important distinction for a cloud environment, where a 1GBit pipe may be split among many users, thereby resulting in lower actual throughput per user. Internal Network Throughput Results The results of the Iperf benchmark show Internap bare-metal cloud with the highest throughput speed and stability. Internap bare-metal cloud has an average network throughput of 913 Mbits/sec, outperforming Amazon s average of 289 Mbits/sec by 3.2x and s average of 288 Mbits/sec by 3.2x. Internap bare-metal cloud has zero instability with a CV of.%. Amazon and are both stable, with CVs of 1.5% and.6% respectively. NETWORK TEST - Iperf Test: Internap bare-metal cloud vs. Amazon vs. Mibits/sec Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Amazon Internap The graph above compares internal network performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Iperf benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. Amazon and s performance are nearly identical, resulting in s trendline to overlap Amazon s in the graph above. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 913 Mbits/sec Mbits/sec.% 913 Mbits/sec 913 Mbits/sec Amazon 289 Mbits/sec 4 Mbits/sec 1.5% 294 Mbits/sec 276 Mbits/sec 288 Mbits/sec 2 Mbits/sec.6% 291 Mbits/sec 283 Mbits/sec 1

12 Ping Test Description In contrast to throughput testing, ping commands can be used to measure the latency between nodes on a private network. Higher latency severely degrades application performance in a distributed computing environment, and can adversely impact user experience, especially with regard to time-sensitive applications. 1 microseconds is 1 millisecond. Cloud Spectator runs the ping test from the Linux Terminal with the following parameters: ping c 1 i.25 s w 3 W 5 Latency Results The results of the Ping benchmark show the relative stability of Internap Bare Metal, while Amazon and s performance was erratic and unstable for the duration. Internap bare-metal cloud has the lowest average latency of 271 microseconds, performing 8% better than s average latency of 1381 microseconds, and 87% better than Amazon s average latency of 2153 microseconds. Internap bare-metal cloud has the most stable latency with a CV of.8%, followed Amazon with 22.2% and with 36.4%. NETWORK TEST - Ping: Internap bare-metal cloud vs. Amazon vs. Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 5 Microseconds Amazon Internap The graph above compares internal network performance on a Linux server between Internap bare-metal, Amazon and. The graph shows the scores from the Ping benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap baremetal has the least amount of latency compared to Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 271 µs 2 µs.8% 274 µs 265 µs Amazon 2153 µs 478 µs 22.2% 3823 µs 1825 µs 1381 µs 52 µs 36.4% 276 µs 699 µs 11

13 Disk Performance IOzone Test Description IOzone measures HDD Performance read and write speed. This test produces measurements from a variety of file operations and is used in the general file system analysis of a system. The average between HDD read and write speeds make up the final score. IOzone performs the following operations: read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread, nmap, aio_read, and aio_write. IOzone Performance Results The results of the IOzone benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, outperforming Amazon and by about 6x. Internap bare-metal cloud leads with an average of 4453 Mb/s, outperforming Amazon s average of 73 Mb/s by 6.3x, and s average of 7 Mb/s by 6.4x. Internap bare-metal cloud has the most stable performance for the testing period with a CV of 1.7%, followed by with 6.9% and Amazon with 13.%. DISK TEST - IOzone Test: Internap bare-metal cloud vs. Amazon vs MiBytes/sec Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Amazon Internap The graph above compares disk performance on a Linux server between Internap bare-metal, Amazon and. The graph shows the scores from the IOzone benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 4453 Mb/sec 74 Mb/sec 1.7% 4536 Mb/sec 4279 Mb/sec Amazon 73 Mb/sec 91 Mb/sec 13.% 938 Mb/sec 619 Mb/sec 7 Mb/sec 48 Mb/sec 6.9% 726 Mb/sec 531 Mb/sec 12

14 MongoPerf Read w/out Cache Test Description Cloud Spectator tests disk read operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. By eliminating the cache functionality, data must always be read directly from the disk, eliminating the added benefit of RAM, for a true measure of disk IO. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:false, r:true, w:false, recsize:4, syncdelay:1 MongoPerf Read w/out Cache Performance Results The results of the Read without Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by Amazon then. Internap bare-metal cloud has the highest average of 245 Ops/sec, outperforming Amazon s average of 5399 Ops/sec by 3.7x, and s average of 693 Ops/sec by 28.9x. While having the lowest performance, has the most stable performance throughout the testing period with a CV of 17.6%, followed by Internap bare-metal cloud with a CV of 22.3%, and Amazon proving to be very unstable with a CV of 75.6% DISK TEST - Read w/out Cache Test: Internap bare-metal cloud vs. Amazon vs. Operations/sec Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal, Amazon and. The graph shows the scores from the Read w/out Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 245 Ops/sec 4469 Ops/sec 22.3% 273 Ops/sec 146 Ops/sec Amazon 5399 Ops/sec 484 Ops/sec 75.6% Ops/sec 57 Ops/sec 693 Ops/sec 122 Ops/sec 17.6% 1136 Ops/sec 563 Ops/sec 13

15 MongoPerf Write w/out Cache Test Description Cloud Spectator tests disk write operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. By eliminating the cache functionality, data must always be written directly to the disk, eliminating the added benefit of RAM, for a true measure of disk IO. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:false, r:false, w:true, recsize:4, syncdelay:1 MongoPerf Write w/out Cache Performance Results The results of the Write without Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by, and then Amazon. Internap bare-metal cloud has an average of 2292 Ops/sec, outperforming s average of 15 Ops/sec by 53%, and Amazon s average of 1157 Ops/sec by 98%. Internap bare-metal cloud also has the most stable performance throughout the testing period with a CV of.5%, followed by with 7.1%, and Amazon with 9.5%. 25 DISK TEST - Write w/out Cache Test: Internap bare-metal cloud vs. Amazon vs. 2 Operations/sec Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Write without Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 2292 Ops/sec 11 Ops/sec.5% 23 Ops/sec 2267 Ops/sec Amazon 1157 Ops/sec 19 Ops/sec 9.5% 136 Ops/sec 78 Ops/sec 15 Ops/sec 16 Ops/sec 7.1% 1583 Ops/sec 981 Ops/sec 14

16 MongoPerf Read w/ Cache Test Description Cloud Spectator tests disk read operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:true, r:true, w:false, recsize:4, syncdelay:6 MongoPerf Read w/ Cache Performance Results The results of the Read with Cache benchmark show Amazon with the highest performance throughout the testing period, followed by Internap Bare Metal, and then. Amazon has an average of Ops/sec, outperforming Internap Bare Metal s average of Ops/sec by 49%, and s average of Ops/sec by 5.9x. While Amazon is the highest performing, it is also the least stable with a CV of 31.9%. Internap bare-metal cloud is the most stable with a CV of 1.3%, followed by with 8.8%. 25 DISK TEST - Read w/ Cache Test: Internap bare-metal cloud vs. Amazon vs. 2 Operations/sec Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Read w/ Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Amazon outperforms Internap bare-metal cloud and over the majority of the test period. Amazon is also the most variable over the test period compared to the other providers. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap Ops/sec 1.3% Ops/sec Ops/sec Ops/sec Amazon Ops/sec 31.9% Ops/sec Ops/sec Ops/sec Ops/sec Ops/sec 8.8% Ops/sec Ops/sec 15

17 MongoPerf Write w/ Cache Test Description Cloud Spectator tests disk write operations per second with Mongoperf, MongoDB s native benchmarking tool for gauging disk IO performance. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nthreads:32, filesizemb:2, mmf:true, r:false, w:true, recsize:4, syncdelay:6 MongoPerf Write w/ Cache Performance Results The results of the Write with Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by, then Amazon. Internap bare-metal cloud has an average of Ops/sec, outperforming s average of Ops/sec by 26%, and Amazon s average of Ops/sec by 11.9x. Internap bare-metal cloud has the most stable performance throughout the test period with a CV of 1.1%, followed by Amazon with 24.5% and proving to be very unstable with a CV of 6.% DISK TEST - Write w/ Cache Test: Internap bare-metal cloud vs. Amazon vs. Operations/sec Amazon Internap Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the Write w/ Cache benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap baremetal cloud outperforms Amazon and over the majority of the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap Ops/sec 2222 Ops/sec 1.1% Ops/sec 2152 Ops/sec Amazon Ops/sec 4231 Ops/sec 24.5% Ops/sec Ops/sec Ops/sec Ops/sec 6.% Ops/sec Ops/sec 16

18 Database Performance MySQL Test Description The MySQL Performance test is MySQL s native benchmarking tool. This tests time to completion of MySQL s database queries/commands (alter, insert, update, and delete). Lower scores are better. MySQL Performance Results The results of the MySQL Performance benchmark show Internap bare-metal cloud outperforming Amazon and throughout the testing period. Internap bare-metal cloud outperformed the other offerings by approximately 3x with an average of 6 seconds, with Amazon and following with average scores of 17 seconds, 18 seconds respectively. has the most stable performance with a CV of 6.3%, followed by Internap bare-metal cloud with 9.5%, and Amazon with 12.%. DATABASE TEST - MySQL Performance Test: Internap bare-metal cloud vs. Amazon vs. Sep 26 Sep 27 Sep 28 Sep 29 Sep 3 Oct 1 Oct 2 Oct 3 Oct 4 Oct 5 Seconds Amazon Internap 2 25 The graph above compares database performance on a Linux server between Internap bare-metal cloud, Amazon and. The graph shows the scores from the MySQL Performance benchmark test for each provider over a period of 1 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and over the test period. PROVIDER AVERAGE STANDARD DEVIATION CV 1-Day HIGH 1-Day LOW Internap 6 Seconds 1 Seconds 9.5% 7 Seconds 4 Seconds Amazon 17 Seconds 2 Seconds 12.% 2 Seconds 12 Seconds 18 Seconds 1 Seconds 6.3% 21 Seconds 16 Seconds 17

19 Price-Performance Measuring Value Raw performance numbers do not always tell the full story. While some users may have performance as their first priority and be willing to pay extra for a quality solution, others have value in mind. These users try to understand which services give the best resource performance for their dollar. Users focused solely on price may not get the best value. If one virtual server has twice the CPU performance of another with an identical configuration, but is priced only 5% more, then this offering provides better value to the user. Instead of paying for two or more servers to get sufficient performance, the user only pays for one and saves money. Extrapolating this idea to a much bigger distributed environment like a Hadoop cluster enhances the importance of accounting for price-performance value. The Calculation After obtaining the correct performance metrics, the calculation for each test is quite simple: Performance/price = price-performance value To further simplify, we take the highest price-performance metric and divide that into all other price-performance metrics for the given test. Then we multiply by 1. This gives the offering providing the most value for that test a score of 1, and pegs all other offerings in relativity to that top score of 1. We call this value out of 1 the CloudSpecs Score. This represents our final price-performance score. For example, this is how we calculate price-performance for the UnixBench test: Offering Score Divided by Price Divided by Highest Score Multiply by 1 UnixBench CloudSpecs Score Internap bare-metal cloud ($.36/hour) 6,415/$.36 = 17,818 17,818/17,818 = 1. 1.*1 = Amazon ($.25/hour) 637/$.25 = 2,548 2,548/17,818 = *1 = ($.49/hour) 917/$.49 = 1,871 1,871/17,818 =.15.15*1 = These scores indicate that for the UnixBench test, Internap s bare-metal cloud provides: 7x more value than Amazon 9.5x more value than The Results The following tables provide price-performance results for all tests involving Internap bare-metal cloud servers outfitted with versus nominal virtual instances of Amazon and. System CPU RAM UnixBench Compression Audio Encoding Video Encoding RAMSpeed INTERNAP

20 Storage IOzone Read(w/o Cache) Write(w/o Cache) Read(w/ Cache) Write(w/ Cache) INTERNAP Internal Network Database Iperf Ping MySQL INTERNAP Results Summary Internap s bare-metal cloud offers the best price-performance ratio in 12 out of 13 total tests. 1/1 system test 3/3 CPU tests 1/1 RAM test 4/5 storage tests 2/2 internal network tests 1/1 database test 19