Cloud IaaS Performance & Price-Performance

Transcription

1 Cloud IaaS Performance & Price-Performance Comparing Linux Compute Performance of 1&1, Amazon AWS, Aruba Cloud, CloudSigma, and Microsoft Azure Prepared for 1&1 on Behalf of SolidFire Commercial Report Published on 6/215

2 Contents 1. Introduction 2 Why Performance Matters 2 2. Executive Summary 4 Findings 4 3. Methodology 6 Process 6 Tests Used 7 VM Configurations and Pricing 8 Understanding Performance Results 1 Understanding The CloudSpecs Score (Price-Performance) 1 Key Considerations Detailed Performance Findings 12 Processor & Memory Bandwidth 12 Disk IOPS: Sequential and Random Operations 14 Internal Network Detailed Price-Performance Findings 19 Processor & Memory Bandwidth 19 Disk IOPS: Sequential and Random Operations 21 Internal Network Conclusion About 26 1 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

3 Introduction In an effort to simulate an end-user experience regarding performance of virtual machines across various cloud providers, Cloud Spectator ran its iterative benchmark suite for 72 hours on each of the following providers: 1&1, AWS, Aruba Cloud, Microsoft Azure, and CloudSigma. SolidFire sponsored this study on behalf of its client, 1&1. In most cases, much of the work was straightforward regarding provisioning and the setup process. Occasionally, vendor-side issues occurred during the provisioning process, and the Cloud Spectator team contacted the corresponding vendor s support team in order to resolve issues such as VM provisioning errors. Three primary VMs of each size were tested on all providers for 24 hours each (72 hours total). This was done sequentially; once one VM had run the test suite for 24 hours, that VM was terminated and a new VM was created. This study not only examined the performance of each vendor, but also tracked performance variability for each of the three 24-hour periods. The methodology allowed Cloud Spectator to capture performance variability both over time on the same VM as well as across different VMs on multiple physical hosts. Some providers, such as 1&1, show strong processor and memory bandwidth performance stability for all of its VMs throughout the course of the study. Other providers, such as AWS, exhibited controlled periods of burst followed by throttled performance on network storage depending on the size of the storage volume. Others, such as CloudSigma, exhibited unstable performance across all resources throughout the study, possibly due to server-side issues at the time of the study, which contributed to provisioning problems as well. Taking performance and stability a step further, price-performance analyses are conducted to help readers understand the value ratio between the cost of the VM and the performance output. While the performance output is limited to the data points collected in the study, by comparing the price-performance ratio, readers can gain better insight into the overall user experience seen on these providers. Why Performance Matters Performance and pricing are both key considerations in the public cloud industry, together having a substantial impact on a company s annual operating costs. Cloud users may need fewer resources on better performing services, which can lower costs. Since many users only consider price and not price-performance, these users may be paying more because they require additional resources to achieve a desired level of performance. While some providers try to differentiate their offerings by cutting prices, others try to differentiate by focusing on improved performance and user experience. Differences in performance outputs of VMs across IaaS providers can greatly impact quality of service as well as annual operating costs. The graph on the right illustrates an example of the average processor performance from a sample of six Cloud Service Providers (CSPs) as studied by Cloud Spectator. CSP 1 has a processor performance three times as high as CSP 6 (names removed), which gives CSP 1 a notable advantage in many processor-intensive workloads. CSPs 2-5 exhibit a closer resemblance in processor performance, but do not offer nearly as much processing power as CSP 1 does. 2 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

4 The table below lists the 3 hardware components studied in this project, and each purpose as a function in the server. CPU & MEMORY PERFORMANCE STORAGE PERFORMANCE NETWORK PERFORMANCE The performance of all applications is highly dependent on the CPU. The CPU is responsible for the processing and orchestration of all applications. The relationship between CPU performance and RAM is also observed by examining RAM bandwidth. While memory performance is not considered one of the key bottlenecks in performance for many applications, a subset of applications particularly HPC and inmemory databases is highly dependent on large sustained memory bandwidth. Because most applications and all data reside on the disk, having fast disk performance is a key consideration for best application performance in many cases. In a cloud environment, network performance is a critical piece. Scalability, in many cases, is dependent on the availability of additional VMs that must maintain a strong network backbone. 3 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

5 Executive Summary On behalf of 1&1, SolidFire commissioned Cloud Spectator to gauge the performance of VMs on five different cloud providers European data centers: 1&1, Amazon AWS, Aruba Cloud, CloudSigma, and Microsoft Azure. Both performance and price-performance were examined to evaluate the value of each provider s VMs. The purpose of the study was to understand, from an end-user perspective, the disparity of performance and value (defined as price-performance) among cloud providers with similarly sized VMs. Overall, 1&1 exceled in performance and price-performance for all component resources of the VMs tested. Its high performance rankings, combined with hourly pricing, introduces powerful, scalable cloud infrastructure at low cost to its users. Findings vcpu & Memory Performance Findings For this study, Cloud Spectator evaluated vcpu and memory bandwidth performance by benchmarking the VMs using Geekench 3, a suite of benchmark tests that simulate tasks such as cryptographic encoding and image processing. Testing occurred over the course of a 72-hour testing period. Pricing was examined in conjunction with the performance tests. vcpu & Memory Performance Key Findings: 1&1 s VMs achieved the highest performance across processor and memory performance in the study. 1&1 s VMs achieved the highest CloudSpecs Score TM in the test group, indicating the strongest price-performance value for processor and memory bandwidth. CloudSigma s processor performance varied the most. Its coefficients of variation (CV), which is a percentage expressing the relationship between the average and standard deviation (useful for determining variability in performance), ranges up to 43.7%. The virtual processors on 1&1, AWS, and Azure exhibited the most stability throughout the study, resulting in less than 3% coefficients of variation. 4 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

6 Storage Performance Findings Storage was evaluated using the FIO disk benchmark, which tests the sequential read/write and random read/write operations of storage. In this study an 8KB block size was used. Testing occurred over a 72-hour test period. Persistent storage (offered as block storage or redundant storage ) was used in all storage tests. Pricing was examined in conjunction with the performance tests. Storage Performance Key Findings: 1&1 VMs offered the highest disk performance among all included in this study. Although AWS offers SSD technology on its block storage offering, EBS, the performance of that offering is more dependent on the size of the block storage volume provisioned. Despite being one of the lower-tier performers in disk IOPS, Azure displayed the most stable disk performance throughout the study. Out of all providers examined in the study, only AWS appeared to provide a period of burst performance for its block storage. On the 2vCPU VMs, which have GB of block storage, AWS volumes displayed a burst behavior. After the period of burst, sequential read/write operations and random read/write operations dropped to become 1% and 2%, respectively, of the initial IOPS achieved during burst. This burst behavior was not seen on VMs with 4GB and of block storage, due to the larger number of IOPS. 1&1 displayed the best price-performance value for disk IOPS. Its high-performance SAN disk offering is designed with SolidFire technology. Internal Network Performance Findings Internal network performance was measured as the throughput between VMs within the same data center of the cloud provider (measured using iperf and ping respectively) over the course of a 72- hour test period. Pricing was examined in conjunction with the performance tests. Internal Network Performance Key Findings: CloudSigma VMs achieved the highest internal network throughput, although the high throughput is unstable and fluctuates between less than 5Mbits to over 1Gbits. CloudSigma displayed the best price-performance value for internal network. Despite its large fluctuation in internal network throughput, its median score range significantly outperformed all other providers. 1&1 achieved the second-highest internal network CloudSpecs Score TM. Other than CloudSigma, only certain Azure VMs exceeded 1 GB/s throughput. 5 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

7 Methodology Cloud Spectator strives to create a transparent and detailed methodology to allow readers to understand the testing process and recreate any studies. If any information is unclear or if you have any questions, please the team at contact@cloudspectator.com or call +1 (617) Process 1. Three iterations of 24-hour test cycles were run for each VM on each provider for a total of 72 hours of testing per VM size. After each 24- hour block, VMs were terminated before beginning another cycle of tests on newly provisioned machines. 2. Each VM was provisioned with a Linux Ubuntu 14.4 OS by default, available from all providers. For AWS, the HVM image was used. 3. Before each 24-hour test period, and after provisioning the VMs, system updates and upgrades were conducted via apt-get. 4. The following dependencies were installed for testing: a. Git. Git was used to clone the test repository on the VM. b. MySQL. For automation purposes, mysql-server was installed to automate data uploads. c. Pip. Used to download the appropriate libraries for Python in order to run the testing. SQL Alchemy was downloaded to interact with MySQL and upload data. d. Libmysqlclient-dev. MySQL database s development files, which are necessary for the SQL Alchemy and MySQL interaction. 5. Each test cycled through in the following sequence: Geekbench 3 (process & memory), fio sequential operations, fio random operations, Iperf internal network throughput (for more information on testing, see Tests Used). a. For fio testing (to measure disk IOPS), sequential operations ran first. Files from the sequential tests were deleted, and fio recreated files before running random operations. Once random operations completed, the files were also deleted. Thus, before each disk IOPS test, the files associated with the tests were deleted and recreated. 6. Internal network testing was conducted in one of the following manners: a. On AWS and Azure, where VMs demonstrated varying internal network throughput depending on size and/or instance type, a clone of that VM was created in the same region/availability zone. The cloned server listened for a TCP connection via Iperf. E.g., two c4.large instances were created in Amazon AWS s EU West 1 region to test throughput. The cloned VM was terminated alongside the tested VMs at the conclusion of each 24-hour test cycle. b. On 1&1, Aruba Cloud, and CloudSigma, where VMs did not demonstrate varying internal network throughput depending on size and/or instance type, a screen session was created on each VM to listen for a TCP connection. Each category of VMs, which contains 2 VMs, connected with each other to perform throughput testing; i.e., 1&1 s 2 vcpu 4GB RAM and 2 vcpu RAM virtual machines conducted network throughput tests across one another. 7. A total of approximately 1.4 million data points were collected throughout the period of the study. 8. At the end of each test iteration, results were uploaded into Cloud Spectator s database through use of SQL Alchemy (Python see 4c in Process). 6 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

8 Tests Used Processor & Memory Bandwidth: Geekbench 3 Geekbench 3, a licensable product created by Primate Labs, is a cross-platform processor benchmark that can measure single-core and multi-core performance by simulating real-world workloads. The Geekbench 3 test suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 1 floating point workloads, and 4 memory-bandwidth tasks. While processor and memory bandwidth are both performance factors that contribute to the final score provided by Geekbench 3, the test suite weighs processing performance much more heavily than memory bandwidth. Also, memory bandwidth is not necessarily affected by the amount of memory available for the VM, so VMs with larger amounts of memory may not exhibit larger bandwidth. For more information on Geekbench 3 and to see its individual workloads, please see Geekbench 3 Tasks (Figure 3.1) TEST TOOL TASK Separate CPU tests that are all aggregated into a final AES, Twofish, SHA1, SHA2, BZip2 Compression, BZip2 Integer Geekbench 3 score. Decompression, JPEG Compression, JPEG Decompression, PNG Compression, Subtests PNG include: Decompression, Integer Math, Sobel, Floating Lua, Dijkstra Point Math Floating Point Geekbench 3 Black Scholes, Mandelbrot, Sharpen Filter, Blur Filter, SGEMM, DGEMM, SFFT, DFFT, N-Body, Ray Trace DESCRIPTION Integer and Floating Point tasks together represent vcpu performance. The performance of all applications is highly dependent on the vcpu since the vcpu is responsible for the processing and orchestration of all applications. Memory Geekbench 3 STREAM Copy, STREAM Scale, STREAM Add, STREAM Triad While memory performance is not considered one of the key bottlenecks in performance for many common applications, a subset of applications particularly HPC and in-memory databases is highly dependent on large sustained memory bandwidth. Sequential and Random Disk IOPS: fio Fio is an open source I/O generator that spawns a number of threads and processes to conduct a particular type of I/O action specified. For the purpose of this study, fio was used to measure disk IOPS by tracking direct I/O to the VM s network storage. 5 x 2mb files were created for sequential operations testing, and 5 x 2 mb files were created for random operations testing. All operations were 5% read and 5% write. Each test iteration used an 8kb block size. Each test iteration lasted 6 seconds. Internal Network Throughput: Iperf Iperf is an open source tool used to measure TCP or UDP network bandwidth performance. In this study, Cloud Spectator used Iperf to measure the network throughput between VMs residing in the same region/availability zone. A screen session was created for Iperf as a server machine on each appropriate VM (see Process 6). Each Iperf test iteration lasted 6 seconds, and data was transferred in one direction, from the test VM to the Iperf server VM. 7 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

9 VM Configurations & Pricing VM configurations were matched to standardize by virtual processors. 2 vcpus, 4 vcpus, and 8 vcpus machines from each provider were used in this study that most closely matched in size. While some providers may offer the option for local storage, none was used and all disk testing was conducted on persistent SAN storage. The storage columns in Figures 3.3 A through C are reflective of the SAN storage provisioned for each VM. Azure uses Blob Storage, which automatically provides the user with as-needed storage. Thus, only the space on the volume that is needed (depending on the OS and installed applications) is given to the user. With Blob Storage, users cannot deploy volumes with a pre-defined amount of storage, although the user can specify how large he or she expects the blob to grow. For more information, see this article: 2 vcpu Virtual Machines (Figure 3.3 A) Provider Instance vcpu RAM (GB) Storage (GB) Monthly ( ) 1&1 4GB & AWS C4.large (EBS Optimized) 95.4 AWS M3.large (EBS Optimized) Aruba Cloud 4GB Aruba Cloud Azure A Blob Storage 68.1 Azure D2 2 7 Blob Storage 14.6 CloudSigma 4GB CloudSigma Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

10 4 vcpu Virtual Machines (Figure 3.3 B) Provider Instance vcpu RAM (GB) Storage (GB) Monthly ( ) 1& & AWS C4.xlarge (EBS Optimized) 28.8 AWS M3.xlarge (EBS Optimized) Aruba Cloud Aruba Cloud Azure A3 4 7 Blob Storage Azure D Blob Storage CloudSigma CloudSigma vcpu Virtual Machines (Figure 3.3 C) Provider Instance vcpu RAM (GB) Storage (GB) Monthly ( ) 1& (2 x 4) &1 3GB (2 x 4) 28.8 AWS C4.2xlarge (EBS Optimized) AWS M3.2xlarge (EBS Optimized) Aruba Cloud Aruba Cloud 3GB Azure A Blob Storage Azure D Blob Storage CloudSigma CloudSigma 3GB Conversion Rates: 1. = 1.4 $1. =.89 9 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

11 Understanding Performance Results The virtual machines performance information was depicted using the minimum, 5 th percentile, median, 95 th percentile, and maximum scores retrieved from all data points collected for each of the tests mentioned above during the testing period. The information was integrated into percentile graphs and value tables designed to visualize performance variation captured while testing over time. An example of a performance percentile graph is displayed below: Sample Performance Graph Legend Maximum: highest score achieved on this VM over the duration of the testing. 95 TH Percentile (High-Score Category): 95% of all scores on this VM achieved this score or lower. Mpixels/sec Median (Median-Score Category): The number separating the higher half of the scores of that VM from the lower half. If the median is closer to the 95 th percentile, then more high scores were observed than low scores; vice versa. 5 AWS Azure Google Rackspace Softlayer 5 TH Percentile (Low-Score Category): 5% of all scores on this provider achieved this score or lower. Minimum: lowest score achieved on this VM over the duration of the testing. Understanding The CloudSpecs Score (Price-Performance) Cloud Spectator uses the CloudSpecs Score TM as an indication of price-performance value for each resource of the VM, separated by 1) processor & memory bandwidth, 2) disk IOPS, and 3) internal network throughput. By definition, the CloudSpecs Score TM provides information on how much performance the user receives for each unit of cost. The CloudSpecs Score TM is an indexed, comparable score ranging from - indicative of value based on a combination of cost and performance. The calculation of the CloudSpecs Score TM is: price-performance_value = [VM performance score] / [VM cost] best_vm_value = max{price-performance_values} CloudSpecs Score TM = *price-performance_value / best_vm_value CloudSpecs Score TM CSP1 CSP2 CSP3 CSP4 CSP5 Cloud Service Provider (CSP) In this report, Cloud Spectator uses the aggregated performance scores as the [provider performance score] to calculate each machine s CloudSpecs Score TM. The graph on the left is an example of how Cloud Spectator s priceperformance analysis is visualized. The closer the score is to, the higher price-performance value it indicates. The score represents the best-value VM among all in the comparison. The value is scaled; e.g., the 1 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

12 VM from Cloud Service Provider 1 (CSP1) with a score of gives 4x the value of the VM from CSP5 with a score of 25. The CloudSpecs Scores TM of any VM can change depending on the participants in the comparison. For example, if the highest score in a comparison changes, the price-performance value represented by score will change accordingly, and so will the other CloudSpecs Score TM values. If you have questions regarding Cloud Spectator s price-performance calculation, please contact us at contact@cloudspectator.com. Key Considerations Pricing used in this study (for price-performance comparisons) is up to date effective June 1, 215. Pricing may change for the specified VMs after the release of this report. Testing was conducted on specific VM types for each provider. Different VM configurations may yield different comparative results between the providers. AWS and Azure offered fixed VM configurations, while 1&1, Aruba Cloud, and CloudSigma offered independently customizable resource configurations. Users may experience different performance across different physical hosts. Factors such as user contention or malfunctions of the physical hardware can cause suboptimal performance. Cloud Spectator terminated and created new VMs for each test iteration to increase the likelihood of testing on different physical hosts. VMs selected were the base offerings across providers; greater performance may be obtained on certain providers by paying for additional features/services. 11 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

13 Detailed Performance Findings Processor and Memory Bandwidth Below are the results of processor and memory bandwidth testing on all providers. Because memory bandwidth is not affected by the amount of provisioned RAM, the VMs with larger amounts of RAM do not necessarily have higher Indexed Scores. The tables on the right specify the scores achieved by each provider s VMs. The lowest score in each category (Min, 5 th, Median, 95 TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables. Indexed Score Processor & Memory PERFORMANCE: 2vCPUs Figure 4.1 A 4GB c4.large m3.large 4GB A2 D2 4GB 2 vcpus Provider VM Min 5TH Median 95TH Max 1&1 4GB AWS c4.large m3.large Aruba 4GB Cloud Azure A D CloudSigma 4GB Indexed Score Processor & Memory PERFORMANCE: 4vCPUs Figure 4.1 B c4.xlarge m3.xlarge A3 D3 4 vcpus Provider VM Min 5TH Median 95TH Max 1& AWS c4.xlarge m3.xlarge Aruba Cloud Azure A D CloudSigma Indexed Score Processor & Memory PERFORMANCE: 8vCPUs Figure 4.1 C 3GB c4.2xlarge m3.2xlarge 3GB A4 D4 3GB 8 vcpus Provider VM Min 5TH Median 95TH Max 1& GB AWS c4.2xlarge m3.2xlarge Aruba Cloud 3GB Azure A D CloudSigma GB Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

14 Observations: Processor & Memory Bandwidth Performance The scores in Figures 4.1 A C are indexed through a combination of floating point and integer performance on processors and memory bandwidth on RAM. CloudSigma s performance varied the most across processors. Its coefficient of variation (CV), which is a percentage expressing the relationship between the average and standard deviation (useful for determining variability in performance), ranges up to 43.7% (on the 8 vcpu, 3GB RAM VM). By contrast, the CVs on 1&1 s tested VMs ranged up to 2.6% (on the 4 vcpu, RAM VM). Additional Observations 1&1 s VMs achieved the highest performance across processor and memory performance in the study. The Indexed Score, which weighs heavily on processor performance, is similar between VMs of the same vcpu count for 1&1, Aruba Cloud, and CloudSigma. All three providers offer independently customizable VMs, which are backed with the same hardware. AWS and Azure exhibit differences in performance results. AWS s C4 Family and M3 Family are provisioned with different processors; Azure s A Series and D Series are provisioned with different processors as well. Although AWS and Azure VMs processor and memory performance scored lower than 1&1 VMs, both providers also achieved similar levels of performance stability in the 72-hour test period, with CVs lower than 3% on all VMs. Aruba Cloud VMs CVs ranged up to 9.3%. CloudSigma VMs large variation in performance, seen in the processor and memory performance results, are present in disk IOPS and network performance as well. CloudSigma and Azure VMs continue to display the lowest processor and memory bandwidth performance in each category (see tables in Figure 4.1 A through C). 13 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

15 SAN Disk IOPS: Sequential Operations Below are the results of disk IOPS testing on all providers, specifically for sequential operations (5% read, 5% write). Disk IOPS was tested with direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the scores achieved by each provider s VMs. The lowest score in each category (Min, 5 th, Median, 95 TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables. IOPS Sequential Read/Write- 2vCPU (GB) Figure 4.2 A 4GB c4.large m3.large 4GB A2 D2 4GB 2 vcpus Provider VM Min 5TH Median 95TH Max 1&1 4GB AWS c4.large m3.large Aruba 4GB Cloud Azure A D CloudSigma 4GB IOPS Sequential Read/Write- 4vCPU (4GB) Figure 4.2 B c4.xlarge m3.xlarge A3 D3 4 vcpus Provider VM Min 5TH Median 95TH Max 1& AWS c4.xlarge m3.xlarge Aruba Cloud Azure A D CloudSigma IOPS Sequential Read/Write - 8vCPU () Figure 4.2 C 3GB c4.2xlarge m3.2xlarge 3GB A4 D4 3GB 8 vcpus Provider VM Min 5TH Median 95TH Max 1& GB AWS c4.2xlarge m3.2xlarge Aruba Cloud 3GB Azure A D CloudSigma GB Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

16 SAN Disk IOPS: Random Operations Below are the results of disk IOPS testing on all providers, specifically for random operations (5% read, 5% write). Disk IOPS was tested with direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the scores achieved by each provider s VMs. The lowest score in each category (Min, 5 th, Median, 95 TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables. IOPS Random Read/Write - 2vCPU (GB) Figure 4.3 A 4GB c4.large m3.large 4GB A2 D2 4GB 2 vcpus Provider VM Min 5TH Median 95TH Max 1&1 4GB AWS c4.large m3.large Aruba 4GB Cloud Azure A D CloudSigma 4GB IOPS Random Read/Write - 4vCPU (4GB) Figure 4.3 B c4.xlarge m3.xlarge A3 D3 4 vcpus Provider VM Min 5TH Median 95TH Max 1& AWS c4.xlarge m3.xlarge Aruba Cloud Azure A D CloudSigma IOPS Random Read/Write - 8vCPU () Figure 4.3 C 3GB c4.2xlarge m3.2xlarge 3GB A4 D4 3GB 8 vcpus Provider VM Min 5TH Median 95TH Max 1& GB AWS c4.2xlarge m3.2xlarge Aruba Cloud 3GB Azure A D CloudSigma GB Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

17 Observations: Sequential and Random Disk IOPS Performance Providers offer unique approaches to SAN disk, from hardware components (SSD vs. traditional magnetic drives) to performance throttling. Even the similarities can be quite different; for example, while 1&1 and CloudSigma both offer SSD-backed storage volumes, the performance difference between the two providers is noticeable (see Figure ), with 1&1 producing more than 6,5 IOPS, while CloudSigma produces less than 7 in the median range. Additional Observations Providers with SSD offerings exhibit little performance difference between random and sequential IOPS, which is expected for SSDs. These providers include 1&1, AWS and CloudSigma. One exception is on the small VMs for AWS, which exhibited different IOPS performance results due to throttling. Although 1&1 and CloudSigma both offer SSDs, 1&1 s volumes achieved x more IOPS than CloudSigma s SSDs when examining 95 TH percentile figures. 1&1, Aruba Cloud, and CloudSigma express large performance variability across the 24 hours of testing. Despite the variability, the low points of 1&1 s disk IOPS in both sequential and random operations still surpass other providers for the majority of tests. Aruba Cloud and CloudSigma s variability results in some of the lowest performance observed during the study. Although AWS offers SSD technology on its block storage offering, EBS, the performance of that offering is more dependent on the size of the block storage volume provisioned. Although the actual IOPS performance varies between sequential and random operations, the pattern of performance remains similar based on the size of the provisioned storage (in this test scenario, GB, 4GB, and sizes were provisioned on 2 vcpu, 4 vcpu, and 8 vcpu machines, respectively). For its General Purpose SSD volumes, AWS offers 3 IOPS per GB with burst up to 3 IOPS. Figures 4.2 A and 4.3 A illustrate AWS s burst for a GB volume. While burst for the GB volume never exceeded 365 IOPS (see Max in the corresponding tables), the non-burst range, expressed by the median, stays at 299 IOPS for sequential operations, but is higher for random operations. The minimum guaranteed IOPS, 3 (GB * 3 IOPS per GB), is sustained for the most part, although minimum values showed lower IOPS, with dips to 227 IOPS. At 8 GB, AWS sustains a fairly stable rate of approximately IOPS. Burst credits are assigned to volumes, and as long as the volume still has credits, it can burst to 3 IOPS. In the period of the study, for 4 vcpu and 8 vcpu machines on AWS, the 4GB and volumes sustained enough credits for a continuous 24-hours of high performance; interestingly, the c4.xlarge VMs dropped in performance for IOPS on sequential operations in all three 24-hour iterations, which suggests that credits for IOPS are being used up on the c4.xlarge VMs much faster than on others. Azure and Aruba Cloud do not offer SSD technology-backed storage. Therefore, both providers offer higher IOPS performance on sequential operations when compared with random operations, which is common behavior for magnetic drives. Figures 4.2 and 4.3 illustrate this behavior for both providers. In one of the sequences for the 8 vcpu with RAM VM, Aruba Cloud did not successfully complete a cycle of random read/write operations, and therefore achieved a minimum of in the total 72-hour study. 1&1 achieved much higher IOPS than all other providers tested, with maximums exceeding 95 IOPS for sequential operations. Even in the median, 1&1 s SolidFire-backed SSD volumes obtained a minimum of 6463 IOPS, which is 2.15x more IOPS than can be achieved per volume on AWS, unless a user decides to purchase provisioned IOPS at an additional charge. For random IOPS, Azure and Aruba Cloud displayed the lowest results in most scenarios. Azure and Aruba Cloud, as mentioned before, are the only providers in this study that use magnetic storage for their storage volumes. 16 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

18 Internal Network Throughput Below are the results of internal network testing on all providers. Internal network was examined by running TCP connections between two servers within the same data center/region/zone. The tables on the right specify the scores achieved by each provider s VMs. The lowest score in each category (Min, 5 th, Median, 95 TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables. Throughput (Mbit/s) Internal Network Throughput PERFORMANCE: 2 vcpus Figure 4.4 A 4GB c4.large m3.large 4GB A2 D2 4GB 2 vcpus Provider VM Min 5TH Median 95TH Max 1&1 4GB AWS c4.large m3.large Aruba 4GB Cloud Azure A D CloudSigma 4GB Throughput (Mbit/s) Internal Network Throughput PERFORMANCE: 4 vcpus Figure 4.4 B c4.xlarge m3.xlarge A3 D3 4 vcpus Provider VM Min 5TH Median 95TH Max 1& AWS c4.xlarge m3.xlarge Aruba Cloud Azure A D CloudSigma Throughput (Mbit/s) Internal Network Throughput PERFORMANCE: 8 vcpus Figure 4.4 C 3GB c4.2xlarge m3.2xlarge 3GB A4 D4 3GB 8 vcpus Provider VM Min 5TH Median 95TH Max 1& GB AWS c4.2xlarge m3.2xlarge Aruba Cloud 3GB Azure A D CloudSigma GB Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

19 Observations: Internal Network Throughput Performance With regards to internal network throughput, three out of five providers exhibited fairly stable performance of less than 1% CV. Azure s A2 and A4 VMs expressed higher variability at 33.3% and 12.1%, respectively. CloudSigma s variability ranged much higher, up to 45.9% on the 4vCPU RAM VM. When examining the median value of the same VM size, CloudSigma achieves much higher throughput (up to almost 7 Gbits) than the other VMs. Comparing with the same VM s minimum value, the almost 7 Gbits drops to 1.5 Gbits. Despite the sharp decrease in performance, CloudSigma s unstable performance still achieved higher throughput, for the most part, than all other providers. Additional Observations AWS s VMs deliver a specific network throughput depending on the size and family of VM. As the size of the VMs increase, the network throughput increases as well. The difference in network performance between the C4 Family and M3 Family are comparable. By contrast, Azure s A Series provides less throughput than its D Series. Azure s VMs, which scale depending on size and Series, exceeds 1GB/s network throughput for D3 and D4s (see Figures 4.4B and C). All other providers, with the exception of CloudSigma, never exceed 1GB/s, although 1&1 VMs and AWS s larger VMs come close. 1&1 provides a continuous and fairly steady throughput of slightly less than 1GB/s (956 GB/s max) regardless of VM size. Similarly, Aruba Cloud provides a lower, continuous throughput at a little less than 9 MB/s, regardless of VM size as well. While AWS displayed some of the lowest internal network throughput numbers for 2 vcpu and 4 vcpu VMs, Aruba Cloud, which does not scale internal network throughput with increased VM size, displayed the lowest throughput numbers for 8 vcpu VMs. 18 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

20 Detailed Price-Performance Findings Processor & Memory Figures 5.1 A through F illustrate the price-performance results for processor and memory bandwidth performance across all tested providers. Rows are organized by vcpu count from lowest to highest. Charts on the left columns compare VMs with less RAM. Processor & Memory Price-Performance: 2 vcpus Figure 5.1 A Processor and Memory Price-Performance: 2 vcpus Figure 5.1 B CloudSpecs Score GB 4 c4.large 26 4GB 2 A2 37 4GB CloudSpecs Score m3.large D2 64 Processor & Memory Price-Performance: 4 vcpus Figure 5.1 C Processor and Memory Price-Performance: 4 vcpus Figure 5.1 D 9 9 CloudSpecs Score c4.xlarge A3 CloudSpecs Score m3.xlarge 43 5 D3 41 Processor & Memory Price-Performance: 8 vcpus Figure 5.1 E Processor and Memory Price-Performance: 8 vcpus Figure 5.1 F CloudSpecs Score c4.2xlarge A4 CloudSpecs Score GB 66 m3.2xlarge 42 3GB 51 D4 28 3GB 19 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

21 Observations: Processor & Memory Price-Performance In all cases illustrated from Figure 5.1 A through F, 1&1 s VMs achieved the highest CloudSpecs Score TM in the test group, indicating the strongest price-performance value for processor and memory bandwidth. AWS s m3.large VM comes closest to matching in value with a 1&1 VM counterpart, at a CloudSpecs Score TM of 87 (see Figure 5.1 B). Additional Observations For VMs with 2 vcpus and approximately RAM (see Figure 5.1 B), all values are most evenly matched, compared with other VM sizes. As VM sizes scale up in processors and RAM, CloudSigma s CloudSpecs Score TM drops, and its rank falls in relativity to other tested providers VMs, due largely to performance (see Figure 4.1). Relatively, Azure s D Series, which has an average CloudSpecs Score TM of 53 across all VM sizes, provides more relative priceperformance value than its A Series, which has an average CloudSpecs Score TM of 27. Because charts cannot be relatively compared, though, the difference between the CloudSpecs Scores TM does not equate to an almost 2x value on Azure s D Series. 2 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

22 SAN Disk IOPS: Sequential Operations Figures 5.2 A through F illustrate the price-performance results for sequential disk IOPS performance across all tested providers. Rows are organized by vcpu count from lowest to highest. Charts on the left columns compare VMs with less RAM. Sequen9al Read/Write: 2 vcpus (GB) Figure 5.2 A Sequen9al Read/Write: 2 vcpus (GB) Figure 5.2 B GB 2 c4.large GB A2 4GB m3.large D2 11 Sequen9al Read/Write: 4 vcpus (4GB) Figure 5.2 C Sequen9al Read/Write: 4 vcpus (4GB) Figure 5.2 D c4.xlarge 7 12 A m3.xlarge 9 12 D3 6 Sequen9al Read/Write: 8 vcpus () Figure 5.2 E Sequen9al Read/Write: 8 vcpus () Figure 5.2 F c4.2xlarge A GB 3 m3.2xlarge GB D4 6 3GB 21 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

23 SAN Disk IOPS: Random Operations Figures 5.3 A through F illustrate the price-performance results for random disk IOPS performance across all tested providers. Rows are organized by vcpu count from lowest to highest. Charts on the left columns compare VMs with less RAM. Random Read/Write: 2 vcpus (GB) Figure 5.3 A Random Read/Write: 2 vcpus (GB) Figure 5.3 B GB 5 3 c4.large 4GB 9 A2 6 4GB m3.large 6 16 D2 11 Random Read/Write: 4 vcpus (4GB) Figure 5.3 C Random Read/Write: 4 vcpus (4GB) Figure 5.3 D c4.xlarge 3 12 A m3.xlarge 4 12 D3 6 Random Read/Write: 8 vcpus () Figure 5.3 E Random Read/Write: 8 vcpus () Figure 5.3 F c4.2xlarge 3 4 A GB 31 m3.2xlarge 4 3GB 12 D4 6 3GB 22 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

24 Observations: Sequential and Random Disk IOPS Price-Performance For sequential and random operations across all VMs, 1&1 offers the highest value due to a combination of high-performance SSDs and low overall cost of the virtual machine (see Figures 5.2 and 5.3). While CloudSigma also offers SSDs, the disk IOPS achieved on CloudSigma VMs display much lower performance (see Figures 4.2 and 4.3). AWS, which also offers SSD-based storage (General Purpose) offering was used in this study, throttles performance at a maximum of 3, IOPS unless users purchase additional provisioned IOPS; therefore, performance on AWS s SSDs did not exceed that of 1&1 either, which had a median range of 6,5-7,5 IOPS. Additional Observations 1&1 achieved the highest CloudSpecs Scores TM as well as most IOPS for each VM tested. Aruba Cloud offered the lowest value for random IOPS on SAN disk on all tested VMs, due to its magnetic drives and cost of the VMs. AWS s c4.large and m4.large (see Figure 5.2 A and B) offered lower price-performance value due to throttled IOPS after burst limits were exceeded for the GB storage volume. Although CloudSigma used SSD technology for its storage volume, the price-performance value of those storage volumes tested are similar to Aruba Cloud and Azure volumes, which use magnetic disks. 23 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

25 Internal Network Throughput Figures 5.4 A through F illustrate the price-performance results for internal network throughput performance across all tested providers. Rows are organized by vcpu count from lowest to highest. Charts on the left columns compare VMs with less RAM. Internal Network Price- Performance: 2 vcpus Figure 5.4 A Internal Network Price- Performance: 2 vcpus Figure 5.4 B GB 13 c4.large GB A2 4GB m3.large D2 Internal Network Price- Performance: 4 vcpus Figure 5.4 C Internal Network Price- Performance: 4 vcpus Figure 5.4 D c4.xlarge 13 A m3.xlarge 9 19 D3 Internal Network Price- Performance: 8 vcpus Figure 5.4 E Internal Network Price- Performance: 8 vcpus Figure 5.4 F c4.2xlarge A GB 26 m3.2xlarge 19 3GB 39 D4 3GB Observations: Internal Network Throughput Price-Performance CloudSigma VMs deliver the highest available throughput out of all providers tested. Despite having higher prices than 1&1, the high network throughput performance resulted in a better CloudSpecs Score TM for CloudSigma. Similar with Processor & Memory price-performance, 24 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

26 Azure also achieved a better CloudSpecs Score TM for its D Series in regards to internal network throughput. The D series offered approximately 1.7x more network throughput on VMs tested in the study. Although AWS s virtual machines increased throughput based on size and family, that increase did not surpass the price-performance ratio of 1&1 VMs throughput and cost; therefore, 1&1 achieved a higher CloudSpecs Score TM on network throughput than AWS. Aruba Cloud, which offered higher throughput than AWS on smaller VMs (see Figure 4.4 A), displayed higher internal network throughput price-performance than AWS until VM sizes scale up. As machine sizes increased, AWS s throughput scaled while Aruba Cloud s did not, which is the reason AWS has higher CloudSpecs Scores TM for the larger VMs. For these VMs, (see Figure 5.4 B and C), Aruba Cloud fell short of AWS, as well as all other competitors in the study. Additional Observations Azure s D3 and D4 VMs exceeded a throughput of 1 GB/s (see Figures 4.4 B and C), giving it better price-performance value in internal network than all other providers for those VM sizes with the exception of CloudSigma (see Figures 5.4 D and F). In all other VM sets, 1&1 achieves the highest CloudSpecs Score TM with the exception of CloudSigma for internal network. Aruba Cloud, which does not scale internal network throughput with the size of the VM (see Figure 4.4), drops in value ranking as VMs are scaled up in size; other providers, such as AWS and Azure, which scale internal network throughput with the corresponding VM size and family/series, surpass Aruba Cloud in value for internal network as the VMs are scaled up. 25 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.

27 Conclusion Selecting the right provider and virtual machines requires a thorough and accurate performance comparison. The study conducted for this report offers a general understanding of performance and price-performance strengths and weaknesses across each included vendor s VMs, and should be considered a source of information to help guide readers in their own testing and analyses. The processor, memory bandwidth, network storage, and internal network were all examined and results clearly show that no single provider can offer everything to fit everyone s needs. Therefore, in order to accurately select the right provider for a business or an application, performance and price-performance analysis is crucial. Results from this study show that, 1&1 s VMs displayed strong performance and price-performance. 1&1 s VMs demonstrated high performance and stability for processing and memory bandwidth both on the same VM, and also across VMs of the same size. Its network-attached storage produced the largest amount of IOPS seen in the study as well, regardless of VM size and without the need for purchasing additional volumes, sizing up, or provisioning IOPS. These high-performance results, combined with the low cost of the VMs, reflect on the value that 1&1 s VMs can deliver to potential users, as seen with the price-performance results. 1&1 s performance and price-performance offers an excellent alternative for high-performance environments such as distributed file systems and data analytics processing. Performance in the industry cannot be assumed to be equal or even similar, as illustrated in this report. When it comes to processor and memory bandwidth performance, tiered providers such as AWS and Azure may offer varying performance depending on the family/series of the VM, despite having equivalent amounts of vcpus and similar amounts of memory. For disk IOPS, although both 1&1 and CloudSigma advertise SSD volumes, 1&1 s SSDs achieved x more IOPS than their CloudSigma counterparts. Internal network performance on CloudSigma, though, exceeds all providers examined in the study. While this study was conducted in the manner of understanding a typical end user experience, it should not be assumed to be accurate for all use cases. Stress testing was conducted to better understand fluctuation and theoretically sustained performance, and should be seen as a general indication of provider performance. For more detailed analysis on any specific use case, please contact Cloud Spectator at contact@cloudspectator.com or by phone at +1 (617) About About Cloud Spectator Cloud Spectator is a cloud analyst agency focused on cloud Infrastructure-as-a-Service (IaaS) performance. The company actively monitors several of the largest IaaS providers in the world, comparing VM performance (i.e., CPU, RAM, disk, internal network, and workloads) and pricing to achieve transparency in the cloud market. The company helps cloud providers understand their market position and helps business make intelligent decisions in selecting cloud providers and lowering total cost of ownership. The firm was founded in early 211 and is located in Boston, MA. For questions about this report, to request a custom report, or if you have general inquiries about our products and services, please contact Cloud Spectator at +1 (617) or contact@cloudspectator.com. For press/media related inquiries, please contact: Ken Balazs VP Sales & Marketing kbalazs@cloudspectator.com 26 Copyright 215 Cloud Spectator, Inc. All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.