CLOUDSPECS PERFORMANCE REPORT LUNACLOUD, AMAZON EC2, RACKSPACE CLOUD AUTHOR: KENNY LI NOVEMBER 2012

Transcription

1 CLOUDSPECS PERFORMANCE REPORT LUNACLOUD, AMAZON EC2, RACKSPACE CLOUD AUTHOR: KENNY LI NOVEMBER 2012

2 EXECUTIVE SUMMARY This publication of the CloudSpecs Performance Report compares cloud servers of Amazon EC2, Rackspace Cloud Servers (not OpenStack Cloud), and Lunacloud. Performance is measured over a period of 30 days to collect data on server performance capabilities as well as consistency in performance. The CloudSpecs system runs open source benchmark tests to measure performance in CPU, RAM, disk, and internal network. The software is set to automatically run 4 times a day for 7 days a week. The report is comprised of: 1 general server performance test 5 CPU performance tests 3 Disk performance tests 2 RAM performance tests 7 network performance tests Key Findings In the general server test, Lunacloud outperforms Amazon and Rackspace consistently over a period of 30 days. Lunacloud disk performance scores up to 8x better than Amazon or Rackspace disk performance. This is an important consideration for database performance. Lunacloud s internal network throughput is 2x faster than Rackspace s internal network throughput, and significantly more stable than Amazon in a period of 30 days. With applications requiring more than one server, internal network speed may be considered a bottleneck in many cases if other components of the server are optimized. METHODOLOGY Amazon EC2 s CPU performance generally scores slightly better than Rackspace and Lunacloud CPU performance. CPU is a vital component of the server, and application performance depends significantly on CPU functionality. Rackspace s RAM is more consistent in performance than Amazon or Lunacloud, though it scores the lowest in performance over a period of 30 days. Though the processor speed is an important factor in server performance, RAM plays an equally important role. Because modern CPUs can perform many of the applications run today, bottleneck instances may actually occur with RAM. Cloud Spectator sets up two servers on each cloud provider that meets the following minimum requirement: Primary Server CPU Cores 4 vcpus RAM in GB 16 Disk in GB 50 Secondary Server CPU Cores 1 vcpu RAM in GB 4 Disk in GB 50 Because Amazon and Rackspace offer packaged cloud servers with preset allocations of CPU, RAM, and disk, the closest package that meets the minimum requirements is selected. AMAZON The servers are set up on Amazon EC2 availability zone US-East-1a. Virtualized server OS is Amazon Linux. Primary Server Secondary Server CPU Cores 4 vcpus RAM in GB 15 Disk in GB 1,690 RACKSPACE The servers are set up in Rackspace s Dallas (DFW) data center. Virtualized server OS is Debian 6. CPU Cores 1 vcpus RAM in GB 3.75 Disk in GB 410 Primary Server CPU Cores 4 vcpus RAM in GB Disk in GB 620 Secondary Server CPU Cores 2 vcpus RAM in GB 4 Disk in GB 160 2

3 LUNACLOUD The servers are set up in Lunacloud s Virtualized server OS is Debian 6. Primary Server CPU Cores 4 vcpus RAM in GB 16 Disk in GB 50 Secondary Server CPU Cores 1 vcpus RAM in GB 4 Disk in GB 50 Cloud Spectator maintains root access to all servers. Using SSH, the CloudSpecs application is installed on to the servers. Both Primary and Secondary servers are located within the same virtual private network. Once the CloudSpecs performance application is installed, it automatically runs 4 times a day, 365 days a year. PERFORMANCE TESTS Cloud Spectator runs basic, open source benchmark tests that are precise and accurate for measuring the performance of a cloud server. Tests included come from the Unixbench Suite ( and Phoronix Test Suite ( The benchmarks for network tests and the disk benchmark for local file copy may be reproduced using the Linux command line interface. The network tests assume $SERVER is the IP of the secondary server and $CLIENT is the IP of the primary server. Command line instructions for the internal network tests are included below in each of the tests within the network description. DATA COLLECTION The CloudSpecs test results for this publication run from October 17, 2012 to November 16, 2012 (a total of 30 days). Cloud Spectator obtains cloud servers by either purchasing the server space directly from the client, or the client reimburses Cloud Spectator for data collected relevant to that active project. CLOUDSPECS SCORE The CloudSpecs Score represents the true value a cloud user receives when using a server. A higher score means better value. The highest value performer receives a score of 100 and other providers are pegged in relativity to value. Value is calculated based on performance over time and the cost of the Primary Server hourly (730 hours per month). Provider Cost per Hour Amazon EC2 East $0.64 Rackspace $0.90 Lunacloud $0.32 Below is the formula process for calculating the CloudSpecs Score (Provider s CloudSpecs Score is the calculation for the final CloudSpecs score of each provider). The CloudSpecs score is rounded off to the nearest whole number. The CloudSpecs score in this report were calculated over a period of 30 days: provider_value(p) = [Provider test score over a period of time] / [Provider price] best_provider_value = max(provider_values) Provider s CloudSpecs Score = 100 * provider_value(p) / best_provider_value The Sectional CloudSpecs Score is calculated by averaging the CloudSpecs scores of each section s available tests. 3

4 PERFORMANCE SUMMARY General Server Performance UnixBench (Score) Lunacloud 1, Amazon 1, Rackspace CPU Performance 7-Zip Compression (MIPS) Lunacloud 4, Amazon 6, Rackspace 6, Linux Kernel Compilation (Seconds to Compile) Lunacloud Amazon Rackspace Video Encoding Using x264 (FPS) Lunacloud Amazon Rackspace Audio Encoding Using LAME MP3 (Seconds to Compile) Lunacloud Amazon Rackspace Apache Kernel Compilation (Seconds to Compile) Lunacloud Amazon Rackspace Disk Performance DBench (MiB/sec) Lunacloud Amazon Rackspace 95 8 Iozone (MiB/sec) Lunacloud 1, Amazon Rackspace File Copy (MiB/sec) Lunacloud Amazon 26 2 Rackspace 39 2 RAM Performance STREAM (MiB/sec) Lunacloud 11, Amazon 15, Rackspace 6, RAMSpeed SMP (MiB/sec) Lunacloud 7, Amazon 7, Rackspace 5,

5 Internal Network Performance Network Throughput with Iperf (Mibit/sec) Lunacloud Amazon Rackspace FTP to RAM (MiBit/sec) Lunacloud Amazon Rackspace FTP to Disk (MiBit/sec) Lunacloud Amazon Rackspace Apache WGET (MiBit/sec) Lunacloud Amazon Rackspace Nginx WGET (MiBit/sec) Lunacloud Amazon Rackspace Network SCP (MiBit/sec) Lunacloud Amazon Rackspace

6 GENERAL SERVER PERFORMANCE Cloud Spectator runs the UnixBench benchmark suite to gauge overall performance of a cloud server. Over the test period of 30 days, Lunacloud outperforms Amazon and Rackspace in the UnixBench test. UNIXBENCH The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system. The UnixBench is a suite of approximately 10 tests that can measure how many instructions per second a CPU can perform. If CPU A performs 2x better than CPU B at only 1.5x the cost, then CPU A provides more performance per unit of cost. Lunacloud 1, Amazon 1, Rackspace

7 CPU PERFORMANCE 7ZIP COMPRESSION This is a test of 7-Zip using p7zip with its integrated benchmark feature, measured in million instructions per second (MIPS). The test consists of compressing a file with random data using the 7-zip program and then dividing the number of CPU instructions executed during the compression by the number of seconds. The result is then divided by 1 million to calculate the value in MIPS. The 7-zip compression test results are important in comparison because the 7-zip compression algorithm uses many kinds of CPU instructions, resulting in an accurate overall score reflecting the performance of the CPU. The fact that it is a compression program is irrelevant in the test. TIMED LINUX KERNEL COMPILATION Lunacloud 4, Amazon 6, Rackspace 6, This test times how long (in seconds) it takes to build the Linux kernel. Unlike the 7-zip compression, this is a compilation source code. Though many compilation tests exist, an advantage of the Linux Kernel compilation is the portability and compatibility with a majority of Linux distributions. Because of the variety of CPU instructions and the length of time for the test, the Linux kernel compilation can accurately measure a CPU s performance. Lunacloud Amazon Rackspace

8 TIMED APACHE COMPILATION This test times how long (in seconds) it takes to build the Apache HTTP Server. Though many compilation tests exist, an advantage of the Apache compilation is the portability and compatibility with a majority of Linux distributions. Because of the variety of CPU instructions and the length of time for the test, the Apache compilation can accurately measure a CPU s performance. X264 VIDEO ENCODING Lunacloud Amazon Rackspace The x264 test benchmarks the speed it takes a server to encode a movie file using the x264 H.264/AVC encoder. When results are shown alone, they have little meaning, but the process requires a large amount of computations, so when compared side-by-side using results from several cloud providers, the test provides an accurate insight of CPU performance. Lunacloud Amazon Rackspace

9 LAME MP3 ENCODING LAME is an MP3 encoder licensed under the LGPL. This test measures the time (in seconds) required to encode a WAV file to MP3 format. Similar to the x264 test, results alone have little meaning, but the process requires a large amount of computations, so when compared side-by-side using results from several cloud providers, the test provides accurate insight of CPU performance. Lunacloud Amazon Rackspace

10 DISK PERFORMANCE DBENCH DBench is a benchmark designed by the Samba project as a free alternative to netbench, but dbench contains only file-system calls for testing the disk performance. DBench can be used to stress a filesystem or a server to see which workload it becomes saturated and can also be used for prediction analysis to determine how many concurrent clients/applications performing this workload the server can handle before response starts to lag. FILE COPY Lunacloud Amazon Rackspace 95 8 This measures the performance of the operating system's 'cp' command, followed by a 'sync' (to flush any cache), executed on the local storage device. The file size is 10GB, and is copied to the same disk measured in mebibytes per second. File copy measures read and write performance of the disk, which is important when processing massive amounts of data. Lunacloud Amazon 26 2 Rackspace

11 IOZONE FILESYSTEM BENCHMARK Iozone benchmarks file IO performance of a server for the operations: Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread,mmap, aio_read, aio_write Iozone can collect data on a broad range of file system performance to locate hot or cold spots in the file systems. Lunacloud 1, Amazon Rackspace

12 RAM TESTS STREAM The STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels. The STREAM benchmark is specifically designed to work with datasets much larger than the available cache on any given system, so that the results are (presumably) more indicative of the performance of very large, vector style applications. STREAM measures memory bandwidth to see how the RAM accommodates to the processes running on the server. RAMSPEED SMP Lunacloud 11, Amazon 15, Rackspace 6, This benchmark tests the system memory (RAM) performance. RAMspeed results correlate highly with STREAM results. Though the algorithms are different, RAMspeed SMP also measures the RAM accommodation to processes running on the server. Lunacloud 7, Amazon 7, Rackspace 5,

13 NETWORK PERFORMANCE TESTS NETWORK THROUGHPUT WITH IPERF Iperf is a commonly used network testing tool that can create TCP and UDP data streams and measure the throughput of a network that is carrying them. Our test consists of transferring as much data as possible through the local network (where possible, private network IPs were used) for 120 seconds over TCP port Iperf tests the quality of the network. Using this test between two cloud servers in the same data center of the same provider, results can be collected on how efficient the internal network speed is, which is important for servers inside the same virtual private network that may be required to transfer data between or among each other quickly and efficiently to effectively run applications. Code wget ' tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fiperf%2F&ts= &use_mirror=ignum' -O iperf.tgz tar zxvf iperf.tgz cd iperf /configure make./src/iperf -s -p 5001 wget ' tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fiperf%2F&ts= &use_mirror=ignum' -O iperf.tgz tar zxvf iperf.tgz cd iperf /configure make./src/iperf -c $SERVER -p t 120 Lunacloud Amazon Rackspace

14 NETWORK FTP TO RAMDISK Using vsftpd (Very Secure FTP Daemon) on Server and lftp on Client, the test transfer a 1GB file from RAM drive to RAM drive. Transmitting from a RAM disk bypasses the disk in the network test, preventing a possible bottleneck at the disk level and enabling a more accurate measurement of the internal network transfer speed. Code mkdir -p /mnt/ramfs chmod 0555 /mnt/ramfs mount -t ramfs -o size=1050mb ramfs /mnt/ramfs dd if=/dev/zero of=/mnt/ramfs/file.big bs=1mib count=1024 time lftpget -d ftp://$server/file.big Lunacloud Amazon Rackspace

15 NETWORK FTP TO HARD DISK Using vsftpd (Very Secure FTP Daemon) on Server and lftp on Client, the test transfer a 5GB file from hard disk to hard disk. The File Transfer Speed of the Disk is similar to the FTP speed of the RAM disk, but in a manner that measures the speed efficiency of a disk-to-disk transfer, rather than RAM-to-RAM. Code dd if=/dev/zero of=/tmp/file.big bs=5mib count=1024 time lftpget -d ftp://$server/file.big Lunacloud Amazon Rackspace

16 NETWORK HTTP Apache Using apache2 as Server and wget as Client, transfer of a 1 GB file. Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage. Code dd if=/dev/zero of=/tmp/file.big bs=1mib count=1024 time wget Lunacloud Amazon Rackspace

17 NETWORK HTTP NGINX Using nginx (a web server application) as server and wget as Client, transfer of a 1 GB file. Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage. Code dd if=/dev/zero of=/tmp/file.big bs=1mib count=1024 time wget Lunacloud Amazon Rackspace

18 NETWORK SCP Copy of a 5GB file using Secure Copy (SCP). Performing a more natural test, the Apache Network HTTP, Nginx Network HTTP, and Network SCP tests all combine the internal network with CPU usage to see how fast servers perform when they have added CPU usage. Code dd if=/dev/zero of=/home/guest/file.big bs=5mib count=1024 time scp guest@$server:file.big. Lunacloud Amazon Rackspace About Cloud Spectator Cloud Spectator is the premier cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud Spectator has monitored the cloud Infrastructure industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 4 times a day, 365 days a year with use of open source benchmark tests. The data is compared relatively with pricing from each provider to allow business decision-makers to smart decisions. Cloud Spectator 800 Boylston Street 16 TH Floor Boston, MA Website: Phone: (USA)