Benchmarking Large Scale Cloud Computing in Asia Pacific

2013 19th IEEE International Conference on Parallel and Distributed Systems ing Large Scale Cloud Computing in Asia Pacific Amalina Mohamad Sabri 1, Suresh Reuben Balakrishnan 1, Sun Veer Moolye 1, Chung Yik Cho 1, Sarinder K. Dhillon 3, Amandeep S. Sidhu 1, 2, SMIEEE 1 School of Engineering & Science, Curtin University, Sarawak, Malaysia 2 Faculty of Health Sciences, Curtin University, Perth, Australia 3 Faculty of Science, University of Malaya, Kuala Lumpur, Malaysia asidhu@ieee.org Abstract - Cloud services allows organizations to run their applications faster, under better manageability and less maintenance so that they can focus on their core business. In this paper we are benchmarking various public clouds (Microsoft, Amazon EC2 and the Australian National eresearch Collaboration Tools and Resources Cloud (NeCTAR)) that are commonly used by academic organizations in Asia Pacific to gauge their performances using variety of tests. Keywords - Cloud Computing; Microsoft ; Amazon EC2; NeCTAR I. INTRODUCTION Cloud computing defines the use of delivering hosted services over the internet [1]. It is also a basic change to an operational model in which applications are not necessarily kept in a specific physical hardware. Computing resources can be easily relocated based on the user s need due to cloud s flexibility [2, 3]. This paper divided into two main parts. The first part is focused on explaining cloud computing concepts and technologies and the second part is focused on the benchmarking for various public clouds commonly used by academic organizations in the Asia Pacific Region - Microsoft, Amazon EC2 and the Australian National eresearch Collaboration Tools and Resources Cloud (NeCTAR). The benchmarking exercise is to gauge their performance by running a variety of tests. II. CLOUD SERVICE MODELS A. Infrastructure as a Service IaaS is the most basic of the cloud service models, which offers extra resources such as virtual machine images, file level storage, IP addresses, load balancers, virtual local area networks (VLANs), firewalls and software bundles [4]. The service provider manages housing, running and maintaining the IaaS model, and the users are responsible to patch and maintain the operating system and the application software. Normally IaaS provides allow dynamic scaling, desktop virtualization, automation of administrative tasks, billing model and computing services [5]. B. Platform as a Service PaaS offers the renting of hardware, storage and network capacity, operating systems [6]. It has many advantages for developers as they can change and upgrade the operating system (OS) features regularly according to their software development lifecycles that may involve many virtual teams across the geographic boundaries. Also using infrastructure from a cloud provider reduces duplication and incompatibility problems. However, if the services offered need exclusive interfaces or development languages, PaaS may have the risk of lock-in [6]. C. Software as a Service SaaS is a software delivery model whereby software and associated data are centrally hosted on the cloud. Users can usually access it by using a thin client via a web browser [7]. Besides web services, SaaS also supports serviceoriented architecture (SOA) and new developmental methods, such as Ajax. The cloud provider will give clients network-based access to a copy of an application produced specifically for SaaS distribution [7]. SaaS is easy to administrate, does automatic updates and patch management and has global accessibility. III. CLOUD IMPLEMENTATION MODELS A. Public Cloud Public cloud is a type of cloud computing model which provides hosted services over the internet [8]. Services provided by public cloud are either free (NeCTAR) or users may need to pay every time they are using the service (, Amazon EC2). It is easy and affordable because as hardware, application and bandwidth are managed by the service provider. Users need to ensure that they have disaster recovery and data backup plans [8]. Some of the commercial public cloud providers are Amazon Elastic Compute Cloud (EC2), HP Cloud, ProfitBricks, and Windows. B. Private Cloud Private cloud is an exclusive computing architecture which provides services to limited number of people behind the firewall. Some organizations (like banks) are highly concerned about the data security so they choose private clouds over public clouds [9]. Like public cloud it still reduces cost and increases efficiency. 1521-9097/13 $31.00 2013 IEEE DOI 10.1109/.122 694 693

C. Hybrid Cloud A hybrid cloud consists of one or more private cloud and one or more public cloud. It is an environment where an organization administers some resources in-house and the rest of resources being managed externally [10]. This way scalability and cost-effectiveness of a public cloud can be fully utilized without exposing sensitive applications and data to third-party vulnerabilities. IV. BENCHMARKING RESULTS We benchmarked three public clouds commonly used by academic organizations in the Asia Pacific Region - Microsoft, Amazon EC2 and the Australian National eresearch Collaboration Tools and Resources Cloud (NeCTAR). The software used to the benchmark the cloud was Roy Longbottom s Linux benchmarking tools [11]. A. Cloud Instance Specifications and Asumptions Processor AMD Opteron Processor 4171 HE Intel Xeon CPU E5-2650 0 @ 2.00GHz Intel Core 2 Duo CPU T7700 @ 2.40GHz Measured Minimum 2095 MHz, Maximum 2095 MHz Minimum 1800 MHz, Maximum 1800 MHz Minimum 2600 MHz, Maximum 2600 MHz CPUs 2 2 2 RAM Size 3.36GB 3.66GB 7.80GB For cloud computing instances, the cloud provider provides a fixed computing instance specification meaning there is no way to adjust the CPU option and memory option to ensure parity between the different providers. The specifications above are all in the medium instance for each provider at the time of running the benchmark. B. Classic For CPU Performance Dhrystone This test is a measure of integer performance. There are two available versions, 1.1 and 2.1. The difference between 1.1 and 2.1 is that 2.1 avoid over-optimization problems that have been encountered in version 1. The table below shows a comparison between the 3 platforms the benchmark tests were run on namely, Amazon, Nectar and. These tests include versions 1.1 and 2.1, with and without optimization. The ratings obtained are that of VAX MIPS where VAX stands for Virtual Address extension and MIPS means Million Instruction per Second. Linpack This test measures the floating point computing power of a system. Floating point shows a way of representing the approximation of a real number in such a way that it can support a wide range of values. The Millions Floating-point Operations per Second is the unit by which the benchmark test is measured. All tests were conducted at Opt 3 and 100 optimization. Livermore Loops Livermore loops is a benchmark test that is usually run for parallel computers. Produced for supercomputers comprising of numerous kernels, three specific types of data sizes are ran and the results obtained are in MFLOPS. The results generated for overall ratings consist of Maximum, Average, Geometric mean (Geomean), Harmonic mean (Harmean) and Minimum whereby Geomean is the official overall rating. All tests for Livermore loops were completed over 24 loops and Opt 3 setting and the geometric mean was the one recorded. Whetstone The Whetstone Single Precision C is related to CPU performance and is meant to check speed ratings in Millions of Whetstone Instructions per Second (MWIPS). Table 1. Classic Results Dhrystone 8155 10455.50 10752.28 (VAX MIPS rating) Linpack Double 1317.95 1603.02 1609.31 Precision Unrolled Livermore Loops 2588.9 2733.8 2634.1 Maximum Rating Whetstone Single Precision C MWIPS 2135.854 2111.706 2644.834 Classic Overall Comments In the Dhrystone Performance NECTAR scores 1.3x better than Windows however between NECTAR and Amazon, NECTAR scores 1.03x better. NECTAR is the best for this benchmark. In the Linpack Performance NECTAR scores 1.22x better than Windows however between NECTAR and Amazon, NECTAR scores 1.004x better. NECTAR is the best for this benchmark. In the Livermore Loops Performance, Amazon scores 1.056x better than Windows however between Amazon and NECTAR, Amazon scores 1.038x better. Amazon is the best for this benchmark. In the Whetstone Single Precision C Performance, NECTAR scores 1.24x better than Windows however between NECTAR and Amazon, NECTAR scores 1.25x better. NECTAR is the best for this benchmark. Overall, NECTAR is the best in this classic benchmarks category followed by Amazon and Windows. 695 694

Figure 2. Disk, USB and LAN results Figure 1. Classic Results C. Disk, USB and LAN s This test makes use of direct I/O for the speed of Local Area Network (LAN) to avoid data from being cached in the main memory of the Operating System. Also involved in the benchmark tests are the read/write speed. In this test, a 64Kb file was written, read and deleted 500 times and the result can be seen below: Table 2. Disk, USB and LAN results Windows Amazon NECTAR Write MB/sec 122.83 25.34 6.23 Read MB/sec 274.74 67.75 92.84 Disk, USB and LAN s Overall Results In the Disk, USB and LAN s, for the write category, Windows is the best followed by Amazon and NECTAR while for read category, Windows is still the best followed by NECTAR and Amazon. Overall, Windows is the best followed by Amazon and NECTAR. D. Multithreading s Simple Add Tests The tests involved in Simple Add Tests execute 32bit and 64bit integer instructions as well as 128bit SSE floating point. The performance is very relative to the amount of CPU cores available in the system. Since the benchmark test is about multithreading, each thread is given an independent code to test for each thread. The values taken for this test is the average of two aggregates tested individually. Whetstone As opposed to the previous whetstone benchmark, this test focuses on multithreading application. Again, the number of cores present is a determinant factor on the speed of the test run. The results taken as reference for the test are based on the time taken for the last thread to finish and measured in Millions of Whetstone Instructions per Second (MWIPS). MP FLOPS Program The purpose of this test is to check for the multiplication of floating point calculations with data from higher level of caches or RAM. These programs can be used as a burnin/reliability test and similar functions can be run on a different segment of data. The last Million Floating Point Instructions per Second value from the test is taken as reference. 696 695

MP Memory Speed Tests This test makes use of single and double precision floating point numbers and integers to test for the speed of the memory. The average value of the read, write and delete were taken individually and then graphed to figure out the best out of the cloud systems. MP Memory Bus Speed Tests The bus/memory speed is tested by reading all the data at the same time. The value taken for this test is the ReadAll value of the largest file. This accounts for a sizeable cache and RAM usage stressing the bus and allowing for an estimation of maximum bus/memory speed. MP Memory Random Access Speed This benchmark test is about read and read/write tests that cover cache and RAM data sizes. The largest file (96MB) is chosen since it uses all the resources and maximises the stress on the cores for the test giving a very relatable value. The average of the serial read, read/write and random read, read/write as well as mutex read, read/write is taken to give a general idea of how it performs on various platforms. Table 3. Multithreading results Bus Speed 5582 7574 4961 MP Memory 5902.44 6183.44 6715.44 Reading Speed Multithreading 13874 6343 14552.50 Add Test (Million Instruction Per Second) MP SSE 18418 10018 27839 MFLOPS Multithreading 4257 1981 5012 Double Precision Whetstones (MWIPS) RandMemMP Speeds (Mbytes Per Second) 2372.33 2608.33 2622.67 1) Multithreading Overall Comments For multithreading benchmarks, NECTAR is the best is in 5 out of 6 categories while Amazon is the best in 1 out of 6 categories. Windows fared badly in all categories however in the multithreading add test and multithreading double precision whetstones, it was close to NECTAR and the overall best performer in multithreading is NECTAR followed by Amazon and Windows. Figure 3. Multithreading results E. OpenMP s for Parallel Processing Performance MemSpeed This test makes use of single and double precision floating point numbers and integers to test for the speed of the memory. The average value of the read, write and delete were taken individually and then graphed to figure out the best out of the cloud systems. Original Open MP Taking the MFLOPS value, this test behaves in a similar way as Windows compilation, meaning the performance gains of the number of cores present is relative to the time taken for the test to complete as compared to a single core. The average value for Data in and out is taken as the comparison value for the different platforms. 697 696

Random Memory This test shows the behavior of the memory with increasing file size in terms of data transfer. The values taken are similar to that of the MP Memory tests. Table 4. OpenMP s Results Memory Reading 4147.33 6457.78 4095 Speed Test OpenMP MFLOPS 4781 10035 13886 OpenMP s Overall Comments For the OpenMP benchmarks, Amazon is the best in memory reading speed test followed by Windows and NECTAR while for OpenMP MFLOPS benchmark, NECTAR is the best followed by Amazon and Windows. Overall, NECTAR and Amazon is the best followed by Windows RAM. The alterations in this test avoid intermediate register to register operations to produce much faster speeds. Again, the largest value is taken as reference and compared across platforms. Table 5. Memory BusSpeed Results Bus Speed Test 5455 7461 2525 Random/Serial 1850.25 3113.625 1910.375 Memory Test SSE & SSE2 Memory Reading Speed Test 4935.25 4267.25 5956.125 BusSpeed Overall Comments In BusSpeed benchmark, Amazon is the best in bus speed test followed by Windows and NECTAR while in random/serial memory test, Amazon is still the best followed by NECTAR and Windows. For SSE & SSE2 memory reading speed test, NECTAR is the best followed by Windows and Amazon. Overall in this BusSpeed benchmark, Amazon is the best followed by NECTAR and Windows. Figure 4. OpenMP Results F. Memory BusSpeed Bus Speed Test This test makes use of single and double precision floating point numbers and integers to test for the speed of the memory. The average value of the read, write and delete were taken individually and then graphed to figure out the best out of the cloud systems. Figure 5. Memory BusSpeed Results SSEfpu This variation of the previous SSE benchmark excludes AMD but measures Single Precision and Double Precision, floating point speeds, data streaming from caches and 698 697

V. CONCLUSIONS We benchmarked various public cloud providers commonly used by academics in Asia Pacific region. The objective was to run a variety of Linux benchmarks on the medium instances of these clouds and then analyze the results. The overall performance rankings all these benchmarks is summarized in the following table. Table 6. Performance Rankings Classic Third Second First s for CPU Performance Disk, USB and First Second Third LAN s Multithreading Third Second First s OpenMP Third Second First for Parallel Processing Performance Memory BusSpeed Third First Second The main aim of this exercise was to identify the best possible candidates for a building a High Performance Computing (HPC) and Cloud Computing model that can be deployed quickly and easily. Due to the nature of this hybrid model performance of Disk and LAN benchmarks will be the key to avoid bottlenecks. We note that Windows is using AMD processors whereas other public clouds are using Intel processors (Table 1) so comparison is not fair. We also note that academic institutions already using Windows HPC will find easier integration with Windows than any other public cloud. In future we plan to construct and benchmark hybrid HPC+Cloud model that is platform agonistic. REFERENCES 1. What is Cloud Computing? 2009 [cited 23 http://aws.amazon.com/what-is-cloudcomputing/. 2. Haff, G. Introduction to Cloud Computing. 2013. 3. Mohammad Hamdaqa, L.T. Cloud Computing Uncovered: A Research Landscape. 2012. 41-85. 4. Infrastructure as a Service (IaaS) [cited 25 http://webcache.googleusercontent.com/search?q=cache:yncglwncnbsj:www.autonomicr esources.com/index.php/services/iaas&hl=en &strip=1. 5. Infrastructure as a Service (IaaS). 2010 [cited 23 http://searchcloudcomputing.techtarget.com/d efinition/infrastructure-as-a-service-iaas. 6. Platform as a Service (PaaS). 2010 [cited 23 http://searchcloudcomputing.techtarget.com/d efinition/platform-as-a-service-paas. 7. Software as a Service (SaaS). 2010 [cited 23 http://searchcloudcomputing.techtarget.com/d efinition/software-as-a-service. 8. What is Public Cloud? 2009 [cited 23 March 2013], Available from: http://webcache.googleusercontent.com/search?q=cache:e4e1ij- 68EsJ:searchcloudcomputing.techtarget.com/d efinition/publiccloud+&cd=3&hl=en&ct=clnk. 9. What is Private Cloud? 2008 [cited 23 March 2013], Available from: http://www.webopedia.com/term/p/private_ cloud.html. 10. What is hybrid cloud? 2013 [cited 30 March 2013], Available from: http://apprenda.com/library/glossary/hybridclouds-a-definition. 11. Longbottom, R. Linux PC s. 2012 [cited 5 May 2013], Available from: http://www.roylongbottom.org.uk/linux%20be nchmarks.htm. 699 698