CLOUD is a metaphor used for the internet and. Network-based Measurements on Cloud Computing Services. Yin Zhang

Transcription

1 Network-based Measurements on Cloud Computing Services Vinod Venkataraman Ankit Shah { vinodv, ankit, yzhang cs.utexas.edu Department of Computer Sciences Yin Zhang The University of Texas at Austin, Austin, TX Abstract Cloud computing is widely believed to be a revolution in computing that could soon become an industry standard, altogether replacing the traditional office setup. Due to the recency of these services, question marks exist over the performance of these systems, and consequently, over the corresponding billing schemes that the service provider issues. This study aims at addressing some of these doubts by introducing a measurement test for the network performance of these services, and comparing them with the performance offered by traditional web-hosting services. To do so, we study the response of one such service - Google App Engine - under light, medium and heavy loads generated by Planetlab nodes, and attempt to infer the overall performance of the systems. Our tests indicate that despite offering better round-trip times and throughputs, App Engine appears to consistently lose large amounts of the data that it is required to send to the clients. We explore this problem, and offer inferences that might explain this erratic behavior. Index Terms cloud computing, App Engine, EC2, measurement, round-trip time, bandwidth, throughput, data loss, packet loss I. Introduction CLOUD is a metaphor used for the internet and is an abstraction for the complex infrastructure that it conceals. Cloud computing [4] is a general concept that incorporates software as a service (SaaS), Web 2.0 and other recent, well-known technology trends, in which the common theme is reliance on the Internet for satisfying the computing needs of the users. A SaaS (Software as a Service) [9] application runs entirely in the cloud, using the servers of the service provider over the internet. The client is a simple browser or some other simple client. For example, GoogleApp Engine provides common business applications online that can be accessed from a browser, while the software and the data are stored on Google s server. The potential of cloud computing is undoubted, if suitably deployed. Speculations are that it may lead to a new revolution in computing that could become the industry standard [7]. Analogous to how very few people today prefer to build a house on their own, but rather prefer to rent one, in the next generation of computing, people may prefer to opt for a scalable and reliable provider for their computing needs, that will actually minimize risks while launching a new application, rather than build an entire new enterprise for the purpose of launching products. Our motivation for this measurement study stems from the hype created around the Cloud Computing concept. Although there is much talk about the advantages of using the cloud, there is no existing measurement study to validate the claims. Also no explicit comparisons have been made between the performance of a cloud computing service and that of a traditional web hosting service. Therefore, we study the performance of the Cloud Computing service under varied load conditions to answer if the hype created is actually justifiable. With regards to Cloud Computing, we can classify measurement studies into two broad categories: computation-based measurements and network-based measurements. The computation-based measurements include CPU/Process cycles, Memory/Storage, language engine performance (for example: Performance of Python Engine in the case of Google App Engine). These measurements can only be made at the server level and hence are taken by the service providers themselves or by authorized third parties. The main focus of our work is on network-based measurements of the Cloud Computing service. The three important metrics that we shall be analyzing for the cloud are Round-trip time (RTT), Network Throughput, and Data Loss, using the measurement tool httperf [3]. We attempt to verify the claims made by the service providers of the cloud computing services with regards to scalability and performance. In order to get a

2 2 better insight, we have used the Planetlab testbed to measure the performance and scalability of the servers of the service providers under the varying load conditions. We deployed our image retrieval application on the Google s AppEngine, and measured the performance in retrieving small (12kb), medium (350kb) and large(1 MB) images*. We then tested the performance by sending multiple parallel requests from the Planetlab nodes. We repeated the same tests on an ordinary web-host to compare the cloud computing service to that of a traditional web host. The detailed implementation and the evaluation of the results is discussed later in the paper. The remainder of the paper is organized as follows. We document the related research in Section II. Section III describes our methodology for the project. Section IV discusses the implementation, followed by the evaluation in Section V. We present the future work in Section VI and our findings are summarized in conclusion in Section VII. *Note that the image size limitation of 1 MB is because of Google App Engine s restriction on file sizes. II. Related Work Cloud computing is a relatively new concept, and the current services are very nascent. As a result, a very limited amount of literature is also available in the area. Furthermore, no clear standards exist in this industry, and hence each service provider has its own definitions for resource usage. As discussed earlier, measurements on a cloud computing service can be broadly classified into emphcomputationbased measurements and emphnetwork-based measurements. Our work falls in the latter category. Complementary to our research is the computationbased measurement which includes CPU/Process cycles, Memory, Storage, language engine performance and these require access to the servers of the service provider, and hence are carried out by the servers or authorized third parties like Hyperic. Hyperic has a subsidiary company called as that are the authorized third party measuring services for the cloud platforms like Googles AppEngine, Amazons EC2. CloudStatus continuously monitors the performance of the Google AppEngines Datastore, Memory Cache and Engine performance and reports the health of the system. III. Methodology This section presents an overview of the current state of cloud computing systems, the kinds of measurement tests that can be performed on these systems, and the techniques we use for structuring our test model. A. Current state of Cloud Computing Services There are a number of cloud computing services in the market today, each offering a variety of services - ranging from limited, but powerful tools like Google App Engine [1] offers, to the complete server solution that Amazon EC2 [5] offers. Amazon was one of the first companies to launch a cloud computing service for the general public, and it continues to have one of the most sophisticated and elaborate set of options. For an application, developers can create virtual machines called Amazon Machine Instances (AMIs) with Elastic Compute Cloud (EC2). For storing data, objects of up to 5GB can be stored in the Simple Storage Service (S3). Amazon has also built a limited database on top of the S3. The AMIs deployed can communicate among themselves with the Simple Queue Service (SQS), a message-passing API. Most of the tasks performed on the Amazon cloud need a command line. Amazon has a large set of tools with sophisticated security options for sending orders to AMI collection, all run from the command line. There are now numerous cloud computing services providing features comparable to Amazon EC2. Some of these competitors include Mosso [10], GoGrid and AppNexus. Google App Engine offers services that are, in some ways, exactly opposite to Amazon s services. While EC2 provides root privileges, App Engine does not even allow file write options, or much flexibility in storing files and folders. Google removes the file write feature out of Python, presumably as a quick measure to avoid security holes. However, App Engine does provide its own datastore, complete with a custom built query language termed Google Query Language (GQL) modeled on SQL. All data writes and accesses are expected to be performed using this datastore. Although at first sight, this seems restrictive, the framework provided by App Engine provides the core features on which powerful, content-rich web applications can be built. Further, App Engine provides users with tools to debug the application on a local machine as well. The API that App Engine provides

3 VENKATARAMAN, SHAH, ZHANG: NETWORK-BASED MEASUREMENTS ON CLOUD COMPUTING SERVICES 3 makes it ideal for individual and small groups of developers who can code simple database front-ends using Python, although they are likely to expand it for larger enterprises eventually. B. Measurement Tests The importance of tests on these services is undebatable - it is the sole means of describing a pricing model for a cloud computing system. Currently, due to the vast differences in standards between each cloud computing service, there is no standard pricing scheme for a system. Most services, however, do use variations of some popular metrics for pricing. These include CPU utilization, storage space, memory usage, network bandwidth, etc. Tests on cloud computing systems can broadly be categorized into two types: 1. Computation-based tests 2. Network-based tests The former category deals with tests on the actual computational performance of the machines used to run the application on the cloud. Some of these are standardized, such as storage space and memory usage. However, each vendor specifies their own limits and mechanisms for computing CPU utilization. Google App Engine specifies this in terms of Megacycles used, a term that remains unclear for us. Amazon EC2 computes this metric in terms of the hours a machine instance has been deployed, and the number of such instances used. However, this category of tests remains out of reach for independent researchers, as it requires root access to the server itself. Such tests are usually performed either by the company itself, or by authorized third parties. One such authorized third party is Hyperic Inc., who have monitor the performance of both EC2 and App Engine in real-time and publish the results on their website CloudStatus [6]. The latter category is the one that our work focuses on. These tests measure the network performance of requests handled by applications deployed on a cloud. These include metrics such as round-trip time, network throughput, data loss, bandwidth, delay, latency, and many others. Of these, the metrics we chose to test on include the first three. A brief description of each of these metrics is provided below: Round-trip Time (RTT): RTT is defined as the time elapsed from the propagation of a message to a remote place and to its arrival back at the source. The choice of this metric is obvious - it provides the exact amount of time that a client accessing a web application would experience as delay in receiving the output of her query from the time of her input. Network Throughput: The average rate of successful data transfer through a network connection is known as network throughput. It is important to distinguish this term from network bandwidth, which is the capacity for a given system to transfer data over a connection. Although providers base their billing on bandwidth and not throughput, from a client s perspective, throughput is more important as it decides the data rate she receives for her request. Packet/Data loss: Packet loss occurs when one or more packets of data traveling across a computer network fail to reach their destination. Loss can be measured either as loss rate - which detects the amount of data in bytes or as packets lost per unit of time - or simply as loss - the amount of data in bytes that were lost during transfer. This metric is important as it places a quantitative test on the data that a client actually received from the server. It is important to note that none of these metrics can alone provide a general picture of the performance of the cloud computing service. This will be demonstrated in section Evaluation where each metric is analyzed individually. C. Testing Model Our test model to measure the network performance of the cloud computing service begins by deploying a web application on both the cloud as well as an ordinary web hosting service. The deployed application may be tailored specific to the service it is deployed on, but the performance of the application is assumed to be similar in both cases, as the time duration of access of the database of the system is negligible on comparison to the transfer times over the network. Next, we test the performance of the two services under light, medium and heavy loads using Planetlab nodes to launch parallel requests. The reasoning behind using Planetlab to generate parallel requests, rather than sending the same number of requests from a single node, is because the latter case tends to be serialized, which does not really test the ability of the service to handle simultaneous requests. Finally, we measure the performance of the service using httperf [3] to obtain the values for the roundtrip time, network throughput, the data loss.

4 4 IV. Implementation This section describes the details involved in the implementation of the test model specified in the previous section. (a) Small Image (12 kb) (b) Medium Size Image (350kB) A. The Web Application The web application deployed on Google App Engine was a simple image retrieval application. The front end, coded using Google App Engine s API [2] in Python, performs the tasks of retrieving a collection of static images from an online source, recoding and storing these images in the datastore, and presenting an HTML page to the user to request an image through an HTTP request, be it from a browser or from any other client. From a regular web host, ordinary GET requests to locations referencing static images are used. B. Measurement Tests The httperf binaries were deployed on 100 Planetlab nodes around the world and were used to perform tests under various different load conditions. These included sending single and multiple requests from individual nodes separately, as well as sending single and multiple requests from multiple nodes parallely. For the statistics presented in the next section, the 100 Planetlab nodes were programmed to send 1, 10 and 100 requests each in parallel and maintain logs for the results of these requests, so as to subject the servers to a wide spectrum of loads, while staying under the limit of the maximum bandwidths allowed by these servers. From the data registered in the logs of each of these servers, the values for round-trip time, average network throughput and percentage of data loss were obtained. V. Evaluation In this section, the results of the experiments conducted are evaluated and interpreted. Fig. 1. Round Trip Time (c) Large Image(1 MB) A. Metrics evaluation A.1 Round-Trip Time (RTT) The Round Trip Time gives the total end to end time, and hence is an important metric in evaluating the performance of the Cloud Computing service. Fig.?? shows that for small sized image (12 kb), the performance of the Google App Engine is a touch better as compared to the traditional web host. However, the picture becomes more clear as we go on to

5 VENKATARAMAN, SHAH, ZHANG: NETWORK-BASED MEASUREMENTS ON CLOUD COMPUTING SERVICES 5 the medium sized image (350kB). The second graph shows that the RTT for Google App Engine is an order of magnitude faster as compared to the traditional web host. The third graph shows that for large images(1 MB), the performance of the App Engine is a couple of orders of magnitudes faster as compared to the traditional web host. Thus, given the results of the RTT only between the App Engine and the traditional web host,the hype created for using the cloud seems to be justifiable. However, we shall evaluate the results over different metrics to get a better understanding of performance and scalability in the cloud. (a) Small Image (12 kb) (b) Medium Size Image (350kB) (c) Large Image(1 MB) Fig. 2. Throughput A.2 Network Throughput Now, we discuss throughput and hence the scalability of an application deployed on the Cloud Computing Service. Scalability means that the throughput available for an application should increase if the load/requests for the application increase. Fig. 1 shows that in the case of small images(12 kb), there is not much of a difference between the Google s App Engine and the traditional web hosting service, Synthasite. However, in the case of a medium sized image (350 kb), Google App Engine clearly seems to have a better bandwidth as compared to Synthasite, however Synthasite seems to scale well under increasing load. The difference in bandwidth and scalability is more pronounced in the case of large images (1 MB), where the traditional web hosting service like Synthasite scales ideally for increased loads whereas Google s App Engine does not seem to scale. From the results we can conclude that as regards bandwidth, Google s Cloud Computing Platform definitely has more bandwidth as compared to a traditional web host but it does not seem to scale as well as the traditional web host, Synthasite in this case. A.3 Data Loss Now, having seen the impressive results of RTT, its time for a reality check. The Data/Packet loss is measured in percentage and gives the amount of data that has not been accounted for when the RTT gets calculated. For example, in the case of transferring a image of X kb and sending 100 request per Planetlab node, we see that each fo the request returned from the Planetlab node as y. The percentage of data loss is given by [1-(x/y)]*100. We expect the data/ packet loss to be as low as possible, ideally 0. Fig.1 shows that even for small image(12 kb), Google App Engine gives an error rate of 12

6 6 B. Result Interpretation - What really happens? (a) Small Image (12 kb) (b) Medium Size Image (350kB) The values in Section A.3 indicate that App Engine performs exceedingly poorly under heavy loads, contrary to claims to the opposite by Google. On consideration, we realized that these results may not be indicative of App Engine s actual performance as our usage of httperf may not have been appropriate for this setting. This is elaborated as follows. At zero load, App Engine will not dedicate much server resource to an application, letting a single server monitor the application. When this server is subjected to an extremely heavy load, the single App Engine server appears to make connection and service every request that arrives to an application at least partially, regardless of the number and size. In the meantime, it appears to be calling for assistance from the other servers in the cluster in order to distribute the load efficiently. This would probably result in a delay in servicing a request for the client. When the client is a measurement tool like emphhttperf, under normal parameters of call, the client assumes that the server has completed the request and effectively times out before the backup servers arrive to continue processing the requests. On the other hand, with a more robust client like a browser, a slightly longer delay is permissible. In order to prove the above conjecture, we would have to conduct further tests using httperf by varying the time-out periods. C. Effect of Geographic Location Fig. 3. Data Loss (c) Large Image(1 MB) To test the effect of geographic location of the client on the performance of App Engine served results over a set of 16 nodes in locations from Europe, Asia, South and North America. These tests indicated that the slowest performers turned out to be from third-world countries with poor network bandwidth. Although these clients fared poorly with the ordinary web-host, they managed to complete the entire transfer of data. On the other hand, their performance with App Engine was so poor, that the requests did not even manage to complete, and timed out essentially. The inference is that countries with lower bandwidth availabilities should stick to ordinary local web-hosts for clients in their own countries, and may use App Engine to serve clients abroad.

7 VENKATARAMAN, SHAH, ZHANG: NETWORK-BASED MEASUREMENTS ON CLOUD COMPUTING SERVICES 7 VI. Future Work We believe that a commercial giant like Google would not market a product that performs so poorly on the critical issue of data loss. Hence a part of the future work would be exploring the data loss phenomenon further by tweaking with the tool emphhttperf so as to adjust the timeout value so as to wait for the Google s servers as they come in to balance the increasing load. Also, there are metrics other than we have studied that can help decide the performance of the Cloud Computing Service, for example, Latency, i.e the time between the image being requested and the image being retrieved. Also, our application has been deployed only on the Google s App Engine. However, we can deploy it on any other cloud computing service by making use of their APIs. This would help us compare the performance of the various service providers. Finally, we need to test the performance of the Cloud Computing services for some real time applications. [6] Hyperic CloudStatus, [7] Rajkumar Buyya, Chee Shin Yeo, and Srikumar Venugopal, Market Oriented Cloud Computing : Vision, Hype and Reality for delivering IT Services as Computing Utilities [8] Twenty One Experts define Cloud Computing, [9] Cloud Computing: The Evolution of Software-as-a-Service, [10] Mosso Cloud Hosting, [11] What cloud computing really means, 1.html VII. Conclusion In this paper, we have introduced the concept of network-based measurement on cloud computing services and justfied their importance from a client point of view. To this end, we have developed a vendor-independent network-based measurement test on cloud computing services, using freely available open-source tools and resources for testing and validation. We have tested the performance of Google App Engine under varying load conditions using Planetlab, and compared it quantatively to the performance of an ordinary web-host handling the same requests. This paper presented a premliminary idea of the kind of tests that can be performed on cloud computing services, and further testing. VIII. Acknowledgments We would like to thank Professor Yin Zhang, our project guide, for his invaluable advice and feedback without which this project would not have been a success. We are also thankful to Professor Mike Dahlin and Han Hee Song for their help with Planetlab. References [1] Google App Engine, [2] Documentation for Google App Engine, [3] Httperf tool, [4] Wikipedia - Cloud Computing computing/ [5] Amazon Elastic Cloud,