Modeling and Simulation Frameworks for Cloud Computing Environment: A Critical Evaluation

1 Modeling and Simulation Frameworks for Cloud Computing Environment: A Critical Evaluation Abul Bashar, Member, IEEE Abstract The recent surge in the adoption of Cloud Computing systems by various organizations has brought forth the challenge of evaluating their performance. One of the major issues faced by the cloud service providers and customers is to assess the ability of Cloud Computing systems to provide the desired services in accordance to the QoS and SLA constraints. To this end, an opportunity exists to develop means to ensure that the desired performance levels of such systems are met under simulated environments. This will eventually minimize the service disruptions and performance degradation issues during the commissioning and operational phase of Cloud Computing infrastructure. However, it is observed that several simulators and modelers are available for simulating the Cloud Computing systems. Therefore, this paper presents a critical evaluation of the state-of-the-art modeling and simulation frameworks applicable to Cloud Computing systems. It compares the prominent simulation frameworks in terms of the API features, programming flexibility, Operating System requirements, supported services, licensing needs and popularity. Subsequently, it provides recommendations regarding the choice of the most appropriate framework for researchers, administrators and managers of Cloud Computing systems. Index Terms Cloud Computing, Modeling Framework, Performance Evaluation and Simulation Tools I. INTRODUCTION THE rapid development in the area of information and computing technology has led to the emergence of new model termed as Cloud Computing. According to the National Institute of Standards and Technology (NIST) definition, Cloud Computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction [1]. This model is realizable through the use of service and deployments models. The three basic service models include provisioning of Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and as a Service (SaaS). Deployment models deal with the way in which the infrastructure of the Cloud Computing system is deployed, owned and managed by the service providers & consumers. This leads to the concept of Private, Public, Community and Hybrid cloud architectures [2]. A typical service architecture of a Cloud Computing system is depicted in Fig. 1. It is seen that, the Cloud Computing system has a layered architecture with the Hardware Layer at the datacenter forming the foundation of the overall system. A. Bashar is with the College of Computer Engineering and Sciences, Prince Mohammad Bin Fahd University, Al-Khobar, Saudi Arabia, 31952 e- mail: (abashar@pmu.edu.sa). It actually consists of the physical hardware devices including the CPU, Memory, Storage and Bandwidth which are physically resident on a server farm. Utilizing the physical hardware the Infrastructure Layer, incorporates the virtualization technology to provide the Infrastructure as a Service (IaaS) (e.g. Amazon Web Services (AWS) [3] and Rackspace [4]) which usually consists of a pool of virtual machines (VMs) that can be provisioned on demand to the IT consumers. Since the IT resources of the physical layer are limited, virtualization is a very important technology which gives the IT consumers a notion of unlimited computing resources at their disposal. Next comes the Platform Layer which provides the platform for creation and development of software which can be later delivered over the web. Hence, this layer facilitates and provides the Platform as a Service (PaaS) (e.g. Google App Engine [5] and Windows Azure [6]) by utilizing the components and services of the infrastructure layer. This includes the services to collaboratively develop, test, deploy, host and maintain software and applications in a common integrated development environment. Finally, the topmost layer is the Layer which provides the avenue for provisioning the ready-to-use software and application for the business needs of the Cloud-based IT consumers. Hence, this layer facilitates and provides the as a Service (SaaS) (e.g. Google Apps [7] and Salesforce [8]) by utilizing the components and services of the platform layer. SaaS is essentially a service which provides a means to access a software or an application through the web. Some of the benefits of such a service is that software is managed from a central location, the users are not required to perform software updates and different pieces of software can be integrated with Application Programming Interfaces (APIs). The benefits of Cloud Computing to the enterprises are the reduction in the CAPEX and OPEX costs of the IT and computing equipments. The physical IT equipments need not be purchased at a high cost, instead they can be purchased as a service on a pay-as-you-go basis. Since the software is purchased from the Cloud Service Provider as a service, the customers need not be concerned about software updates and upgrades. The storage and memory are no longer a limited resource in a Cloud Computing model, as the customers now have the notion of unlimited resources on-demand. The data of the customers is now more reliably stored on the Cloud as there is no fear of losing the data in case of hardware failures at customer premises. Since the customer data is stored centrally, it allows for a easier group collaboration among the employees of the organization. Also, the data can be accessed from various independent devices as the data is not tied to a particular access device.

2 Fig. 1. Cloud Computing Service Architecture The various benefits mentioned above have to be properly tested before they can be used by the customers. At the same time, the service providers need to test the various service models so that they can achieve maximum profit. Moreover, Cloud-based applications have varying composition, configuration and deployment requirements. Hence, quantifying the performance of these Cloud applications under varying demands is a challenging task. This problem is especially escalated when the Cloud datacenter is in the operational state. This could be risky from the point of view of service providers and customers alike, since any degraded performance will result in the customers moving away from the service providers. The solution to this problem is to utilize simulation tools that can evaluate the performance of Cloud applications before being deployed in a real setup. The benefit of this approach is that services can be tested in a repeatable and controlled environment free of cost. Also, this gives the opportunity to tune the performance bottlenecks before deploying on real Clouds [9]. However, due to the availability of numerous simulation tools for simulating Cloud Computing environments it is essential that a critical evaluation of such tools be done in order to choose an appropriate tool for proper simulation and evaluation. The work presented in this paper basically answers the following research questions, which, to our knowledge, have not been addressed before, and hence we claim the novelty of this research. What are the prominent Cloud Computing simulation tools available? What is the possible set of criteria to evaluate a Cloud Computing simulation tool? How to make a choice of a particular simulation tool based on a set of requirements? The remainder of this paper is structured as follows. In Section II we provide the required background and survey some related work in this research domain. In Section III we present the evaluation framework for Cloud simulation tools. We then present the results and discussion in Section IV. Section V concludes the paper by suggesting possible future work. II. BACKGROUND AND RELATED WORK In order to make sure that the Cloud computing infrastructure is able to provide the services according to the agreed standards and the services are provisioned with the desired quality, it is essential that they be tested before being deployed. This can be achieved by using simulation and modeling tools which can test the infrastructure, services and applications under varying demand conditions. In the presence of numerous simulators and modeling tools, it is necessary to perform a comprehensive survey and critical evaluation of them. This section therefore presents the work which already exists in the literature in the area of survey and review of modeling and simulation tools for testing the performance of Cloud Computing systems. Zhao et al. indicate that there exists two types of cloud computing simulators, namely, simulators based only on software and simulators based on both software and hardware [10]. They consider eleven simulators and mention their brief description along with their basic features. Finally, they provide a comparison based on the criteria of underlying platform, programming language and hardware/software composition. A very similar work by Oujani et al. provides a comparison among eight Cloud simulators (which are the same as in the paper of Zhao et al.) [11]. Their comparison criteria is also based on three features, namely, underlying platform programming language and hardware/software composition. In another review by Malhotra et al., the CloudSim simulator and all of its variants (CloudAnalyst, GreenCloud, NetworkCloudSim, EMUSIM and MDCSim) are described and compared [12]. They compare the variants of CloudSim based on the criteria of platform, programming language, networking feature, simulator type (event/packet based) and availability (open source/commercial). Another interesting paper by Sakellari et al., classifies the Cloud Computing study into three categories, namely, Mathematical Modeling of Cloud Systems, Cloud Simulation and Cloud Testbeds [13]. They clearly demonstrate the relevance and application of these approaches and their relative merits and drawbacks. For the Mathematical Modeling category, they evaluate the solutions based on the criteria of QoS/Performance and energy efficiency features. The Cloud Simulation are compared based on the criteria of energy efficiency feature, QoS, Programming language (Java/C++) and availability (open source/commercial). Finally, the Cloud Testbeds have been classified as Commercial (e.g. Amazon EC2 and S3, Google Apps), Scientific (e.g. Open Cirrus, Open Cloud) and Frameworks (e.g. Eucalyptus, OpenStack). These Cloud Testbeds are evaluated for the services which they can support, namely, IaaS, PaaS and SaaS and their availability (open source/commercial). Based on this survey, it is seen that there is limited research on the review of modeling and simulation approaches for testing the performance of Cloud Computing systems. Another point which is to be noted is that all the review generally provide the information about various available approaches and their features. They do not however provide clear

3 Criteria Provider License Category API OS Services Popularity Comments TABLE I DEFINITION OF EVALUATION CRITERIA Description Organization(s) involved in the development of the simulator License requirements (e.g. Open source, Commercial, Proprietary Membership) The category of the simulation tool (e.g. Simulation, Testbed (Commercial, Scientific or Framework)) Type of Application Programming Interface provided in the simulator The Operating Systems which support the installation of the simulator The type of Cloud Services supported by the simulator (e.g. IaaS, PaaS, SaaS) Number of search results on Google Scholar. Simulator with maximum count has score of 10 Special notable feature or property of the simulator guidelines as to which approach is suitable for a particular situation. Hence we consider a complementary review of the existing approaches, the novelty of which is to provide a ranking of these approaches based on their popularity in the research community. The following section describes the various modeling and simulation approaches and proposes a new evaluation framework. III. PROPOSED EVALUATION FRAMEWORK In order to critically evaluate and compare the existing Cloud Simulation and Modeling tools, it is proposed to develop an evaluation framework. This framework consists of a set of criteria which will be used to compare the various approaches and the classification categories. Due to the existence of many simulation/modeling tools, it was decided to shortlist and make a reduced set of such tools. In this paper only 10 such tools are presented. They are Eucalyptus, ns2, CloudSim, Opnet, GreenCloud, OpenStack, Open Cloud, Open Cirrus, CloudAnalyst and icancloud. A. Criteria Description Considering the criteria in the existing literature from the related work section (Section II) and proposing new criteria, the criteria set in this research is given in Table I. The description for most of the criteria (Provider, License, Category, API, OS, Services and Comments) are clear and self explanatory. However, the Popularity criteria is explained here in detail. The various tools (see Table II) were first searched on Google Scholar and the number of search results were noted for each of them. To bring out a relative comparison, a Popularity Index was proposed on the scale of 0 to 10. The simulator which had the maximum count was assigned a score of 10 (which in this case was Eucalyptus). The Popularity Index of other simulators were calculated as ((Search Count/Maximum Search Count)*10). B. Simulation Tools Description This subsection provides a concise description of 10 Cloud Computing simulators and modeling solutions which have been chosen for critical evaluation and comparison. 1) Eucalyptus: It is an open source software framework for cloud computing that implements Infrastructure as a Service (IaaS) [14]. Eucalyptus gives users the ability to run and control virtual machine instances deployed across a variety of physical computing resources. It provides compatibility with popular Amazon Web Services (AWS) APIs including Elastic Cloud Compute (EC2) and Simple Storage Service (S3). 2) ns2: It is a discrete-event network simulator, primarily used in research and teaching [15]. It is a free software, publicly available under the GNU GPLv2 license for research, development, and use. Even though ns2 is a very accurate and popular simulator in the area of computer network simulations, it has been found that it is not very much suitable for simulating Cloud Computing systems. However, it is found that some researchers have used ns2 in their work on Cloud Computing simulations, but it is observed that they only have limited features. It is to be noted that, a Cloud simulator named GreenCloud is based on ns2 platform. 3) CloudSim: It is an extensible simulation toolkit that enables modeling and simulation of Cloud computing environments [9]. The CloudSim toolkit supports modeling and creation of one or more virtual machines (VMs) on a simulated node of a Data Center, jobs and their mapping to suitable VMs [16]. It also allows simulation of multiple Data Centers to enable a study on federation and associated policies for migration of VMs for reliability and automatic scaling of applications [17]. 4) Opnet: It is a commercial network simulator which performs application performance management in order to deliver the application performance to the users and to satisfy business demands [18]. From the point of view of Cloud computing systems, Opnet does not have the capability to implement and test Infrastructure as a Service, however, it does have the capability for application testing for Cloud systems [19]. 5) GreenCloud: It is a simulation environment for energyaware cloud computing data centers based on the ns2 platform. The simulator is designed to capture details of the energy consumed by data center components (servers, switches, and links) as well as packet-level communication patterns in realistic setups and workload distributions [20]. 6) OpenStack: It is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface [21]. It has been developed as a Linux-based collaborative project with three strands, namely, compute, object storage and image service. It provides compatibility with popular commercial cloud providers such as Amazon Web Services, Rackspace and HP Cloud. 7) Open Cloud: It is a cloud testbed with over 30 members including Cisco, NASA, and universities from the United States and Japan, running projects primarily in the areas of Big Data cloud computing middleware [22]. Open

4 Cloud is using wide area high performance networks to connect their four data centers located in the United States. It is a system which provides all the three Cloud services, namely, IaaS, PaaS and SaaS. 8) Open Cirrus: It is a very large cloud testbed comprising of federated heterogeneous distributed data centers [23]. Open Cirrus is a joint initiative sponsored by Hewlett Packard, Intel and Yahoo in collaboration with more than eight other organizations and universities around the world [24]. It is considered to be the largest testbed composed of ten sites in North America, Europe and Asia and consists of several thousand cores and associated storage. 9) CloudAnalyst: It is a tool to simulate large-scale Cloud applications with the purpose of studying the behavior of such applications under various deployment configurations [25]. CloudAnalyst helps developers with insights in how to geographically distribute applications among Cloud infrastructures and value added services, such as optimization of applications performance. It is based on the CloudSim platform and provides GUI feature to perform easier study of Cloud applications. 10) icancloud: It is a simulation platform which is oriented towards the simulation of a wide range of cloud computing systems and their underlying architectures [26]. It has the ability to model and simulate large environments (thousands of nodes) and distributed applications with a customizable level of detail. icancloud simulator uses the popular OMNET++ framework used for simulating computer networks. The following section provides the results on the comparison and evaluation of the above mentioned Cloud simulation and modeling tools based on the criteria discussed in the earlier part of this section. IV. RESULTS AND DISCUSSION This section provides the details of the evaluation of the Cloud Computing simulation and modeling tools under the proposed evaluation framework as described in the previous section. The main results and their explanation are provided below and the comparison summary is presented in Table II. (a) Category Distribution: As seen from Table II, the 10 tools are distributed in three categories, namely, Simulation, Scientific Testbed and Framework. About 60% of these tools belong to the Simulation category, and 20% each are in the Scientific Testbed and Framework categories. It is seen that Simulation category is the most popular way of studying the Cloud Computing environment, since it is very economical and easiest tool to do research. (b) Academic Initiatives: Another observation is that about 70% of the tools have been academic initiatives by researchers in a university environment. However, only 30% of the initiatives have been from the industry. The reason for this distribution is that it takes less investment and time to initiate an university project and the industrial projects on the other hand can be very expensive and require more time to setup because of collaboration and legal constraints. (c) Licensing Issues: It is interesting to note that 70% of the tools are open source and the remaining 30% are either commercial or require propriety memberships to use the tool. The reason for this is that open source tools have wide applicability in the research community and promote collaboration environment. However, commercial tools have the advantage that they are more sophisticated and have full functionality to study the Cloud Computing environment. (d) Application Programming Interface: In order to gauge the level of programming required to investigate a simulation scenario, it is important to see the type of APIs supported in these tools. It is observed that about 30% of the tools use Java, 40% use C++ and the remaining use a mixture of Java, C++, Python etc. It is concluded that object oriented programming languages constitute a majority of the share. (e) Operating System: It is observed that almost 80% of the tools have been tested on the various flavors of Linux operating system. This is because Linux is available free of cost, much more secure than Windows OS, flexible in terms of choosing the hardware, software (open source) and applications. (f) Cloud Service: The services distribution is that all the tools support IaaS, which includes all the three categories of the tools. However, only about 20% of them support all the three services (IaaS, PaaS and SaaS). It is to be noted that only the tools which are implemented as testbeds have the features to support all the three services. (g) Recommendations: Based on the results in Table II and the discussion in the points above, it is recommended to use Eucalyptus software framework as the first choice in studying the performance of Cloud Computing systems. This is because it has been found to be the most popular among the 10 tools evaluated in this research. However, it is to be noted that this is only a software framework and arrangements for hardware (PCs, servers and network equipments) need to be made. Even though, ns2 comes at second position in popularity, it was found that it does actually support any of the three Cloud Computing service models. So, the second popular choice is CloudSim which is a simulation software and is a viable solution if access to hardware is limited. The third choice is to go for GreenCloud which is a simulation software specialized in energy efficiency of Cloud Computing environments. Opnet was ruled out for the same reason as that of ns2. This study has concluded that CloudSim is the best choice if research has to be done by using a simulation software and Eucalyptus is the best choice if enough hardware is available to setup a private cloud and use the software framework to manage it. V. CONCLUSION This paper addressed the need for providing a framework for evaluation and comparison of modeling and simulation tools for Cloud Computing environments. To this end, it proposed

5 TABLE II COMPARISON SUMMARY OF CLOUD SIMULATORS Simulator Provider License Category API OS Services Popularity Comments Eucalyptus Eucalyptus Systems Open source Java/C Linux, IaaS 10.00 AWS compatible Pri- Inc. Framework Windows vate Cloud ns2 University of Open source Simulation C++/OTcl Linux N/A 2.78 Extended in Green- South California Cloud simulator CloudSim University of Open source Simulation Java Linux, IaaS 2.50 Generalized and extensible Melbourne Windows, framework Mac Opnet Opnet Inc. Paid Simulation C/C++ Windows, N/A 2.35 Limited application Linux testing for Cloud GreenCloud University of Open source Simulation C++/OTcl Linux IaaS 1.79 Based on ns2, focus OpenStack Luxembourg Group of Companies Open Cloud Group of US Universities Open Cirrus Group of Universities and Companies CloudAnalyst University of Melbourne icancloud University of Madrid Open source Propriety Membership required Propriety Membership required Open source Open source Framework Scientific Testbed Scientific Testbed Simulation Simulation on energy efficiency Python Linux IaaS 1.36 Compatible with Amazon Web Services Hardware Hardware IaaS, 1.35 Appropriate for testing based based PaaS, Cloud applica- Hardware based Hardware based SaaS IaaS, PaaS, SaaS tions 0.34 Focus on testing Federated Cloud Datacenters Java Linux, IaaS 0.23 Extension of Windows, CloudSim with Mac GUI C++ Linux IaaS 0.09 Uses OMNET++ framework and implemented a framework which consisted of six criteria, namely, API feature, Operating Systems requirements, supported services, tool category, licensing needs and popularity in the research community. The paper considered ten most popular simulation software, software frameworks and testbed solutions and performed the evaluation and comparison based on the six criteria. A novel aspect of this work was to rank the simulation tools in order of their popularity and acceptability in the Cloud Computing research community. One of the conclusions was to choose Eucalyptus as the simulation tool, if there is an adequate availability of computing and networking hardware and use Eucalyptus software framework to manage the private cloud testbed. Also, in the absence of hardware, CloudSim simulation software is the best choice to simulate various performance evaluation experiments. As a future work, it is planned to investigate the features of CloudSim and Eucalyptus and compare them in detail by simulating Cloud Computing services scenarios. ACKNOWLEDGMENT The authors would like to acknowledge the support of Prince Mohammad Bin Fahd University, KSA for providing the facilities to perform this research work. REFERENCES [1] A. Mell and T. Grance, The NIST Definition of Cloud Computing, Recommendations of the National Institute of Standards and Technology, vol. 800-145, pp. 1 7, 2011. [2] Q. Zhang, L. Cheng, and R. Boutaba, Cloud computing: state-of-the-art and research challenges, Journal of Internet Services and Applications, vol. 1, no. 1, pp. 7 18, May 2010. [3] Amazon Inc., Amazon Web Services, http://aws.amazon.com/, (Last accessed : Jan. 2014). [4] Rackspace Inc., Rackspace: The open Cloud Company, http://www.rackspace.com/, (Last accessed : Jan. 2014). [5] Google Inc., Google App Engine, http://code.google.com/appengine/, (Last accessed : Jan. 2014). [6] Microsoft Inc., Windows Azure, http://www.microsoft.com/windowsazure/, (Last accessed : Jan. 2014). [7] Google Inc., Google Apps, http://www.google.com/enterprise/apps/business/, (Last accessed : Jan. 2014). [8] Salesforce Inc., Salesforce, http://www.salesforce.com, (Last accessed : Jan. 2014). [9] R. Buyya, R. Ranjan, and R. N. Calheiros, Modeling and simulation of scalable cloud computing environments and the CloudSim toolkit: challenges and opportunities, in 7th International Conference on High Performance Computing and Simulation, June 2009, pp. 1 11. [10] W. Zhao, Y. Peng, F. Xie, and Z. Dai, Modeling and simulation of cloud computing: A review, in IEEE Asia Pacific Cloud Computing Congress (APCloudCC 2012), Nov. 2012, pp. 20 24. [11] A. Oujani and R. Jain, A Survey on Cloud Computing Simulations and Cloud Testing, http://students.cec.wustl.edu/ azinoujani/, (Last accessed : Jan. 2014). [12] R. Malhotra and P. Jain, Study and Comparison of CloudSim Simulators in the Cloud Computing, SIJ Transactions on Computer Science Engineering and its Applications (CSEA), vol. 1, no. 4, pp. 111 115, Sep. 2013. [13] G. Sakellari and G. Loukas, A survey of mathematical models, simulation approaches and testbeds used for research in cloud computing, Simulation Modelling Practice and Theory, vol. 39, no. 0, pp. 92 103, Dec. 2013. [14] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov, The Eucalyptus open-source cloud-computing system, in 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID 2009, May 2009, pp. 124 131. [15] S. McCanne and S. Floyd, Network Simulator ns-2, http://www.isi.edu/nsnam/ns/, 1997, (Last accessed : Jan. 2014). [16] R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. F. D. Rose, and R. Buyya, CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, : Practice and Experience, vol. 41, no. 1, pp. 23 50, Jan 2011. [17] R. N. Calheiros, R. Ranjan, C. A. F. D. Rose, and R. Buyya, Cloudsim: a novel framework for modeling and simulation of cloud computing infrastructure and services, University of Melbourne, Tech. Rep., 2009.

[18] OPNET Technologies Inc., Opnet Modeler 16.0, http://www.opnet.com, (Last accessed : Aug 2013). [19] A. Bashar, Autonomic Scaling of Cloud Computing Resources using BN-based Prediction Models, in IEEE International Conference on Cloud Networking (CLOUDNET 2013), Nov. 2013, pp. 200 204. [20] D. Kliazovich, P. Bouvry, Y. Audzevich, and S. U. Khan, GreenCloud: A Packet-Level Simulator of Energy-Aware Cloud Computing Data Centers, in IEEE GLOBECOM 2010, Dec. 2010, pp. 1 5. [21] X. Wen, G. Gu, Q. Li, Y. Gao, and X. Zhang, Comparison of open-source cloud management platforms: Openstack and Opennebula, in IEEE International Conference on Fuzzy Systems and Knowledge Discovery (FSKD 2012), May 2012, pp. 2457 2461. [22] R. L. Grossman, Y. Gu, M. Sabala, C. Bennet, J. Seidman, and J. Mambretti, The Open Cloud Testbed: a wide area testbed for cloud computing utilizing high performance network services, in Gridnets 2009, Sep. 2009, pp. 89 97. [23] A. I. A. et al., Open Cirrus: a global cloud computing testbed, IEEE Computer, vol. 43, no. 4, pp. 35 43, 2010. [24] R. C. et al., Open Cirrus Cloud Computing testbed: federated data centers for open source systems and services research, in USENIX Hotcloud 09, June 2009, pp. 1 5. [25] B. Wickremasinghe, R. N. Calheiros, and R. Buyya, CloudAnalyst: A CloudSim-Based Visual Modeller for Analysing Cloud Computing Environments and Applications, in 24th IEEE International Conference on Advanced Information Networking and Applications (AINA 2010), Apr. 2010, pp. 446 452. [26] A. Nez, J. L. Vzquez-Poletti, A. C. Caminero, G. G. Casta, J. Carretero, and I. M. Llorente, icancloud: A Flexible and Scalable Cloud Infrastructure Simulator, Journal of Grid Computing, vol. 10, no. 1, pp. 185 209, Mar. 2012. 6