Simulation of Cloud Computing Eco-Efficient Data Centre

Simulation of Cloud Computing Eco-Efficient Data Centre Ibrahim Alzamil MSc Computing and Management Session (2011/2012) The candidate confirms that the work submitted is their own and the appropriate credit has been given where reference has been made to the work of others. I understand that failure to attribute material which is obtained from another source may be considered as plagiarism. (Signature of student) Ibrahim Alzamil

Abstract Cloud Computing has created a new era in the IT industry, offering businesses and individuals technical and economical benefits, such as utilising computing resources through virtualisation; reducing operational costs by moving the cost of ownership to the Cloud providers, and paying only for what they use. However, the data centres that underpin the Clouds consist of huge computing resources that consume large amounts of power. A data centre can even consume the same amount of electricity that is consumed by a city. Thus, maintaining data centres can incur huge operational costs for the providers. Moreover, this would increase the impact on the environment in terms of CO2 emissions. Therefore, this project will evaluate some techniques, namely DVFS and VM migration, which can be deployed to enhance the eco-efficiency of data centres. Through using the CloudSim simulator, a number of experiments will be conducted to investigate the efficiency of these techniques in terms of reducing energy consumption. I

Acknowledgement First and foremost, I thank Allah for granting me knowledge, health, and the patience to successfully complete this project. I would also like to show my gratitude and appreciation to the following people: Dr. Karim Djemame, my supervisor, for his generous and constructive feedback and continuous encouragement throughout the project. Dr. Raymond Kwan, my assessor, for his critical and constructive feedback for the interim report and valuable comments in the progress meetings. I also wish to take this opportunity to thank Dr. Mariam Kiran for her advice about where to start with learning to use CloudSim. Special thanks and gratitude go to all my colleagues and friends, especially Abdulaziz Albalti and Khalid Alhammad, who have been there cheering me on and supporting me. Finally, I would like to express my indebtedness to my parents, my brothers, and my sisters who have always been supportive and encouraged me morally. II

List of Acronyms DVFS: Dynamic Voltage and Frequency Scaling WBS: Work Breakdown Structure NS2: Network Simulator IT: Information Technology SOA: Service-Oriented Architecture SIP: Strategic Inflection Point PC: Personal Computer SLAs: Service Level Agreements SaaS: Software as a Service PaaS: Platform as a Service IaaS: Infrastructure as a Service OS: Operating System DDoS: Distributed Denial of Service TCO: Total Cost of Ownership CO2: Carbon Dioxide ICT: Information and Communication Technology QoS: Quality of Service PUE: Power Usage Effectiveness CCTV: Closed Circuit Television ERE: Energy Reusable Efficiency III

CPU: Central Processing Unit VM: Virtual Machine PTM: Power and Thermal Management WM: Workload Monitor DPM: Dynamic Power Management IDEs: Integrated Development Environments MIPS: Millions of Instructions per Second RAM: Random Access Memory IV

List of Figures Figure 1. Four-Layer Cloud Architecture... 7 Figure 2. Layered Cloud Computing Architecture... 8 Figure 3. Initial Layered CloudSim Architecture... 17 Figure 4. Current Layered CloudSim Architecture... 18 Figure 5. An iterative approach for experiments development... 23 Figure 6. Comparison between energy consumption mean values for each experiments - Scenario 1... 27 Figure 7. A screenshot sample of using Excel to calculate mean values, standard deviation, and confidence interval... 71 Figure 8. Comparison between energy consumption mean values for each experiments - Scenario 2... 28 Figure 9. Comparison between energy consumption mean values for each experiments - Scenario 3... 30 Figure 10. Comparison between energy consumption mean values for each experiments - Scenario 4... 31 Figure 11. Comparison between energy consumption mean values for each experiments - Scenario 5... 32 Figure 12. Comparison between energy consumption mean values for each experiments - Scenario 6... 34 Figure 13. Comparison between energy consumption mean values for each experiments - Scenario 7... 36 Figure 14. Comparison between energy consumption mean values for each experiments - Scenario 8... 37 V

List of Tables Table 1. Typical Characteristics of Host, VMs, and Cloudlets... 25 Table 2. Results of Scenario 1 experiments with the deployment of DVFS mechanism... 67 Table 3. Results of Scenario 1 experiments without the deployment of power aware mechanism... 69 Table 4. Results of Scenario 2 Experiments with the deployment of DVFS mechanism... 83 Table 5. Results of Scenario 2 experiments without the deployment of power aware mechanism... 85 Table 6. Results of Scenario 3 Experiments with the deployment of DVFS mechanism... 87 Table 7. Results of Scenario 3 experiments without the deployment of power aware mechanism... 89 Table 8. Results of Scenario 4 Experiments with the deployment of DVFS mechanism... 91 Table 9. Results of Scenario 4 experiments without the deployment of power aware mechanism... 93 Table 10. Results of Scenario 5 Experiments with the deployment of DVFS mechanism... 95 Table 11. Results of Scenario 5 experiments without the deployment of power aware mechanism... 96 Table 12. Results of Scenario 6 Experiments with the deployment of DVFS mechanism... 98 Table 13. Results of Scenario 6 experiments without the deployment of power aware mechanism... 100 Table 14. Results of Scenario 7 Experiments with the deployment of VM migration mechanism with DVFS enabled... 102 Table 15. Comparison for energy consumption mean values between scenario 7 and 1... 35 Table 16. Results of Scenario 7 Experiments with the deployment of VM migration mechanism with DVFS enabled... 103 Table 17. Benchmarking of energy consumption between the deployment of DVFS mechanism and non power aware (NPA) mechanism in a data centre... 38 VI

Table of Contents Abstract... I Acknowledgement... II List of Acronyms... III List of Figures... V List of Tables... VI Table of Contents... VII 1. Introduction... 1 1.1. Overview... 1 1.2. Aim and Objectives... 1 1.3. Minimum Requirements and Further Enhancements... 2 1.4. Deliverables... 2 1.5. Motives and Relevance to the Degree Programme... 2 1.6. Methodology... 3 1.7. Initial Project Plan... 4 1.7.1. Key Activities... 4 1.7.2. Milestones... 5 2. Review of Literature and Technologies... 6 2.1. Introduction to Cloud Computing... 6 2.1.1. What is Cloud Computing?... 6 2.1.2. Architecture of Clouds... 7 2.1.3. Types of Clouds Services... 9 2.1.4. Types of Clouds Deployment... 9 2.1.5. Virtualisation... 10 2.2. Issues Associated with Clouds... 11 2.2.1. Cost of Clouds Maintenance... 12 2.2.2. Eco-efficient Data Centre Management... 12 2.2.2.1. A Scientific Model of Eco-efficient Data Centre... 13 2.2.2.2. Management of Power Consumption... 14 2.2.2.2.1. VM Allocation... 15 2.2.2.2.2. Power and Thermal Management... 15 2.2.2.2.3. Dynamic Voltage and Frequency Scaling... 15 2.3. CloudSim Toolkit... 16 2.3.1. Architecture of CloudSim... 16 2.3.2. Usability... 18 2.3.3. Capabilities... 18 2.3.4. Limitations... 19 3. Methodology... 21 3.1. Project Management... 21 3.1.1. Revised Project Plan... 21 3.1.1.1. Key Activities... 21 3.1.1.2. Milestones... 21 VII

3.1.2. Progress Meetings... 22 3.2. Problem Summary... 22 3.3. Project Methodology... 23 3.4. Analysis and Design of Simulated Experiments... 24 4. Implementation of Experiments... 26 4.1. DVFS Mechanism... 26 4.1.1. Scenario 1: Increasing the No. Of Hosts, VMs and Cloudlets... 26 4.1.1.1. Objectives and Design... 26 4.1.1.2. Discussion of the Results... 26 4.1.2. Scenario 2 : Increasing the No. Of Hosts... 28 4.1.2.1. Objectives and Design... 28 4.1.2.2. Discussion of the Results... 28 4.1.3. Scenario 3: Increasing the No. Of VMs... 29 4.1.3.1. Objectives and Design... 29 4.1.3.2. Discussion of the Results... 29 4.1.4. Scenario 4: Increasing the No. Of Cloudlets... 30 4.1.4.1. Objectives and Design... 30 4.1.4.2. Discussion of the Results... 31 4.1.5. Scenario 5: Increasing the No. Of Hosts and VMs... 32 4.1.5.1. Objectives and Design... 32 4.1.5.2. Discussion of the Results... 32 4.1.6. Scenario 6: Increasing the No. Of Hosts and Cloudlets... 33 4.1.6.1. Objectives and Design... 33 4.1.6.2. Discussion of the Results... 33 4.2. VM Migration... 34 4.2.1. Scenario 7: Increasing the No. Of Hosts, VMs and Cloudlets... 34 4.2.1.1. Objectives and Design... 34 4.2.1.2. Discussion of the Results... 35 4.2.2. Scenario 8: Increasing the No. Of Hosts... 36 4.2.2.1. Objectives and Design... 36 4.2.2.2. Discussion of the Results... 37 4.3. Overall Discussion and Results Comparison... 38 5. Evaluation... 40 5.1. Evaluation According to Minimum Requirements... 40 5.2. Evaluating the Methodology... 41 5.3. Evaluating the Results... 42 5.4. Limitations... 43 5.5. Future Work... 43 6. Conclusion... 45 6.1. Summary... 45 6.2. Problems Encountered... 45 7. Bibliography... 46 8. Appendices... 50 8.1. Appendix A: Project Reflection... 50 8.2. Appendix B: Interim Report... 52 VIII

8.3. Appendix C: Initial Project Plan... 54 8.4. Appendix D: Revised Project Plan... 56 8.5. Appendix E: Weekly Progress Report... 58 8.6. Appendix F: Work in Progress Presentation... 64 8.7. Appendix G: Collected Results of Scenario 1: Increasing the No. Of Hosts, VMs and Cloudlet DVFS Mechanism... 67 8.8. Appendix H: A Source Code Sample of the CloudSim Java Classes... 72 8.9. Appendix I: Collected Results of Scenario 2: Increasing the No. Of Hosts DVFS Mechanism... 83 8.10. Appendix J: Collected Results of Scenario 3: Increasing the No. Of VMs DVFS Mechanism... 87 8.11. Appendix K: Collected Results of Scenario 4: Increasing the No. Of Cloudlets DVFS Mechanism... 91 8.12. Appendix L: Collected Results of Scenario 5: Increasing the No. Of Hosts and VMs DVFS Mechanism... 95 8.13. Appendix M: Collected Results of Scenario 6: Increasing the No. Of Hosts and Cloudlets DVFS Mechanism... 98 8.14. Appendix N: Collected Results of Scenario 7: Increasing the No. Of Hosts, VMs and Cloudlets VM Migration... 102 8.15. Appendix O: Collected Results of Scenario 8: Increasing the No. Of Hosts VM Migration... 103 IX

1. Introduction 1.1. Overview Cloud Computing has become an extremely attractive topic of research and application these days. NIST has defined Cloud Computing as a model for enabling convenient, on-demand network access to a shared pool of configurable Computing resources [1, p. 1]. Cloud security and storage have grasped lots of attention in this field [2]. In terms of storage, it acts as the lower layer of cloud infrastructure that supports the other layers functionality. Also, since it is important to make data storage redundant in different locations as contingency planning to ensure availability of cloud services, it would incur huge operating costs for the providers [2]. Therefore, this raises an important issue of how to improve the eco-efficiency of data centres to add value to the operational and maintenance costs of the cloud storage. 1.2. Aim and Objectives Cloud Computing has gained huge popularity for what it can offer for business and individuals in terms of cost effectiveness. However, maintaining the Clouds data centres can cost lots of money and energy. Therefore, the overall aim of this project is to identify and evaluate a technique, such as DVFS, that can be used to improve the eco-efficiency and performance of Clouds data centres. In order to achieve this aim, a number of objectives have been outlined: - Understand the importance of cloud computing, architecture of clouds, types of clouds, models of clouds, and virtualization. - Understand some issues associated with Cloud Computing in terms of data centre management. - Understand Cloud simulation, in particular CloudSim toolkit in terms of its usability, capability and limitations to offer a feasible evaluation of the eco-efficiency techniques used in data centres. 1

1.3. Minimum Requirements and Further Enhancements The minimum requirements have been identified as follows: - Design a simulated scenario of a cloud data centre using a DVFS mechanism - Implement the scenario - Collect and evaluate the simulation results - Evaluate the mechanism used to improve the eco-efficiency of data centre management in Cloud Computing Further enhancements are: - Design and implement additional scenarios of a Cloud data centre using a DVFS mechanism, and others without using a DVFS mechanism - Design and implement additional scenarios of a Cloud data centre using VM Allocation mechanism - Collect and evaluate the simulation results 1.4. Deliverables The main deliverable of this project is to provide an evaluation and recommend the ecoefficiency technique/s that can be used to improve the performance of cloud data centres. Also, in order to do that, there will be the implementation of a set of CloudSim java classes to simulate the experiments (see Appendix H for a source code example). 1.5. Motives and Relevance to the Degree Programme The primary motive is to find a challenging project that will allow me to develop new skills and enhance my knowledge in the field of Cloud Computing. Cloud Computing is currently a popular topic and is believed to be the future of the IT industry. Consequently, identifying and evaluating the eco-efficiency of cloud data centres is essential to its development. This project builds upon the knowledge and skills gained from several modules taken during the course of studying the MSc in Computing and Management. One of the modules, Advanced Distributed Systems, has provided a fundamental understanding of Web Services and Cloud Computing, from which this project will primarily be built. The knowledge and skills gained from Problem Solving with Computers, in terms of programming with Java and understanding the logic of 2

attempting to solve some IT problems, are very useful. These skills can be further enhanced to add value to this project. Business Information Systems increased the awareness of some key topics, such as choosing the appropriate methodology for this project. Techniques for Knowledge Management introduced key knowledge management topics; these topics will be investigated to evaluate the experimental results of this project. Additionally, knowledge learned from the business modules, which are Understanding Organisation, Marketing for Management, and Strategic Management, will be useful in enhancing the researching, critiquing, documenting, and management skills needed to construct this project. 1.6. Methodology To achieve the aim of this project, the chosen methodology should offer a way to design and build an experimental scenario of cloud computing, implement the scenario, and evaluate its performance, and more specifically, the eco-efficiency, of the data centres of the cloud. There are some methodologies that can be adopted for this project, but it is important to determine which one would be more applicable. Direct experiments, mathematical modelling, and simulation methodologies are going to be investigated next. Firstly, direct experiments can be used to implement the scenario of a Cloud data centre in reality and obtain accurate results. However, it may be time consuming to build and implement the Cloud. It also requires many resources. This project, however, is restricted with a limited time scale. As such, this method is not feasible for this project. Secondly, mathematical modelling, involving mathematical equations, can be used to evaluate the energy and performance trade-off of cloud computing environments [3]. Therefore, this method may be appropriate for this project as it can offer benchmarking of the results. Nonetheless, thirdly, it would be better to use a simulation that not only offers mathematical equations to measure the energy and performance, but also provides an entire environment to simulate a scenario, starting from designing the desired experiments, running experiments, and showing the results after the calculations. More justification of using a simulation as a methodology for this project will be discussed in Section 2.3. There are different simulation tools that can be used for the experimental development of Cloud Computing (e.g. GreenCloud, CloudSim, and CloudAnalyst). GreenCloud, built on top of NS2, can be used to determine the detail of the consumed energy in the components of a data centre and 3

in the packet-level communications between these components [4]. However, because its simulation takes a long time and requires high memory usage, its scalability is limited only to small data centres [5]. CloudSim is an extensible toolkit used to model and simulate cloud infrastructures, including data centres, users, user workloads, and application provision environments [6] [7]. CloudSim can be scalable and should have low simulation overheads [5] [6]. CloudAnalyst is built on top of the CloudSim toolkit; it provides visual modelling, easy to use graphical user interfaces, and large-scale application simulations deployed on the cloud infrastructure. Application developers can use CloudAnalyst to determine the best approaches for allocating resources among available data centres to serve specific requests and determine the related costs for these operations [7]. After reviewing the options, it was determined that the CloudSim toolkit will be used for this project, due to the support of the specifications that come with it, as well as the support of some of the eco-efficiency techniques which will be addressed throughout the project. More details about CloudSim will be presented in Section 2.3. Briefly, the development approach of this project will start with analysing and designing a scenario of the simulated experiment, running the scenario, verifying the results, and finally, evaluating the results. More details about the development approach will be explained in Section 3.3. 1.7. Initial Project Plan In order to control and manage the project, a Work Breakdown Structure (WBS) and a Gantt chart have initially been created during the planning stage, as shown in Appendix C. Also, some changes may take place during the implementation phase in the case of encountering any challenges. This project plan will help to track the completion of each development stage within the time-scale to ensure the delivery of a successfully completed project. 1.7.1. Key Activities A. Initiating: this stage covers the initial meeting with the supervisor to discuss the project idea and set up the aim and minimum requirements for the project. B. Planning: in this stage, a feasible planned project structure has been set and initially agreed on with the supervisor. C. Executing: this stage includes the main activities of the project s development; the literature review and background research about Cloud Computing and eco-efficiency issues in data 4

centres; analysing and gaining a deep understanding of the issues and how to provide an evaluated solution; designing a simulated scenario of cloud computing to evaluate the ecoefficiency technique as solution, and finally, implementing the scenario. D. Evaluation: this stage involves evaluating the results of using the eco-efficiency technique. E. Controlling: this stage covers the progression status of the project including weekly progress reports and a work in progress presentation. F. Closing: this stage is made up of the write-up of the dissertation, finalising, and the submission of the dissertation. 1.7.2. Milestones In order to measure the progress of the project and ensure meeting the aim and minimum requirements, milestones, known as self-goals, have been included in the project plan to indicate the starting point and deadline for each stage [8]. These milestones are: 1. Decide on Project (10/02/2012) 2. Decide on Planning (09/03/2012) 3. Analysis and Implementation (01/03/2012) 4. Monitoring (28/05/2012) 5. Evaluation (09/07/2012) 6. Preparing for the Final Project and Dissertation (16/07/2012) 7. End of Project (30/08/2012) 5

2. Review of Literature and Technologies 2.1. Introduction to Cloud Computing To begin with, as the way people live and work together has evolved over time, so Information Technology (IT) has come to play a significant role in adapting to and meeting changing business requirements. Software systems have been developed from being monolithic one-tier systems to more complex and decoupled n-tier systems, which emphasises, along with the Internet evolution, the importance and need to have suitable integration technologies to enable businesses to communicate over networks [9]. Therefore, a significant evolution of integration technology has been the development of Service-Oriented Architecture (SOA), which is a pattern used for understanding and maintaining business activities that bridge large distributed systems [10]. Additionally, this development has introduced middleware technologies that enable communication and the sharing of resources between heterogeneous systems. Thus, having all these above mentioned technologies available, along with Grid Computing which is already in place, has led to the development of a new era of IT, which is Cloud Computing. Cloud Computing is considered to be a Strategic Inflection Point (SIP is a time of change in the life of an economy, a company, a society, or a person) in IT, as it introduced software technology for virtualisation, metalevels, and virtual integration in 2010. The SIP of Cloud Computing has arrived after a 15 year gap since the last SIP, which was the wide adoption of the internet in 1994 [11]. Also, Buyya et al [12] argue that based on the increasingly common perceived vision of computing, it will become the fifth utility (after electricity, gas, water, and telephony), providing the general public with the basic level of computing services used in their daily routines. The following subsections will provide further background information about Cloud Computing. 2.1.1. What is Cloud Computing? Instead of individuals and businesses using their own PCs and other IT resources to perform their daily work, they can use the services offered by Cloud Computing at a reasonable cost, and leave the maintenance and cost of ownership to the cloud providers. There are many definitions of Cloud Computing suggested by different IT experts; each definition focuses on certain aspects of the Cloud. Vaquero et al [13] argue that to offer a comprehensive definition of Cloud Computing is like a large pool of easily usable and accessible virtualised resources, such as hardware, development 6

Cloud Architecture platforms, and/or services. These resources can be dynamically re-configured to adjust to a variable load (scale), allowing also for an optimum resource utilisation. This pool of resources is typically exploited by a pay-per-use model in which guarantees are offered by the Infrastructure Providers by means of customised Service Level Agreements (SLAs) (p. 51). Therefore, the main Cloud Computing characteristics and features, which can be derived from the definition above, are virtualisation, dynamic scalability, resource utilisation, and a business model of on-demand services based on SLAs. Therefore, Cloud Computing can be seen as providing useful opportunities for some businesses. Basically, it can offer economical and technological benefits. In terms of economic benefits, organisations can reduce the huge costs of large capital investments in IT resources by simply paying only for offered on-demand services (during peak time) from a Cloud provider, as well as lowering the cost of labour and IT resources maintenance [14] [17]. In terms of technological benefits, Cloud Computing can offer easier group collaboration, such as sharing documents between users from anywhere, regardless of their operating systems. Also, it can offer on-demand scaling of resources, such as acquiring unlimited storage capacity [14]. However, Cloud Computing comes with some disadvantages as well. For example, it requires constant connection to the Internet in order to access the services. Also, it mostly requires high bandwidth for data transfer [14]. More issues with Cloud Computing will be discussed in Section 2.2. 2.1.2. Architecture of Clouds Cloud Computing architecture consists of multiple different layers, as is the case with most distributed systems. Foster et al [15] categorise Cloud Computing architecture, based on a comparison with Grid architecture, into four layers, which are fabric, unified resource, platform and application, as shown in Figure 1 below: Application Platform Unified Resource Fabric Figure 1. Four-layer Cloud Architecture [15] 7

Firstly, the fabric layer is the lower level layer of the architecture and includes the raw hardware resources, like storage resources. Secondly, the unified resource layer contains abstract resources, through virtualisation, that form integrated resources exposed to the upper layer and end users. Thirdly, the platform layer depends on the unified resources layer and includes an additional set of dedicated tools, middleware, and services in order to provide an environment for the applications development and deployment. Fourthly, the applications that run in the clouds are contained in the application layer [15]. Moreover, another study by Buyya et al [16] shows that Cloud architecture mainly consists of user-level middleware, core-middleware, and system level, as shown in Figure 2 below: Figure 2. Layered Cloud Computing Architecture [16] Beginning from the top of the architecture, Cloud application layers contain applications which can be accessed by end-users directly. Alternatively, users own applications can be deployed in this layer [6]. In addition, the user-level middleware layer contains software frameworks that assist the developer to create an environment for applications to be developed, deployed and executed in Clouds. Moreover, the platform level services that set the run-time environment are performed in the core-middleware layer to host and control services at user-level application. Finally, the system level layer is where massive physical resources, like servers, exist, and these resources are managed by the virtualisation services set above this layer [16]. 8

2.1.3. Types of Clouds Services Organisations can access different levels of services from Clouds with a pay-per-use policy. These types of Cloud services can be categorised based on the layers of Cloud architecture mentioned earlier. So, there are mainly three Cloud services, namely SaaS (Software as a Service), PaaS (Platform as a Service), and IaaS (Infrastructure as a Service). SaaS service is about offering applications, like Google docs, to end users to use on the Cloud vendors servers. Cloud vendors offer PaaS services, like Google App Engine, to allow users to host their applications instead of buying new resources to house them. Furthermore, IaaS service is about using Cloud providers virtualised low level resources; and this service can be named Everything as a Service. For example, if the user wants to use the storage of the vendors data centres, it can be named Storage as a Service [17]. Based on the architecture of Cloud Computing, SaaS is supplied at the user level, and PaaS is provided in the core middleware, and IaaS is supplied at the system level [6]. Data centres form the low level foundation of Cloud Computing, and these centres consist of thousands of inter-connected servers, usually located in areas with low probability of natural disasters and cheaper energy rates [18]. SaaS, PaaS, and IaaS services are sometimes provided by different cloud vendors. 2.1.4. Types of Clouds Deployment There are many different types of Cloud deployment, which are mainly public, private, hybrid, community, and federated Clouds. Users can have open access, at fairly cheap cost, to use public Clouds services through the Internet, while access to private Cloud services is restricted to only those who own the Clouds and their subsidiaries [19]. Public Clouds offer IT resources and services to individual and businesses to save the overheads of having in-house built resources. Private Clouds allow organisations to utilise their IT resources efficiently (through virtualisation) as well as increase security since it can be managed internally. Moreover, the concept of hybrid Clouds is about connecting internal Clouds (private) to external Clouds (public) [11]. This combination usually happens when businesses using private Clouds for their business-critical systems need to scale up their resources at low cost by outsourcing more IT services from public Cloud providers to fulfil their non-critical system needs. Furthermore, a specific community of organisations who have common concerns can share their Clouds infrastructure to collaborate by forming a community Cloud [1]. Thus, a community Cloud can be formed through a combination of public, private, or hybrid Clouds. 9

Additionally, federated Clouds usually occur when two private Clouds owned by different organisations have a form of contract to share some of their Clouds resources, as forming an alliance can better utilise their IT resources effectively and gain the advantage of using others capabilities. 2.1.5. Virtualisation Virtualisation is a vital technology of Cloud Computing which offers two important featuresabstraction and encapsulation [15]. It is about creating an abstract layer between hardware and software. Usually, the virtualisation layer is set above the physical layer of the Cloud s architecture. Virtualisation technology is used widely in Cloud Computing data centres owing to the benefits offered, such as utilising resources, lowering costs, easier management of servers, server consolidation, and live migration of virtual machines [20]. Virtualisation is mostly used in Cloud Computing platforms as means to optimise resource usage [21]. Through virtualisation, the number of hardware resources used in Clouds can be reduced to minimise the capital cost as well as the cost of power consumption and cooling systems [22]. For instance, through server consolidation, multiple (virtual) servers can be allowed to run simultaneously on a single physical server. Also, live migration of the virtual machine to the not fully utilised physical servers would allow more and more physical servers to be turned off, which would lead to better achievement of energy efficiency for data centres [20] [15]. Furthermore, visualisation in Cloud Computing can offer dynamic configurations for different applications resource requirements, and aggregate these resources for different needs. Additionally, it can improve responsiveness by monitoring, maintaining and provisioning resources automatically. Therefore, all these features offered by virtualisation are used in Clouds in order to meet the criterion of the business requirements of SLAs [15]. Moreover, there are mainly two ways of virtualisation deployment, as stated by Hardy et al [22]. Firstly, when the virtualisation layer is installed on an existing operating system as an application, it is called hosted system virtualisation or application virtualisation. Secondly, when the virtualisation layer is installed directly on clean x86-based hardware, instead of a conventional operating system, it is called hypervisor systems, and is usually used for server consolidation [22]. Furthermore, there are several types of virtualisation, some of which are full virtualisation, hardware assisted virtualisation, partial virtualisation, paravirtualisation, hybrid virtualisation and operation system-level virtualisation [23]. Full virtualisation allows a guest operating system to be run unmodified and in isolation by simulating, almost completely, actual hardware. Hardware assisted virtualisation is about improving the efficiency of a guest environment by improving the 10

computational performance of a virtual machine without interpreting and isolating the instructions passed directly to the host processor. Partial virtualisation is about simulating most but not the entire underlying host environment. Paravirtualisation, hybrid virtualisation, and operating systemlevel virtualisation can be deployed as a way of utilising partial virtualisation. Paravirtualisation provides a similar software interface or API s of virtual machines to that of the underlying host hardware. Hybrid virtualisation is about combining the advantages of both hardware assisted virtualisation and paravirtualisation to obtain better performance than with software-only paravirtualisation [24]. Finally, operating system-level virtualisation is about creating multiple isolated virtual machines with the requirement that the OS of the virtual machine should be the same as the host [23]. 2.2. Issues Associated with Clouds As mentioned earlier, Cloud Computing, as the latest technology in the IT industry, has attracteds lots of businesses nowadays owing to its promising benefits, such as cost reduction for capital investment, scalability, on-demand services, and energy saving. However, there are some issues associated with clouds, such as security, availability of services, reliability of data, unpredictable performance, and the risk of data lock-in [25]. Firstly, since the data of consumers is stored somewhere in the Cloud and not on their site anymore, security issues have become one of the major concerns for consumers use of Cloud Computing technology. Also, private data leakage would raise a concern, as consumers data has to be moved from a single-tenant to a multi-tenant environment [26]. Moreover, businesses logically would not be willing to be in the position of putting all their eggs in one basket by moving all their business data to the Cloud where it could be vulnerable to hacking attacks. For example, Distributed Denial of Service (DDoS) attacks can be set by a hacker by hosting a malicious application in a cloud which his victim, another user, is subscribing to. Launching these attacks with a large number of zombie machines from multiple resources can pass heavy flood of packets that may affect the performance or availability of the service. However, Sabahi [26] argues that Clouds can be used as a solution against these DDoS attacks because the Cloud can offer the flexibility of providing more resources instantly, to tolerate such attacks and avoid site shutdown. Even so, more investigations need to be carried out to discover to what extent Cloud technology can handle this type of attack in order to ensure the availability and reliability of the services. 11

Moreover, the key strategy of Cloud providers offering services at cheap cost is to attract more customers till they gradually move their core business requirements to the Cloud, which would then increase the risk of data lock-in. After relying on the Cloud, it would be very difficult for the consumer to move back to their own system because it would require some additional costs. Therefore, the Cloud providers can take advantage of consumers being bound to their services and include more add-on services. Furthermore, there are some cost issues associated with the Clouds in terms of maintenance and management of the resources, especially the data centres, which will be discussed next. 2.2.1. Cost of Clouds Maintenance To begin with, Total Cost of Ownership (TCO) is not only about the capital expenditure required to own an IT infrastructure, but also about the ongoing cost of managing and operating the IT infrastructure in a business [27]. Therefore, in addition to buying hardware and software assets to build the infrastructure of the Cloud and provide services, there are more ongoing costs, like having highly skilled IT experts employed to maintain and ensure its availability and performance all the time. Also, as a Cloud provides more and more services to a large consumer base, enhancement of resource utilisation is needed to meet the demand and gain maximum economies of scale by producing more output with less expensive resource consumption. Besides, since Cloud providers follow a strategy of offering Cloud services at cheap cost to attract more consumers to use their businesses, the added-value activities for maintaining the Clouds should be at the lowest cost possible in order enable the Cloud providers to achieve their strategy as well as gain a competitive advantage. Thus, such techniques should be implemented to reduce the cost of maintaining the data centres, which will be discussed more in the next Section. 2.2.2. Eco-efficient Data Centre Management Eco-efficiency can be directly linked to being environmentally friendly. It is about how to manage the data centres of the Clouds in ways that have less impact on the energy consumed, as well as on carbon dioxide (CO2) emissions. Therefore, mechanisms and policies should be put in place to help understand how green the data centre of a Cloud is. The dramatic increase in greenhouse gas emissions is having a detrimental effect on the global climate, like increasing temperatures, dryness, and floods. Also, the ICT industry has been 12

contributing to this growth; as has been stated by Smarr, who explains that the carbon emission from this sector is expected to triple from 2002 to 2020 [28]. Cloud infrastructure is likely embodied by enormous and power-consuming data centres that generate large amounts of heat when processing. So, in order to effectively remove this heat, data centres are equipped with great cooling systems that consume huge amounts of energy in order to work [29]. Therefore, there is great pressure on Cloud providers to improve such mechanisms for the energy efficiency of Cloud data centres to both reduce costs and support the prices offered to customers, as well as to ensure meeting QoS to satisfy the customers expectations [29]. 2.2.2.1. A Scientific Model of Eco-efficient Data Centre There are some metrics that can be used in order to measure and understand how eco-efficient a Cloud data centre is. For example, Power Usage Effectiveness (PUE) is a well known metric that can be used to define how eco-efficient the infrastructure of a data centre is. PUE has been defined by The Green Grid organisation as a measurement for identifying how a data centre uses its power efficiently by actually calculating how much of the power is consumed by the servers in contrast to other overheads, like cooling [30] [31]. To illustrate, PUE is calculated by dividing the Total Data Centre Energy Usage by the IT Energy Usage (PUE = Total Energy/IT Energy). The total energy includes IT energy plus the energy consumed for lighting, security, CCTV, heating and cooling, and all other facilities in the data centre. As has been stated by Gillan [32], the average PUE efficiency in typical data centres these days is about 2.0. So, as the PUE increases, the more inefficient the data centre infrastructure is, and the closer the PUE is to 1.0, the more efficient the data centre infrastructure is considered to be. Thus, PUE optimisation aims to lower the total energy consumed by all facilities, to be almost equal to the energy consumed by IT equipment. In other words, it means minimising the energy used by the facilities, other than IT equipment, to almost zero consumption. However, using PUE metrics alone is not enough to consider data centre infrastructure to be very eco-efficient. Therefore, a new metric, Energy Reusable Efficiency (ERE), has been defined by the Green Grid. This metric measures how efficiently the wasted energy that leaves the data centre is reused in adjacent, non-data centre spaces [33]. For example, the heat generated from the data centre can be converted beneficially to electricity by using reverse heat engines and technologies. The energy reproduced should be for use somewhere else on the campus, but outside the control volume boundary of the data centre, so that it will not affect PUE measurement [33]. So, both PUE and ERE are useful in analysing the efficiency of energy usage in a data centre, but the former 13

assesses the efficiency of the energy used by the data centre infrastructure, and the latter assesses the efficiency of altering the wasted energy that leaves the data centre to be reused in other spaces outside the boundary of the data centre [34]. Moreover, it has been argued that PUE may be a poor operational metric because it does not consider IT efficiency or IT load distribution [35] [36]. In fact, more optimal energy savings would be achieved theoretically when there is utilisation of the IT resources to reduce the running machines and further reduce the energy consumed [36]. Virtualisation is a very important part of utilising the resources, as discussed before in Section 2.1.5, but more utilisation for resources like CPU utilisation can add more value to energy efficiency and cost reduction. Also, as the technology advances, producing new servers that run faster and consume the same energy as legacy servers, it can be beneficial to have a life cycle strategy for replacement. For instance, replacing the old servers with new powerful servers would help to increase the energy efficiency of data centres by reducing the number of servers required to deliver the same IT workload [36]. Thus, the definition of a scientific model of eco-efficient data centre depends on how efficiently the energy is consumed in a data centre with minimum costs and less, or most favourably no, impact on the environment in terms of CO2 emissions. There are a number of metrics that should be used to measure the efficiency of energy usage in the data centre from different aspects. As new metrics are being developed, it would help to drive the data centres towards being more eco-efficient. 2.2.2.2. Management of Power Consumption The energy supplied in data centres is consumed for computational operation, cooling systems, networks, and other overheads. In terms of computational operations, there are some energy-saving techniques that can be deployed to monitor and control energy consumption. These eco-efficient energy techniques have become one of the hot topics in the IT business these days because of the benefits gained, not only from an economic perspective, but also from an environmental perspective. Cloud service providers can save huge costs by efficiently utilising their data centre to its maximum capacity. Also, governments pressure companies to conduct their businesses with less impact on the environment, in terms of the emission of CO2. Hence, Clouds service providers can also improve their SLAs by being friendly to the environment to attract more customers. Furthermore, in terms of power efficiency, it can be said that data centre A is better than data centre B if A can consume less power and process the same workload as B, or A can consume the same power but with more workload compared with B [37]. Thus, lots of research has been 14

conducted to suggest some techniques for reducing power consumption without degrading the performance in the data centres, some of which will be discussed next. 2.2.2.2.1. VM Allocation Firstly, a study by Corradi et al states that Virtual Machine (VM) consolidations can be used as a means of reducing the power consumption of cloud data centres [37]. To illustrate, this technique tries to allocate more VMs on less physical machines as far as possible to allow maximum utilisation of the running of physical machines. For instance, when there are two VMs, instead of allocating each one to a physical server that has not been fully utilised, this technique tries to allocate both VMs on one physical server and switch the other server off to save energy. Therefore, using this technique in a data centre can reduce the operational costs and increase the efficiency of energy usage. However, it is important to note that the number of VMs in one physical machine should not be too high to the extent that it may degrade the performance of VMs [37]. 2.2.2.2.2. Power and Thermal Management Furthermore, another study conducted by Pakbaznia et al emphasises the importance of developing power and thermal management (PTM) to improve the energy efficiency of data centres [38]. Basically, this approach supports making the decision for the number and placement of ON servers, as well as maintaining the supplied cold air temperature, in order to reduce the power consumption for both the servers and cooling units. However, in order for this approach to achieve its goal, it requires a prediction of the incoming workload in terms of requests per second, which can be achieved by using Workload Monitoring (WM). Thus, as stated in this study, this approach endeavours to use a combination of chassis consolidation and efficient cooling to reduce the overall power consumption [38]. 2.2.2.2.3. Dynamic Voltage and Frequency Scaling Dynamic Voltage and Frequency Scaling (DVFS) can be used as one of the eco-efficiency techniques to reduce the energy consumed in computing servers. This technique observes the workload offered and then alters the CPU power consumption accordingly, which would then change the performance level as well [39]. Also, DVFS along with Dynamic Power Management (DPM) have been widely deployed and extensively studied [4]. Usually, the idle servers consume two-thirds of the peak load in order to keep the other components, like memory and disks, up and running; and the remaining one-third of the consumption is based on the CPU utilisation, which can be managed by DVFS. On the other hand, the DPM technique may be more efficient by powering 15

down the servers, including all the components. Nonetheless, DPM would consume a greater amount of energy compared to DVFS when there is a need to turn these servers back on to run [39]. It can be deduced from the simulated experiments conducted by Kim, Beloglazov, and Buyya [40] that data centre providers can increase their profits by reducing power consumption through the deployment of a DVFS technique. A DVFS technique is designed in a way that decreases the energy consumption of the CPU by controlling and reducing the supply of voltage and frequency. 2.3. CloudSim Toolkit IT companies who are willing to offer some services in the Cloud can use a simulation-based approach to perform some benchmarking experiments with the services to be offered in dependable, scalable, repeatable, and controllable environments before real deployment in the Cloud [6]. Therefore, they can test their services in a controlled environment free of cost, and through a number of iterations, with less effort and time. Also, by using simulation, they can carry out different experiments and scenarios to identify the performance bottlenecks of resources and develop provisioning techniques before real deployment in commercial Clouds [6] [41]. Therefore, CloudSim has been developed to fulfil these requirements by simulating and modelling extensible Clouds; more details about CloudSim will be discussed next. 2.3.1 Architecture of CloudSim CloudSim can be defined as a new, generalised, and extensible simulation framework that allows seamless modelling, simulation, and experimentation of emerging Cloud Computing infrastructure and application services (p.24) [6]. Initially, the framework of CloudSim consists of multiple layers starting from the lowest layer of SimJava up to the top layer of User Code, as shown in Figure 3. 16

Figure 3. Initial Layered CloudSim Architecture [16] At the lowest layer, SimJava provides the base engine of the simulation that supports the implementation of core functionalities essential for the higher-level frameworks of the simulation, like queuing and processing of events; formation of system components (services, hosts, brokers, VMs); interaction between these components, and administration of the simulation clock [16]. On top of that layer is the GridSim layer which supports modelling high-level and fundamental Grid components, such as networks, resources, data sets, and information services [16]. Then, the CloudSim layer forms the next level of the architecture that extends the core functionalities of the GridSim layer. This layer supports modelling virtualised Cloud-based data centre environments, including VMs, memory, storage and bandwidth. Also, this layer can manage instantiating and simultaneously implementing a large scale Cloud infrastructure composed of thousands of system components (VMs, hosts, data centres, and application) [16]. Finally, User Code is the top-most layer of the simulation toolkit, which reveals the configuration of functionality for the system components, such as the number and specification of hosts and the scheduling policies of the broker [16]. At this layer, a developer can model and perform robust experiments and scenarios of Cloud environments based on custom policies and 17

configurations already supported by the CloudSim, in order to evaluate and tackle some Cloud issues like the complexities of Cloud infrastructure and application. Moreover, the architecture of the CloudSim toolkit has been reviewed and changed with its new release, as can be seen in Figure 4. Figure 4. Current Layered CloudSim Architecture [6] Therefore, based on Figure 4, the current layers of the simulation architecture are the CloudSim core simulation engine, CloudSim, and User Code. The SimJava layer in the previous release has some limitations, like not allowing the simulation to programmatically reset at run-time, not supporting a new simulation entity to be created at run-time, and the SimJava nature of being multithreaded can cause performance overheads with the increasing size of the system. Thus, SimJava has been removed and replaced with CloudSim core simulation engine in the new release of CloudSim to overcome these limitations and enable more advanced operations that were not backed up by SimJava [6]. 2.3.2 Usability In order to use the CloudSim toolkit, users need to have a basic background in Java programming language because it is written in Java [6]. Also, it requires users to write some code to use the components from its library in order to simulate the desired scenarios. Therefore, it is not 18

just about setting the parameters, running the program, and collecting the results, but it also requires a deep understanding of how the program works. In addition, a little knowledge about Integrated Development Environments (IDEs), like NetBeans or Eclipse, will be useful to ease installing the toolkit and the development of scenarios. Furthermore, CloudSim provides a library that can be used to build a ready-to-use solution, such as CloudAnalyst which is built on top of CloudSim, to offer an easy to use graphical user interface [7]. 2.3.3 Capabilities CloudSim has some compelling features and capabilities that can be extended to model a custom Cloud Computing environment. According to Calheiros et al [6], CloudSim can offer flexibility and applicability and with less time and effort to support initial performance testing. It can support simulating, from small-scale up to large-scale cloud environments containing data centres, with little or almost no overheads in terms of initialisation and consumption of memory. Also, it has a virtualisation engine that allows the creation of multiple virtualised services that can be independently managed on a single node of the data centre. Moreover, it supports, in addition to other features, energy-awareness provisioning techniques at resource, VM, and application level, such as VM allocation and DVFS [6]. For managing the energyconscious techniques in a data centre, CloudSim architecture contains the key components CloudCoordinator, Sensor, and VMM. The Sensor component, which is attached to every host, is used by the CloudCoordinator to monitor particular performance parameters, like energy consumption and resource utilisation. Thus, through the attached Sensors, CloudCoordinator passes real-time information, like load conditions and processing share, of the active VMs to the VMM. Then, VMM uses this information to perform the appropriate application of DVFS and resizing of VMs. Also, according to VMs policy and current utilisation of resources, CloudCoordinator constantly issues VM migration commands and changes the power state of nodes to adapt the allocation of VMs [6]. 2.3.4 Limitations CloudSim is a powerful tool for modelling and simulating Cloud computing, but it has some limitations. Firstly, it is not a ready-to-use tool that that would just require setting parameters only. Actually, it does require writing some Java code to use its library, as discussed earlier. Also, the capabilities of CloudSim are sometimes limited and require some extensions. For instance, CloudAnalyst has been developed as an extension of CloudSim capabilities to offer a separation of 19

the simulation experimentation exercise from the technicalities of programming, using the library in order to ease modelling by simply focusing on the complexity of the simulated scenario, without spending much effort and time on the language in which the simulator is interpreted [7]. 20

3. Methodology 3.1. Project Management Managing the project appropriately by using the right tools can be a key factor to its success [42]. Project management should involve the activities of planning, follow-up and control, and ensure the project is being completed according to the specification and within the permitted budget and timescale. 3.1.1. Revised Project Plan For the project undertaken, a project plan was initially created, as discussed earlier in Section 1.7. But, because of uncertainties about the hurdles encountered in the early stages, the project plan was revised during the Implementation stage because it took longer than initially expected. The reasons for revising the plan are the lack of a user manual for how to use the CloudSim toolkit and time spent on running the experiments. There was no user guide or online tutorial for how to use the toolkit; therefore, it took a while to perform reverse engineering to understand the simulation source code and how it works and behaves. Also, running the experiments took a very long time. More details of these issues and how they were solved will be explained in Section 6.2. In addition, more time was allocated for the Implementation stage to add more novelty to the project by implementing further scenarios instead of having only one. Thus, the project plan has been revised without exceeding the limited timescale for the project deadline (see the revised plan in Appendix D). 3.1.1.1. Key Activities The key activities for the project are still the same as initially discussed in Section 1.7.1. 3.1.1.2. Milestones The start dates for some of the milestones have been updated, as follows: 1. Decide on Project (10/02/2012) 2. Decide on Planning (09/03/2012) 3. Analysis and Implementation (01/03/2012) 21

4. Monitoring (28/05/2012) 5. Evaluation (09/08/2012) 6. Preparing for the Final Project and Dissertation (30/07/2012) 7. End of Project (30/08/2012) 3.1.2. Progress Meetings In order to track the progression status of the project, weekly progress meetings with the supervisor have been conducted almost from the beginning of the project till the end. During these meetings, it was very useful to discuss the progress of the project, reveal any issues encountered, and set plans based on the discussion output. After the submission of the interim report, weekly progress reports were documented and brought to the meetings for tracking and discussing the tasks accomplished for that week, and setting the tasks to be completed for the following week (see these weekly progress reports in Appendix E). In addition, a work in progress presentation was conducted with both the supervisor and assessor to show and discuss the overall progress of the project and obtain feedback to enhance the project accordingly (see the presentation slides in Appendix F). 3.2. Problem Summary Cloud infrastructure is equipped with large and power consuming data centres that bring challenges to managing the power consumption eco-efficiently [29] [40]. As has been stated by Scheihing [43], data centres consume about 10 to 100 times more energy per square foot than typical office buildings. They can even consume the same amount of electricity that is used by a city [44]. Also, high energy consumption in a data centre can lead to extreme heat dissipation which would raise the cooling costs and make the servers vulnerable to risks and failures. All these issues would also inflate the operational costs in order to maintain and keep the data centre functioning [45]. Additionally, governmental policies across the world push data centre providers to reduce energy waste and become greener with regard to the environment. Thus, using certain eco-efficient mechanisms, like DVFS and VM allocation, to reduce the energy consumption can help to enhance the sustainability and profitability of the data centres and make them greener. 22

3.3. Project Methodology As discussed previously, the methodology will be based on using a simulation to evaluate the mechanisms that can be used to improve the eco-efficiency of data centres. Thus, the development approach of this project will start with analysing and designing a scenario of the simulated experiment, running the scenario, verifying the results, and finally, evaluating the results. The designing, running and verifying of the stages will follow an iterative approach, as shown in Figure 5, to ensure the appropriateness of the simulated scenario for this project. Figure 5. An iterative approach for experiments development The main reason for following an iterative approach is to give more flexibility to quickly altering the settings of the design of the experiments in case of any mistake or error that would terminate running and completing the experiments. Also, after completing each experiment successfully, the iterative approach allows going back and changing the objectives of the experiment in order to investigate more aspects of Cloud data centres. The analysis stage is more about understanding the background to the CloudSim toolkit, how it has been used by others to simulate some experiments, and setting the objectives for the experiments to be simulated. The design stage is more about forming the settings for the experiments according to the objectives. Running the experiments and collecting the results then takes place. After that, verifying and interpreting the results and making sure that the experiment is completed successfully is done. Finally, evaluating and contrasting the results with the other work already carried out in the literature takes place. 23

3.4. Analysis and Design of Simulated Experiments As discussed previously, the methodology adopted for this project is based on simulating some experiments by using the CloudSim toolkit to evaluate eco-efficiency mechanisms in Cloud data centres. Therefore, as mentioned in Section 2.3, CloudSim has been analysed and reviewed in terms of its usability and capabilities of offering an environment for experimenting with some scenarios of Cloud data centres. The main objective of the experiments is to evaluate the energy consumption in data centres. In terms of the design, CloudSim version 2.1.1 has been used because it has more features than the earlier release, as discussed in Section 2.3.1., and it provides examples of power provisioning techniques that can be used for conducting the experiments in this project. CloudSim consists of significant classes that form the building blocks of the simulator. For example, Cloudlet is the class that forms Cloud-based applications services like business workflow, content delivery and social networking [6]. For each life cycle of the Cloudlet execution, it has to undertake the pre-assigned instruction length and data transfer overhead including its creation, allocation to VM, scheduling in VM, and termination [46]. Also, Host class is responsible for modelling the physical resources like storage or compute servers. This class encapsulates essential details about the host, such as the amount of memory and storage, and lists and types of the processing cores. Furthermore, Vm class models the VM component which is hosted and managed by a Cloud host component that stores the related characteristics to the VM, including accessible memory, storage size, and processor [6]. Therefore, as a design for the scenarios to be experimented, they will focus on altering the settings of three parameters, which are the number of hosts, VMs, and Cloudlets. The characteristics of these three parameters will be kept as the typical values already set in CloudSim as it has already been experimented and validated by Calheiros et al to quantify CloudSim efficiency in simulating Cloud Computing environments [6]. Also, the values of these characteristics represent the typical settings of the machines found in a data centre. These typical characteristics of hosts, VMs, and Cloudlets are shown in Table 1. 24

Table 1. Typical characteristics of hosts, VMs, and Cloudlets Hosts Max Power 250 Watts Static Power 0.7 (70%) MIPS { 1000, 2000, 3000 } RAM Storage Bandwidth VMs 10000 MB 1000000 MB 100000 Mbps MIPS Rating { 250, 500, 750, 1000 } Number of CPUs 1 RAM Bandwidth Image Size Cloudlets 128 MB 2500 Mbps 2500 MB Length 150000 (equivalent to 10 mins execution on 250 MIPS) Number of CPUs 1 File Size 300 Output Size 300 Thus, provisioning requests for each VM would demand 1 CPU core (with variation of 250 MIPS to 1000 MIPS), 128 MB of RAM, and 2.5 GB of storage. Moreover, the simulation has some randomness with the results of the experiments. Therefore, the experiments will be run N-times to increase the reliability of the results by having a sample size and then calculating the mean values, standard deviation, and confidence interval with a confidence level of 95%- a commonly used level that will give very conservative results, as stated by Payton et al [47]. 25

4. Implementation of Experiments This chapter will show some simulated scenarios of a Cloud data centre using CloudSim, with the overall objective of evaluating the techniques based on the variation of energy consumption metric. Firstly, it will include several scenarios conducted with the deployment of a DVFS mechanism. Furthermore, it will show two more simulated scenarios using VM migration with DVFS enabled. Each scenario will have a number of experiments that will vary based on the value settings of the key parameters, hosts, VMs and Cloudlets, as discussed in the previous Section 3.4. They will start with the default values of 10 hosts, 20 VMs, and 20 Cloudlets as the minimum values, and then they will be incremented to represent the higher scales of a data centre for each succeeding experiment. The maximum values for each scenario will vary from one to another based on the objectives, as will be stated later on. 4.1. DVFS Mechanism 4.1.1. Scenario 1: Increasing the No. of Hosts, VMs, and Cloudlets 4.1.1.1. Objectives and Design The objectives of the following experiments are to evaluate how increasing the number of resources and services would reflect on the energy consumption; how the deployment of a DVFS mechanism would contribute towards reducing energy consumption compared to an environment without the deployment of this mechanism, and what the maximum parameters values that CloudSim can handle are. The parameter settings for the number of hosts, VMs, and Cloudlets will initially be according to the default values already set in CloudSim, and then they will be doubled each time in order to observe the variation in energy consumption. 4.1.1.2 Discussion of the Results The experiments for the two scenarios, with and without the deployment of DVFS, have been run and the results have been collected and shown in Table 2 and Table 3 in Appendix G. Thus, in terms of the parameter settings of this scenario, experiment 1 has the minimum values of 10 hosts, 26

Experiment 1 Experiment 2 Experiment 3 Experiment 4 Experiment 5 Experiment 6 Experiment 7 Experiment 8 Experiment 9 Experiment 10 Experiment 11 Experiment 12 Experiment 13 Experiment 14 Experiment 15 20 VMs, and 20 Cloudlets, and experiment 15 has the maximum values of 4603 hosts, 9206 VMs, and 9206 Cloudlets. From the results collected and calculated, Figure 6 below has been produced to explain the difference between the two scenarios based on the mean values of the energy consumption metric. 600 500 400 300 200 100 With DVFS Mechanism Without Power Aware Mechanism 0-100 Figure 6. Comparison between energy consumption mean values for each experiment Based on the results shown in Figure 6, doubling the number of resources and services would also double the energy consumption for both scenarios, with and without the deployment of DVFS. However, the energy consumed with the deployment of a DVFS mechanism is about three times less than the energy consumption with the scenario with no deployment of DVFS. Therefore, the deployment of DVFS would save about 2/3 of the consumed energy because the DVFS mechanism seeks to decrease the dynamic energy consumption by reducing the supplied voltage and frequency [40]. Also, this scenario should have stopped at experiment 10 where it cannot accept doubling the parameters values any more, but it was carried out for the purpose of finding out the maximum parameter values that CloudSim can handle, which took approximately 31 hours to complete. As can be observed from experiment 10 to 15, the scenario has changed, since the values of the parameter settings are not doubling this time because of the limitation of CloudSim s ability to handle them. As a result of this particular scenario, the purpose of finding the maximum values that CloudSim can handle has been achieved, and they are 4603 Hosts, 9206 VMs, and 9206 Cloudlets, as shown in experiment 15 in Appendix G. 27

4.1.2. Scenario 2: Increasing the No. Of Hosts 4.1.2.1. Objectives and Design The objectives of the following experiments are to evaluate how increasing the number of hosts would reflect on the energy consumption, and how the deployment of a DVFS mechanism would contribute towards reducing energy consumption compared to an environment without the deployment of this mechanism. The parameter settings for the number of hosts, VMs, and Cloudlets will initially be according to the default values already set in CloudSim, and then the number of hosts will be doubled each time to observe the variation in energy consumption. 4.1.2.2. Discussion of the Results Based on the results collected from the two scenarios, with and without the deployment of DVFS, shown in Table 4 and 5 in Appendix I, Figure 8 below has been produced to show the difference between the two scenarios based on the mean values of the energy consumption. For this scenario, experiment 1 has the minimum values of the parameter settings with 10 hosts, 20 VMs, and 20 Cloudlets, and experiment 14 has the maximum values of 81920 hosts, 20 VMs, and 20 Cloudlets. 9000 8000 7000 6000 5000 4000 3000 2000 With DVFS Mechanism Without Power Aware Mechanism 1000 0-1000 Figure 8. Comparison between energy consumption mean values for each experiment 28

Based on the results shown in Figure 8, doubling the number of machines would also double the energy consumption for the scenario without the deployment of DVFS, because more machines would be running and waste energy if there is no power control mechanism. But with the deployment of DVFS, the energy consumed is almost unchanged and stays the same for each experiment because only the number of machines required are running to provide the services, while the rest of the machines are not running, which would be reflected greatly in saving energy. 4.1.3. Scenario 3: Increasing the No. Of VMs 4.1.3.1. Objectives and Design The objectives of the following experiments are similar to scenario 2, but this time the number of VMs will be increased, while the Hosts and Cloudlets remain constant. Thus, the design of the experiments will involve doubling the value only of the VMs parameter each time to observe the variation in energy consumption. 4.1.3.2. Discussion of the Results Based on the results collected from the two scenarios, with and without the deployment of DVFS, shown in Table 6 and 7 in Appendix J, Figure 9 below has been produced to show the difference between the two scenarios based on the mean values of energy consumption. For this scenario, experiment 1 has the minimum values of the parameter settings with 10 hosts, 20 VMs, and 20 Cloudlets, and experiment 14 has the maximum parameter values of 10 hosts, 163840 VMs, and 20 Cloudlets. 29

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 With DVFS Mechanism Without Power Aware Mechanism Figure 9. Comparison between energy consumption mean values for each experiment Based on the results shown above in Figure 9, doubling the number of VMs from experiment 1 to 2 increased the energy consumption by about three times for the scenario with the deployment of DVFS, and then the energy consumption stayed the same for the subsequent experiments. Interestingly, this scenario may indicate that the number of under-utilised VMs used has increased from experiment 1 to 2, since the number of services and physical machines has stayed the same. In terms of the scenario without the deployment of DVFS, it remained almost constant and unchanged for all the experiments. However, the deployment of DVFS in experiment 1 saved about 70% of energy consumption compared to the other scenario. Also, for the remaining experiments, it still saved about 16% of the consumed energy compared to the scenario without the deployment of DVFS. 4.1.4. Scenario 4: Increasing the No. Of Cloudlets 4.1.4.1. Objectives and Design The objectives of the following experiments are similar to the previous scenarios, but this time it will increase the number of services while the Hosts and VMs remain constant. Thus, only the value of the Cloudlets parameter will be doubled each time to observe the variation in energy consumption. 30

4.1.4.2. Discussion of the Results Based on the results collected from the two scenarios, with and without the deployment of DVFS, shown in Table 8 and 9 in Appendix K, Figure 10 below has been produced to show the difference between the two scenarios based on the mean values of the energy consumption. For this scenario, experiment 1 has the minimum values of the parameter settings with 10 hosts, 20 VMs, and 20 Cloudlets, and experiment 13 has the maximum parameter values of 10 hosts, 20 VMs, and 61440 Cloudlets. 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 With DVFS Mechanism Without DVFS Mechanism Figure 10. Comparison between energy consumption mean values for each experiment Based on the results shown above in Figure 10, doubling the number of services has not, surprisingly, increased the energy consumption for both scenarios. They almost stayed the same for all the experiments. Also, the figure above clearly shows that the deployment of a DVFS mechanism saves about 70% of energy consumption compared to the other scenario without the deployment of this mechanism. 31

4.1.5. Scenario 5: Increasing the No. Of Hosts and VMs 4.1.5.1. Objectives and Design The objectives of the following experiments are similar to the previous scenarios, but this time it will involve increasing the number of Hosts and VMs only, while the number of Cloudlets remains constant. Thus, the value of Host and VM parameters only will be doubled each time to observe the variation in energy consumption. 4.1.5.2. Discussion of the Results Based on the results collected from the two scenarios, with and without the deployment of DVFS, shown in Table 10 and 11 in Appendix L, Figure 11 below has been produced to show the difference between the two scenarios based on the mean values of the energy consumption. For this scenario, experiment 1 has the minimum values of the parameter settings with 10 hosts, 20 VMs, and 20 Cloudlets, and experiment 10 has the maximum parameter values of 3840 hosts, 7680 VMs, and 20 Cloudlets. 400 350 300 250 200 150 100 With DVFS Mechanism Without Power Aware Mechanism 50 0-50 -100 Figure 11. Comparison between energy consumption mean values for each experiment Based on the results shown in Figure 11 and tables 10 and 11 in Appendix L, doubling the number of Hosts and VMs would also double the energy consumption for both scenarios, with and 32

without the deployment of DVFS. But, interestingly, with the deployment of DVFS, the energy consumed has increased three times from experiment 1 to 2, and for the subsequent experiments it just doubled. Similar to scenario 3, this increase may indicate that the number of under-utilised VMs used has increased from experiment 1 to 2 since the number of services stayed the same. The energy consumed in experiment 1 with the deployment of a DVFS mechanism is almost three times less than for the same experiment without the deployment of this mechanism. Also, for the remaining experiments, the energy consumption for the DVFS scenarios was almost twice as less than the consumption with the other scenario. 4.1.6. Scenario 6: Increasing the No. Of Hosts and Cloudlets 4.1.6.1. Objectives and Design The objectives of the following experiments are similar to the previous scenarios, but this time it involves increasing the number of Hosts and Cloudlets only, while the number of VMs remains constant. Thus, the value of Hosts and Cloudlet parameters will be doubled each time to observe the variation in energy consumption. 4.1.6.2. Discussion of the Results Based on the results collected from the two scenarios, with and without the deployment of DVFS, shown in Table 12 and 13 in Appendix M, Figure 12 below has been produced to show the difference between the two scenarios based on the mean values of the energy consumption. For this scenario, experiment 1 has the minimum values of the parameter settings with 10 hosts, 20 VMs, and 20 Cloudlets, and experiment 13 has the maximum parameter values of 30720 hosts, 20 VMs, and 61440 Cloudlets. 33

3500 3000 2500 2000 1500 1000 Without Power Aware Mechanism With DVFS Mechanism 500 0-500 Figure 12. Comparison between energy consumption mean values for each experiment Based on the results shown in Figure 12 and the tables in Appendix M, doubling the number of Hosts and Cloudlets would also double the energy consumption for the scenario without the deployment of DVFS. But, with the deployment of DVFS, the energy consumed has remained the same. These results are exactly the same as the results shown in the previous scenario 2. However, the only difference is that in this scenario, CloudSim accepts doubling the values till experiment 12 and in experiment 13 the values have been increased by just 50%, while in scenario 2 it continues by handling two more experiments till experiment 14 without crashing the simulation. This may suggest that increasing the number of Cloudlets would not affect the energy consumption, as this can also be observed in scenario 4 as well. 4.2 VM Migration 4.2.1. Scenario 7: Increasing the No. Of Hosts, VMs, and Cloudlets 4.2.1.1. Objectives and Design The objectives of the following experiments are to evaluate how increasing the number of resources and services would reflect on the energy consumption in an environment with the deployment of a VM migration mechanism with DVFS enabled, compared to those results shown in scenario 1 with DVFS mechanism deployment only. 34

The parameter settings for the number of hosts, VMs, and Cloudlets will initially be according to the default values already set in CloudSim, and then they will be doubled each time in order to observe the variation of energy consumption. Also, it is important to note that the utilisation threshold will be set as the typical value of 0.8 (80%) in order to prevent allocating too great a number of VMs on one physical host to the extent that it may degrade the performance of VM. 4.2.1.2. Discussion of the Results Based on the results collected from the two scenarios, with the deployment of VM migration and DVFS enabled and the deployment of DVFS only, shown in Table 14 in Appendix N and Table 2 in Appendix G, Table 15 and Figure 13 below have been produced to show the difference between the two scenarios based on the mean values of the energy consumption. For this scenario, experiment 1 has the minimum values of the parameter settings with 10 hosts, 20 VMs, and 20 Cloudlets, and experiment 9 has the maximum values of 2560 hosts, 5120 VMs, and 5120 Cloudlets. Table 15. Comparison for energy consumption mean values between scenario 7 and 1 VM Migration and DVFS enabled (Scenario 7) DVFS Mechanism Only (Scenario 1) Percentage of Energy Reduction Achieved Experiment 1 0.221429 0.245714 9.88% Experiment 2 0.41 0.50 18% Experiment 3 0.757143 1.005714 24.71% Experiment 4 1.435714 2.094286 31.45% Experiment 5 2.96 4.187143 29.30% Experiment 6 5.99 8.428571 28.93% Experiment 7 11.87 16.91 29.80% Experiment 8 23.24857 33.87 31.36% Experiment 9 46.80571 67.69571 30.86% 35

80 70 60 50 40 30 20 With VM Migration and DVFS Enabled With DVFS Mechanism Only 10 0-10 -20 Figure 13. Comparison between energy consumption mean values for each experiment Thus, based on the results shown in the previous Table 15 and Figure 13, doubling the number of resources and services would also double the energy consumption for both scenarios, with the deployment of VM migration and DVFS enabled and with the deployment of DVFS mechanism only. But the energy consumed with the deployment of VM migration and DVFS enabled is about 30% less than the energy consumption with the deployment of DVFS, except for only the first three experiments which have fewer reduction values, as can be seen in Table 15 above. Therefore, the deployment of VM migration along with DVFS being enabled would save more energy because the VM migration mechanism would further reduce energy by allocating more VMs to less physical machines to allow maximum utilisation of the running of physical machines [37]. 4.2.2. Scenario 8: Increasing the No. Of Hosts 4.2.2.1. Objectives and Design The objectives of the following experiments are to evaluate how increasing the number of hosts would reflect on the energy consumption in an environment with the deployment of VM migration with DVFS enabled, compared to those results in scenario 2 with DVFS mechanism deployment only. The parameter settings for the number of hosts, VMs, and Cloudlets will initially be according to the default values already set in CloudSim, and then the number of hosts only will be doubled each time to observe the variation in energy consumption. Also, the utilisation threshold will remain at the typical value of 0.8 for all the experiments to prevent performance degradation. 36

Experiment 1 Experiment 2 Experiment 3 Experiment 4 Experiment 5 Experiment 6 Experiment 7 Experiment 8 Experiment 9 Experiment 10 Experiment 11 Experiment 12 Experiment 13 Experiment 14 4.2.2.2. Discussion of the Results Based on the results collected from the two scenarios, with the deployment of VM migration and DVFS enabled, and the deployment of DVFS only, shown in Table 16 in Appendix O and Table 4 in Appendix I, Figure 14 below has been produced to show the difference between the two scenarios based on the mean values of the energy consumption. For this scenario, experiment 1 has the minimum values of the parameter settings with 10 hosts, 20 VMs, and 20 Cloudlets, and experiment 14 has the maximum values of 81920 hosts, 20 VMs, and 20 Cloudlets. 0.3 0.25 0.2 0.15 0.1 0.05 With VM Migration and DVFS Enabled With DVFS Mechanism Only 0 Figure 14. Comparison between energy consumption mean values for each experiment Based on the results shown in Figure 14, doubling the number of machines would mostly not affect the energy consumption for both scenarios because DVFS is enabled for both to turn on only the required number of machines, and the rest are turned off. But with the deployment of VM migration and DVFS enabled in scenario 7, the energy consumed is less than the energy consumption in scenario 2 with DVFS deployment only, because the VM migration would further reduce the energy consumption by utilising the current running physical machines [37]. 37

4.3 Overall Discussion and Results Comparison Overall, as can be seen from the results of the experiments, deploying DVFS and VM migration mechanisms can enhance the eco-efficiency of a data centre by lowering the performance overheads. Based on the scenarios from 1 to 6, a data centre with the deployment of DVFS mechanism would consume significantly less energy than a data centre with no deployment of a power aware mechanism, which reflects how effectively the DVFS mechanism has been designed to dynamically control the input voltage and clock frequency of a CPU to decrease the power consumption, as stated by Raj and Shriram [45]. A summarised comparison of the energy consumption between the deployment of the DVFS mechanism and no deployment of a power aware mechanism for these scenarios can be seen in Table 17 below. Table 17. Benchmarking of energy consumption between the deployment of DVFS mechanism and non power aware (NPA) mechanism in a data centre Scenario 1: Doubling the number of hosts, VMs, and Cloudlets Scenario 2: Doubling the number of hosts Scenario 3: Doubling the number of VMs Scenario 4: Doubling the number of Cloudlets Scenario 5: Doubling the number of hosts and VMs Scenario 6: Doubling the number of hosts and Cloudlets Variation of Energy consumption for each subsequent experiment with DVFS Variation of Energy consumption for each subsequent experiment with NPA Average energy savings Double Double About 70% Stable Almost stable Double Stable Varies (Dramatic increase in energy savings as the scale of the experiments goes higher) About 70% in experiment 1 About 16% for the remaining experiments Stable Stable About 70% Almost double Stable Double Double About 70% in experiment 1 About 55% for the remaining experiments Varies (Dramatic increase in energy savings as the scale of the experiments goes higher) 38

Based on the results shown in the table above, scenario 2 and 6 are almost the same, which suggests along with scenario 4 that increasing the number of cloudlets does not influence energy consumption. Moreover, the results of scenario 1 are similar to those of scenario 5. Although the energy saving with the deployment of DVFS compared to NPA in scenario 5 is about 70% only for the first experiment, and for the remaining it is about 55%; while in scenario 1 it is about 70% for all the experiments. Furthermore, scenario 3 suggests that increasing the number of VMs only would not influence the energy consumption while the number of hosts and cloudlets are constant. Similarly, scenario 4 suggests that increasing the number of cloudlets only, would not influence the energy consumption while the number of hosts and VMs remain constant. However, increasing the number of hosts only would influence the energy consumption, as can be seen in scenario 2 with NPA mechanism. Thus, it can be concluded that the key parameter that influences energy consumption most significantly is the number of hosts. Also, the number of VMs would also influence the energy consumption only if this is increased with the number of hosts. Additionally and most importantly, the deployment of the DVFS mechanism shows significant energy savings, as can be seen in the last column in Table 17 above. Therefore, the DVFS mechanism does a good job saving energy, and this can be justified by the way it has been designed to control the voltage supply and frequency scaling of a CPU. Moving on, for the deployment of VM migration with DVFS enabled, a further reduction of energy consumption has been shown in scenarios 7 and 8 compared to scenarios 1 and 2 respectively with the deployment of DVFS mechanism only. For scenario 7 with doubling the number of hosts, VMs and Cloudlets, it shows about a 30% further reduction in energy consumption, while in scenario 8 with doubling the number of hosts only it shows about 20% further reduction in energy consumption. This can be justified by the way that the VM migration mechanism is designed to allocate more VMs on less physical machines to enable maximum utilisation of the running of physical machines. 39

5. Evaluation This chapter will evaluate the project from three different aspects. Firstly, it will evaluate the project in terms of meeting the aim and minimum requirements. Secondly, it will evaluate the appropriateness of the chosen methodology. Thirdly, it will evaluate the results of the experiments, which will include a comparison of the two techniques investigated and an overall comparison of the results of the experiments conducted in this project with those already contained in the literature. 5.1. Evaluation According to Minimum Requirements The aim of this project is to evaluate DVFS mechanisms that can be used to improve the ecoefficiency of Clouds data centres. The minimum requirements for this project, as previously stated in Section 1.3, are: - Design a simulated scenario of a cloud data centre using DVFS mechanism - Implement the scenario - Collect and evaluate the simulation results - Evaluate the mechanism used to improve the eco-efficiency of data centre management in Cloud Computing In order to achieve the aim and minimum requirements, the literature review in Section 2.1 has thoroughly investigated some important aspects of Cloud Computing, including the architecture of Clouds, types of Cloud services, types of Cloud deployment, and virtualisation. Also, in Section 2.2, it has investigated some issues associated with Clouds, including the cost of Cloud maintenance and, most importantly, management of eco-efficiency in data centres. In Section 2.3, it has investigated the CloudSim toolkit in terms of its architecture, usability, capability, and limitations. Moreover, in Section 3.4 and throughout Chapter 4, this project has not only designed and implemented one simulated scenario of a Cloud data centre using a DVFS mechanism successfully, but also another five scenarios, along with comparisons with other scenarios in an environment without the deployment of a power aware mechanism, in order to compare and evaluate the energy reduction. In addition, two more scenarios with the deployment of VM migration and DVFS enabled have been designed and implemented to add more novelty to the project, by allowing further 40

energy reduction and improvement of the eco-efficiency of a data centre. Thus, the aim and minimum requirements have been achieved with further enhancements as well. 5.2. Evaluating the Methodology The methodology used in this project is based on using a simulation for experimenting with scenarios of a Cloud data centre to evaluate the mechanisms that can be used to improve ecoefficiency. Using the simulation as a methodology for this project has been effective for the reasons that it offers an environment that is scalable, reliable, repeatable, and controllable for performing some benchmarking experiments, with low overheads in terms of cost and effort [6]. In terms of the simulator, CloudSim toolkit has been validated and investigated with regard to its efficiency in modelling some scenarios of Clouds by the people who developed it from the University of Melbourne. Therefore, CloudSim has been widely used by researchers and organisations, such as HP labs in the US, to model some scenarios for supporting their investigation into different aspects of Clouds, like the energy-efficient management of data centres [6]. For this project, CloudSim has been effective in terms of its flexibility and ability to support modelling a Cloud data centre, from small-scale to large-scale experiments. Additionally, it supports modelling some energy aware provisioning techniques, such as VM allocation and DVFS, as discussed in Section 2.3.3. Even so, it has some limitations, such as the findings of the experiments conducted in scenario 1 which reveal the maximum limit of the parameter values that CloudSim can handle. Also, running the experiments sometimes takes a long time to complete, especially when the scale of the experiment is high. Because of the randomness of CloudSim simulation, each experiment had to be run seven times in order to obtain more reliable results, which also increases the time required for running the experiments. In terms of the development approach mentioned in Section 3.3, the iterative approach has been found suitable and effective for the scenarios conducted in this project because it provides clear and easy to follow stages for the scenarios development. Also, it allows speedy adjustment of the design settings in the case of the experiments that have not been run successfully. 41

5.3. Evaluating the Results In this section, an evaluation will be presented from two dimensions. First, it will evaluate and compare the results of the two mechanisms, DVFS and VM migration, in terms of how each one would contribute towards enhancing the eco-efficiency of data centres. Second, it will contrast the results of the experiments conducted in this project with those already included in the literature. Firstly, as has been stated by Raj and Shriram [45], a DVFS mechanism focuses on utilising the CPU by manipulating the voltage and clock frequency of a CPU to decrease the power consumed. Based on the results of the experiments simulated in Section 4.1, scenarios 1 to 6 show significant energy reduction with the deployment of a DVFS mechanism in a data centre, ranging from 16% reduction up to 70%, and an even greater reduction for scenarios 2 and 6. Furthermore, the VM allocation mechanism maximises the utilisation of the physical machines running by dynamically migrating more VMs to less physical machines as much as possible, but to the extent that it does not downgrade its performance [30]. Based on the results from the simulated experiments in Section 4.2, scenarios 7 and 8 show further energy reduction with the deployment of VM migration with DVFS enabled in a data centre, ranging from 20% up to 30%. Thus, based on the results from all of the scenarios, the deployment of both DVFS and VM migration in a data centre would reduce energy consumption significantly, and it can be deduced that it would also contribute greatly to enhancing the eco-efficiency of data centres. However, it would have added more novelty to the project if the experiments had investigated the efficiency of mechanisms based on variant metrics instead of only the energy consumption metric. Secondly, one study has concluded with the overall evaluation that VM migration with the goal of server consolidation is always suitable for reducing power consumption; however, it can lead to performance degradation when the number of consolidated VMs exceeds the specified limit of a service threshold [37]. Similarly, the experiments conducted with VM migration in this project have also shown the applicability of the mechanism to reduce power consumption without performance degradation, by setting the threshold to 0.8 (80% utilisation or less). Moreover, another study involved conducting an experiment to evaluate VM migration with the application of DVFS [6]. The results show that such an energy conscious technique can considerably decrease the power consumption of the hosts in a data centre by up to 50%. Whereas an evaluation of the VM migration with DVFS enabled conducted in this project, shows the results to be a reduction in power consumption of up to 30%, from which it can be deduced that the difference 42

may be because of the different settings of the parameter values for the experiments. Also, this project has included scenarios consisting of multiple experiments, starting from modelling small scale data centres up to the large scale; whereas the experiment conducted in the other study referred to, has only modelled one scenario representing one scale of data centre. 5.4. Limitations The assessment of the two experimental mechanisms, DVFS and VM migration, in this project has been limited to measuring the metric of energy consumption only. If another metric, such as SLAs violation, was included, it would have added more novelty to the project in terms of measuring the efficiency of the mechanisms by representing some trade-offs. However, due to the lack of knowledge concerning the architecture of CloudSim before conducting this project, it was difficult to extend the CloudSim to include another metric for measuring the SLAs violation, especially due to the limited time-scale allocated to the project. Therefore the evaluation of these two mechanisms could be enhanced by measuring their efficiency based on variant metrics including energy consumption and SLAs violation. Some of the scenarios simulated may possibly have had similar results with repeated points, and this can be justified due to each scenario not being designed and implemented based on specific needs and real requirements; instead, they were all based on generic requirements and objectives using typical value settings. This could be avoided in the future by studying and analysing a specific real data centre, and then performing the simulation experiments to evaluate the efficiency of deploying the DVFS and VM migration techniques to enhance the eco-efficiency of that particular data centre. 5.5. Future Work In addition to overcoming the limitations mentioned in the previous Section 5.4, direct experiments involving the scenarios could be set up in a laboratory to obtain more accurate results for measuring the performance of energy-conscious techniques, such as DVFS and VM allocation, in a real environment. However, the development approach would be different to that used in this project. In essence, it would start with planning and setting up the experiments in terms of the computing resources, including machines and networks. Then, there should be a device put in place for measuring the consumed energy. Finally, after the implementation comes analysing and evaluating the results. 43

This project has focused on evaluating the techniques that could be used in a data centre as a way of improving the eco-efficiency of Cloud Computing in terms of the infrastructure layer. However, the eco-efficiency of Cloud Computing can be improved for different layers as well. For example, for the top layer, designing a framework to develop applications that would consume less energy when running on the Cloud infrastructure would also contribute towards the eco-efficiency of Cloud Computing. 44

6. Conclusion 6.1. Summary As the usage of Cloud Computing is continually increasing, the energy consumption of the computing resources, such as data centres, which underpin the Clouds, will increase as well. This would increase the burden of the operational costs and the environmental impact if there is no control over the energy consumption. Therefore, controlling and reducing the energy consumption in data centres would not only lower the operational costs, which in return would increase the profits, but also lower the environmental impact in terms of CO2 emissions. Thus, this project has investigated and evaluated two techniques, namely DVFS and VM migration, which can be deployed in data centres to control the energy consumption and improve their eco-efficiency. The experiments explained in Chapter 4 have shown that significant energy reduction in data centres could be achieved with the deployment of DVFS and VM migration. 6.2. Problems Encountered A number of problems have been faced while conducting the project, particularly when conducting the experiments. Since CloudSim is open source software, there is no user manual for it, which made it difficult to understand how to use the toolkit. This meant it took longer than expected to carry out further research and reverse engineering in order to understand how the tool works for modelling some Cloud scenarios. The Google group for CloudSim community was very useful when attempting to tackle this issue. Also, the time for running the experiments was very long, especially when the scale of the scenario is high. Also, because of the randomness of the simulation, each experiment had to be run multiple times. For example, scenario one had 15 experiments and each experiment was run seven times, which means that 105 runs were conducted through eclipse in order to complete scenario 1 only. Thus, it was very difficult and time consuming to complete the rest of the scenarios in the same way; therefore, by using shell scripting, a number of experiments were run simultaneously to speed up conducting the remaining scenarios. 45

7. Bibliography [1] Mell, P. and Grance, T. (2011) The NIST Definition of Cloud Computing. [Internet], National Institute of Standards and Technology, September, p.1-3. Available from: <http://csrc.nist.gov/publications/nistpubs/800-145/sp800-145.pdf> [Accessed 8 th June 2012] [2] Deng, J et al (2010) Research and Application of Cloud Storage. 2 nd International Workshop on Intelligent Systems and Applications (ISA), 22-23 May, pp. 1-5. [3] Yeo, S. and Lee, H. (2011) Using Mathematical Modelling in Provisioning a Heterogeneous Cloud Computing Environment. Computer, 44(8), August, pp. 55-62. [4] Kliazovich, D. et al (2010) GreenCloud: A Packet-level Simulator of Energy-aware Cloud Computing Data Centers. IEEE Global Telecommunication Conference (GLOBECOM 2010), 6-10 December, pp.1-5. [5] Grag, S. and Buyya, R. (2011) NetworkCloudSim: Modelling Parallel Applications in Cloud Simulations. Fourth IEEE International Conference on Utility and Cloud Computing (UCC), 5-8 December, pp. 105-113. [6] Calheiros, R. et al (2010) CloudSim: A Toolkit for Modelling and Simulation of Cloud Computing Environments and Evaluation of Resources Provisioning Algorithms. Software Practice and Experience 41, 24 August, pp. 23-50, John Wiley & Sons, Ltd. [7] Wickremasinghe, B., Calheiros, R. and Buyaa, R. (2010) CloudAnalyst: a CloudSim-based Visual Modeller for Analysing Cloud Computing Environments and Applications. 24 th IEEE International Conference on Advanced Information Networking and Applications, 20-23 April, pp. 446-452. [8] Doest, L. Et al (2006) Personal Goal Facilitation Through Work: Implications for Employee Satisfaction and Well-Being. Applied Psychology: An International Review, 55(2), pp. 192-219. [9] Krafzig, D. et al (2005) Enterprise SOA: Service-Oriented Architecture: Best Practice. Upper Saddle River, Pearson Education, Inc. [10] Josuttis, N. (2007) SOA in Practice: The Art of Distributed System Design. California, O Reilly Media, Inc. [11] Chorafas, D. (2011) Cloud Computing Strategies. United States of America: CRS Press, Taylor and Francis Group. [12] Buyya, R. et al (2009) Cloud Computing and emerging IT Platforms: Vision, Hype, and Reality for Delivering Computing as the 5 th Utility. Future Generation Computer Systems 25, pp. 599-616. [13] Vaquero, L. et al (2009) A Break in the Clouds: Towards a Cloud Definition. ACM SIGCOMM Computer Communication Review, 39(1), January, pp. 50-55. 46

[14] Miller, M. (2008) Cloud Computing: Web-Based Applications That Change the Way You Work and Collaborate Online. United States of America: Que Publishing. [15] Foster, I. et al (2008) Cloud Computing and Grid Computing 360-Degree Compared. Grid Computing Environment Workshop, GCE 0. 12-16 November, pp. 1-10. [16] Buyya, R., Ranjan, R., and Calheiros, R. (2009) Modelling and Simulation of Scalable Cloud Computing Environment and the CloudSim Toolkit: Challenges and Opportunities. International Conference on High Performance Computing and Simulation, HPCS 09. 21-24 June, pp. 1-11. [17] Velte, A., Velte, T., and Elsenpeter, R. (2010) Cloud Computing: A Practical Approach. United States of America: McGraw-Hill. [18] Tsai, W., Sun, X., and Balasooriya, J. (2010) Service-Oriented Cloud Computing Architecture. Seventh International Conference on Information Technology. 12-14 April, pp. 684-689. [19] Rehman, Z., Hussain, F., and Hussain, O. (2011) Towards Multi-Criteria Cloud Service Selection. Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing. 30 June 2 July, pp. 44-48. [20] Ye, K. et al (2010) Virtual Machine Based Energy-Efficient Data Center Architecture for Cloud Computing: A Performance Perspective. IEEE/ACM International Conference on Green Computing and Communications & IEEE/ACM International Conference on Cyber, Physical and Social Computing. 18-20 Devember, pp. 171-178. [21] Duran-Limon, H. et al (2011) Using Lightweight Virtual Machines to Run High Performance Computing Applications: The Case of Weather Research and Forecasting Model. Fourth IEEE International Conference on Utility and Cloud Computing. 5-8 December, pp. 146-153. [22] Hardy, J. et al (2012) Assessment and Evaluation of Internet-based Virtual Computing Infrastructure. 15 th IEEE International Symposium on Object/Component/Service-Oriented Real- Time Distributed Computing. 11-13 April, pp. 39-46. [23] Armstrong, D. and Djemame, K. (2009) Towards Quality of Service in the Cloud. Proceedings of the 25 th UK Performance Engineering Workshop, pp. 226-240. [24] Nakajima, J. and Mallick, A. (2007) Hybrid Virtualisation Enhanced Virtualisation for Linux. Proceedings of Linux Symposium, 2, 27 th -30 th June, pp. 87-96. [25] Tsaravas, C. and Themistocleous, M. (2011) Cloud Computing and EGovernment: A Literature Review. European, Mediterranean & Middle Eastern Conference on Information Systems. 30-31 May, pp. 154-164. [26] Sabahi, F. (2011) Cloud Computing Security Threats and Responses. IEEE 3 rd International Conference on Communication Software and Networks. 27-29 May, pp. 245-249. [27] Li, X. et al (2009) The Method and Tool of Cost Analysis for Cloud Computing. IEEE International Conference on Cloud Computing, CLOUD 09. 21-25 September, pp. 93-100. 47

[28] Smarr, L. (2010) Project GreenLight: Optimizing Cyber-Infrastructure for a Carbon-Constrained World. Computer, 43(1), January, pp. 22-27. [29] Moreno, I., and Xu, J. (2011) Energy-Efficiency in Cloud Computing Environments: Towards Energy Savings without Performance Degradation. International Journal of Clouds Applications and Computing, 1, pp. 16-33. [30] Jaureguialzo, E. (2011) PUE: The Green Grid metric for evaluating the energy efficiency in DC (Data Center). Measurement method using the power demand. IEEE 33rd International Telecommunications Energy Conference (INTELEC). 9-13 October, pp.1-8. [31] Tellbach, D. (2012) What are Next Steps to Make PUE Statements more Helpful? Datacentre Transformation Conference, [Internet], at the University of Leeds, 26-27 June. Available from: <http://www.dtconference.com/presentations> [Accessed 30 th July 2012] [32] Gillan, P. (2012) Fresh Air The Coolest Way to Save Energy: Direct Free Cooling Utilising Outside Air. Datacentre Transformation Conference, [Internet], at the University of Leeds, 26-27 June. Available from: <http://www.dtconference.com/presentations> [Accessed 30 th July 2012] [33] Patterson, M. et al (2010) WP 29-ERE: A Metric for Measuring the Benefits of Reuse Energy from a Data Center. [Internet], The Green Grid, 22 September. Available from: <http://www.thegreengrid.org/en/global/content/white-papers/ere> [Accessed 30 th July 2012] [34] VanGeet, O. (2010) FEMP Best Practice Guide for Energy-Efficient Data Center Design. [Internet], Federal Energy Management Program (FEMP), prepaid by the National Renewable Energy Laboratory (NREL), US. Available from: <http://www.nrel.gov/docs/fy11osti/47201.pdf> [Accessed 30 th July 2012] [35] Brady, E. (2012) A Critical Evaluation of the PUE Metric. Datacentre Transformation Conference, [Internet], at the University of Leeds, 26-27 June. Available from: <http://www.dtconference.com/presentations> [Accessed 30 th July 2012] [36] Rudgyard, M. (2012) End-To-End Monitoring and Management in Data Centre; a Fresh Approach. Datacentre Transformation Conference, [Internet], at the University of Leeds, 26-27 June. Available from: <http://www.dtconference.com/presentations> [Accessed 30 th July 2012] [37] Corradi, A., Fanelli, M., and Foschini, L. (2011) Increasing Cloud Power Efficiency through Consolidation Techniques. IEEE Symposium on Computers and Communications. 28 June 1 July, pp. 129-134. [38] Pakbaznia, E., Ghasemazar, M., and Pedram, M. (2010) Temperature-Aware Dynamic Resource Provisioning in a Power-Optimized Datacenter. Design, Automation & Test in Europe Conference & Exhibition. 8-12 March, pp. 124-129. [39] Kliazovich, D., Bouvry, P., and Khan, S. (2010) DENS: Data Center Energy-Efficient Network- Aware Scheduling. IEEE/ACM International Conference on Green Computing and Communications (GreenCom) & International Conference on Cyber, Physical and Social Computing (CPSCom). 18-20 December, pp. 69-75. 48

[40] Kim, K., Beloglazov, A., and Buyya, R. (20009) Power-aware Provisioning of Cloud Resources for Real-time Services. Proceedings of the 7 th International Workshop on Middleware for Grids, Clouds and e-science MGC 09, 30 th November, pp. 1-6. [41] Quiroz, A. et al (2009) Towards autonomic workload provisioning for enterprise grids and clouds. 10 th IEEE/ACM International Conference on Grid Computing, 13-15 October, pp. 50-57. [42] Jessen, S. (2011) The Impact on Project Success of Using Technology in Modern Project Planning and Control. IEEE International Technology Management Conference (ITMC), 27-30 June, pp. 801-805. [43] Scheihing, P. (2007) Creating Energy-Efficient Data Centers. Data Center Facilities and Engineering Conference, Washington, DC, 18 th May. [44] Markoff, J. and Lohr, S. (2002) Intel s Huge Bet Turns Iffy. The New York Times, [Internet], 29 th September. Available from: <http://www.nytimes.com/2002/09/29/business/intel-s-huge-bet-turnsiffy.html?pagewanted=all&src=pm> [Accessed 6 th August 2012] [45] Raj, V. and Shriram, R. (2011) Power Aware Provisioning in Cloud Computing Environment. International Conference on Computer, Communication and Electrical Technology ICCCET, 18 th - 19 th Marsh, pp. 6-11. [46] Shiraz, M. and Gani, A. (2012) Mobile Cloud Computing: Critical Analysis of Application Deployment in Virtual Machines. International Proceedings of Computer Science and Information Technology, vol.27, February, pp. 11-16. [47] Payton, M., Greenstone, M., and Schenker, N. (2003) Overlapping Confidence Intervals or Standard Error Intervals: What Do They Mean in Terms of Statistical Significance. Journal of Insect Science, 3(34), 30 October, pp. 1-6. [48] Beloglazov, A. and Buyya, R. (2011) Optimal Online Deterministic Algorithms and Adaptive Heuristics for Energy and Performance Efficient Dynamic Consolidation of Virtual Machines in Cloud Data Centres. Concurrency and Computation: Practice and Experience, [Internet] 7 th October. Available from: <http://beloglazov.info/papers/2012-optimal-algorithms-ccpe.pdf> [Accessed 1 st June 2012] 49

8. Appendices 8.1. Appendix A: Project Reflection This project has been very interesting and challenging as it provided me with vital skills and knowledge about Cloud Computing and the techniques used to improve the eco-efficiency of data centres. Having completed the project, I would like to share some experiences and lessons learnt with others interested in doing a similar project. First of all, once you have chosen a topic that is interesting to you, you should set your objectives and create a thorough project plan as early on as possible. It might be difficult to foresee everything at the beginning, but planning will help you to think ahead about how to effectively manage your time in order to complete the project successfully. Moreover, it is important to complete the background reading and some of the writing before May, when exams start, so that you have enough time to write a solid interim report, which has to be submitted about three weeks after the exams. Indeed, if the topic chosen is new to you, it would be very difficult to complete both the initial reading and writing within three weeks. Choosing an appropriate methodology for the project is another key success factor to achieving a better outcome. As my project was aimed at investigating several techniques through experiments, I found that using simulations and applying an iterative approach was most suitable for conducting the experiments because it gave me more flexibility. Personally, I expected the CloudSim simulator to be easy to use, but once I started exploring it, I faced numerous challenges, especially in terms of how to use it. If you use CloudSim for your project, it is important to start learning about the tool by studying and running simple examples available in the program; then explore it more deeply until you understand how it works and how you can employ it for your project. Also, it is important to explore the tool while carrying out the background reading before May so that you do not need to waste time learning it once you start the implementation, which was the most important lesson I learnt. In addition, if you encounter any issues while using the tool, visit the online discussions for CloudSim, which are available on Google group; you may find the answers to your queries there, or you can post your own queries to be answered by the people who developed CloudSim, who manage the group. 50

In terms of project management, ideally, have one project plan and stick to it. This will help to ensure that everything goes smoothly and as planned. However, sometimes you may have to change the project plan due to unexpected issues or the underestimation of some tasks. Thus, revise the project plan regularly to ensure successful completion of the project within the limited timescale. Furthermore, weekly meetings with the supervisor gave me the opportunity to discuss the issues encountered as well as gain more advice on improving the project. To make the meetings effective, prepare an agenda for each meeting to discuss the tasks you have achieved during that week and to set tasks to be completed in the following week. In addition, detailed and constructive feedback from both the supervisor and assessor for the interim report and the work in progress presentation was extremely useful, as it directed me to where improvements should be made in the project. If I were to do the project again, I would have prepared and included more work in the interim report and the presentation in order to obtain more feedback. Finally, it has been my experience that discussing the project with friends can help identify some interesting points for improvements, although not necessarily from a technical perspective. This approach may, however, help with the overall organisation of the project. Overall, I have learnt a lot from completing this project and have truly enjoyed the experience. 51

8.2. Appendix B: Interim Report 52

53

8.3. Appendix C: Initial Project Plan 54

55

8.4. Appendix D: Revised Project Plan 56

57

8.5. Appendix E: Weekly Progress Report Weekly Progress Report 1 Date: 22/June/2012 Tasks undertaken and progress accomplished for this week - Read more about how to use CloudSim toolkit - Understand the examples provided with CloudSim Planned tasks for next week - Learn more about using CloudSim - Analyse and design a scenario to simulate DVFS mechanism Supervisor Name: Dr. Karim Djemame Student Name: Ibrahim Alzamil Student ID: 200627581 58

Weekly Progress Report 2 Date: 29/June/2012 Tasks undertaken and progress accomplished for this week - Read and learn more about using CloudSim, and particularly understand the DVFS example provided in CloudSim-2.1.1. - Analyse and design four scenarios to simulate DVFS mechanism. - Understand how to calculate the mean, standard deviation, and confidence interval for the energy consumption statistics of each experiment. - Run the experiments for each scenario and collect the results. - Calculate the results and find out the mean, standard deviation, and confidence interval for each experiment. - Analyse and discuss the results Planned tasks for next week - Find out the maximum values of the parameters setting for each scenario that CloudSim can handle. - Do two more scenarios by only increasing: 1 - the no of hosts and VMs, 2 - the no of hosts and cloudlets, - Look at VM allocation mechanism if there is enough time. Supervisor Name: Dr. Karim Djemame Student Name: Ibrahim Alzamil Student ID: 200627581 59

Weekly Progress Report 3 Date: 6/July/2012 Tasks undertaken and progress accomplished for this week - Find out the maximum values of the parameters setting that CloudSim can handle in scenario 1, and run the experiments, collect the results, and then calculate the results to get the mean, standard deviation, confidence interval for each experiment. - Analyse and discuss the results - Try to complete scenario 2 and do the same as for scenario 1. But still uncompleted because the simulation takes a very long time! Planned tasks for next week - Learn how to use shell scripting to speed up the simulation time. - Extend the scenarios 2, 3, and 4 - Do two more scenarios by only increasing: 1 - the no of hosts and VMs, 2 - the no of hosts and cloudlets Supervisor Name: Dr. Karim Djemame Student Name: Ibrahim Alzamil Student ID: 200627581 60

Weekly Progress Report 4 Date: 12/July/2012 Tasks undertaken and progress accomplished for this week - Learn how to use shell scripting to speed up the simulation time. - Extend the scenarios (2, 3, and 4) up to experiment 14 or to a point where CloudSim cannot handle doubling the parameter values. Run and collect the results. - Calculate the results to find out the mean, standard deviation, and confidence interval. - Use the built-in function in Excel to calculate the confidence interval instead of the previous way, and adjust all the results accordingly. - Do two more scenarios by only increasing: 1 - the no of hosts and VMs, 2 - the no of hosts and cloudlets. - Analyse and discuss the results Planned tasks for next week - Prepare table of contents and a draft chapter of the report to submit on 20 th of July. - Design and implement two scenarios of VM migration mechanism Supervisor Name: Dr. Karim Djemame Student Name: Ibrahim Alzamil Student ID: 200627581 61

Weekly Progress Report 5 Date: 20/July/2012 Tasks undertaken and progress accomplished for this week - Review Project Plan - Review aim and minimum requirements - Prepare table of content - Analyse and designing two scenarios of VM migration mechanism - Run experiments for each scenario and collect results, and fixing the bug found to continue running - Calculate the results and find out the mean, standard deviation, and confidence interval for each experiment - Analyse and discuss the results Planned tasks for next week - Complete scenario 2 - Prepare a draft chapter of Implementation - Plan for Evaluating the results Supervisor Name: Dr. Karim Djemame Student Name: Ibrahim Alzamil Student ID: 200627581 62

Weekly Progress Report 6 Date: 27/July/2012 Tasks undertaken and progress accomplished for this week - Complete scenario two of VM migration - Write a draft chapter of implementation - Review table of contents - Plan for evaluation Planned tasks for the remaining weeks - Start writing up: - Review and adjust the Interim Report (first two chapters) based on the feedback from supervisor and assessor - Write up the Methodology chapter - Review the Implementation chapter based on the supervisor feedback - Evaluate the project - Write up the Conclusion - Finalise the project and write Personal Reflection Supervisor Name: Dr. Karim Djemame Student Name: Ibrahim Alzamil Student ID: 200627581 63

8.6. Appendix F: Work in Progress Presentation Work in Progress Problem summary What is the problem? Cloud infrastructure is likely embodied by enormous data centres that consume huge energy in order to work [29]. Managing cloud data centres eco-efficiently Why is it important? Reduce energy consumption Reduce operational costs Increase profit Reduce CO2 emissions and be more friendly with the environment Ibrahim Alzamil Applying some power aware mechanisms to control and reduce power consumption Dynamic Voltage and Frequency Scaling (DVFS) Alters the CPU power consumption based on the offered workload. Virtual Machine (VM) Allocation Migrates more VMs on less physical machines to allow maximum utilisation of running physical machines. Aim: Evaluate a technique that can be used to improve the eco-efficiency and operational costs. Objectives: Understand the importance of cloud computing, architecture of clouds, types of clouds, models of clouds, and virtualization. Understand some issues associated with cloud computing. Understand CloudSim toolkit in terms of its usability, capability and limitations to offer a feasible evaluation of the eco-efficiency techniques used in data centres. 64

Energy Consumption kwh Analysis Evaluation Minimum Requirements: Designing and implementing a simulated scenario of a cloud data centre using DVFS mechanism Collecting and evaluating the results Providing eco-efficiency evaluation of the mechanism used to handle some of the issues associated with data centre management of cloud computing Further enhancements: Designing and implementing five more scenarios of a cloud data centre using DVFS mechanism and others without using DVFS mechanism Designing and implementing two more scenarios of a cloud data centre using VM Allocation mechanism Collecting and evaluating the results Run Design Verify Results Six simulated scenarios with DVFS mechanism compared to an environment with no deployment of power aware mechanism Two simulated scenarios with the deployment of VM migration and DVFS-enabled compared to an environment with DVFS deployment only Objectives: How increasing the No. of resources and services would affect energy consumption How the deployment of DVFS would contribute to reduce energy compared to an environment without this mechanism Design: Based on the parameter settings of hosts, VMs, and cloudlets Doubling the number of hosts, VMs and cloudlets for each experiment Run: Now, will go through demonstration of CloudSim Results: 600 500 400 300 200 100 With DVFS Mechanism Without Power Aware Mechanism 0-100 Figure 1: Comparison between energy consumption mean values for each experiment 65

Writing up the dissertation Reviewing Introduction and LR chapters based on the interim report feedback Writing up the methodology chapter Reviewing Implementation chapter based on the feedback Evaluating the project Writing up the Conclusion Finalising and writing personal reflection Thank you 66

8.7. Appendix G: Collected Results of Scenario 1: Increasing the No. Of Hosts, VMs, and Cloudlets DVFS Mechanism Table 2. Shows 15 experiments with the deployment of DVFS mechanism. Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 20 No of VMs = 40 = 40 40 No of VMs = 80 = 80 80 No of VMs = 160 = 160 160 No of VMs = 320 = 320 320 No of VMs = 640 = 640 640 No of VMs = Results of Energy Consumption (EC) for each run Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 0.25 kwh 0.25 kwh 0.24 kwh 0.25 kwh 0.24 kwh 0.25 kwh 0.24 kwh 0.50 kwh 0.49 kwh 0.50 kwh 0.50 kwh 0.49 kwh 0.49 kwh 0.50 kwh 1.00 kwh 1.02 kwh 1.01 kwh 1.03 kwh 0.99 kwh 0.98 kwh 1.01 kwh 2.12 kwh 2.10 kwh 2.09 kwh 2.08 kwh 2.11 kwh 2.07 kwh 2.09 kwh 4.20 kwh 4.18 kwh 4.17 kwh 4.21 kwh 4.14 kwh 4.19 kwh 4.22 kwh 8.44 kwh 8.41 kwh 8.43 kwh 8.39 kwh 8.44 kwh 8.45 kwh 8.44 kwh 16.94 kwh 16.93 kwh 16.92 kwh Mean Value of EC 0.245714 0.50 1.005714 2.094286 4.187143 Standard Deviation of EC 0.005345 0.005345 0.017182 0.017182 0.026904 8.428571 0.021157 16.91 0.041519 Confidence Interval of EC 0.249674 0.241755 0.499674 0.491755 1.018443 0.992986 2.107014 2.081557 4.207073 4.167213 8.444244 8.412898 16.94361 67

1280 Run 4 16.86 kwh Run 5 16.85 kwh 16.8821 = 1280 Run 6 16.96 kwh Run 7 16.93 kwh Exp.8 Run 1 33.86 kwh Run 2 33.83 kwh 1280 Run 3 33.91 kwh 33.89671 No of VMs = Run 4 33.91 kwh 33.87 0.036056 2560 Run 5 33.90 kwh Run 6 33.83 kwh 33.84329 = 2560 Run 7 33.85 kwh Exp.9 Run 1 67.68 kwh Run 2 67.66 kwh 2560 Run 3 67.75 kwh 67.72853 No of VMs = Run 4 67.69 kwh 67.69571 0.044293 5120 Run 5 67.64 kwh Run 6 67.69 kwh 67.6629 = 5120 Run 7 67.76 kwh Run 1 101.46 kwh Run 2 101.60 kwh 3840 Run 3 101.69 kwh 101.6961 Exp.10 No of VMs = Run 4 101.51 kwh 101.61 0.11619 +%50 7680 Run 5 101.53 kwh Run 6 101.77 kwh 101.5239 = 7680 Run 7 101.71 kwh Run 1 114.28 kwh Run 2 114.39 kwh 4320 Run 3 114.31 kwh 114.3747 Exp.11 No of VMs = Run 4 114.21 kwh 114.2857 0.120119 +%12.5 8640 Run 5 114.26 kwh Run 6 114.46 kwh 114.1967 = 8640 Run 7 114.09 kwh Run 1 121.54 kwh Run 2 121.47 kwh 4590 Run 3 121.41 kwh 121.5032 Exp.12 No of VMs = Run 4 121.44 kwh 121.45 0.075656 +%6.25 9180 Run 5 121.42 kwh Run 6 121.32 kwh 121.3911 = 9180 Run 7 121.53 kwh Run 1 121.77 kwh Run 2 121.65 kwh 4598 Run 3 121.75 kwh 121.782 Exp. 13 No of VMs = Run 4 121.80 kwh 121.7171 0.087505 +%0.2 9196 Run 5 121.58 kwh 121.6523 = 9196 Run 6 121.81 kwh Run 7 121.66 kwh Run 1 121.90 kwh Run 2 121.91 kwh 4602 Run 3 121.74 kwh 121.8553 Exp.14 No of VMs = Run 4 121.77 kwh 121.7686 0.117108 +%0.1 9204 Run 5 121.56 kwh Run 6 121.75 kwh 121.6818 = 9204 Run 7 121.75 kwh Exp.15 Run 1 121.81 kwh 121.82 0.095917 68

+%0.02 4603 No of VMs = 9206 = 9206 Run 2 121.96 kwh 121.8911 Run 3 121.83 kwh Run 4 121.75 kwh Run 5 121.79 kwh 121.7489 Run 6 121.92 kwh Run 7 121.68 kwh Table 3. Shows 15 experiments without the deployment of power aware mechanism Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Parameters Setting 10 No of VMs = 20 No of Cloudlets 20 No of VMs = 40 No of Cloudlets = 40 40 No of VMs = 80 No of Cloudlets = 80 80 No of VMs = 160 No of Cloudlets = 160 160 No of VMs = 320 No of Cloudlets = 320 320 No of VMs = 640 No of Cloudlets = 640 Results of Energy Consumption (EC) for each run Run 1 0.87 kwh Run 2 0.88 kwh Run 3 0.89 kwh Run 4 0.86 kwh Run 5 0.90 kwh Run 6 0.84 kwh Run 7 0.85 kwh Run 1 1.83 kwh Run 2 1.74 kwh Run 3 1.81 kwh Run 4 1.78 kwh Run 5 1.84 kwh Run 6 1.75 kwh Run 7 1.77 kwh Run 1 3.63 kwh Run 2 3.51 kwh Run 3 3.56 kwh Run 4 3.50 kwh Run 5 3.60 kwh Run 6 3.63 kwh Run 7 3.61 kwh Run 1 7.28 kwh Run 2 7.31 kwh Run 3 7.14 kwh Run 4 7.47 kwh Run 5 7.19 kwh Run 6 7.39 kwh Run 7 7.25 kwh Run 1 14.39 kwh Run 2 14.72 kwh Run 3 15.06 kwh Run 4 14.50 kwh Run 5 14.44 kwh Run 6 14.61 kwh Run 7 14.33 kwh Run 1 29.33 kwh Run 2 29.11 kwh Run 3 29.78 kwh Run 4 29.78 kwh Run 5 30.40 kwh Run 6 29.44 kwh Run 7 30.11 kwh Mean Value of EC Standard Deviation of EC 0.87 0.021602 1.788571 0.038914 3.577143 0.054685 7.29 0.113284 14.57857 0.250162 29.70714 0.451505 Confidence Interval of EC 0.886003 0.853997 1.817399 1.759744 3.617654 3.536632 7.373921 7.206079 14.76389 14.39325 30.04162 29.37267 69

Exp.7 Exp.8 Exp.9 Exp.10 +%50 Exp.11 +%12.5 Exp.12 +%6.25 Exp. 13 +%0.2 Exp.14 +%0.1 640 No of VMs = 1280 No of Cloudlets = 1280 1280 No of VMs = 2560 No of Cloudlets = 2560 2560 No of VMs = 5120 No of Cloudlets = 5120 3840 No of VMs = 7680 No of Cloudlets = 7680 4320 No of VMs = 8640 No of Cloudlets = 8640 4590 No of VMs = 9180 No of Cloudlets = 9180 4598 No of VMs = 9196 No of Cloudlets = 9196 4602 No of VMs = 9204 No of Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 59.33 kwh 59.56 kwh 58.67 kwh 59.56 kwh 59.33 kwh 60.44 kwh 60.67 kwh 117.33. kwh 120.89 kwh 119.56 kwh 118.22 kwh 120.0 kwh 119.56 kwh 120.44 kwh 240.0 kwh 238.22 kwh 250.67 kwh 244.44 kwh 239.11 kwh 241.78 kwh 241.78 kwh 362.67 kwh 360.00 kwh 368.00 kwh 364.00 kwh 357.33 kwh 372.00 kwh 365.33 kwh 409.50 kwh 418.50 kwh 414.00 kwh 406.50 kwh 409.50 kwh 409.50 kwh 408.00 kwh 436.69 kwh 433.50 kwh 428.72 kwh 441.47 kwh 438.28 kwh 436.69 kwh 441.47 kwh 434.26 kwh 431.06 kwh 432.66 kwh 442.24 kwh 432.66 kwh 432.66 kwh 439.05 kwh 442.62 kwh 441.03 kwh 437.83 kwh 442.62 kwh 436.23 kwh 59.65143 0.688705 119.4286 1.250632 242.2857 4.22864 364.19 4.895134 410.7857 4.101393 436.6886 4.507806 434.9414 4.108813 440.34 2.584944 60.16162 59.14124 120.355 118.5021 245.4183 239.1532 367.8163 360.5637 413.824 407.7474 440.0279 433.3492 437.9852 431.8976 442.2549 438.4251 70

Exp.15 +%0.02 Cloudlets = 9204 4603 No of VMs = 9206 No of Cloudlets = 9206 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 439.43 kwh 442.62 kwh 431.53 kwh 429.93 kwh 439.52 kwh 441.12 kwh 436.33 kwh 437.92 kwh 434.73 kwh 435.8686 4.09711 438.9037 432.8334 The mean values, standard deviation and confidence interval have been calculated based on using the built-in functions in Excel sheet. As a sample, Figure 7 below shows a screen shot of the calculation made for this scenario with the deployment of DVFS. The calculations for the remaining scenarios are calculated in the same way. Figure 7. A sample of using Excel to calculate mean values, standard deviation, and confidence interval. 71

8.8. Appendix H: A Source Code Sample of the CloudSim Java Classes The following sample is adapted from the example provided in CloudSim by the people from the University of Melbourne [48], and this code was particularly implemented for experiment 1 of scenario 1 with the DVFS mechanism. package org.cloudbus.cloudsim.examples.power; /* * Title: CloudSim Toolkit * Description: CloudSim (Cloud Simulation) Toolkit for Modeling and Simulation * of Clouds * Licence: GPL - http://www.gnu.org/copyleft/gpl.html * * Copyright (c) 2009, The University of Melbourne, Australia */ import java.text.decimalformat; import java.util.arraylist; import java.util.calendar; import java.util.linkedlist; import java.util.list; import org.cloudbus.cloudsim.cloudlet; import org.cloudbus.cloudsim.cloudletschedulerdynamicworkload; import org.cloudbus.cloudsim.datacenterbroker; import org.cloudbus.cloudsim.datacentercharacteristics; import org.cloudbus.cloudsim.log; import org.cloudbus.cloudsim.storage; import org.cloudbus.cloudsim.utilizationmodelstochastic; import org.cloudbus.cloudsim.vm; import org.cloudbus.cloudsim.vmschedulertimeshared; import org.cloudbus.cloudsim.core.cloudsim; import org.cloudbus.cloudsim.power.powerdatacenter; 72

import org.cloudbus.cloudsim.power.powerhost; import org.cloudbus.cloudsim.power.powerpe; import org.cloudbus.cloudsim.power.powervmallocationpolicysinglethreshold; import org.cloudbus.cloudsim.power.models.powermodellinear; import org.cloudbus.cloudsim.provisioners.bwprovisionersimple; import org.cloudbus.cloudsim.provisioners.peprovisionersimple; import org.cloudbus.cloudsim.provisioners.ramprovisionersimple; /** * An example of a heterogeneous DVFS-enabled data center: the voltage * and clock frequency of the CPU are adjusted ideally according * to current resource requirements. * * Modified and Implemented by Ibrahim Alzamil (IA) */ public class ScenariosWithDVFS { /** The cloudlet list. */ private static List<Cloudlet> cloudletlist; /** The vm list. */ private static List<Vm> vmlist; // IA.. here we adjust the settings of these three parameters based on the objective and design for each experiment private static double hostsnumber = 10; private static double vmsnumber ; private static double cloudletsnumber ; /** * Creates main() to run this example. * 73

* @param args the args */ public static void main(string[] args) { experiments // IA.. the next three lines were used when using shell scripting to run the hostsnumber = Double.valueOf(args[0]).doubleValue(); vmsnumber = Double.valueOf(args[1]).doubleValue(); cloudletsnumber = Double.valueOf(args[2]).doubleValue(); Log.printLine("Starting DVFS example..."); try { // First step: Initialize the CloudSim package. It should be called // before creating any entities. We can't run this example without // initializing CloudSim first. We will get run-time exception // error. int num_user = 1; // number of cloud users Calendar calendar = Calendar.getInstance(); boolean trace_flag = false; // mean trace GridSim events // Initialize the CloudSim library CloudSim.init(num_user, calendar, trace_flag); // Second step: Create Datacenters // Datacenters are the resource providers in CloudSim. We need at // list one of them to run a CloudSim simulation PowerDatacenter datacenter = createdatacenter("datacenter_0"); datacenter.setdisablemigrations(true); 74

// Third step: Create Broker DatacenterBroker broker = createbroker(); int brokerid = broker.getid(); // Fourth step: Create one virtual machine vmlist = createvms(brokerid); // submit vm list to the broker broker.submitvmlist(vmlist); // Fifth step: Create one cloudlet cloudletlist = createcloudletlist(brokerid); // submit cloudlet list to the broker broker.submitcloudletlist(cloudletlist); // Sixth step: Starts the simulation double lastclock = CloudSim.startSimulation(); // Final step: Print results when simulation is over List<Cloudlet> newlist = broker.getcloudletreceivedlist(); Log.printLine("Received " + newlist.size() + " cloudlets"); CloudSim.stopSimulation(); printcloudletlist(newlist); Log.printLine(); 75

results // IA.. to check the parameters settings were set correctly when getting the System.out.println("Number of Hosts are: "+ hostsnumber); System.out.println("Number of VMs are: "+ vmsnumber); System.out.println("Number of Cloudlets are: "+ cloudletsnumber); Log.printLine(String.format("Total simulation time: %.2f sec", lastclock)); Log.printLine(String.format("Energy consumption: %.2f kwh", datacenter.getpower() / (3600 * 1000))); Log.printLine(); } catch (Exception e) { e.printstacktrace(); Log.printLine("Unwanted errors happen"); } } Log.printLine("DVFS experiment finished!"); /** * Creates the cloudlet list. * * @param brokerid the broker id * * @return the cloudlet list */ private static List<Cloudlet> createcloudletlist(int brokerid) { List<Cloudlet> list = new ArrayList<Cloudlet>(); long length = 150000; // 10 min on 250 MIPS 76

int pesnumber = 1; long filesize = 300; long outputsize = 300; for (int i = 0; i < cloudletsnumber; i++) { Cloudlet cloudlet = new Cloudlet(i, length, pesnumber, filesize, outputsize, new UtilizationModelStochastic(), new UtilizationModelStochastic(), new UtilizationModelStochastic()); } cloudlet.setuserid(brokerid); cloudlet.setvmid(i); list.add(cloudlet); } return list; /** * Creates the vms. * * @param brokerid the broker id * * @return the list< vm> */ private static List<Vm> createvms(int brokerid) { List<Vm> vms = new ArrayList<Vm>(); // VM description int[] mips = { 250, 500, 750, 1000 }; // MIPSRating int pesnumber = 1; // number of cpus int ram = 128; // vm memory (MB) long bw = 2500; // bandwidth 77

long size = 2500; // image size (MB) String vmm = "Xen"; // VMM name for (int i = 0; i < vmsnumber; i++) { vms.add( new Vm(i, brokerid, mips[i % mips.length], pesnumber, ram, bw, size, vmm, new CloudletSchedulerDynamicWorkload(mips[i % mips.length], pesnumber)) } ); } return vms; /** * Creates the datacenter. * * @param name the name * * @return the datacenter * * @throws Exception the exception */ private static PowerDatacenter createdatacenter(string name) throws Exception { // Here are the steps needed to create a PowerDatacenter: // 1. We need to create an object of HostList2 to store // our machine List<PowerHost> hostlist = new ArrayList<PowerHost>(); double maxpower = 250; // 250W double staticpowerpercent = 0.7; // 70% 78

int[] mips = { 1000, 2000, 3000 }; int ram = 10000; // host memory (MB) long storage = 1000000; // host storage int bw = 100000; for (int i = 0; i < hostsnumber; i++) { // 2. A Machine contains one or more PEs or CPUs/Cores. // In this example, it will have only one core. // 3. Create PEs and add these into an object of PowerPeList. List<PowerPe> pelist = new ArrayList<PowerPe>(); pelist.add(new PowerPe(0, new PeProvisionerSimple(mips[i % mips.length]), new PowerModelLinear(maxPower, staticpowerpercent))); // need to store PowerPe id and MIPS Rating machines // 4. Create PowerHost with its id and list of PEs and add them to the list of hostlist.add( new PowerHost( i, new RamProvisionerSimple(ram), new BwProvisionerSimple(bw), storage, pelist, new VmSchedulerTimeShared(peList) ) ); // This is our machine } // 5. Create a DatacenterCharacteristics object that stores the // properties of a Grid resource: architecture, OS, list of 79

// Machines, allocation policy: time- or space-shared, time zone // and its price (G$/PowerPe time unit). String arch = "x86"; // system architecture String os = "Linux"; // operating system String vmm = "Xen"; double time_zone = 10.0; // time zone this resource located double cost = 3.0; // the cost of using processing in this resource double costpermem = 0.05; // the cost of using memory in this resource double costperstorage = 0.001; // the cost of using storage in this // resource double costperbw = 0.0; // the cost of using bw in this resource DatacenterCharacteristics characteristics = new DatacenterCharacteristics( costperstorage, costperbw); arch, os, vmm, hostlist, time_zone, cost, costpermem, // 6. Finally, we need to create a PowerDatacenter object. PowerDatacenter powerdatacenter = null; try { the nonpoweraware powerdatacenter = new PowerDatacenter( // IAA this is the difference from name, characteristics, new PowerVmAllocationPolicySingleThreshold(hostList, 1.0), new LinkedList<Storage>(), 5.0); } catch (Exception e) { e.printstacktrace(); } return powerdatacenter; } // We strongly encourage users to develop their own broker policies, to // submit vms and cloudlets according 80

// to the specific rules of the simulated scenario /** * Creates the broker. * @return the datacenter broker */ private static DatacenterBroker createbroker() { DatacenterBroker broker = null; try { broker = new DatacenterBroker("Broker"); } catch (Exception e) { e.printstacktrace(); return null; } return broker; } /** * Prints the Cloudlet objects. * * @param list list of Cloudlets */ private static void printcloudletlist(list<cloudlet> list) { int size = list.size(); Cloudlet cloudlet; String indent = "\t"; Log.printLine(); Log.printLine("========== OUTPUT =========="); Log.printLine("Cloudlet ID" + indent + "STATUS" + indent + "Resource ID" + indent + "VM ID" + indent + "Time" + indent + "Start Time" + indent + "Finish Time"); 81

DecimalFormat dft = new DecimalFormat("###.##"); for (int i = 0; i < size; i++) { cloudlet = list.get(i); Log.print(indent + cloudlet.getcloudletid()); if (cloudlet.getcloudletstatus() == Cloudlet.SUCCESS) { Log.printLine(indent + "SUCCESS" + indent + indent + cloudlet.getresourceid() + indent + cloudlet.getvmid() + indent + dft.format(cloudlet.getactualcputime()) + indent + dft.format(cloudlet.getexecstarttime()) + indent + indent + dft.format(cloudlet.getfinishtime()) ); } } } } 82

8.9. Appendix I: Collected Results of Scenario 2: Increasing the No. Of Hosts DVFS Mechanism Table 4. Shows 14 experiments with the deployment of DVFS mechanism. Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 20 40 80 160 320 640 Results of Energy Consumption (EC) for each run Run 1 0.26 kwh Run 2 0.25 kwh Run 3 0.25 kwh Run 4 0.24 kwh Run 5 0.25 kwh Run 6 0.24 kwh Run 7 0.26 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.25 kwh Run 4 0.24 kwh Run 5 0.24 kwh Run 6 0.24 kwh Run 7 0.25 kwh Run 1 0.24 kwh Run 2 0.25 kwh Run 3 0.24 kwh Run 4 0.23 kwh Run 5 0.24 kwh Run 6 0.25 kwh Run 7 0.25 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh Run 4 0.24 kwh Run 5 0.25 kwh Run 6 0.23 kwh Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.25 kwh Run 4 0.24 kwh Run 5 0.23 kwh Run 6 0.25 kwh Run 7 0.25 kwh Run 1 0.25 kwh Run 2 0.24 kwh Run 3 0.25 kwh Run 4 0.24 kwh Run 5 0.24 kwh Run 6 0.23 kwh Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh Mean Value of EC Standard Deviation of EC 0.25 0.008165 0.242857 0.00488 0.242857 0.007559 0.24 0.005774 0.242857 0.007559 0.241429 0.006901 0.241429 0.00378 Confidence Interval of EC 0.256049 0.243951 0.246472 0.239242 0.248457 0.237257 0.244277 0.235723 0.248457 0.237257 0.246541 0.236317 0.244229 83

Exp.8 Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 Exp.14 1280 2560 5120 10240 20480 40960 81920 Run 4 0.24 kwh Run 5 0.24 kwh 0.238629 Run 6 0.24 kwh Run 7 0.25 kwh Run 1 0.24 kwh Run 2 0.23 kwh Run 3 0.24 kwh 0.244277 Run 4 0.24 kwh 0.24 0.005774 Run 5 0.24 kwh Run 6 0.25 kwh 0.235723 Run 7 0.24 kwh Run 1 0.25 kwh Run 2 0.24 kwh 0.246472 Run 3 0.25 kwh Run 4 0.24 kwh 0.242857 0.00488 Run 5 0.24 kwh 0.239242 Run 6 0.24 kwh Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.25 kwh Run 3 0.24 kwh 0.244229 Run 4 0.24 kwh 0.241429 0.00378 Run 5 0.24 kwh Run 6 0.24 kwh 0.238629 Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.23 kwh Run 3 0.24 kwh 0.241371 Run 4 0.24 kwh 0.238571 0.00378 Run 5 0.24 kwh Run 6 0.24 kwh 0.235771 Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.25 kwh 0.244229 Run 4 0.24 kwh 0.241429 0.00378 Run 5 0.24 kwh Run 6 0.24 kwh 0.238629 Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh 0.244229 Run 4 0.25 kwh 0.241429 0.00378 Run 5 0.24 kwh Run 6 0.24 kwh 0.238629 Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh 0.246472 Run 4 0.24 kwh 0.242857 0.00488 Run 5 0.25 kwh Run 6 0.25 kwh 0.239242 Run 7 0.24 kwh 84

Table 5. Shows 14 experiments without the deployment of DVFS mechanism Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 20 40 80 160 320 640 Results of Energy Consumption (EC) for each run Run 1 0.89 kwh Run 2 0.89 kwh Run 3 0.85 kwh Run 4 0.88 kwh Run 5 0.88 kwh Run 6 0.87 kwh Run 7 0.84 kwh Run 1 1.78 kwh Run 2 1.74 kwh Run 3 1.76 kwh Run 4 1.71 kwh Run 5 1.67 kwh Run 6 1.76 kwh Run 7 1.74 kwh Run 1 3.46 kwh Run 2 3.40 kwh Run 3 3.47 kwh Run 4 3.49 kwh Run 5 3.43 kwh Run 6 3.58 kwh Run 7 3.36 kwh Run 1 6.94 kwh Run 2 7.28 kwh Run 3 6.64 kwh Run 4 7.06 kwh Run 5 6.78 kwh Run 6 7.03 kwh Run 7 6.89 kwh Run 1 14.22 kwh Run 2 13.89 kwh Run 3 14.44 kwh Run 4 13.83 kwh Run 5 14.11 kwh Run 6 13.56 kwh Run 7 14.28 kwh Run 1 27.33 kwh Run 2 27.67 kwh Run 3 28.22 kwh Run 4 27.44 kwh Run 5 28.11 kwh Run 6 27.67 kwh Run 7 28.33 kwh Run 1 56.22 kwh Run 2 58.00 kwh Run 3 56.89 kwh Run 4 55.11 kwh Run 5 54.89 kwh Run 6 54.67 kwh Run 7 57.56 kwh Mean Value of EC Standard Deviation of EC 0.871429 0.019518 1.737143 0.036839 3.455714 0.070441 6.945714 0.206548 14.04714 0.303189 27.82429 0.394456 56.19143 1.342304 Confidence Interval of EC 0.885887 0.85697 1.764433 1.709852 3.507897 3.403532 7.098724 6.792705 14.27174 13.82254 28.1165 27.53207 57.1858 55.19705 85

Exp.8 Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 Exp.14 1280 2560 5120 10240 20480 40960 81920 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 111.11 kwh 112.44 kwh 108.89 kwh 110.22 kwh 106.67 kwh 114.67 kwh 115.11 kwh 220.44 kwh 226.67 kwh 223.11 kwh 226.67 kwh 227.56 kwh 228.44 kwh 221.33 kwh 446.22 kwh 444.44 kwh 451.56 kwh 448.00 kwh 439.11 kwh 428.44 kwh 424.89 kwh 910.22 kwh 896.00 kwh 860.44 kwh 885.33 kwh 917.33 kwh 878.22 kwh 888.89 kwh 1728.00 kwh 1728.00 kwh 1763.56 kwh 1870.22 kwh 1777.78 kwh 1777.78 kwh 1792.00 kwh 3754.67 kwh 3498.67 kwh 3697.78 kwh 3584.00 kwh 3498.67 kwh 3612.44 kwh 3669.33 kwh 7253.33 kwh 7139.56 kwh 7111.11 kwh 7253.33 kwh 7025.78 kwh 6712.89 kwh 7367.11 kwh 111.3014 3.04467 224.8886 3.207093 440.38 10.14694 890.9186 19.24177 1776.763 48.12904 3616.509 97.74878 7123.301 212.7669 113.5569 109.0459 227.2644 222.5128 447.8968 432.8632 905.1728 876.6643 1812.417 1741.109 3688.921 3544.097 7280.918 6965.684 86

8.10. Appendix J: Collected Results of Scenario 3: Increasing the No. Of VMs DVFS Mechanism Table 6. Shows 14 experiments with the deployment of DVFS mechanism. Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 = 20 10 No of VMs = 40 = 20 10 No of VMs = 80 = 20 10 No of VMs = 160 = 20 10 No of VMs = 320 = 20 10 No of VMs = 640 = 20 10 No of VMs = 1280 = Results of Energy Consumption (EC) for each run Run 1 0.24 kwh Run 2 0.25 kwh Run 3 0.25 kwh Run 4 0.25 kwh Run 5 0.24 kwh Run 6 0.25 kwh Run 7 0.25 kwh Run 1 0.71 kwh Run 2 0.75 kwh Run 3 0.73 kwh Run 4 0.75 kwh Run 5 0.72 kwh Run 6 0.74 kwh Run 7 0.71 kwh Run 1 0.72 kwh Run 2 0.73 kwh Run 3 0.73 kwh Run 4 0.71 kwh Run 5 0.71 kwh Run 6 0.75 kwh Run 7 0.74 kwh Run 1 0.74 kwh Run 2 0.76 kwh Run 3 0.73 kwh Run 4 0.73 kwh Run 5 0.71 kwh Run 6 0.71 kwh Run 7 0.74 kwh Run 1 0.72 kwh Run 2 0.73 kwh Run 3 0.77 kwh Run 4 0.72 kwh Run 5 0.76 kwh Run 6 0.75 kwh Run 7 0.74 kwh Run 1 0.74 kwh Run 2 0.72 kwh Run 3 0.74 kwh Run 4 0.73 kwh Run 5 0.75 kwh Run 6 0.75 kwh Run 7 0.74 kwh Run 1 0.74 kwh Run 2 0.74 kwh Run 3 0.71 kwh Mean Value of EC Standard Deviation of EC 0.247143 0.00488 0.73 0.017321 0.727143 0.01496 0.731429 0.017728 0.741429 0.019518 0.738571 0.01069 0.725714 0.011339 Confidence Interval of EC 0.250758 0.243528 0.742831 0.717169 0.738225 0.71606 0.744561 0.718296 0.755887 0.72697 0.746491 0.730652 0.734114 87

Exp.8 Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 Exp.14 20 Run 4 0.72 kwh Run 5 0.72 kwh 0.717314 Run 6 0.73 kwh Run 7 0.72 kwh Run 1 0.73 kwh Run 2 0.76 kwh 10 Run 3 0.75 kwh 0.756914 No of VMs = 2560 Run 4 0.77 kwh 0.745714 0.015119 = Run 5 0.73 kwh 20 Run 6 0.74 kwh 0.734514 Run 7 0.74 kwh Run 1 0.72 kwh Run 2 0.73 kwh 10 Run 3 0.73 kwh 0.7411 No of VMs = 5120 Run 4 0.74 kwh 0.732857 0.011127 = Run 5 0.72 kwh 20 Run 6 0.74 kwh 0.724614 Run 7 0.75 kwh Run 1 0.76 kwh Run 2 0.74 kwh 10 Run 3 0.72 kwh 0.750229 No of VMs = 10240 Run 4 0.75 kwh 0.738571 0.015736 = Run 5 0.75 kwh 20 Run 6 0.73 kwh 0.726914 Run 7 0.72 kwh Run 1 0.74 kwh Run 2 0.74 kwh 10 Run 3 0.73 kwh 0.741543 480 Run 4 0.73 kwh 0.735714 0.007868 = Run 5 0.73 kwh 20 Run 6 0.73 kwh 0.729886 Run 7 0.75 kwh Run 1 0.72 kwh Run 2 0.75 kwh 10 Run 3 0.73 kwh 0.73514 No of VMs = 40960 Run 4 0.72 kwh 0.725714 0.012724 = Run 5 0.72 kwh 20 Run 6 0.73 kwh 0.716288 Run 7 0.71 kwh Run 1 0.74 kwh Run 2 0.74 kwh 10 Run 3 0.74 kwh 0.742743 No of VMs = 81920 Run 4 0.72 kwh 0.737143 0.007559 = Run 5 0.74 kwh 20 Run 6 0.74 kwh 0.731543 Run 7 0.74 kwh Run 1 0.73 kwh 10 Run 2 0.73 kwh No of VMs = Run 3 0.72 kwh 0.744244 163840 Run 4 0.76 kwh 0.728571 0.021157 = Run 5 0.71 kwh 20 Run 6 0.70 kwh 0.712898 Run 7 0.75 kwh 88

Table 7. Shows 14 experiments without the deployment of Power Aware mechanism Parameters Setting Results of Energy Consumption (EC) for each run Mean Value of EC Standard Deviation of EC Confidence Interval of EC Run 1 0.87 kwh Run 2 0.85 kwh 10 Run 3 0.89 kwh 0.891635 Exp.1 Run 4 0.88 kwh 0.875714 0.021492 Run 5 0.91 kwh Run 6 0.88 kwh 0.859793 Run 7 0.85 kwh Run 1 0.86 kwh Run 2 0.86 kwh 0.897463 10 Run 3 0.92 kwh Exp.2 No of VMs = 40 Run 4 0.86 kwh 0.875714 0.029358 Run 5 0.91 kwh 0.853966 Run 6 0.84 kwh Run 7 0.88 kwh Run 1 0.89 kwh Run 2 0.86 kwh 10 Run 3 0.87 kwh 0.896274 Exp.3 No of VMs = 80 Run 4 0.88 kwh 0.884286 0.016183 Run 5 0.89 kwh Run 6 0.91 kwh 0.872297 Run 7 0.89 kwh Exp.4 Run 1 0.86 kwh Run 2 0.89 kwh 10 Run 3 0.83 kwh 0.875887 No of VMs = Run 4 0.85 kwh 0.861429 0.019518 160 Run 5 0.86 kwh Run 6 0.86 kwh 0.84697 Run 7 0.88 kwh Exp.5 Run 1 0.91 kwh Run 2 0.88 kwh 10 Run 3 0.84 kwh 0.899127 No of VMs = Run 4 0.91 kwh 0.88 0.02582 320 Run 5 0.86 kwh Run 6 0.89 kwh 0.860873 Run 7 0.87 kwh Exp.6 Run 1 0.85 kwh Run 2 0.87 kwh 10 Run 3 0.88 kwh 0.893755 No of VMs = Run 4 0.91 kwh 0.867143 0.035923 640 Run 5 0.91 kwh Run 6 0.82 kwh 0.840531 Run 7 0.83 kwh Exp.7 Run 1 0.87 kwh Run 2 0.88 kwh 10 Run 3 0.89 kwh 0.888696 No of VMs = Run 4 0.90 kwh 0.872857 0.021381 1280 Run 5 0.88 kwh Run 6 0.85 kwh 0.857018 Run 7 0.84 kwh Exp.8 Run 1 0.85 kwh 0.857143 0.017995 89

Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 Exp.14 10 No of VMs = 2560 10 No of VMs = 5120 10 No of VMs = 10240 10 No of VMs = 20480 10 No of VMs = 40960 10 No of VMs = 81920 10 No of VMs = 163840 Run 2 0.85 kwh 0.870473 Run 3 0.86 kwh Run 4 0.86 kwh Run 5 0.86 kwh 0.843812 Run 6 0.89 kwh Run 7 0.83 kwh Run 1 0.91 kwh Run 2 0.87 kwh Run 3 0.89 kwh 0.91463 Run 4 0.88 kwh 0.891429 0.03132 Run 5 0.93 kwh Run 6 0.84 kwh 0.868227 Run 7 0.92 kwh Run 1 0.89 kwh Run 2 0.86 kwh Run 3 0.88 kwh 0.891704 Run 4 0.89 kwh 0.878571 0.017728 Run 5 0.90 kwh Run 6 0.85 kwh 0.865439 Run 7 0.88 kwh Run 1 0.86 kwh Run 2 0.87 kwh Run 3 0.88 kwh 0.909632 Run 4 0.87 kwh 0.88 0.04 Run 5 0.96 kwh Run 6 0.89 kwh 0.850368 Run 7 0.83 kwh Run 1 0.87 kwh Run 2 0.89 kwh Run 3 0.90 kwh 0.888537 Run 4 0.88 kwh 0.878571 0.013452 Run 5 0.86 kwh Run 6 0.88 kwh 0.868606 Run 7 0.87 kwh Run 1 0.87 kwh Run 2 0.86 kwh Run 3 0.90 kwh 0.878443 Run 4 0.85 kwh 0.865714 0.017182 Run 5 0.86 kwh Run 6 0.87 kwh 0.852986 Run 7 0.85 kwh Run 1 0.91 kwh Run 2 0.88 kwh Run 3 0.88 kwh 0.897102 Run 4 0.87 kwh 0.881429 0.021157 Run 5 0.86 kwh Run 6 0.91 kwh 0.865756 Run 7 0.86 kwh 90

8.11. Appendix K: Collected Results of Scenario 4: Increasing the No. Of Cloudlets DVFS Mechanism Table 8. Shows 13 experiments with the deployment of DVFS mechanism. Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting No of Host = 10 No of Host = 10 = 40 No of Host = 10 No of VMs =20 = 80 No of Host = 10 = 160 No of Host = 10 = 320 10 = 640 10 Results of Energy Consumption (EC) for each run Run 1 0.25 kwh Run 2 0.25 kwh Run 3 0.25 kwh Run 4 0.26 kwh Run 5 0.25 kwh Run 6 0.24 kwh Run 7 0.26 kwh Run 1 0.25 kwh Run 2 0.25 kwh Run 3 0.26 kwh Run 4 0.25 kwh Run 5 0.25 kwh Run 6 0.26 kwh Run 7 0.24 kwh Run 1 0.26 kwh Run 2 0.25 kwh Run 3 0.25 kwh Run 4 0.24 kwh Run 5 0.24 kwh Run 6 0.25 kwh Run 7 0.25 kwh Run 1 0.25 kwh Run 2 0.24 kwh Run 3 0.24 kwh Run 4 0.25 kwh Run 5 0.25 kwh Run 6 0.25 kwh Run 7 0.25 kwh Run 1 0.25 kwh Run 2 0.26 kwh Run 3 0.25 kwh Run 4 0.25 kwh Run 5 0.25 kwh Run 6 0.25 kwh Run 7 0.25 kwh Run 1 0.25 kwh Run 2 0.25 kwh Run 3 0.25 kwh Run 4 0.24 kwh Run 5 0.25 kwh Run 6 0.24 kwh Run 7 0.25 kwh Run 1 0.24 kwh Run 2 0.25 kwh Run 3 0.25 kwh Mean Value of EC Standard Deviation of EC 0.251429 0.006901 0.251429 0.006901 0.248571 0.006901 0.247143 0.00488 0.251429 0.00378 0.247143 0.00488 0.247143 0.00488 Confidence Interval of EC Upper bound = 0.256541 0.246317 Upper bound = 0.256541 0.246317 Upper bound = 0.253683 0.243459 Upper bound = 0.250758 0.243528 Upper bound = 0.254229 0.248629 Upper bound = 0.250758 0.243528 Upper bound = 0.250758 91

Exp.8 Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 +%50 = 1280 10 = 2560 10 = 5120 10 = 10240 10 480 10 = 40960 10 = 61440 Run 4 0.25 kwh Run 5 0.25 kwh 0.243528 Run 6 0.25 kwh Run 7 0.24 kwh Run 1 0.25 kwh Run 2 0.26 kwh Upper bound = Run 3 0.26 kwh 0.256472 Run 4 0.25 kwh 0.252857 0.00488 Run 5 0.25 kwh Run 6 0.25 kwh 0.249242 Run 7 0.25 kwh Run 1 0.26 kwh Upper bound = Run 2 0.25 kwh 0.255237 Run 3 0.26 kwh Run 4 0.24 kwh 0.248571 0.008997 Run 5 0.25 kwh 0.241906 Run 6 0.24 kwh Run 7 0.24 kwh Run 1 0.25 kwh Upper bound = Run 2 0.24 kwh 0.251371 Run 3 0.25 kwh Run 4 0.25 kwh 0.248571 0.00378 Run 5 0.25 kwh 0.245771 Run 6 0.25 kwh Run 7 0.25 kwh Run 1 0.24 kwh Run 2 0.25 kwh Upper bound = Run 3 0.25 kwh 0.248245 Run 4 0.24 kwh 0.244286 0.005345 Run 5 0.25 kwh Run 6 0.24 kwh 0.240326 Run 7 0.24 kwh Run 1 0.25 kwh Run 2 0.25 kwh Upper bound = Run 3 0.24 kwh 0.254277 Run 4 0.25 kwh 0.25 0.005774 Run 5 0.25 kwh Run 6 0.25 kwh 0.245723 Run 7 0.26 kwh Run 1 0.24 kwh Run 2 0.25 kwh Upper bound = Run 3 0.24 kwh 0.253683 0.248571 Run 4 0.25 kwh 0.006901 Run 5 0.25 kwh Run 6 0.26 kwh 0.243459 Run 7 0.25 kwh 92

Table 9. Shows 13 experiments without the deployment of Power Aware mechanism Parameters Setting Results of Energy Consumption (EC) for each run Mean Value of EC Standard Deviation of EC Confidence Interval of EC Run 1 0.88 kwh Exp.1 Run 2 0.83 kwh Upper bound = No of Host = 10 Run 3 0.87 kwh 0.884244 0.868571 0.021157 Run 4 0.88 kwh Run 5 0.88 kwh Run 6 0.85 kwh 0.852898 Run 7 0.89 kwh Run 1 0.88 kwh Upper bound = Run 2 0.87 kwh No of Host = 10 0.903416 Run 3 0.92 kwh Exp.2 Run 4 0.87 kwh 0.881429 0.029681 Run 5 0.83 kwh = 40 0.859441 Run 6 0.91 kwh Run 7 0.89 kwh Run 1 0.86 kwh Exp.3 Run 2 0.88 kwh Upper bound = No of Host = 10 Run 3 0.87 kwh 0.871394 No of VMs =20 Run 4 0.84 kwh 0.861429 0.013452 Run 5 0.86 kwh = 80 Run 6 0.85 kwh 0.851463 Run 7 0.87 kwh Run 1 0.89 kwh Exp.4 Run 2 0.86 kwh Upper bound = No of Host = 10 Run 3 0.84 kwh 0.881781 Run 4 0.85 kwh 0.867143 0.01976 Run 5 0.89 kwh = 160 Run 6 0.88 kwh 0.852504 Run 7 0.86 kwh Run 1 0.85 kwh Exp.5 Run 2 0.86 kwh Upper bound = No of Host = 10 Run 3 0.86 kwh 0.866914 Run 4 0.88 kwh 0.855714 0.015119 Run 5 0.86 kwh = 320 Run 6 0.83 kwh 0.844514 Run 7 0.85 kwh Run 1 0.88 kwh Exp.6 Run 2 0.84 kwh Upper bound = 10 Run 3 0.86 kwh 0.875483 Run 4 0.86 kwh 0.862857 0.017043 Run 5 0.86 kwh = 640 Run 6 0.89 kwh 0.850231 Run 7 0.85 kwh Run 1 0.87 kwh Exp.7 Run 2 0.89 kwh Upper bound = 10 Run 3 0.85 kwh 0.896003 Run 4 0.87 kwh 0.88 0.021602 Run 5 0.88 kwh = 1280 Run 6 0.92 kwh 0.863997 Run 7 0.88 kwh Exp.8 10 Run 1 0.87 kwh 0.871429 0.017728 Upper bound = 93

Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 +%50 = 2560 10 = 5120 10 = 10240 10 480 10 = 40960 10 = 61440 Run 2 0.88 kwh 0.884561 Run 3 0.85 kwh Run 4 0.87 kwh Run 5 0.88 kwh 0.858296 Run 6 0.90 kwh Run 7 0.85 kwh Run 1 0.88 kwh Run 2 0.87 kwh Upper bound = Run 3 0.84 kwh 0.889807 Run 4 0.89 kwh 0.875714 0.019024 Run 5 0.88 kwh Run 6 0.87 kwh 0.861622 Run 7 0.90 kwh Run 1 0.93 kwh Run 2 0.85 kwh Upper bound = Run 3 0.87 kwh 0.889249 Run 4 0.86 kwh 0.867143 0.029841 Run 5 0.84 kwh Run 6 0.85 kwh 0.845037 Run 7 0.87 kwh Run 1 0.89 kwh Run 2 0.88 kwh Upper bound = Run 3 0.86 kwh 0.887014 Run 4 0.87 kwh 0.874286 0.017182 Run 5 0.85 kwh Run 6 0.87 kwh 0.861557 Run 7 0.90 kwh Run 1 0.90 kwh Run 2 0.88 kwh Upper bound = Run 3 0.86 kwh 0.880442 Run 4 0.86 kwh 0.865714 0.019881 Run 5 0.87 kwh Run 6 0.84 kwh 0.850987 Run 7 0.85 kwh Run 1 0.86 kwh Run 2 0.87 kwh Upper bound = Run 3 0.88 kwh 0.881394 Run 4 0.87 kwh 0.871429 0.013452 Run 5 0.85 kwh Run 6 0.88 kwh 0.861463 Run 7 0.89 kwh 94

8.12. Appendix L: Collected Results of Scenario 5: Increasing the No. Of Hosts and VMs DVFS Mechanism Table 10. Shows 10 experiments with the deployment of DVFS mechanism. Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 20 No of VMs = 40 40 No of VMs = 80 80 No of VMs = 160 160 No of VMs = 320 320 No of VMs = 640 640 No of VMs = Results of Energy Consumption (EC) for each run Run 1 0.24 kwh Run 2 0.25 kwh Run 3 0.24 kwh Run 4 0.25 kwh Run 5 0.25 kwh Run 6 0.24 kwh Run 7 0.24 kwh Run 1 0.75 kwh Run 2 0.73 kwh Run 3 0.74 kwh Run 4 0.74 kwh Run 5 0.73 kwh Run 6 0.74 kwh Run 7 0.72 kwh Run 1 1.48 kwh Run 2 1.51 kwh Run 3 1.55 kwh Run 4 1.52 kwh Run 5 1.51 kwh Run 6 1.47 kwh Run 7 1.57 kwh Run 1 3.27 kwh Run 2 3.18 kwh Run 3 3.08 kwh Run 4 3.10 kwh Run 5 3.20 kwh Run 6 3.02 kwh Run 7 3.06 kwh Run 1 6.39 kwh Run 2 6.39 kwh Run 3 6.38 kwh Run 4 6.86 kwh Run 5 6.16 kwh Run 6 6.43 kwh Run 7 6.68 kwh Run 1 12.90 kwh Run 2 12.75 kwh Run 3 13.00 kwh Run 4 13.00 kwh Run 5 12.30 kwh Run 6 13.35 kwh Run 7 13.05 kwh Run 1 25.89 kwh Run 2 26.29 kwh Run 3 25.08 kwh Mean Value of EC Standard Deviation of EC 0.244286 0.005345 0.74 0.009759 1.515714 0.035523 3.13 0.088882 6.47 0.229056 12.90714 0.323301 25.37 0.590972 Confidence Interval of EC 0.248245 0.240326 0.742944 0.728485 1.54203 1.489399 3.195843 3.064157 6.639684 6.300316 13.14664 12.66764 25.80636 95

Exp.8 Exp.9 Exp.10 +50% 1280 1280 No of VMs = 2560 2560 No of VMs = 5120 3840 No of VMs = 7680 Run 4 25.08 kwh Run 5 25.69 kwh 24.93078 Run 6 24.78 kwh Run 7 24.77 kwh Run 1 50.08 kwh Run 2 49.67 kwh Run 3 48.86 kwh 50.8028 Run 4 50.08 kwh 50.16571 0.859997 Run 5 51.70 kwh Run 6 50.49 kwh 49.52863 Run 7 50.28 kwh Run 1 102.59 kwh Run 2 102.59 kwh Run 3 98.93 kwh 102.8646 Run 4 100.96 kwh 101.6014 1.705153 Run 5 101.37 kwh Run 6 100.56 kwh 100.3383 Run 7 104.21 kwh Run 1 155.15 kwh Run 2 148.44 kwh Run 3 153.33 kwh 154.3286 Run 4 146.61 kwh 151.67 3.588876 Run 5 155.77 kwh Run 6 153.33 kwh 149.0114 Run 7 149.06 kwh Table 11. Shows 10 experiments without the deployment of power aware mechanism Exp.1 Exp.2 Exp.3 Parameters Setting 10 20 No of VMs = 40 40 No of VMs = 80 Results of Energy Consumption (EC) for each run Run 1 0.84 kwh Run 2 0.88 kwh Run 3 0.88 kwh Run 4 0.84 kwh Run 5 0.85 kwh Run 6 0.86 kwh Run 7 0.94 kwh Run 1 1.74 kwh Run 2 1.76 kwh Run 3 1.74 kwh Run 4 1.72 kwh Run 5 1.72 kwh Run 6 1.77 kwh Run 7 1.78 kwh Run 1 3.39 kwh Run 2 3.49 kwh Run 3 3.60 kwh Run 4 3.32 kwh Run 5 3.39 kwh Run 6 3.44 kwh Run 7 3.38 kwh 96 Mean Value of EC Standard Deviation of EC 0.87 0.035119 1.747143 0.023604 3.43 0.091652 Confidence Interval of EC 0.896016 0.843984 1.764629 1.729657 3.497895 3.362105 Exp.4 Run 1 6.86 kwh 6.982857 0.14127

Exp.5 Exp.6 Exp.7 Exp.8 Exp.9 Exp.10 +50% 80 No of VMs = 160 160 No of VMs = 320 320 No of VMs = 640 640 No of VMs = 1280 1280 No of VMs = 2560 2560 No of VMs = 5120 3840 No of VMs = 7680 Run 2 6.94 kwh 7.087509 Run 3 7.06 kwh Run 4 6.83 kwh Run 5 7.00 kwh 6.878205 Run 6 6.94 kwh Run 7 7.25 kwh Run 1 13.39 kwh Run 2 14.11 kwh Run 3 13.56 kwh 14.46986 Run 4 13.22 kwh 13.93714 0.719113 Run 5 14.00 kwh Run 6 13.89 kwh 13.40443 Run 7 15.39 kwh Run 1 27.22 kwh Run 2 27.67 kwh Run 3 28.67 kwh 28.15288 Run 4 27.89 kwh 27.7 0.611337 Run 5 27.56 kwh Run 6 26.78 kwh 27.24712 Run 7 28.11 kwh Run 1 56.00 kwh Run 2 55.78 kwh Run 3 58.00 kwh 56.9871 Run 4 54.67 kwh 55.81 1.588962 Run 5 53.11 kwh Run 6 56.00 kwh 54.6329 Run 7 57.11 kwh Run 1 111.11 kwh Run 2 114.67 kwh Run 3 113.87 kwh 113.385 Run 4 113.33 kwh 111.1871 2.966832 Run 5 106.67 kwh Run 6 110.22 kwh 108.9893 Run 7 108.44 kwh Run 1 226.67 kwh Run 2 223.11 kwh Run 3 224.00 kwh 229.1784 Run 4 227.56 kwh 225.7786 4.589486 Run 5 230.22 kwh Run 6 231.11 kwh 222.3787 Run 7 217.78 kwh Run 1 334.67 kwh Run 2 344.00 kwh Run 3 340.00 kwh 338.91 Run 4 326.67 kwh 334.4771 5.983894 Run 5 332.00 kwh Run 6 329.33 kwh 330.0443 Run 7 334.67 kwh 97

8.13. Appendix M: Collected Results of Scenario 6: Increasing the No. Of Hosts and Cloudlets DVFS Mechanism Table 12. Shows 13 experiments with the deployment of DVFS mechanism. Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 20 = 40 40 = 80 80 = 160 160 = 320 320 = 640 640 Results of Energy Consumption (EC) for each run Run 1 0.25 kwh Run 2 0.25 kwh Run 3 0.26 kwh Run 4 0.25 kwh Run 5 0.25 kwh Run 6 0.25 kwh Run 7 0.26 kwh Run 1 0.24 kwh Run 2 0.25 kwh Run 3 0.25 kwh Run 4 0.24 kwh Run 5 0.23 kwh Run 6 0.24 kwh Run 7 0.24 kwh Run 1 0.25 kwh Run 2 0.24 kwh Run 3 0.24 kwh Run 4 0.25 kwh Run 5 0.24 kwh Run 6 0.24 kwh Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh Run 4 0.24 kwh Run 5 0.24 kwh Run 6 0.24 kwh Run 7 0.23 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.23 kwh Run 4 0.24 kwh Run 5 0.24 kwh Run 6 0.24 kwh Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.25 kwh Run 3 0.24 kwh Run 4 0.25 kwh Run 5 0.24 kwh Run 6 0.24 kwh Run 7 0.23 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh Mean Value of EC Standard Deviation of EC 0.252857 0.00488 0.241429 0.006901 0.242857 0.00488 0.238571 0.00378 0.238571 0.00378 0.241429 0.006901 0.241429 0.00378 Confidence Interval of EC 0.256472 0.249242 0.246541 0.236317 0.246472 0.239242 0.241371 0.235771 0.241371 0.235771 0.246541 0.236317 0.244229 98

Exp.8 Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 +50% = 1280 1280 = 2560 2560 = 5120 5120 = 10240 10240 480 20480 = 40960 30720 = 61440 Run 4 0.25 kwh Run 5 0.24 kwh 0.238629 Run 6 0.24 kwh Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.25 kwh 0.246472 Run 4 0.24 kwh 0.242857 0.00488 Run 5 0.24 kwh Run 6 0.25 kwh 0.239242 Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.23 kwh 0.244277 Run 4 0.25 kwh 0.24 0.005774 Run 5 0.24 kwh Run 6 0.24 kwh 0.235723 Run 7 0.24 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh 0.244229 Run 4 0.24 kwh 0.241429 0.00378 Run 5 0.24 kwh Run 6 0.24 kwh 0.238629 Run 7 0.25 kwh Run 1 0.25 kwh Run 2 0.24 kwh 0.249674 Run 3 0.25 kwh Run 4 0.24 kwh 0.245714 0.005345 Run 5 0.25 kwh 0.241755 Run 6 0.24 kwh Run 7 0.25 kwh Run 1 0.24 kwh Run 2 0.24 kwh Run 3 0.24 kwh 0.244229 Run 4 0.24 kwh 0.241429 0.00378 Run 5 0.24 kwh Run 6 0.24 kwh 0.238629 Run 7 0.25 kwh Run 1 0.24 kwh Run 2 0.24 kwh 0.244229 Run 3 0.24 kwh Run 4 0.24 kwh 0.241429 0.00378 Run 5 0.24 kwh 0.238629 Run 6 0.24 kwh Run 7 0.25 kwh 99

Table 13. Shows 13 experiments without the deployment of power aware mechanism Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 20 = 40 40 = 80 80 = 160 160 = 320 320 = 640 640 = 1280 Results of Energy Consumption (EC) for each run Run 1 0.86 kwh Run 2 0.89 kwh Run 3 0.86 kwh Run 4 0.91 kwh Run 5 0.86 kwh Run 6 0.87 kwh Run 7 0.87 kwh Run 1 1.79 kwh Run 2 1.74 kwh Run 3 1.76 kwh Run 4 1.73 kwh Run 5 1.69 kwh Run 6 1.76 kwh Run 7 1.80 kwh Run 1 3.51 kwh Run 2 3.44 kwh Run 3 3.68 kwh Run 4 3.51 kwh Run 5 3.46 kwh Run 6 3.58 kwh Run 7 3.68 kwh Run 1 7.39 kwh Run 2 6.72 kwh Run 3 6.78 kwh Run 4 6.81 kwh Run 5 6.64 kwh Run 6 6.97 kwh Run 7 6.67 kwh Run 1 13.83 kwh Run 2 13.89 kwh Run 3 13.78 kwh Run 4 13.89 kwh Run 5 13.78 kwh Run 6 13.44 kwh Run 7 14.67 kwh Run 1 28.78 kwh Run 2 29.56 kwh Run 3 28.33 kwh Run 4 28.00 kwh Run 5 27.22 kwh Run 6 28.89 kwh Run 7 27.22 kwh Run 1 54.44 kwh Run 2 54.67 kwh Run 3 55.56 kwh Run 4 56.44 kwh Run 5 55.33 kwh Run 6 54.44 kwh Run 7 56.44 kwh Mean Value of EC Standard Deviation of EC 0.874286 0.019024 1.752857 0.037289 3.551429 0.098392 6.854286 0.260055 13.89714 0.373796 28.28571 0.874488 55.33143 0.869567 Confidence Interval of EC 0.888378 0.860193 1.780481 1.725234 3.624317 3.47854 7.046934 6.661638 14.17405 13.62024 28.93353 27.6379 55.9756 54.68726 100

Exp.8 Exp.9 Exp.10 Exp.11 Exp.12 Exp.13 +50% 1280 = 2560 2560 = 5120 5120 = 10240 10240 480 20480 = 40960 30720 = 61440 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 118.67 kwh 109.78 kwh 108.44 kwh 114.67 kwh 110.67 kwh 116.00 kwh 114.22 kwh 219.56 kwh 224.00 kwh 214.22 kwh 222.22 kwh 224.00 kwh 222.22 kwh 229.33 kwh 449.78 kwh 437.33 kwh 444.44 kwh 451.56 kwh 451.56 kwh 455.11 kwh 453.33 kwh 888.89 kwh 910.22 kwh 864.00 kwh 906.67 kwh 888.89 kwh 853.33 kwh 853.33 kwh 1777.78 kwh 1728.00 kwh 1827.56 kwh 1813.33 kwh 1834.67 kwh 1699.56 kwh 1763.56 kwh 2688.00 kwh 2730.67 kwh 2602.67 kwh 2656.00 kwh 2613.33 kwh 2560.00 kwh 2709.33 kwh 113.2071 3.690455 222.2214 4.618161 449.0157 6.147864 880.7614 24.00389 1777.78 51.27857 2651.429 62.15335 115.941 110.4733 225.6425 218.8003 453.57 444.4614 898.5434 862.9794 1815.767 1739.793 2697.472 2605.386 101

8.14. Appendix N: Collected Results of Scenario 7: Increasing the No. Of Hosts, VMs, and Cloudlets - VM Migration Mechanism Table 14. Shows 9 experiments with the deployment of VM migration mechanism with DVFS enabled. Exp.1 Exp.2 Exp.3 Exp.4 Exp.5 Exp.6 Exp.7 Parameters Setting 10 20 No of VMs = 40 = 40 40 No of VMs = 80 = 80 80 No of VMs = 160 = 160 160 No of VMs = 320 = 320 320 No of VMs = 640 = 640 640 Results of Energy Consumption (EC) for each run Run 1 0.23 kwh Run 2 0.23 kwh Run 3 0.22 kwh Run 4 0.21 kwh Run 5 0.23 kwh Run 6 0.21 kwh Run 7 0.22 kwh Run 1 0.40 kwh Run 2 0.42 kwh Run 3 0.41 kwh Run 4 0.41 kwh Run 5 0.41 kwh Run 6 0.43 kwh Run 7 0.39 kwh Run 1 0.74 kwh Run 2 0.74 kwh Run 3 0.75 kwh Run 4 0.77 kwh Run 5 0.78 kwh Run 6 0.76 kwh Run 7 0.76 kwh Run 1 1.39 kwh Run 2 1.45 kwh Run 3 1.40 kwh Run 4 1.47 kwh Run 5 1.46 kwh Run 6 1.42 kwh Run 7 1.46 kwh Run 1 2.98 kwh Run 2 2.90 kwh Run 3 2.86 kwh Run 4 2.87 kwh Run 5 2.98 kwh Run 6 3.03 kwh Run 7 3.10 kwh Run 1 5.91 kwh Run 2 5.96 kwh Run 3 5.91 kwh Run 4 5.85 kwh Run 5 6.11 kwh Run 6 5.98 kwh Run 7 6.21 kwh Run 1 12.05 kwh Run 2 11.76 kwh Mean Value of EC Standard Deviation of EC 0.221429 0.008997 0.41 0.01291 0.757143 0.01496 1.435714 0.032071 2.96 0.088506 5.99 0.126623 11.87 0.211401 Confidence Interval of EC 0.228094 0.214763 0.419564 0.400436 0.768225 0.74606 1.459473 1.411956 3.025565 2.894435 6.083802 5.896198 12.02946 102

Exp.8 Exp.9 No of VMs = 1280 = 1280 1280 No of VMs = 2560 = 2560 2560 No of VMs = 5120 = 5120 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 12.15 kwh 12.00 kwh 11.71 kwh 11.55 kwh 11.89 kwh 22.94 kwh 22.73 kwh 23.45 kwh 24.01 kwh 22.94 kwh 23.39 kwh 23.28 kwh 46.55 kwh 46.44 kwh 46.66 kwh 46.71 kwh 47.09 kwh 46.74 kwh 47.45 kwh 23.24857 0.428308 46.80571 0.348753 11.71625 23.56586 22.93128 47.06407 46.54736 8.15. Appendix O: Collected Results of Scenario 8: Increasing the No. Of Hosts - VM Migration Mechanism Table 16. Shows 14 experiments with the deployment of VM migration mechanism with DVFS enabled. Exp.1 Exp.2 Exp.3 Parameters Setting 10 20 40 Results of Energy Consumption (EC) for each run Run 1 0.25 kwh Run 2 0.23 kwh Run 3 0.30 kwh Run 4 0.26 kwh Run 5 0.27 kwh Run 6 0.26 kwh Run 7 0.23 kwh Run 1 0.18 kwh Run 2 0.17 kwh Run 3 0.21 kwh Run 4 0.22 kwh Run 5 0.18 kwh Run 6 0.21 kwh Run 7 0.20 kwh Run 1 0.19 kwh Run 2 0.23 kwh Run 3 0.20 kwh Run 4 0.24 kwh Run 5 0.20 kwh Run 6 0.21 kwh Run 7 0.19 kwh Mean Value of EC Standard Deviation of EC 0.257143 0.0243 0.195714 0.019024 0.208571 0.019518 Confidence Interval of EC 0.275144 0.239142 0.209807 0.181622 0.22303 0.194113 103

Exp.4 Exp.5 Exp.6 Exp.7 Exp.8 Exp.9 Exp.10 Exp.11 80 160 320 640 1280 2560 5120 10240 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 0.18 kwh 0.18 kwh 0.22 kwh 0.22 kwh 0.21 kwh 0.21 kwh 0.18 kwh 0.21 kwh 0.20 kwh 0.19 kwh 0.19 kwh 0.20 kwh 0.21 kwh 0.22 kwh 0.17 kwh 0.22 kwh 0.19 kwh 0.20 kwh 0.21 kwh 0.21 kwh 0.19 kwh 0.18 kwh 0.19 kwh 0.23 kwh 0.19 kwh 0.21 kwh 0.17 kwh 0.21 kwh 0.19 kwh 0.19 kwh 0.22 kwh 0.20 kwh 0.21 kwh 0.20 kwh 0.19 kwh 0.19 kwh 0.20 kwh 0.19 kwh 0.17 kwh 0.20 kwh 0.18 kwh 0.22 kwh 0.18 kwh 0.20 kwh 0.20 kwh 0.20 kwh 0.21 kwh 0.18 kwh 0.24 kwh 0.19 kwh 0.18 kwh 0.19 kwh 0.19 kwh 0.18 kwh 0.2 0.019149 0.202857 0.011127 0.198571 0.016762 0.20 0.020587 0.20 0.011547 0.192857 0.016036 0.201429 0.020354 0.188571 0.006901 0.214185 0.185815 0.2111 0.194614 0.210988 0.186154 0.212393 0.181892 0.208554 0.191446 0.204736 0.180978 0.216507 0.18635 0.193683 0.183459 104

Exp.12 Exp.13 Exp.14 20480 40960 81920 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 Run 1 Run 2 Run 3 Run 4 Run 5 Run 6 Run 7 0.20 kwh 0.19 kwh 0.19 kwh 0.21 kwh 0.19 kwh 0.20 kwh 0.18 kwh 0.19 kwh 0.19 kwh 0.21 kwh 0.18 kwh 0.18 kwh 0.19 kwh 0.21 kwh 0.24 kwh 0.19 kwh 0.20 kwh 0.19 kwh 0.19 kwh 0.19 kwh 0.19 kwh 0.20 kwh 0.19 kwh 0.192857 0.009512 0.2 0.021602 0.192857 0.00488 0.199904 0.185811 0.216003 0.183997 0.196472 0.189242 105