Monitoring and Analysis of CPU load relationships between Host and Guests in a Cloud Networking Infrastructure

Thesis no: MEE-2015-NN Monitoring and Analysis of CPU load relationships between Host and Guests in a Cloud Networking Infrastructure An Empirical Study Krishna Varaynya Chivukula Faculty of Computing Blekinge Institute of Technology SE 371 79 Karlskrona, Sweden

This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering. The thesis is equivalent to 20 weeks of full time studies. Contact Information: Author(s): Krishna Varaynya Chivukula E-mail: krch13@student.bth.se University advisor: Prof.Dr. Kurt Tutschku Department of Telecommunication Systems Faculty of Computing Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE 371 79 Karlskrona, Sweden Fax : +46 455 38 50 57

Abstract Cloud computing has been a fast-growing business of the IT sector in the recent years as it favors hardware resource sharing by reducing the infrastructure maintenance costs and promising improved resource utilization and energy efficiency to the service providers and customers. Cloud Service Providers, CSP, implement load management techniques for effective allocation of resources based on need, enabling them to maintain costs alongside meeting the SLAs. Understanding the impact and behavior of variable workloads in a cloud system is essential for achieving load management. CPU load is a principle computational resource that plays an important role in resource management. This thesis work aims to monitor and analyze load in cloud infrastructure by applying load collection and evaluation techniques. The aim is also to investigate CPU load relationship between host and guest machines under varying workload conditions. In the thesis, along with a cloud environment built using OpenStack, we also consider a system with KVM hypervisor to achieve the goals. The methodology applied is empirical, that is pure experimental examination. This study is about performing measurements to make an assessment about load behavior in the system. The experiments are designed to fulfill the objectives of the thesis. We also employ visual correlation analysis to understand the strength of association between host and guest CPU load. Results of the initial experimental study include distinction between CPU load of OpenStack compute device and a device with KVM hypervisor. Further experimental runs are based on these observations. The succeeding results show quite remarkable association between PM and VM under 100% workload conditions. However, few other variations in workload do not resemble similar results. CPU load results obtained from cloud and a standalone virtualization system differ, not drastically though. 100% workload situations have shown negligible distortion in the visual correlation and usually reported linearity. Lower workloads showed distortions in correlation analysis. It is anticipated that more iterations can likely refine the observations. Further investigation of these relationships by using other applications commonly used in cloud is potential. Keywords: Cloud, CPU load, Measurements, OpenStack, Virtualization

ii Dedicated to my family

Acknowledgements I am forever indebted to my supervisor, Prof. Dr. Kurt Tutschku for his valuable guidance, patience and continuous support throughout my thesis. I could not have imagined a better advisor and mentor for my master thesis. I sincerely thank Dr. Patrik Arlos for his constant encouragement and suggestions in between his busy schedule. I extend my heartfelt thanks to Dr. Dragos Ilie for his invaluable guidance when approached. Words cannot express my gratitude for Anders Carlsson, my father figure, who gave me never ending support and opportunities to excel. I am grateful to Svenska Institutet for embracing me as a deserving candidate for scholarship and enabling me to fulfill my dream of Master s education. I acknowledge City Network Hosting AB for letting us perform tests in their infrastructure. I am thankful to god, my wonderful parents and sister for their immense support and motivation. Without you, it would not be the same. Last but not least, a huge thanks to Vida and all my friends who made this journey worthwhile. iii

List of Abbreviations NIST IaaS SaaS PaaS PM VM SLA PCPU vcpu VMM KVM QEMU OS National Institute of Standards and Technology Infrastructure-as-a-Service Software-as-a-Service Platform-as-a-Service Physical Machine Virtual Machine Service-Level Agreement Physical CPU virtual CPU Virtual Machine Monitor Kernel-based Virtual Machine Quick Emulator Operating System(s) iv

List of Figures 3.1 Minimal architecture and services offered by OpenStack s Controller node(left), Networking node(center) and Compute node(right) 13 3.2 Experimental Methodology portrayed as a sprial model....... 15 3.3 Example of graph showing scatter plots and linear correlation as a relation between x and y attributes.................. 16 3.4 Anticipated graphical plots of possibly attainable correlation between host and guest in terms of CPU load............. 16 3.5 Server, Virtualization and Network components that are related to causing or effecting load in the system................ 18 3.6 example of output of uptime command on terminal........ 18 3.7 example of output of top command on terminal.......... 19 4.1 An OpenStack environment can have n number of compute nodes based on requirement and a controller manages the compute nodes. 22 4.2 Abstraction of hardware, software and virtualization layers in a system. Nova-compute is not present in a normal virtualization system.................................. 22 4.3 Visual representation of the PMs, VMs and tools used in the implementation of experiments. The figure resembles the question as to what could be the relationship between load on host and guest machines................................ 23 4.4 Depiction of the experimental setup on OpenStack platform... 25 v

4.5 Stages of experiments performed with stress............ 26 4.6 Depiction of on-premise device setup................ 27 4.7 Depiction of on-premise device experimental setup for Single guest 29 4.8 Stress applied initially on 1 vcpu.................. 30 4.9 Stess configured to load 2 vcpus.................. 31 4.10 Stess configured to load 3 vcpus.................. 31 4.11 Stess configured to load 3 or more vcpus. The dotted lines indicate that the number of vcpus being stressed is increased in each experimental run............................ 32 4.12 Stess-ng configured to impose load om 1 or more vcpus with 10, 20, 30, 40, 50 and 100% load. The dotted lines indicate that the number of vcpus being stressed is increased in each experimental run................................... 33 5.1 CPU load observed in OpenStack compute node and on-premise device in multiple guest scenario................... 35 5.2 Scatter plots showing the relationship between CPU load average on host and guest in different vcpu stress conditions........ 37 5.3 Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool............ 39 5.4 Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool............ 40 5.5 Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool............ 41 5.6 Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool............ 42 5.7 Scatter plots showing the CPU load relationships between host and guest in varying load conditions - top tool.............. 43 vi

List of Tables 3.1 Minimal hardware required to install a controller and a compute node.................................. 13 4.1 Specifications of the OpenStack Compute Node used for experiments 24 4.2 Specifications of the on-premise device used for experiments... 27 5.1 Stress-ng and uptime results on centos host............ 44 5.2 Stress-ng and uptime results on Ubuntu host............ 44 5.3 Stress-ng and top results on CentOS host.............. 44 5.4 Stress-ng and top results on Ubuntu host.............. 45 vii

Contents Abstract i Acknowledgements iii List of Abbreviations iv List of Figures vi List of Tables vii 1 Introduction 1 1.1 Background.............................. 1 1.2 Aims and Objectives......................... 3 1.3 Research Questions.......................... 4 1.4 Expected Contribution........................ 4 2 Related Work 5 3 Methodology 8 3.1 Introduction to Underlying Technologies.............. 8 3.1.1 Virtualization......................... 8 viii

3.1.2 Hypervisors.......................... 9 3.1.3 Cloud Computing and OpenStack.............. 10 3.2 Methodology............................. 14 3.2.1 Experimental Research.................... 14 3.2.2 Visual Correlation Analysis................. 15 3.3 Measurement Tools.......................... 17 4 General Experimental Setup 21 4.1 Experimental Modeling........................ 21 4.2 Experimental setup.......................... 24 4.2.1 OpenStack Cloud Test-bed.................. 24 4.2.2 On-premise Test-bed..................... 25 4.2.3 Stress tests on Single Guest................. 28 5 Results and Analysis 34 5.1 Results from OpenStack and on-premise Test-beds......... 35 5.2 Results of Stress tests......................... 36 5.3 Results of Stress-ng tests....................... 38 5.3.1 Uptime tool.......................... 38 5.3.2 Top tool............................ 40 5.4 Discussions.............................. 43 6 Conclusions and Future Work 46 6.1 Conclusions.............................. 46 6.2 Future Work.............................. 47 ix

References 48 x

Chapter 1 Introduction This thesis document consists of 6 chapters. Chapter 1 introduces thesis concepts, problems statements and motivation in the background section, aims and objectives, research questions and expected contribution of the thesis in the later sections. Research work associated with the thesis in general is presented in Chapter 2. Chapter 3 exhibits the main thesis concepts by briefly discussing the underlying technologies like virtualization, cloud computing, standard tools used in experimentation and the methodology adopted. Experimental modeling and setup is highlighted in Chapter 4. Chapter 5 presents the results obtained from the experimental runs with a detailed analysis and discussions. Conclusions derived from the analysis along with intended future work are highlighted in Chapter 6. 1.1 Background Cloud computing is a predominant phenomenon in telecommunication, which allows sharing of hardware resources with scalability and flexibility by eliminating the constraints on distance. According to NIST, cloud model is classified into three service models based on the resources provided: Software-as-a-Service, SaaS, Platform-as-a-Service, PaaS and Infrastructure-as-a-Service, IaaS. Application or software itself is offered to the customer as service in SaaS model, while a platform for building or developing customer s applications is provided as service in PaaS. On the other hand, IaaS renders pools of computational, storage or network resources and permits the consumers to provision them as per need. These cloud solutions are utilized on a pay-per-use basis, thereby saving the initial investment and maintenance costs for the customers. [1,2,3] 1

Chapter 1. Introduction 2 Similar to traditional systems, monitoring system performance with respect to computational resources and applications in cloud computing is important. Performance monitoring and resource management in cloud infrastructures lies on a higher level of complexity due to lack of standards in these service models, where the customers do not have access to the underlying hardware machines. Cloud service providers, CSP, monitor the resources to ensure quality in their services as well as to bill their customers.[4,5] The costs faced by CSP depend on the CPU utilization and the costs of the user are based on the time of lease of resources. Higher CPU utilization requires more electricity and cooling, which amount to around 40% costs in datacenters. Although cloud services promise improved resource utilization, it is complicated to determine the adequate amount of resources in order to satisfy variable workload. Load management techniques address this issue of managing computational resources according to the varying workloads, thus help in avoiding headroom or hotspot and minimizing the costs. Our study focuses on CPU load as metric of monitoring and analysis in cloud networking infrastructures.[6,7,8,9 ] CPU load is the demand for computational resources, in other words, is the number of processes running or waiting for the resources. CPU load is determined by adding the CPU utilization and saturation. Utilization is the time a processor is busy and is indicated in percentage. Saturation is the number of processes that are waiting for the CPU at a position where the CPU is 100% utilized.[10,11] Cloud computing is built on virtualization technologies that provide a structure where multiple instances can be run on one PM. Since the customers cannot access the hardware, the responsibility of monitoring the resources lies with the service provider to meet the SLAs. It is however complex since there is no possibility for the service provider or user to ensure that the resources are either over used or under used, which is violation of SLAs [5,12]. In such a case, monitoring Virtual Machine, VM, data is a challenge since the CSP and its customer have a different perspective of system performance. In this thesis, we aim at investigating CPU load as viewed from both CSP and customer perspective to identify the relationship between CPU load on host to that on the guests. This relationship can help the CSP in billing their customers based on time as well as resource usage and can also be applied for initiating load prediction for load management techniques. The thesis mainly focuses on modeling a framework for CPU load generation, collection and evaluation and on analyzing the guest and host CPU load relationships. The experiments are conducted on OpenStack compute node and

Chapter 1. Introduction 3 an on-premise virtualization device using a set of standard Linux tools. This research is carried out in collaboration with a second master thesis An investigation of CPU utilization relationship between host and guests in cloud infrastructure. While this thesis solely focuses on methodologies for obtaining, monitoring and analyzing CPU load relationships; the second thesis focuses on methodologies in obtaining CPU utilization. The experimental scenarios of both the theses coincide, yet the tools used for measurements and contributions made from the observation and analysis of results, differ. [13] 1.2 Aims and Objectives The aim of the thesis is to establish a framework for CPU load characterization in federated cloud environment. The experiments outline the performance of load as defined by a set of standard Linux tools that are assist in obtaining the CPU metrics. This is achieved by imposing well-known stress on the vcpus and extracting the load values from the Physical Machine, PM and VMs, there by identifying the relation between the PCPU and vcpu metrics by a visual correlation analysis. Study related to the working mechanism of these tools is beyond the scope of this thesis. Cloud operators need to scrutinize the resources available regularly in order to ensure proper load management and meet the SLAs set with cloud customers. Main objectives of the thesis include: Study of commercially off the shelf performance evaluation tools Study of available tools or applications for generating workload Modelling of experimental platform in OpenStack Modeling of standalone virtualization test platform Implementation and experimental runs on both the platforms Use of standard tools for CPU load collection Iterations of the experiments to ensure robust result Analysis of the results Observation of correlation values Visual correlation analysis of obtained host and guest CPU load

Chapter 1. Introduction 4 1.3 Research Questions As discussed in section1.2, our goal is to design a model for load collection and evaluation and to investigate host and guest CPU load relationships for better load management. The following are the research questions formulated: 1. How to model ways of collecting physical and virtual CPU load from a cloud infrastructure? 2. How do the physical and virtual systems react when load is changed? 3. How to identify relationship between host CPU load and Guest CPU load? 4. In what way is the relationship of host and guest useful in load management? 1.4 Expected Contribution The expected outcome of this thesis comprises: Design, modeling and implementation of cloud as well as virtualization platforms for experiments. Data collection and observation of the measurements over iterations to ensure robust results. Method for analyzing load relationships between PM and VM in cloud and virtualization environment. A detailed mathematical and visual correlation analysis. Identifying association between physical and virtual CPU load for better load management.

Chapter 2 Related Work Relevant research work associated with this thesis in multiple aspects is introduced in this section. A comparative study is conducted to identify the research gaps with proposals of other applicable methods and tools. As mentioned in previous sections, monitoring resource usage for proper load management in cloud is an important and on-going topic of research. In [14], Mour, et al., presented a novel approach towards load management in cloud by migrating VMs from a burdened hardware machine to other under loaded machines to help achieve substantial improvement in performance. In their model, they have considered a cloud environment comprising diversified range of physical as well as virtual machines. The model comprises of several modules handling individual tasks of load collection, load evaluation, load prediction, load balancing and VM migration. In order to manage the load it is necessary to collect and evaluate the existing load and make a proper prediction based on current load as well as load expected in near future. Our thesis concentrates on load collection and evaluation in OpenStack cloud and virtualization system aiming at finding a relationship between load on VM and PM, which can be of use in load management models. Another paper, [15], characterized a dynamic resource management system for cloud computing with focus on VM CPU mapping to the physical CPU because PM capacity should be adequate to fulfill the resource needs of all VMs running on it. They tried to estimate future resource usages without looking inside the VMs, using trend of resource usage patterns. The cloud environment they have used is based on Xen hypervisor. Few research works include empirical study on VM performance in cloud, i.e.; deducing from practical observations rather than believing in theory [16]. 5

Chapter 2. Related Work 6 One such work can be found in [17], where the authors have characterized and analyzed server utilization with diverse workloads encountered in a real cloud systems. Their analysis states that workload variability across different time spans in a cloud environment is relatively high and they further intended to carry out a more fine-grained analysis on the correlation and effects of workload types on server. In [18], an empirical study on OpenStack cloud s scheduling functionality is portrayed. The paper aimed at evaluating behavior of OpenStack scheduler with respect to memory and CPU cores of PM and VMs and their interactions. Their findings include that CPU cores requested by instances is an important factor at the time of resource allocation to VMs in all types of OpenStack schedulers. Acquiring the concepts and research gaps from these papers, we set out to perform empirical study applying black-box approach similar to theirs, in Open- Stack cloud networking infrastructures to identify load relationships between host and guest. Authors of [19] designed a system to measure CPU overhead in the virtual computing system to obtain VM and physical CPU utilization to map a relationship between them. However this does not deal with the impact on PCPU when a VM utilizes more than the allocated resources i.e.; vcpus, which is addressed in our experiments. Corradi et al., in their work VM consolidation: A real case based on OpenStack Cloud, conclude that power consumption can be exceptionally reduced with VM consolidation and they desire to investigate further in the direction of workload effects with either CPU or network to understand the VM consolidation and role of SLAs in service decision process. They also wished to deploy larger OpenStack cloud for further testing [20]. This is related to our thesis work from the perspective of VM consolidation in real cloud infrastructures, which is experimented and tested in our thesis on a real OpenStack cloud while multiple guests share resources. In [21], CPU utilization while running web service application on cloud and a normal virtualization system is compared which would be helpful for the user in specifying the capacity to server, based on computational needs. They performed experiments on cloud and virtualization environments and the results show that web service CPU utilization in cloud is higher than that of on-site device. Their scenarios of experimentation are similar to ours, but our goal is to identify load relationships of host and guest in variable workloads rather than running a single application. These excerpts from the related work show that load management and workload impact on PCPU and vcpus are indeed current important topics of research. After careful review and comparison of these works and future targets

Chapter 2. Related Work 7 that stand as motivation to our thesis in multiple ways, we proceed to perform our experiments aiming at comparing and identifying PCPU relation with respect to varying workload on vcpus which could be advantageous in load prediction and management techniques.

Chapter 3 Methodology This chapter provides a brief description of the underlying technologies along with the research methodology and measurement tools used in this thesis. 3.1 Introduction to Underlying Technologies A concise description of fundamental technologies required to grasp the main idea of the thesis is provided in this section. The fundamentals include virtualization, hypervisor and cloud computing principles and standard measurement tools, which detailed in the forthcoming sections respectively. 3.1.1 Virtualization Virtualization facilitates abstraction of physical servers into virtual machines with their OS. The virtual machines can share the resources at the same time. Virtualization is efficient as it reduces the need for physical resources by hosting multiple servers on a single physical machine. Two main techniques of virtualization are OS virtualization and Hardware Virtualization.[10] OS virtualization In OS virtualization, the operating system is partitioned to multiple instances, which behave like individual guests. These guests can be run, rebooted, administered independent to the host machine. These instances can be virtual servers of high performance to cloud customers and of high density to the operators. The disadvantage of this technique is that the guests cannot run different kernel 8

Chapter 3. Methodology 9 versions and is overcome in hardware virtualization technique.[22] Hardware Virtualization This technique of virtualization involves creation of virtual machines with entire operating system including kernels. This means that they can run different kernel versions unlike OS virtualization. Hardware virtualization supplies an entire system of virtual hardware components where an OS can be installed. This in turn has the following types involved: Full virtualization binary translations: The instructions passed to the guest kernel are translated during the run time. Full virtualization hardware assisted: The guest kernel instructions are not translated or modified and are operated by hypervisor running a Virtual Machine Monitor, VMM. Para virtualization: This type of virtualization provides a virtual system with an interface to virtual OS for using physical resources through hypercalls. This is mostly prevalent in network interfaces and storage controllers.[10, 23] 3.1.2 Hypervisors Hypervisor is a computer software or hardware or firmware that creates, provisions, runs and monitors virtual machines. The following are two types of hypervisors: Type 1 This hypervisor runs on the processor directly but not as a kernel software. The supervision is taken care of by the first guest on the hypervisor, which runs on ring 0. This performs the administration work like creating and running new guests. This is also called as bare-metal hypervisor and provides scheduling for VMs. eg: Xen. Type 2 Host OS kernel executes and supervises the hypervisor and the guests existing on it. This does not come with a scheduler of it s own but uses the host kernel scheduler. Eg: KVM.[24] KVM, Kernel-based Virtual Machine, is a type 2 open source hypervisor used widely in cloud computing. This hypervisor, coupled with a user process called QEMU Quick Emulator, creates hardware assisted virtual instances [25]. It is also used in Joyent public cloud and Google Compute Engine [26]. Guests are at first provisioned by allocating CPU resources as vcpus and then provided

Chapter 3. Methodology 10 scheduling by hypervisor. The vcpus allocation is limited to the physical CPU resource. When it comes to observability, physical resource usage cannot be observed from the virtual instances. Hardware support is limited to 8 virtual cores for each physical core on the virtual machine and once the maximum number of CPUs exceeds, QEMU provides software virtualization to KVM. In case of multiple VMs hosted by a physical machine, better performance can be attained by assigning 1 virtual core per VM.[27] 3.1.3 Cloud Computing and OpenStack As summarized in section 1.1, cloud computing is a popular technology supporting physical resource sharing by multiple tenant servers. The services of cloud: IaaS provides compute; storage, network resources and the consumers are allowed to provision them based on their needs. PaaS allows users to run their applications on provided platform and SaaS, on the other hand, allows the customer to utilize the application via a user interface.[1] In the Future Internet architectures, federation of such public and private clouds is an interesting feature. One such project working towards the goal of reaching a cloud federation is FI-PPP, Future Internet Public Private Partnership framework. This framework consists of several smaller projects and XiFi is the project that concentrates on building and providing smart infrastructures for this cloud federation. These Infrastructures facilitate the deployment of several applications and instances in a unified market place, where, business logic is instantiated as VMs. BTH is one among the current operational nodes across Europe.[28] These nodes of XiFi are interconnected through networking infrastructure. These architectures are beneficial since they are not implemented at one place and hence are resilient. If hardware at a place is crashed or out of storage, the virtual instances can be moved to other places depending on the already existing load on them. The XiFi nodes are heterogeneous clouds built on OpenStack and provide tools and services such as Generic Enablers for the deployment of various applications. They encompass the Cloud principles namely, on demand self-service, resource pooling, scalability and security.[29] Another example for such federated platform is the infrastructure at City Network hosting AB. City Network AB is a corporation that provides cloud

Chapter 3. Methodology 11 domain services and hosting to its customers. Similar to XiFi, the services are delivered via OpenStack user interface, where the users can create and provision their VMs and utilize the storage and other high quality services offered by City Network. This web interface is called City Cloud and is similar to FIWARE cloud lab of XiFi, which is built on the OpenStack dashboard. However, unlike XiFi, City Network upgrades regularly and keeps track the latest OpenStack releases. Currently, they provide hosting on their datacenters in UK, Stockholm and Karlskrona.[30] City Network architectures have considerable number of customers utilizing their services. Identifying and comparing the host and guest CPU load will be of great value in these operational clouds for better customer service and load balancing. Cloud Networking focuses on providing the VMs with static or dynamic IP addresses, firewalls to make them available to reach from elsewhere. Cloud Networking provides control and security for the network functions and services delivered as a service over global cloud computing infrastructure. The word global, here, resembles the federation of the local clouds through network infrastructures. Cloud Networking can be of two types Cloud Enabled Networking (CEN) and Cloud Based Networking (CBN). In the criterion of CEN, the management and control characteristics are moved into the cloud while the network functions such as Routing, Switching and Security services remain in the hardware. In the second principle CBN, the requirement for physical hardware is abolished and the network functions are transferred to the Cloud. Yet, the networking infrastructure needed for fulfilling the networking demands of physical devices remains in the hardware.[31,32] OpenStack is open source cloud solution software that provides and manages IaaS. The infrastructures offered by OpenStack include compute, storage and networking resources. OpenStack comes with a combination of core and optional services that can be implemented in its cloud architecture. The minimal architecture of OpenStack consists of its core components that can be realized in either three-node or two-node architectures. Figure 3.1 shows the core components and services in a three-node architecture. The two-node architecture is similar to figure 3.1 eliminating the Network node, whose services are moved to compute node instead.[33,34] Controller Openstack comes with a cloud controller that controls or administers other core and optional components or nodes of openstack. Controller node is built to serve as central management system for OpenStack deployments. Controller can be deployed in a single node or various nodes depending on

Chapter 3. Methodology 12 the requirement. The main services managed by controller include authentication and authorization services for identity management, databases, user dashboard and image services.[35] Compute OpenStack s compute is known as Nova by its project name. The compute node is the device that comprises the software to host virtual instances, thus providing IaaS cloud platform. Nova does not come with virtualization software but comprises drivers to interact with the virtualization layer underneath. Its main components include object storage component called as swift and block storage component called as cinder that provide storage services.[36] Networking This aims at providing network services to the instances and enables communication between the compute nodes and the virtual instances. It is not necessary to have a separate node for networking and one of the compute nodes can be utilized for the this purpose. The project name of Networking component in OpenStack is Neutron.[34] Dashboard OpenStack provides a user interface for the users to create and provision virtual instances as per need. It is formally named as Horizon in Open- Stack.[35] Telemetry As shown in figure 3.1, Telemetry is an optional service in OpenStack. Telemetry, or Ceilometer in OpenStack, monitors the OpenStack cloud environment for providing billing services. Ceilometer has an agent to collect CPU and network metrics that are used for billing, auditing and capacity planning. It has meters to collect duration of instance, CPU time used, number of disk io requests. The CPU utilization reported by this agent is based on CPU ticks but not workload of the VMs.[37,38,39] The hardware required for OpenStack installation depends upon the number of virtual instances needed or the types of services provided. Table 3.1 displays the minimal requirements for a small cloud platform using Open- Stack.[40] It is evident that cloud networking aims at increasing efficiency. Efficiency can be defined as the ability to do something without wastage of material, time and energy. Smart Load management can improve efficiency in cloud infrastructures with focus on how best the resources can be shared. In cloud, we place as many VMs as we can in the system at different locations to increase efficiency;

Chapter 3. Methodology 13 Figure 3.1: Minimal architecture and services offered by OpenStack s Controller node(left), Networking node(center) and Compute node(right) Table 3.1: Minimal hardware required to install a controller and a compute node Processor RAM Hard Disk Controller 1 2GB 5GB Compute 1 2GB 10GB on the other hand, we need quality and speed required for running the VMs. The system is not efficient if it is not completely loaded and we need to know about load on the system to characterize efficiency [14,15]. Our hypothesis is to find and understand the relation between CPU load inside the VMs and outside the VMs (host machine), which can be applied to smart load management techniques to ensure efficient usage of resources.

Chapter 3. Methodology 14 3.2 Methodology In this section, we describe the methodology applied to our thesis. In order to reach the aim of our thesis, our methodology includes two subsequent stages - Experimental Research and Visual Correlation Analysis. Experimental Research section resembles our approach in performing the experiments. Visual Correlation Analysis is our analysis strategy to find relationship between host and guest CPU load. 3.2.1 Experimental Research Prior to experimentation, we have conducted a thorough study of appropriate literature and tools and modeled experiments. The approach in carrying out this thesis is Experimental methodology, which is broadly used while evaluating solutions for questions. In an experimental research, the researchers take measurements and further make evaluations to answer the questions.[41] In an experimental approach, it is important to implement the following three stages, which are followed in our approach as well. 1. Record keeping a good documentation of the experiments and configuration is important in experimental work for future reference and relevance. 2. Experimental setup design the platform for running experiments need to be modeled and designed. At the end of this phase, researcher should document hardware and software components and the experimental findings. 3. Reporting experimental results the results obtained should be stated clearly without distortion and discussed.[41] We have adopted spiral methodology, a well-known method of software development, in performing our experimental research [42]. As resembled in figure 3.2, our spiral model consists of 4 dimensions Configuration, experimentation, observation and analysis. First, we make necessary configurations of PM and VMs and then perform the experiments in that configured scenario. The results are then observed and analyzed. Based on the analysis, the complexity of the configurations is increased for further experimentation and observation, justifying the spiral model approach.

Chapter 3. Methodology 15 Figure 3.2: Experimental Methodology portrayed as a sprial model. Configuration Analysis Experimentation Observation 3.2.2 Visual Correlation Analysis There are various possible analysis methods viz. statistical analysis but we start with a simple approach, where we do a visual correlation analysis of the data obtained from experimentation. Visual correlation analysis shows association between a set of x and y attributes as depicted in figure 3.3. This visual representation to identify any correlation between 2 sets of values is called as scatter plots [43]. Figure 3.3 is an illustration of the scattered plots and linear correlation line mapping the x and y axes values. We have applied bivariate analysis to the results to determine empirical relationship between CPU load reported by host and guest in the form of scatter plots and correlation co-efficient table. Bivariate analysis is used to predict value of dependent variable when the value of independent variable is known [44]. However, finding the reason for such a behavior is beyond the scope of this thesis. We set out to find a graphical representation that shows the relationship between host and guest CPU load. Since, we apply a black-box approach as discussed under chapter 2, we anticipate that the correlation we obtain from the bivariate analysis can be linear or exponential or highly varying curves as shown in figure 3.4. Identifying the behaviour of this relation is of interest here. In case of linear correlation obtained, we identify the strength of the relationship from correlation coefficient. Correlation coefficient (ρ) is calculated using the following formula: ρ (H,G) = cov(h, G) σ H σ G (3.1)

Chapter 3. Methodology 16 Figure 3.3: Example of graph showing scatter plots and linear correlation as a relation between x and y attributes. 8 7 6 5 Y attributes 4 3 2 1 0 0 1 2 3 4 5 6 7 8 X attributes Figure 3.4: Anticipated graphical plots of possibly attainable correlation between host and guest in terms of CPU load where: cov(h, G) = (H H)(G G) n (3.2) H: host G: guest

Chapter 3. Methodology 17 cov: covariance σ H : Standard deviation of host σ G : Standard deviation of guest As we have gone through the overview of our methodology, we proceed to the measurement tools used in our experiments in the next section. 3.3 Measurement Tools There are several simple and standard tools that help obtain CPU load metrics in a Linux system. We have followed Tools methodology in system performance, which embraces the use of available performance tools, followed by checking the metrics they provide and finally interpreting the metrics obtained [10]. Various standard tools used for this experimental purpose are described in this section. The scope of this thesis is limited to the use of these tools for experiments and it does not include the study related to the mechanisms of these tools. In a cloud environment, load can be caused or affected by the factors as depicted in figure 3.5. Load exists in three dimensions in physical server, in the network and in the virtualization technology [45,46]. This thesis is about CPU load observation in VMs and the physical server hosting them. Hence, our focus is on the hardware and resource allocation on server and virtualization dimensions respectively. CPU load is the number of processes utilizing the computational resources. It is calculated based on exponentially moving average and is updated every 1 min, 5 min and 15 min interval [47]. There are various CPU performance analysis tools in LINUX OS. Few such tools that provide CPU statistics are top and uptime that fetch the values from /proc/loadavg. Load average implies the demand for CPU and is computed by summing the number of threads running and the number that are waiting to run. When there are processes that are queued for compute resources, the CPU is said to be in saturation. The load average should between the numbers of the CPU cores and if load average is higher than maximum number of CPU cores, the CPU is in saturation state since there are processes that are queued.[48,49]

using them. The tools in this section are listed in Table 6.6. Table 6-6 CPU Analysis Tools Linux Chapter 3. Solaris Methodology Description 18 uptime uptime load averages Figure vmstat 3.5: Server, vmstat Virtualization includes system-wide and Network CPU components averages that are related to causing or effectingload load in management the system. mpstat mpstat per-cpu statistics sar sar historical Server statistics ps ps process status top prstat Application monitor per-process/thread Virtualization CPU usage pidstat prstat per-process/thread CPU breakdowns Database System time ptime time a command, with CPU Resource breakdowns allocation DTrace, perf Hypervisor Operating DTraceSystem CPU profiling and tracing perf cpustat CPU performance Virtualization counter analysis type Server Hardware Network The list begins with tools for CPU statistics, and then drills down to tools for deeper analysis including code-path profiling and CPU cycle analysis. This is a selection of tools and capabilities to support Section 6.5, Methodology. See the documentation for each tool, including its man pages, for full references of its features. Description of the measurement tools used: While you may be interested in only Linux or only Solaris-based systems, consider looking at the other operating system s tools and the observability that they UPTIME This is a command tool used for printing the load averages of the provide for a different perspective. system for the last 1-, 5- and 15-minute. As indicated above, load average is the amount of work waiting for the processor and we can compare these three load averages in order to identify the trends in CPU load [50]. Figure 3.6 shows the output 6.6.1 of uptime uptime command on Linux terminal.[10] Hardware Protocols Functions uptime(1) is one of several commands that print the system load averages: Figure 3.6: example of output of uptime command on terminal. $ uptime 9:04pm up 268 day(s), 10:16, 2 users, load average: 7.76, 8.32, 8.60 TOP Another popular tool for monitoring the top running processes is top. The top command, when issued, prints a system-wide summary and processes running on the terminal. The top results are updated at regular intervals on Linux. System-wide summary is the CPU statistics which includes From the Library Load average, of Kurt Tutschku percentage utilization of the user, system, nice, idle waiting, hardware interrupt, software interrupt and stealing levels. These values are obtained by taking the mean across all CPUs. [10,51] Top is a well-accepted tool by Beginners of performance analysis due to its features. In spite of this significance it has a minor drawback due to its impact on the performance by applying more load. That is, when top command

as the sum across all CPUs. A single-threaded CPU-bound process will report 100%. A two-thread CPU-bound process will report 200%. On Solaris, %CPU is normalized for the CPU count. For example, a single CPUbound thread will be shown as 12.5% for an eight-cpu system. This metric also shows recent CPU usage, using similar decayed averages as with load averages. Chapter Various 3. other Methodology options are available for ps(1), including -o to customize 19the output and columns shown. is issued on a system, the process or thread that is running top is visible in the %usr column. This implies that top command consumes the CPU itself for extracting 6.6.6 topthe statistics, which is not favorable in performance analysis. This is due to the existence of open, read and close function system calls which have their top(1) ownwas costs created of extracting by William statslefebvre from /proc in 1984 entries. for Top BSD. also He has was another inspired minor by the drawback VMS command of eliminating MONITOR short-living PROCESS/TOPCPU, processes, which sinceshowed the stats the are top provided CPU-consuming after taking jobs with a snapshot CPU percentages of the /proc and entities. an ASCII Hence, bar chart it does histogram not consider (but the not processes columns of that data). lived in between the interval of these snapshots [10]. Figure 3.7 shows the output of top command on terminal. The top(1) command monitors top running processes, updating the screen at regular intervals. Figure 3.7: For example, ofon output Linux: of top command on terminal. $ top top - 01:38:11 up 63 days, 1:17, 2 users, load average: 1.57, 1.81, 1.77 Tasks: 256 total, 2 running, 254 sleeping, 0 stopped, 0 zombie Cpu(s): 2.0%us, 3.6%sy, 0.0%ni, 94.2%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 49548744k total, 16746572k used, 32802172k free, 182900k buffers Swap: 100663292k total, 0k used, 100663292k free, 14925240k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11721 web 20 0 623m 50m 4984 R 93 0.1 0:59.50 node 11715 web 20 0 619m 20m 4916 S 25 0.0 0:07.52 node 10 root 20 0 0 0 0 S 1 0.0 248:52.56 ksoftirqd/2 51 root 20 0 0 0 0 S 0 0.0 0:35.66 events/0 11724 admin 20 0 19412 1444 960 R 0 0.0 0:00.07 top 1 root 20 0 23772 1948 1296 S 0 0.0 0:04.35 init Although top has these minor drawbacks mentioned above, this standardized tool is used in well-known performance monitoring applications like Nagios [52]. We have chosen both uptime and top tools for our experiments after keenly studying them, to be able to make a comparison and validation of our experiments on CPU load of the system. This helps us find out the how the From the Library of Kurt Tutschku results of top differ from uptime, even though both collect data from the /proc entries [10]. Our aim however, is not to compare these tools but to use them for our experiments. Stress is a popular Linux tool that is used to literally stress the CPU with proper computational workload. It comes with various other options of CPU cores, memory, time etc. In our experiments, we have stressed the vcpus for a period of 5 minutes. Scripts for uptime and top commands are written to run after every 5-minute interval. Stress comes with various options and we have used -c in this experiment. Option -c is for increasing the number of CPUs that you wish to be stressed. This is equal to vcpus if used on VMs and CPUs if used on host machine.[53,54] Stress-ng is a tool similar to stress tool with additional load customizing option.

Chapter 3. Methodology 20 With stress-ng tool, we can impose desired amount of load in percentage using -l option [55]. In our experiments, we started with the well-known tool stress widely used in system performance tests viz. [56] and moved on to smarter tool stress-ng for a refined analysis of system behavior under varying workloads.

Chapter 4 General Experimental Setup Now that we have given a clear direction of our methodology and tools selected, this chapter gives brief description of our experimental design and it is essential to have a clear picture of basic concepts and our approach from the previous chapters, in order to be able to connect the dots and to interpret the motivation behind our experimental model. The following sections explain our conceptual model and different experimental structures with figures. 4.1 Experimental Modeling We have modeled our experimental test bed in an OpenStack cloud to observe load relationship between host and guest. As stated earlier, in OpenStack, nova compute is the agent that provisions and runs the VMs (refer section 3.1.3) and depending on the requirement there can be multiple compute nodes. Figure 4.1 is a notion of having multiple compute nodes that are managed my one controller in an OpenStack environment. A compute node requires a hypervisor for interacting and managing the VMs. Our OpenStack cloud environment uses KVM hypervisor, which is a type-2 hypervisor. Type-2 hypervisors use the host s kernel scheduler to process requests from VMs and do not have their own schedulers. Figure 4.2 is an abstraction of the layers involved in a virtualization system with and without OpenStack compute agent. As shown in the figure 4.2, nova compute interacts with the virtualization mechanism using APIs but does not have virtualization mechanism of its own. [36] Hence, our hypothesis is that simple virtualization system and a system 21

Chapter 4. General Experimental Setup 22 Figure 4.1: An OpenStack environment can have n number of compute nodes based on requirement and a controller manages the compute nodes. Controller Compute Node1 Compute Node2 Compute Node3 Compute NodeN Figure 4.2: Abstraction of hardware, software and virtualization layers in a system. Nova-compute is not present in a normal virtualization system. with OpenStack compute agent will show similar load behavior. Since we apply spiral development approach to our experimentation, we initially have similar experimental setup on two devices OpenStack compute node and an on-premise device with basic configurations to compare the results. If the observed results support our hypothesis, we increase the complexity in our experiments and continue them on our on-premise setup. The main reason for setting up an on-site device is the difficulty in collecting information from the VMs as a CSP is devoid of access to VMs and also to eliminate the impact of other services running on operational cloud, on our test environment and avoid inconvenience for customers.

Chapter 4. General Experimental Setup 23 Working accordingly towards our aim of finding relationship between host and guest CPU load, we collect the CPU load from both PM and VMs as portrayed in figure 4.3. The main subject of interest lies in investigating if we can interpolate and extrapolate from host to guest and vice-versa by observing load behavior on the PM with varying workloads on the VM. Figure 4.3: Visual representation of the PMs, VMs and tools used in the implementation of experiments. The figure resembles the question as to what could be the relationship between load on host and guest machines. Host Guest1 Guest2 Guest3 Guest4 uptime/top? stress/stress/ng uptime/top stress/stress/ng uptime/top stress/stress/ng uptime/top stress/stress/ng uptime/top GuestOS Kernel GuestOS Kernel GuestOS Kernel GuestOS Kernel Hypervisor(KVM) HostKernel HostOS CPU I/O Memory Apart from this, we have performed the experiments also on two different host OS - CentOS and Ubuntu, popularly used in cloud architectures. As KVM is type 2 hypervisor and runs on host OS as software, the VMs on KVM are processes that can be manipulated. OS scheduling is provided by host kernel and our hypothesis leads us to observe behavior of these two different kernel versions of Linux distribution system. CentOS and Ubuntu are chosen to examine our hypothesis. The tools used for load generation are stress and stress-ng and for CPU load collection are uptime and top. All experimental scenarios and configurations are described in detail in the succeeding sections of this chapter along with figures. The observations of the experiments are presented in chapter 5.

Chapter 4. General Experimental Setup 24 4.2 Experimental setup In the initial setup, we modeled the experimental platform on operational cloud Infrastructure and an on-premise device, as described below. 4.2.1 OpenStack Cloud Test-bed This experiment was executed on one of many compute nodes of City Cloud infrastructure owned by City Network AB [57]. Features of the compute node used for the experiments are listed in Table 4.1. Table 4.1: Specifications of the OpenStack Compute Node used for experiments Criteria Specification CPU Intel Xeon CPU E5-2697 v2 @ 2.70GHz 48 cores RAM 32GB Hard disk 2x300 GB SAS Hypervisor KVM Host OS CentOS 6.6 Guest OS Ubuntu 14.04 LTS server Four virtual machines were created on this compute node with CentOS as host machine. Ubuntu 14.04 LTS server is the OS used for all the 4 VMs on this node. Each VM was provisioned with 2 vcpus as KVM hypervisor uses core numbers for provisioning [27]. The next step in the process is stress testing VMs and simultaneously collecting the CPU load metrics from host. There is no other application running on the guests or hosts while stress tests are in proceeding, to make sure that the CPU is free of any other load infliction than produced by stress. The setup is as presented in figure 4.4. The four virtual instances present on the host device are up and running all through the experiment. At first, stress command is issued to impose load on 1 VM for 5 minutes while the other VMs are idle. The stress command is customized to stress 2 vcpu cores, which means that the VM is imposed with 100% load, as each VM has 2 vcpus assigned to it. Simultaneously, the load values are obtained from the host with the help of command line tool uptime. The experiment is conducted in similar manner for 2 VMs. This means,

Chapter 4. General Experimental Setup 25 Figure 4.4: Depiction of the experimental setup on OpenStack platform GuestOS VM1 Kernel GuestOS VM2 Kernel GuestOS VM3 Kernel GuestOS VM4 Kernel KVM/Libvirt OpenStackNOVACompute HostKernel UbuntuorCentOS Hardware(Processors) 2 vcpus each of 2 VMs were stressed. In other words, 100% load was imposed on 2 VMs each and load metrics were collected once again from host. The rest 2 VMs are idle while 2 guests are being stressed. The next stage involves conducting similar stress tests on 3 VMs with 1 VM idle. After collecting the load from host machine, all the 4 VMs are imposed with stress on their dedicated vcpus, i.e, 8 vcpus in total. This again is simultaneous with uptime running on host. Figure 4.5 provides a clear picture of the 4 stages of this experimental run. 4.2.2 On-premise Test-bed This test setup is installed on a standalone device with KVM hypervisor. This setup helps us in distinguishing between the OpenStack existent and non-existent cases for supporting our hypothesis as stated in section 4.1. This test bed is built on the attributes as listed in Table 4.2. This experimental setup imitates the City Network s compute node chosen for the previous experiment, with minor difference being the number of CPU cores. The hypervisor used is KVM as it is highly used by popular public and private clouds viz. joyant and google [10], which is also evident from City Network s infrastructure.

Chapter 4. General Experimental Setup 26 Figure 4.5: Stages of experiments performed with stress (a) stage 1 - stress imposed on 1VM (b) stage 2 - stress imposed on 2VMs stress VM1 VM2 VM3 VM4 stress VM1 stress VM2 VM3 VM4 CPU (c) stage 3 - stress imposed on 3VMs CPU (d) stage 4 - stress imposed on 4VMs stress stress VM1 stress VM1 stress VM2 stress VM2 stress VM3 VM3 stress VM4 VM4 CPU CPU Virtual machine provisioning in KVM is based on the cores available on the PCPU [27]. For the experimental purposes, 4 guests were created and provisioned with 2 vcpus each. That is, out of the 8 total cores available in the PCPU, 2 vcpus each are allocated to one VM each. Figure 4.6 provides a pictorial representation of this test-bed setup. The host OS used is CentOS 6.6 and it should be noted that the guest OS is Ubuntu

Chapter 4. General Experimental Setup 27 Table 4.2: Specifications of the on-premise device used for experiments Criteria Specification CPU Intel Xeon CPU E3-1230 v2 @ 3.30GHz 8 cores RAM 8 GB Hard disk 500 GB SAS Hypervisor KVM Host OS CentOS 6.6 Guest OS Ubuntu 14.04 LTS server 14.04 LTS server in all the cases that are considered for experimentation. Also, similar to the previously described OpenStack test case, all the VMs are running all through the experimentation. Moving on to the performance tests, stress tool is used for generating load on the guests. Stress command is first issued on VM 1 for producing 100% load by stressing the 2 vcpus available on the VM. The CPU load is collected from the Centos host using uptime. Then, this is repeated for 2 VMs, stressed with 2 vcpus each. The host CPU load behavior when 2 vcpus are stressed fully is observed via uptime command. Figure 4.6: Depiction of on-premise device setup Guest$OS$ Guest$OS$ Guest$OS$ Guest$OS$ VM1$ VM2$ VM3$ VM4$ kernel$ kernel$ kernel$ kernel$ KVM$ Host$Kernel$ Ubuntu$or$Centos$ Hardware$(Processors)$

Chapter 4. General Experimental Setup 28 The next step is to stress test 3 VMs, while 1 VM is still idle, and observe load characteristics of the hardware CPU through the host with the help of once again. The last run is stress testing 7 vcpus of the 4 VMs and collecting the CPU load from the host machine. Stress tests on these setups are performed for 5 min on each VM while CPU load values are retrieved from the respective hosts as explained. We reiterated the experiments ten times to ensure robustness in the results. The data obtained is stored for analysis later on and is produced in Chapter 5. 4.2.3 Stress tests on Single Guest Way through our spiral approach, the results of this first stage of experiments convinced us to contiue our experimentation on the on-site virtualization testbed with increased complexity. This second main framework of experimentation is described this section. This experiment is set on the standalone device with virtualization environment but no OpenStack platform. The structure of this experiment is quite different from the previous experimental configuration except for the hardware used, which is similar in both the cases. Besides, a notable difference is the number of VMs that are hosted by the hypervisor, KVM. The specification is as shown is Table 4.2 and is summarized below once again Intel Xeon CPU E3-1230 v2 @ 3.30 GHz 8 cores CPU KVM hypervisor Ubuntu 14.04 server for guest OS Ubuntu 14.04 LTS Desktop and CentOS 6.6 for Host OS This is performed on two different Linux distribution OS Ubuntu and CentOS due to their wide usage in cloud infrastructures. In contrast to the former arrangement, only 1 VM is created on KVM and is allocated only 2 vcpus. In other words, out of the total available 8 cores of PCPU, only 2 vcpus are assigned to the only existing VM. This VM is later stress tested in multiple ways for recognizing the impact on system load. The experimental setup is as depicted in figure 4.7.

Chapter 4. General Experimental Setup 29 Figure 4.7: Depiction of on-premise device experimental setup for Single guest Guest&OS& VM& Kernel& KVM& Host&Kernel& Ubuntu&or&CentOS& Hardware&(Processors)& 4.2.3.A. Testing with Stress tool Initially, stress tool is operated for inserting load on 1 vcpu of the VM. Uptime command is run on both the host and guest, to understand the CPU behavior of both physical and virtual systems. Load information of the guest systems was not collected earlier in the multiple (four) guest scenario. This experiment is also run for 5 min. Figure 4.8 demonstrates this initial step clearly. In our experiments, we have used uptime command to collect cpu load average every 1 second in this 5 min period. Next, the load is applied to 2 vcpus on the VM. To achieve this, stress on the VM is issued along with the option of 2 (v)cpus to be stressed. At the same time, uptime is once again passed to host and guest for collecting the load values over this 5 min time period. Likewise, stress is now given an option to load 3 vcpus from the VM. Even though the VM is bound to run with only 2 vcpu resources, stress is a tool that can be used for enforcing load on more CPUs than that are actually available on the compute machines [53]. It is possible to enter a number higher than the available vcpus and this would make the OS unstable. We chose to do this experiment to observe the impact on PM load behavior when the guest utilizes more than the allocated resources. Henceforth, after generating workload on 1 and 2 vcpus, the tool

Chapter 4. General Experimental Setup 30 stress Figure 4.8: Stress applied initially on 1 vcpu VM 1VCPU2VCPU Cores01234567 stresses 3 vcpus, while the load retrieval from host and Guest is on going simultaneously. Thereafter, the rest of the vcpus are tested for stressing, one after the other sequentially including the former vcpus, that means, 4 vcpus stressed at once, 5 vcpus at once, then 6 vcpus followed by 7 vcpus. All these vcpus are tested subsequently for 5 min and the respective load is retrieved for that particular period of 5 min. Figures 4.9, 4.10 and 4.11 present the architecture and test sequences in a clear manner. Similar tests are conducted on a different host OS, both CentOS and Ubuntu and the obtained load values are stored for further analysis. One should not confuse that these tools are used at the same time on the same experimental run. The results are discussed in the forthcoming Chapter while we move on to the next and last test case in the upcoming section. 4.2.4.B. Tests with Stress-ng tool The observations from the previous experiments gave us an impression that there is need for furthur refinement in the experiments. Following our spiral model, we have used the tool stress-ng for a fine-grain analysis of the system behavior under varying workload conditions. This test case constitutes the same environment and device as the preceding experiment. Given, this time, the load infliction is not

Chapter 4. General Experimental Setup 31 stress VM Figure 4.9: Stess configured to load 2 vcpus 1VCPU2VCPU Cores01234567 stress VM Figure 4.10: Stess configured to load 3 vcpus 1VCPU2VCPU Cores01234567 carried out by stress but by stress-ng. As described in section 3.3, stress-ng is an updated version of the stress tool and provides the facility of producing customized percentage of load with the help of option -l. Provided that there is only 1 VM present on the host machine and that this VM is provisioned with 2 vcpus, stress-ng tool is used for generating certain

Chapter 4. General Experimental Setup 32 Figure 4.11: Stess configured to load 3 or more vcpus. The dotted lines indicate that the number of vcpus being stressed is increased in each experimental run. stress VM 1VCPU2VCPU Cores01234567 varying amounts of load on the VM. In the initial stage of this experimental run, the stress-ng command is modified to generate 10% of load on 1 vcpu of the guest. The tool is set to run for 5 min; meanwhile uptime and top are used for fetching CPU load from host and guest. In similar fashion as the previous case, the number of vcpus being stressed is increased to 2 and more in sequentially. That means, stressing 2 vcpus, 3 vcpus and so on until 7 vcpus, with 10% load through stress-ng and the load values are retrieved in all the cases over the 5 min experimental run. After the 10% load generation, the load is doubled to 20%. So, 1 vcpu of the guest is inflicted with 20% of load by stress ng and load behavior on host is observed. After the experiment is conducted for the rest of the vcpus in order, the load is now increased to 30%. In simpler terms, this can be explained as 1 vcpu is loaded by 30%, then 2 vcpus on whole are loaded by 30%, after which, 3 vcpus are loaded by 30%, followed by 4vCPUs by 30%, 5 vcpus by 30% load, 6 vcpus by 30% of load using stress-ng and lastly all the 7 vcpus loaded by 30% using stress-ng tool. This is as usual for 5 min with a 5-minute interval between runs. The load is retrieved simultaneously from the host and guest using top and uptime in separate runs. The fourth stage includes imposing 40% load following the same strategy

Chapter 4. General Experimental Setup 33 of 0 vcpu to 7 vcpus sequentially. The last two stages involve workload of 50% and 100% respectively on the vcpus. Refer to figure 4.12 for visual abstraction and a rather better understanding of this complex scenario. Figure 4.12: Stess-ng configured to impose load om 1 or more vcpus with 10, 20, 30, 40, 50 and 100% load. The dotted lines indicate that the number of vcpus being stressed is increased in each experimental run. Stressng 10/20/30/40/50/100 VM 1VCPU2VCPU Cores01234567 Once again, it is noteworthy to mention that individual runs are performed for each OS - CentOS and Ubuntu, each load collection tool - uptime and top. On the whole, we have collected immense data from these experimental runs, which, we anticipate will provide the relationship between host and guest. The results obtained were stored for further analysis and are discussed in the next Chapter.

Chapter 5 Results and Analysis All the experiments were conducted in multiple scenarios as described in section 4.2. At first, the experiments were run on compute node of operational OpenStack and the later scenarios were performed on a test bed with just virtualization environment. The runs on the test bed were of different combinations of tools, VMs, vcpus and load values to observe the change in load behavior on the PCPU with respect to vcpus. The experimental runs are for a time period of 5 min with a 5-minute interval between each run. The results that were stored for further analysis are presented and discussed in this Chapter. Summarizing the experimental runs, each run was for a period of 5 min. The 1st min and the last min of the experiments are considered as warm-up phase and are eliminated from analysis. All experiments are iterated 10 times and an average of these values is plotted and considered for further analysis. As we know from section 3.3, top and uptime present load-average of the system over 1-min, 5-min and 15-min period. 1-min load average values are retrieved during the experiment in order to have stable load values. One should not confuse this 5-min load average with our 5 min experimental run time period. Following sections present the results with appropriate analysis, from each scenario - OpenStack and on-premise setup, stress and stress-ng tests respectively. 34

Chapter 5. Results and Analysis 35 5.1 Results from OpenStack and on-premise Testbeds In this experiment, 1-4 VMs on a cloud and on a device with KVM hypervisor are stress tested and the host CPU load values are collected from the OpenStack s compute node and our on-site device respectively. Uptime tool is used for load collection. Before stressing the VMs, in order to determine the CPU performance of the system, uptime is issued for 5 minutes. Then, VM 1 is stressed after a 5-minute interval post termination of the previous uptime command. Then the experiments are continued for rest of the VMs with a 5-minute between each run. In short, 5 min experimental runs with 5-minute interval between each run. The host OS in both cases is CentOS and each VM is assigned 2vCPUs. The results obtained from this experiment are presented in the graphs in figure 5.1. Figure 5.1: CPU load observed in OpenStack compute node and on-premise device in multiple guest scenario (a) CPU load-average collected from compute node 10 CPU load OpenStack compute node 9 8 7 CPU Load Average 6 5 4 3 2 1 0 60 80 100 120 140 160 180 200 220 240 Time(s) 0VM 1VM 2VM 3VM 4VM (b) CPU load-average collected from on-premise host 10 CPU Load Average in CentOS Host 9 8 7 CPU Load Average 6 5 4 3 2 1 0 60 80 100 120 140 160 180 200 220 240 Time(s) 0VM 1VM 2VM 3VM 4VM

Chapter 5. Results and Analysis 36 The x-axis on the graphs in figure 5.1 represents the time of the experimental run in seconds, 300s (5 min) in our case. We present the results after removing warm up phase of 60 s at the start and end. The y-axis presents the CPU load average observed from compute node - figure 5.1(a) and our on-site device - figure 5.1(b). The results show that when we increase the load in each VM, the load on the respective hosts increased. The CPU load average of our on-site device shows exponential behavior over the 60s to 240s time period - Figure 5.1(b), while compute node reported constant load over this period - figure 5.1(a). When observed keenly, these graphs show that there is difference between the CPU loads observed on both the devices. The load average on the cloud device observed is higher by a small margin when compared. This load difference is not a substantial difference as we contemplate it to be due to the impact of several other services running in the operational cloud environment. This result, in a way, supports our hypothesis that the load behavior on both systems should be similar as stated clearly above in section 4.1. After analyzing this, we have decided to continue our experiments on the on-site device, aiming to reach our goal of finding a relation between host and guest CPU load. The following sections present our observations in this context. 5.2 Results of Stress tests This section presents the observations made as a result of experiments from our stress tests on a single guest scenario. This setup was implemented in a virtualization device on our premises with two host OS - CentOS and Ubuntu. The host machine had only 1 VM and this VM was provisioned with 2vCPUs. The stress was initially imposed on 1 vcpu, then on 2 vcpus and so on until 7vCPUS. We collected the load from both PM and VM in this case to map a relationship between them. We have applied visual correlation analyis to these results to investigate CPU load relationships between host and guest. The scatter plots in figure 5.2 show visual correlation between host and guest CPU loads on CentOS device in figure 5.2(a) and on Ubuntu device in figure 5.2(b). The figures 5.2 (a) and (b) are scatter plots that are used to show relation between x-axis - CPU load average on guest and y-axis CPU load average on CentOS and Ubuntu host respectively. The scatter plots above show that in case of CentOS host, figure 5.2 (a), the trends in correlation plots between PM and VM CPU loads are distorted

Chapter 5. Results and Analysis 37 Figure 5.2: Scatter plots showing the relationship between CPU load average on host and guest in different vcpu stress conditions. (a) Correlation between CPU loads on host and guest in CentOS device 3 Correlation between CPU load of Host and Guest in CentOS 2.5 Load Average on Host 2 1.5 1 0.5 0 0 1 2 3 4 5 6 7 8 Load Average on Guest 1 vcpu 2 vcpu 3 vcpu 4 vcpu 5 vcpu 6 vcpu 7 vcpu (b) Correlation between CPU loads on host and guest in Ubuntu device 3 Correlation between CPU load of Host and Guest in Ubuntu 2.5 Load Average on Host 2 1.5 1 0.5 0 0 1 2 3 4 5 6 7 8 Load Average on Guest 1 vcpu 2 vcpu 3 vcpu 4 vcpu 5 vcpu 6 vcpu 7 vcpu with slight linearity observed, on the range of 1-7vCPUs that are stressed. On the other hand, plots on Ubunut host - figure 5.2 (b), report better correlation than CentOS host until 6vCPUs while the trend falls in case of 7vCPUs. This behavior is considered strange provided that the guest machines in both cases are Ubuntu servers. We believe that a refinement in the load observation by varying the workloads is required and hence we proceeded to perform more experiments for

Chapter 5. Results and Analysis 38 a fine-grain analysis of varying workload behavior. 5.3 Results of Stress-ng tests In order to have deeper insights into the CPU load behavior, these experiments are performed using stress-ng tool in varying load conditions ranging from 10, 20, 30, 40, 50 and 100 %. This load variability is applied to 1-7 vcpus on the only existing VM, which is provisioned with 2 vcpus in this setup. Uptime and top commands are used for load collection from PM and VM in both CentOS and Ubuntu host. To analyze the relationship between PM and VM, we present the scatter plots mapping load average of VM and PM. The following sections show the scatter plots. 5.3.1 Uptime tool Figures 5.3, 5.4, 5.5 and 5.6 show the uptime load average values obtained from PM and VM in case of 1, 2, 3 and 7 vcpus respectively. Their sub-figures (a) and (b) present the observations from CentOS and Ubuntu hosts respectively. In all the scatter plots, x-axis is the load average observed on guest and y-axis represents load average observed on host. The scatter plots of load ranging from 10, 20, 30, 50 and 100 % are are plotted. As an inital approach of analyzing results, a linear trendline as an overall interpolation and extrapolation referrence, in case of 100% load is drawn. This comes handy in showing the deviation of the scatter plots from linearity. However, the results of 4, 5 and 6 vcpus are omitted due to similarity in most cases but their correlation co-efficient values are shown in section 5.4. The following are the observations made based on these graphs. In all the cases - figures 5.3 to 5.6, scatter plots on Ubuntu host followed the linear trendline better than the CentOS scatter plots. The load behavior in the conditions below 100% load varies to a great extent, even though most of the plots observed positive correlation. The systems correlated in a proper trend with slight deviation at 100% load compared to the conditions under 100% load.

Chapter 5. Results and Analysis 39 Figure 5.3: Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool. (a) Correlation between CPU loads on host and guest - 1vCPU - CentOS device Correlation between CPU load of Host and Guest CentOS Host, uptime, stress ng on 1 vcpu 1.2 1 Load Average on Host 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 1.2 Load Average on Guest 10% 20% 30% 40% 50% 100% linear (b) Correlation between CPU loads on host and guest - 1vCPU - Ubuntu device Correlation between CPU load of Host and Guest Ubuntu Host, uptime, stress ng on 1 vcpu 1.2 1 Load Average on Host 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 1.2 Load Average on Guest 10% 20% 30% 40% 50% 100% linear However, the overall view shows that the plots do not completely obay the linear trendline. In some cases, it is slightly exponential too - figure 5.5 (a), 5.6 (b). It is anticipated that there is need for more experimentation and research in this regard. More iterations may be needed to make conclusions based on these observations. We can observe the top results from the 7vCPUs in the same scenario in the next section for comparison of the load behavior.

Chapter 5. Results and Analysis 40 Figure 5.4: Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool. (a) Correlation between CPU loads on host and guest - 2vCPU - CentOS device 2.5 Correlation between CPU load of Host and Guest CentOS Host, uptime, stress ng on 2 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 Load Average on Guest 10% 20% 30% 40% 50% 100% linear (b) Correlation between CPU loads on host and guest - 2vCPU - Ubuntu device 2.5 Correlation between CPU load of Host and Guest Ubuntu Host, uptime, stress ng on 2 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 Load Average on Guest 10% 20% 30% 40% 50% 100% linear 5.3.2 Top tool Figure 5.7 shows CPU load behavior as characterized by the tool top. Even though we have extracted top results in all the scenarios, we believe this one scenario would be sufficient for comparison and hence, omitted the other plots. The discussions section - 5.4, will present correlation co-efficient values from all

Chapter 5. Results and Analysis 41 Figure 5.5: Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool. (a) Correlation between CPU loads on host and guest - 3vCPU - CentOS device 2.5 Correlation between CPU load of Host and Guest CentOS Host, uptime, stress ng on 3 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 3 3.5 Load Average on Guest 10% 20% 30% 40% 50% 100% linear (b) Correlation between CPU loads on host and guest - 3vCPU - Ubuntu device 2.5 Correlation between CPU load of Host and Guest Ubuntu Host, uptime, stress ng on 3 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 3 3.5 Load Average on Guest 10% 20% 30% 40% 50% 100% linear the cases. As we can interpret from figures 5.6 (a) and 5.7 (a), the load average observed by top tool differs from uptime tool to a major extent. However, the behavior shown in figures 5.6 (b) and 5.7 (b) on Ubuntu hosts highly coincides. From the above presented results and observations, we understand that CPU

Chapter 5. Results and Analysis 42 Figure 5.6: Scatter plots showing the CPU load relationships between host and guest in varying load conditions - uptime tool. (a) Correlation between CPU loads on host and guest - 7vCPU - CentOS device 2.5 Correlation between CPU load of Host and Guest CentOS Host, uptime, stress ng on 7 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 1 2 3 4 5 6 7 8 Load Average on Guest 10% 20% 30% 40% 50% 100% linear (b) Correlation between CPU loads on host and guest - 7vCPU - Ubuntu device 2.5 Correlation between CPU load of Host and Guest Ubuntu Host, uptime, stress ng on 7 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 1 2 3 4 5 6 7 8 Load Average on Guest 10% 20% 30% 40% 50% 100% linear load relationships between host and guest show likeliness towards a linear relationship in few cases, while few cases show the opposite. However, it can be seen that the gaps observed between the scatter plots are due to the interval between experimental runs.

Chapter 5. Results and Analysis 43 Figure 5.7: Scatter plots showing the CPU load relationships between host and guest in varying load conditions - top tool. (a) Correlation between CPU loads on host and guest - 7vCPU - CentOS device 2.5 Correlation between CPU load of Host and Guest CentOS Host, top, stress ng on 7 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 1 2 3 4 5 6 7 8 Load Average on Guest 10% 20% 30% 40% 50% 100% (b) Correlation between CPU loads on host and guest - 7vCPU - Ubuntu device 2.5 Correlation between CPU load of Host and Guest Ubuntu Host, top, stress ng on 7 vcpu 2 Load Average on Host 1.5 1 0.5 0 0 1 2 3 4 5 6 7 8 Load Average on Guest 10% 20% 30% 40% 50% 100% 5.4 Discussions As we have seen the visual correlation analysis of the various experimental scenarios in the previous section, we present correlation co-efficient of the relationship values in this section to know the strength of the association. The strength of correlation is said to be large if the correlation co-efficient lies between 0.5 to 1

Chapter 5. Results and Analysis 44 for positive and -0.5 to -1 for negative correlations. The strength is medium if the co-efficient ranges between 0.3 to 0.5 and -0.3 to -0.5 for positve and negative correlations respectively. Tables 5.1 and 5.2 present the correlation co-efficient values obtained from uptime - CentOS and uptime - Ubuntu combinations respectively. Tables 5.3 and 5.4 show the correlation co-efficient values of top - CentOS and top - Ubuntu Combinations. Table 5.1: Stress-ng and uptime results on centos host 1vCPU 2vCPU 3vCPU 4vCPU 5vCPU 6vCPU 7vCPU 10% 0.763-0.461 0.738-0.562-0.459 0.867 0.408 20% 0.152-0.703 0.349-0.072 0.354 0.205-0.589 30% -0.411-0.490 0.865 0.568 0.851 0.451 0.612 40% -0.326-0.239 0.886 0.135 0.778 0.203 0.620 50% 0.722 0.391 0.025 0.826 0.561 0.557 0.842 100% 0.970 0.959 0.965 0.985 0.972 0.963 0.953 Table 5.2: Stress-ng and uptime results on Ubuntu host 1vCPU 2vCPU 3vCPU 4vCPU 5vCPU 6vCPU 7vCPU 10% 0.124-0.172-0.366-0.319-0.413-0.663-0.463 20% 0.540-0.549-0.504-0.048 0.638 0.835 0.594 30% 0.416-0.129-0.421-0.383 0.056 0.547 0.567 40% -0.181-0.150 0.357 0.760 0.135 0.589 0.445 50% 0.347 0.484 0.339 0.040 0.585 0.206 0.656 100% 0.973 0.987 0.981 0.998 0.989 0.952 0.969 Table 5.3: Stress-ng and top results on CentOS host 1vCPU 2vCPU 3vCPU 4vCPU 5vCPU 6vCPU 7vCPU 10% -0.129 0.401 0.603 0.477-0.488-0.097-0.551 20% 0.510 0.953 0.752-0.384 0.520 0.098 0.761 30% 0.252 0.704 0.535-0.253 0.167 0.622-0.300 40% 0.367-0.031 0.162 0.230 0.417 0.163-0.347 50% 0.131 0.754 0.839 0.891 0.140 0.303 0.186 100% 0.892 0.962 0.969 0.822 0.947 0.955 0.853 From these tables of correlation co-efficient values, we observe that when the system reaches 100% load and thereby becomes saturated, the strength of

Chapter 5. Results and Analysis 45 Table 5.4: Stress-ng and top results on Ubuntu host 1vCPU 2vCPU 3vCPU 4vCPU 5vCPU 6vCPU 7vCPU 10% 0.003-0.259 0.485-0.020 0.581 0.011-0.041 20% 0.157 0.026-0.107 0.381 0.474 0.170 0.032 30% -0.295-0.551 0.057 0.521 0.806 0.167 0.141 40% 0.241 0.207 0.737 0.509-0.052 0.871 0.491 50% 0.653 0.679 0.408 0.533-0.595 0.774 0.572 100% 0.943 0.742 0.979 0.963 0.996 0.972 0.993 association between CPU load of PM and VM is very high. The strength of this association under 50% load condition is between medium to large in most of the cases. Other lower load conditions showed weaker relationships. These differences observed can be either an inherent feature of the tools that needs to be investigated further or a measurement shortcoming that needs more iterations to have a more robust observation. Since this thesis concentrates on investigating methods to load relationships only, we were successful in accomplishing this task of our experimental research. These association strengths and more experimental measurements can lead us to predict the load conditions in a system accordingly. Furthermore, in an experimental research, validity threats have to be addressed to validate the experimentation. In this thesis, internal and external validity threats addressed. Internal validity is the extent to which one can be sure that no other variables except the one we study causes the result. To ensure that we meet this, we have thoroughly understood the setup and design and performed 10 iterations per experiment. External validity is the extent to which results of a study can be generalized to other environments and we have addressed this by performing experiments in multiple scenarios like VMs, vcpus, OS ensuring that it can be generalized.[58]

Chapter 6 Conclusions and Future Work 6.1 Conclusions The purpose of this research work is to study the CPU load behavior of host and guest machines under complex workload conditions. This investigation is based on experiments and measurements in various complex scenarios. We have performed numerous experiments that are rarely found elsewhere and tried to define a methodology for doing this. Initially, we have modeled the experiments in two scenarios OpenStack cloud at City Network and an on-premise system. This initial setup is a multiple guest scenario with 4 VMs running on each host. We observed that the results supported our hypothesis about OpenStack not having huge impact on system load when compared to a virtualization system. Convinced by these observations, following our spiral model (refer to section 3.2.1), we further performed the experiments in more complex scenarios in the virtualization system. Here, we observed CPU load values from both PM and VM. The application of these scenarios is not feasible in the cloud environment since the CSP does not have access to the VMs (refer to section 1.1) and biding to this, we have continued our complex configurations and experiments on virtualization system using different tools. Working towards the aim of finding relationship between host and guest CPU loads, we collected the results from both PM and VM and presented a visual correlation analysis of the results as scatter plots. We also applied bivariate analysis to study the strength of this association. Our results showed that the correlation between guest and host in terms of CPU load is stronger at higher load levels, especially near 100% load. Other load conditions showed weaker 46

Chapter 6. Conclusions and Future Work 47 correlation. We contribute to the understanding of the CPU load relationships between host and guest and defined methods for achieving it using standardized tools. The task here is to use the available simple tools in identifying these relationships but not the inherent mechanisms of these tools. The purpose of having several experimental scenarios is not to have a comparison of the tools or OS but to understand the load behavior. Understanding these load relationships can prove effective in load prediction and management in datacenters. 6.2 Future Work This research work focused on measurements and analysis under varying workload conditions using standard tools. There is potential scope in this area to perform these measurements while running real-time business critical applications using similar load collection tools. Also, working towards a load prediction model using similar approach and tools can be considered as future research interest. Apart from CPU load, memory and I/O are also the aspects considered in load management techniques, which can also be covered in our future work.

References [1] P. Mell and T. Grance, The NIST definition of cloud computing, 2011. [2] R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility, Future Gener. Comput. Syst., vol. 25, no. 6, pp. 599 616, 2009. [3] A. Lenk, M. Klems, J. Nimis, S. Tai, and T. Sandholm, What s inside the Cloud? An architectural map of the Cloud landscape, in Proceedings of the 2009 ICSE Workshop on Software Engineering Challenges of Cloud Computing, 2009, pp. 23 31. [4] Q. Zhang, L. Cheng, and R. Boutaba, Cloud computing: state-of-the-art and research challenges, J. Internet Serv. Appl., vol. 1, no. 1, pp. 7 18, 2010. [5] G. Katsaros, G. Gallizo, R. Kübert, T. Wang, J. O. Fitó, and D. Henriksson, A Multi-level Architecture for Collecting and Managing Monitoring Information in Cloud Environments., in CLOSER, 2011, pp. 232 239. [6] G. Vilutis, L. Daugirdas, R. Kavaliunas, K. Sutiene, and M. Vaidelys, Model of load balancing and scheduling in Cloud computing, in Information Technology Interfaces (ITI), Proceedings of the ITI 2012 34th International Conference on, 2012, pp. 117 122. [7] M. Gusev, S. Ristov, M. Simjanoska, and G. Velkoski, CPU Utilization while Scaling Resources in the Cloud, presented at the CLOUD COMPUTING 2013, The Fourth International Conference on Cloud Computing, GRIDs, and Virtualization, 2013, pp. 131 137. [8] A. Beloglazov and R. Buyya, Energy Efficient Allocation of Virtual Machines in Cloud Data Centers, in Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, Washington, DC, USA, 2010, pp. 577 578. 48

References 49 [9] A. Berl, E. Gelenbe, M. Di Girolamo, G. Giuliani, H. De Meer, M. Q. Dang, and K. Pentikousis, Energy-efficient cloud computing, Comput. J., vol. 53, no. 7, pp. 1045 1051, 2010. [10] B. Gregg, Systems Performance: Enterprise and the Cloud. Pearson Education, 2013. [11] http://www.cyberciti.biz/tips/how-do-i-find-out-linux-cpu-utilization.html. [12] Q. Huang, L. Ye, X. Liu, and X. Du, Auditing CPU Performance in Public Cloud, in 2013 IEEE Ninth World Congress on Services (SERVICES), 2013, pp. 286 289. [13] V. Ahmadi and K. Tutschku, An investigation of CPU utilization relationship between host and guests in cloud infrastructure. [14] S. Mour, P. Srivastava, P. Patel, H. Ram, B. N. Gohil, and D. Patel, Load management model for cloud computing, in Internet Technology and Secured Transactions (ICITST), 2014 9th International Conference for, 2014, pp. 178 184. [15] Z. Xiao, W. Song, and Q. Chen, Dynamic resource allocation using virtual machines for cloud computing environment, Parallel Distrib. Syst. IEEE Trans. On, vol. 24, no. 6, pp. 1107 1117, 2013. [16] G. Dodig-Crnkovic, Scientific methods in computer science, Proc. Conf. Promot. Res. IT New Univ. Univ. Coll. Swed. Skövde Suec., pp. 126 130, 2002. [17] P. Garraghan, P. Townend, and J. Xu, An analysis of the server characteristics and resource utilization in google cloud, in Cloud Engineering (IC2E), 2013 IEEE International Conference on, 2013, pp. 124 131. [18] O. Litvinski and A. Gherbi, Experimental evaluation of openstack compute scheduler, Procedia Comput. Sci., vol. 19, pp. 116 123, 2013. [19] G. Tong, H. Jin, X. Xie, W. Cao, and P. Yuan, Measuring and Analyzing CPU Overhead of Virtualization System, in Services Computing Conference (APSCC), 2011 IEEE Asia-Pacific, 2011, pp. 243 250. [20] A. Corradi, M. Fanelli, and L. Foschini, VM consolidation: A real case based on OpenStack Cloud, Future Gener. Comput. Syst., vol. 32, pp. 118 127, 2014. [21] M. Gusev, G. Velkoski, S. Ristov, and M. Simjanoska, Web service CPU overutilization in the cloud, in ICIT 2013 The 6th International Conference on Information Technology. Cloud Computing, 8-10 May 2013, 2013, pp. 459 66.

References 50 [22] B. Golden, Virtualization for dummies. John Wiley & Sons, 2011. [23] D. Marshall, Understanding Full Virtualization, Paravirtualization, and Hardware Assist. Copyright, 2007. [24] M. Portnoy, Virtualization essentials, vol. 19. John Wiley & Sons, 2012. [ [25] http://www.linux-kvm.org/page/main_page.. [26] Frequently Asked Questions, Google Cloud Platform. [Online]. Available: https://cloud.google.com/compute/docs/faq. [Accessed: 09-Sep-2015].. [27] KVM Performance Limits for virtual CPU cores, Open- Source Routing and Network Simulation. [Online]. Available: http://www.brianlinkletter.com/kvm-performance-limits-for-virtual-cpucores/. [Accessed: 09-Sep-2015]. [28] XIFI-D1.5-Federated Platform Architecture v2.pdf. Available at http://wiki.fi-xifi.eu/main.. [29] HP, Seven design principles for building a converged cloud. Retrieved 7January 2015. http://www8.hp.com/h20195/v2/getdocument.aspx?docname=4aa4-6524enw.. [30] City Cloud - Cloud Computing, City Network.. [31] T. Benson, A. Akella, A. Shaikh, and S. Sahu, CloudNaaS: A Cloud Networking Platform for Enterprise Applications, in Proceedings of the 2Nd ACM Symposium on Cloud Computing, New York, NY, USA, 2011, pp. 8:1 8:13. [32] Cloud-based networking, Wikipedia, the free encyclopedia. 05-Aug-2015. [33] OpenStack: The Open Source Cloud Operating System availiable at http://www.openstack.org/software/.. [34] Openstack installation guide availiable at http://docs.openstack.org/juno/install-guide/install/apt/content/ch_overview.html.. [35] Designing for Cloud Controllers and Cloud Management availiable at http://docs.openstack.org/openstackops/content/cloud_controller_design.html.. [36] Nova Concepts and Introduction availiable at http://docs.openstack.org/developer/nova/nova.concepts.html#novaconcepts-and-introduction..

References 51 [37] OpenStack Metering Using Ceilometer - Pure Play OpenStack.. [38] Openstack system architecture availiable at http://docs.openstack.org/developer/ceilometer/architecture.html.. [39] Billing for openstack availiable at http://docs.openstack.org/developer/ceilometer/glossary.h billing.. [40] http://docs.openstack.org/icehouse/install-guide/install/yum/content/basicsprerequisites.html.. [41] J. N. Amaral, About computing science research methodology, 2011. [42] B. W. Boehm, A spiral model of software development and enhancement, Computer, vol. 21, no. 5, pp. 61 72, 1988. [43] Bivariate analysis availiable at http://sociologyindex.com/bivariate_analysis.htm.. [44] E. Babbie, The practice of social research. Cengage Learning, 2015. [45] Physical Servers vs. Virtual Machines, Backup Academy. [Online]. Available: http://www.backupacademy.com/blog/physical-servers-vs-virtualmachines.html. [Accessed: 03-Oct-2015]. [46] Network congestion, Wikipedia, the free encyclopedia. 03-Sep-2015. [47] R. Walker, Examining Load Average, Linux J, vol. 2006, no. 152, p. 5, Dec. 2006. [48] M. Richard, Solaris" Performance And Tools: Dtrace And Mdb Techniques For Solaris 10 And Opensolaris. Pearson Education India, 2007. [49] T. Myer and J. Barnaby, Tenex Executive Manual. Bolt Beranek and Newman, Inc April, 1973. [50] uptime Linux man page availaible at http://linux.die.net/man/1/uptime.. [51] top Linux man page availaible at http://linux.die.net/man/1/top.. [52] https://exchange.nagios.org/directory/plugins/system-metrics/cpu- Usage-and-Load.. [53] http://people.seas.harvard.edu/ apw/stress/.. [54] http://manpages.ubuntu.com/manpages/lucid/man1/stress.1.html.. [55] http://manpages.ubuntu.com/manpages/utopic/man1/stress-ng.1.html..

References 52 [56] H. Cheng, Z. Chen, N. Sun, F. Chen, and M. Wang, Evaluation framework of virtualization systems for cloud computing, in 2012 IEEE Asia Pacific Cloud Computing Congress (APCloudCC), 2012. [57] High performance public OpenStack cloud, City Cloud. [Online]. Available: https://www.citycloud.com/. [Accessed: 03-Oct-2015]. [58] Steven Taylor and Gordon J.G. Asmundson, "Ineternal and External Validity In Clinical Research", [Online].Available: http://www.sagepub.com/upmdata/19352_chapter_3.pdf.