The Cost of Reconfiguration in a Cloud

Transcription

1 The Cost of Reconfiguration in a Cloud Akshat Verma IBM Research - India akshatverma@in.ibm.com Gautam Kumar Ricardo Koller IIT Kharagpur Florida International University gautamkumar.iit@gmail.com rkoll001@cs.fiu.edu ABSTRACT Emerging clouds promise enterprises the ability to increase or decrease their resource allocation on demand using virtual machine resizing and migration. These dynamic reconfiguration actions lead to performance impact during the reconfiguration duration. In this paper, we study the cost of reconfiguring a cloud-based IT infrastructure in response to workload variations. We observe that live migration requires a significant amount of spare CPU on the source server (but not on the target server). If spare CPU is not available, it impacts both the duration of migration and the performance of the application being migrated. Further, the amount of CPU required for live migration varies with the active memory of the VM being migrated. Finally, we show that live migration may impact any co-located VMs based on the cache usage pattern of the co-located VM. We distill all our observations to present a list of practical recommendations to cloud providers for minimizing the impact of reconfiguration during dynamic resource allocation. 1. INTRODUCTION Cloud computing has emerged as one of the most promising compute platforms for the future [3, 4, 9]. Clouds have a distinct advantage over traditional data centers in providing elasticity and achieving higher resource utilization. Elasticity from a customer s perspective is the ability to increase or decrease the amount of resources it wants to reserve for itself. Elasticity from a providers perspective is the ability to seamlessly move resources from one customer to another in response to variation in demand, thus allowing the cloud to operate at high resource utilization. Virtualization is the key technology that enables both elasticity and high resource utilization in clouds. To achieve the twin objectives, clouds host diverse applications as virtual machines on a shared physical server to achieve higher resource utilization. Further, the resources assigned to any set of applications hosted on the cloud or even the complete cloud can shrink/expand based on workload intensity. The virtu- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Middleware 10 Companion Bangalore, India Copyright 2010 ACM X-XXXXX-XX-X/XX/XX...$5.00. alization layer provides the required isolation layer between applications running on the same physical server. Frequent reconfiguration or dynamic resource allocation from a shared pool is one of the defining attributes of a cloud. Moving resources from one customer to another or from a shared pool is achieved using either (i) Dynamic Virtual Machine (VM) resizing and VM Live Migration or (ii) Creating new VM instances. Dynamic VM resizing allows the resources assigned to a VM to be changed on the fly and Live Migration allows a VM to move to a different physical server if enough resources are not available on the server it is currently hosted on. Live migration also allows a cloud provider to consolidate VMs to a small number of physical servers during periods of low load, thus achieving power savings. Increasing or decreasing the number of VMs for a customer also provides the same functionality, albeit at a coarser granularity. Such horizontal scaling is only applicable to clustered applications with a gateway that distributes requests to nodes in the cluster. Further, horizontal scaling incurs additional cost due to increase in the number of software licenses. Finally, data center management cost is typically directly proportional to the number of VMs and horizontal scaling leads to an increased labour cost. As a consequence, VM resizing and Live Migration have been preferred as a tool for dynamic consolidation in virtualized data centers [1, 5, 11, 16, 18]. Hence, in this study, we focus on the combination of Live VM Resizing and Live Migration technologies, which provide a more flexible and cost-effective alternative for dynamic resource allocation or reconfiguration in virtualized data centers or clouds. In this paper, we use the terms virtualized data center and clouds interchangeably to denote data centers that support dynamic resource allocation for virtual machines. 1.1 Motivation Dynamic resource management in a typical cloud setting consists of periodic reconfiguration in a cloud (Fig. 1). A Monitoring Engine collects the resource usage for various applications in the cloud, which is used by a Prediction Engine to estimate the resource demand for the next reconfiguration period. A Placement Manager uses the demand to come up with a reconfiguration of the cloud. The Placement Manager uses models for performance, power and reconfiguration to come up with a reconfiguration that minimizes the total cost. There has been a lot of work on Placement Manager, Performance Modeling and Power Modeling [7, 11, 13, 15, 16, 17, 18]. In this work, we focus on Reconfiguration Cost Modeler that estimates the reconfiguration cost for a

2 Performance Modeler Reconfiguration Cost Modeler Power Modeler reconfiguration impact due to consolidation. Monitoring Engine Prediction Engine Workload Demand Managed Cloud Environment Placement Manager Virtualization Manager Figure 1: Reconfiguration Cost Models in Cloud Resource Management specific consolidation period. Migration Time (seconds) 25 bt is ua sp lu daxpy Benchmark Figure 2: Variation in Live Migration time Dynamic resource reconfiguration using VM Resizing and Live Migration comes with associated costs. The cost of live migration is significant and needs to be factored during dynamic resource allocation [15, 6, 1, 5]. Existing approaches for dynamic consolidation assume the cost to be proportional to active memory [15] or use a application-specific model that is oblivious to other co-located VMs and server utilization [6]. We observe that the duration of a live migration for an application running identical workload can vary by % or more depending on the server utilization and other co-located VMs (Fig. 2). Thus it is necessary to take these factors into account while administering dynamic resource allocation. 1.2 Contribution In this work, we address the following questions: (i) What are the important factors that determine the cost of reconfiguration? (i) How can we reduce the impact of reconfiguration? We conduct a detailed experimental study and identify the parameters that influence the cost of migrating a VM. We show that reconfiguration cost depends heavily on the application type and can not be captured solely by high level system parameters. We show that live migration requires spare CPU on the source server but not on the target server. We also show that this CPU overhead varies with the number of dirty pages of the VM being migrated. Our most interesting observation is that the impact of a colocated VM is the same as the impact of reduced resources on the server. Finally, we use our experimental studies to come up with a list of practical recommendations that may be employed by administrators of virtualized data centers or clouds. Our recommendations can be used to refine existing dynamic consolidation methodologies and minimize the 2. BACKGROUND We now present a background of the mechanisms for reconfiguration in a cloud. 2.1 Reconfiguration Mechanisms in cloud The reconfiguration mechanisms considered in this paper are (i) Dynamic VM resizing or Dynamic Logical Partition (DLPAR) resizing (ii) Live VM Migration. We provide a background of both these actions next. Modern hypervisors allow fairly low overhead implicit and explicit VM resizing. A VM is defined with a minimum resource reservation min, a maximum capacity allowed max and a share to capture its priority for shared resources. The range between the min and max provides implicit VM resizing in a hypervisor. When a VM is idle, its allocation is given to any other VM that may require resources (within its max). This allows the physical machine to provide resource to any virtual machine that sees an increase in workload as long as the server has spare capacity. Explicit VM resizing is performed by changing the entitlements (min or max or both) for each VM. Hypervisors now support DL- PAR resizing, which allows the entitlements to be changed on a running VM. Our experiments on DLPAR resizing for on both IBM Power6 platform with phyp hypervisor and IBM HS21 BladeServer with VMWare ESX 3 showed that the duration for VM resizing was less than 1 second. Further, for applications as diverse as Rubis and NAS Parallel Benchmark, there is no perceptible performance impact during VM resizing. Hence, we focus only on the impact of live migration for this study. Live VM migration is a very powerful mechanism allowing workloads to be spread around a larger number of machines during high load and to be consolidated to fewer servers during periods of low load. The most important aspect in terms of the performance impact of a live migration activity is the copying of in-memory state from the source hypervisor to the target hypervisor [12, 2, 10]. The copying of in-memory state consists of the following phases: 1. Pre-Copy Phase: The applications keep running in this phase. The phase works in rounds. In the first round, all the active pages in memory are copied to the target server. In any subsequent round, all pages that were made dirty in the previous round are copied. The Pre-Copy Phase typically terminates when either the number of dirty pages are small (less than some constant C small ) or the decrease in the number of dirty pages between two subsequent iterations is small, i.e. no progress (less than C noprog). 2. Stop and Copy Phase: The application is stopped in this phase and all the remaining dirty pages are copied to the target server to complete migration. The Stop and Copy Phase is small for typical applications, usually less than 1 second. It is the Pre-Copy Phase that is much longer [12, 2] and increases with the size of memory being copied. In this work, we try to characterize the duration and impact of this phase. 2.2 Model Parameters

3 The goal of this work is to characterize the impact of a reconfiguration activity. We use the following parameters to determine the performance impact of migrating a VM V M i. 1. Duration (τ(i)): The time taken to complete the live migration. 2. Self-Impact (π s(i)): Ratio between the drop in throughput during migration and the throughput without migration of the application on V M i. 3. Co-Impact (π c(j)): Ratio between the drop in throughput for a co-located VM V M j during migration and the throughput of V M j without migration The first parameter determines the duration for which the performance impact of the reconfiguration would be observed. The second and third parameters capture the quantitative impact during the migration. 3. IMPACT OF VIRTUAL MACHINE LIVE MIGRATION We conducted a large number of experiments to obtain insights that could be used to predict the cost of live migration actions. We first describe the experimental setup used for our study. 3.1 Experimental Setup Our experimental testbed consisted of a small virtualized server farm and a SAN environment. Our server farm consisted of 7 IBM HS 21 Bladecenter servers hosted on an IBM Bladecenter-H Chassis. To simulate a cloud infrastructure, all the servers ran VMWare ESX Server Enterprise 3.5 with VMotion enabled. One network port for each server was dedicated as a vkernel port for VMotion. The other port on the server was used for communicating with a client that was used to drive workloads on our testbed. Every server had a dedicated 2GBps Fibre Channel port, which was connected to an IBM DS4800 Storage Controller via a Cisco MDS Fiber Channel Switch. All the virtualization management actions (VM resizing, VM migration) were handled via the VMWare Virtual Center 2.5 hosted on a separate dedicated server. We used the daxpy and dcopy benchmarks from the BLAS- 1 library [14] for our experiments. daxpy is a compute intensive benchmark that uses most of the compute components whereas dcopy is a purely memory copy benchmark. The benchmarks allowed us to vary the size of the vector used during execution, the number of iterations, and allowed us to control the throughput by sleeping between iterations. All the experiments were repeated 8 times and the means are reported. Hypervisors from different vendors have their own strengths and weaknesses based on their current implementation. Our goal is to come up with a model that captures the inherent design of live migration and is oblivious of certain weaknesses of the current implementation, which may be easily sorted out in future. Hence, we also experimented with a parallel data center using the phyp hypervisor on IBM Power6 JS-22 blades and validated our observations. 3.2 Observations We divided our data center into two clusters. The first cluster (Cluster 1) had servers running 1 3.2GHz Xeon processor with 2MB cache. The second cluster (Cluster 2) had servers with 2 3.2GHz Xeon processors. Servers from both clusters had 8GB RAM each. We also ensured that our virtual machines always required less than 3.2GHz to run their applications. Hence, the second cluster captures server farms where there is spare CPU capacity available for any migration related overheads whereas the first cluster captures server farms running at high utilization. Migration Time e+06 Dcopy_10_2 Dcopy_20_2 Daxpy_10_2 Daxpy_2000_2 Daxpy_20_2 Figure 3: Impact of Active Memory on Duration of Migration with spare CPU In our first experiment, we investigate the linear relationship between active memory and duration of migration. We increase the memory footprint of both daxpy and dcopy benchmarks and migrate them from the high capacity cluster 2. Further, we change their throughput so that they use from 10MHz to 20MHz and observe their duration of migration (Fig. 3). We note that there is indeed a linear relationship between memory actively used and the duration of migration for a fixed application. However, two applications with the same active memory may have different migration duration. Further, we note that the throughput of an application does not impact migration duration (at constant memory footprint). Since the write rate and dirty rate vary with throughput, the above observation implies that write rate and dirty rate do not have a significant impact on the duration of migration. The minimal impact of dirty rate on the duration of migration implies that the expensive logging of memory traces is not required in clouds. We capture the results of this experiment in the following observation. Observation 1. If there are no resource constraints, the duration of migration for an application varies linearly with the active memory of the VM. The migration duration varies across applications with same memory footprint. The impact of dirty rate can be ignored without significant loss in accuracy. In our next experiment, we investigate if CPU utilization does play a role in determining the impact of migration. We run the daxpy benchmark at different CPU and active memory utilization on Cluster 1 (low capacity cluster) and observe the duration of migration in Fig. 4(a). It is evident that as the application reaches 2000MHz, the migration thread gets throttled. Further, we note that the impact of migration may reach as high as %. One may also observe that the migration duration increases at very low CPU allocation of the VM being migrated. The current implementation of VMWare ESX 3.5 requires some resources from the VM s quota to carry out the migration. However, on the Power6 platform using phyp, we did not see any impact of lower VM entitlement as long as the server had spare resources. We conclude that increase in the duration of migration for a VM with a small resource entitlement (even when the server

4 Duration of Migration Normalized Job Duration MB 80MB 0MB CPU Allocated (a) Cluster1 (No Migration) Cluster2 (No Migration) Cluster1 (Migration to 2) Cluster2 (Migration to 1) 0.9 (b) Figure 4: (a) Impact of CPU on Migration Duration. (b) Performance Impact of source and target server spare CPU has spare resources) is an artifact of a specific implementation and not a design constraint. Hence, we ignore this impact in our work. Our previous experiment established the need for incorporating CPU utilization in a model to estimate the impact of live migration. The migration of a VM involves two servers, a source server and a target server. We need to understand if high CPU utilization at source or target server impacts migration. Hence, in our next experiment, we migrate the daxpy benchmark at CPU utilization of 20MHz from high capacity cluster 2 to low capacity cluster 1 and vice versa. We log the time taken to run the benchmark with and without migration and plot the self-impact of migration π s(i) in Fig. 4(b). We note that migration from cluster 2 to cluster 1 leads to no performance impact whereas migration from cluster 1 to cluster 2 leads to an increase in running time upto %. This behaviour is a direct consequence of the fact that the source hypervisor needs to maintain the list of active pages, which requires processing cycles. On the other hand, the target hypervisor only needs to receive memory pages sent by the source server, requiring minimal CPU resources. The high CPU need for live migration and the contrasting impact of source and target server CPU utilization is captured in our next observation. Observation 2. Live migration requires spare CPU resources on the source server but not on the target server. If spare CPU is not available, it impacts the duration of migration and the performance of the VM being migrated. Our early experiments hint that the amount of CPU required for live migration is not a constant and varies based on the application footprint. Since the hypervisor needs to maintain a dirty bit for each active page, the amount of CPU required by the hypervisor for live migration may be related to the number of active pages used by the VM. We confirm this intuition by estimating the self-impact of migration for the daxpy benchmark at different values of dirty pages and CPU utilization (Fig. 5(a)). We note that for a CPU allo- Migration Time Throughput Drop MHz 2000MHz 10MHz (a) Dcopy_10_1 Dcopy_20_1 Daxpy_10_1 Daxpy_2000_1 Daxpy_20_1 Daxpy_20_ e+06 (b) Figure 5: Impact of dirty memory on (a) CPU required for Migration and (b) duration of migration cation of 20MHz, the application incurs performance impact for the entire range of active memory. Further, the performance impact increases with an increase in active memory. The run with CPU allocation of 2000MHz is more revealing as it shows no performance impact at low memory footprint and significant performance impact at high memory footprint. The same observation is confirmed for a CPU allocation of 10MHz as well. The impact of memory footprint on CPU requirement exhibits itself in the duration of migration as well. The CPU bottleneck throttles the migration activity as well, leading to an increase in Migration Time Vs Active Memory slope in Fig. 5(b). One may observe that the slope is lower for the high capacity cluster 2 (daxpy 20 2) than for cluster 1 (daxpy 20 1) at 20MHz. One may also note that the increase in migration slope for daxpy 10 and daxpy 2000 coincides with the first time non-zero performance impact is observed in Fig. 5(a). The above experiment confirms our intuition and leads to the following observation. Observation 3. The amount of CPU required for live migration increases with an increase in the number of active pages of the VM being migrated. Migration Duration 80 Migration (VM2 = 0) Migration (VM2 = 0) Migration (VM2 = 800) Figure 6: Impact of Secondary VM on Duration of Migration We next investigate the impact of a secondary VM on the performance of a VM being migrated and on the migration process. We run the daxpy benchmark on a background V M V M2 with a memory footprint of MB and vary its CPU allocation. The foreground VM has a CPU allocation of 1200MHz. The memory footprint of the foreground VM

5 VM1 Throughput Drop VM2 Duration VM2 = 0 VM2 = 0 VM2 = 800 VM2=0 (baseline) VM2=0 (Migration) VM2=800 (baseline) VM2=800 (Migration) 85 Figure 7: Impact of Secondary VM on Performance is varied and it is migrated from the low capacity cluster 1. The duration of migration is plotted in Fig. 6. The throughput drop of the foreground VM and the running time of the background VM are plotted in Fig. 7. We observe that the secondary VM has a significant performance impact on both the foreground VM as well as the migration thread. On the other hand, the secondary VM seems to be unaffected by the migration of the foreground VM. Further, there is no co-impact even at increased server utilization. The above observation can be explained by the fact that the secondary VM gets its entitlement independent of the resource contention triggered by the migration activity. The migration activity takes resources from the free resource pool and in case of contention, takes resource from the quota of the VM being migrated. Hence, the impact of a secondary VM is the same as operating the foreground VM on a lower capacity server. We capture these insights in the following observation. Observation 4. A co-located VM impacts a VM being migrated by taking away resources from the physical server. The co-located VM does not suffer from CPU contention but may suffer from cache contention based on its cache usage pattern. Our experimental study highlights the importance of incorporating CPU utilization of the source server in any model for live migration. Further, our study indicates that the impact of write rate WR i and dirty rate DR i on migration is fairly small and can be ignored. We also confirm the observation made earlier that the model should take the active memory of the application into account. Our most important observation is that the impact of migration varies based on the application. The daxpy benchmark has more instructions per memory access and lower CPI than dcopy, leading to higher processor activity. We excluded these parameters from our study but observed that he duration of migration and performance impact for these two benchmarks differ for the same active memory, CPU utilization and write/dirty rate (Fig. 3). We assess the implications of our insights for reconfiguration actions in the next section. 4. IMPLICATIONS FOR DYNAMIC RESO- URCE ALLOCATION IN CLOUDS We now leverage our observations to come up with practical recommendations for cloud or virtualized data center administrators during dynamic resource allocation. Our recommendations can also be used to enhance the vast body of dynamic consolidation methodologies, allowing them to minimize costs with minimum reconfiguration overheads. 1. Scale up needs to be pro-active as opposed to reactive: Scaling up or down is often associated with live migration of a VM. Further, scale up is often associated with high source server and low target server utilization. Combining this with the fact that live migration requires significant CPU resources (Observation 2), we conclude that scale up in reaction to high CPU utilization may lead to prolonged reconfiguration, while significantly impacting the performance of the application being migrated. Further, for the (extended) duration of live migration, all the co-located VMs would also incurr the co-impact of migration. Hence, a scale up action be performed based on a predicted resource bottleneck in advance. If an accurate prediction is not possible, a lower resource threshold should be set for triggering scale up to accommodate for resource bottlenecks during migration. 2. Scale down can be either proactive or reactive: Using the same observations as scale up, we conclude that scale down is often associated with a low source server utilization and high target server utilization. Since target server utilization has no impact on the duration or performance impact of live migration, scale down can be performed in a reactive fashion as well. 3. Prefer applications with small active memory for migration: We note that the duration of a migration does not depend only on the size of the active memory. However, Observation 1 concludes that as long as there is no resource contention, the duration of migration is directly proportional to the size of active memory. Further, as noted in Observation 3, the amount of CPU required for live migration also increases with the number of active memory pages. Hence, if a provider has a choice between migrating two applications of equal priority, (s)he should prefer applications with a smaller active memory. 4. Prefer low priority applications for migration: Drawing from Observation 4, we know that a co-located VM is not impacted by CPU contention. However, the VM being migrated has to incur both CPU contention as well as cache contention due to migration traffic. Hence, given a choice between a low priority and high priority application, one should migrate lower priority application to shield the high priority application from CPU contention. 5. While migrating low priority applications colocated on a server with high priority application, resize the VM to a smaller size: We know from the design of live migration techniques that migrating a VM with low resource entitlement would not impact the duration of migration. On the other hand, resizing a VM to a smaller size would free up more spare CPU capacity for the migration thread. Hence, the duration of migration would be reduced by resizing a low priority VM that is migrated. This would also reduce the length of time during which any co-located high priority applications suffer from co-impact π c of migration. 6. Live Migration is not suitable to handle hotspots: Hotspots are short-term overloads on servers. Since a hotspot

6 by definition is temporary and associated with high source server utilization, live migration may take an inordinately long amount of time and significantly impact the performance of the VM being migrated. Further, since the hotspot is temporary, the hotspot may disappear by the time the reconfiguration is complete. Hence, live migration may not be beneficial for handling hotspots. 5. RELATED WORK AND DISCUSSION Cloud computing has emerged as an exciting new paradigm for enterprises to achieve high resource utilization. This paradigm is based on efficient and dynamic allocation of resources between applications, facilitated by the underlying virtualization layer. Hence, dynamic consolidation of a virtualized server farm has attracted a lot of attention in the recent past. Dynamic resource consolidation techniques use either only VM resizing [13] or VM resizing in combination with live migration [16, 15, 1, 11, 5]. However, we are not aware of any study that investigates the impact of various dynamic allocation mechanisms. Anecdotal evidence as well as the recent findings in [6] identify live migration as the dynamic resource allocation mechanism with significant performance impact. Dynamic consolidation techniques, following this conventional wisdom, aim to minimize the number of migrations [1, 5, 15]. Live migration technology is available on most popular virtualization platforms including VMWare ESX, Xen, and IBM phyp [2, 12, 10]. The power of live migration as a system tool for performance management has encouragement development of live migration on non-virtualized servers as well [8]. Other than dynamic consolidation, live migration is often used for server or application maintenance as well as to eliminate performance hotspots [19]. Designers of the technology do provide empirical evidence to suggest that the performance impact of live migration is manageable. However, to our best knowledge, there is no systematic study of the parameters that impact the duration and performance impact of live migration. Further, there is no study on the frequency of reconfiguration actions in a typical virtualized server farm that employs dynamic allocation of resources to maximize resource utilization. Existing work in this area takes a very simplistic view of live migration and attaches a constant cost for migrating any VM [1, 5, 19]. Verma et al. [15] take this model a step further by linking migration cost with the active memory of the VM. In [6], Jung et al. note that live migration can significantly impact both foreground and background applications. However, none of the existing work presents a rigorous study that details out system parameters that impact applications during live migration. Our work presents this missing piece for the elaborate work on dynamic consolidation by providing an estimate on the duration and the performance impact of reconfiguration actions. We also conduct a detailed study of the performance impact of live migration on the VM being migrated as well as on co-located VMs. We distill our observations to present a list of recommendations that can be directly employed by virtualized data center administrators as well as cloud providers to minimize the impact of reconfiguration during dynamic resource allocation. Autonomous learning for efficient resource utilization of dynamic vm migration. In Proc. ACM ICS [2] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In Proc. Usenix NSDI [3] Amazon Elastic Compute Cloud. [4] Google App Engine. [5] D. Gmach, J. Rolia, L. Cherkasova, G. Belrose, T. Turicchi, and A. Kemper. An integrated approach to resource pool management: Policies, efficiency and quality metrics. In Proc. DSN, [6] G. Jung, K. Joshi, M. Hiltunen, R.Schlichting, and Calton Pu. A cost-sensitive adaptation engine for server consolidation of multitier applications. In Proc. IFIP/ACM/Usenix Middleware [7] R. Koller, A. Verma, and A. Neogi. Wattapp: An application aware power meter for shared data centers. In ICAC, [8] M. Kozuch, M. Kaminsky, and M. Ryan. Migration without virtualization. In HotOS, [9] LotusLive. [10] G. McLaughlin, L. Liu, D. DeGroff, and K. Fleck. Ibm power systems platform: Advancements in state of the art in it availability. In IBM Systems Journal, [11] Ripal Nathuji and Karsten Schwan. Virtualpower: coordinated power management in virtualized enterprise systems. In Proc. ACM SOSP, [12] M. Nelson, B-H. Lim, and G. Hutchins. Fast transparent migration for virtual machines. In Proc. Usenix ATC [13] J. Stoess, C. Lang, and F. Bellosa. Energy management for hypervisor-based virtual machines. In Proc. Usenix ATC, [14] BLAS (Basic Linear Algebra Subprograms). [15] A. Verma, P. Ahuja, and A. Neogi. pmapper: Power and migration cost aware application placement in virtualized systems. In Proc. Middleware, [16] A. Verma, P. Ahuja, and A. Neogi. Power-aware dynamic placement of hpc applications. In Proc. ACM ICS, [17] A. Verma, G. Dasgupta, T. Nayak, P. De, and R. Kothari. Server workload analysis for power minimization using consolidation. In Proc. Usenix ATC, [18] A. Verma, P. De, V. Mann, T. Nayak, A. Purohit, and R. Kothari. Brownmap: Enforcing power budget in shared data centers. In ACM/IFIP/Usenix Middleware, [19] T. Wood, P. Shenoy, A. Venkataramani, and M. Yousif. Black-box and gray-box strategies for virtual machine migration. In Proc. NSDI REFERENCES [1] H. W. Choi, H. Kwak, A. Sohn, and K. Chung.