Comparison of Memory Balloon Controllers

Comparison of Memory Balloon Controllers Presented by: PNVS Ravali Advisor: Prof. Purushottam Kulkarni June 25, 2015 Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 1/34

Memory Overcommitment I Server Consolidation Give each VM only the memory it needs Memory Overcommit: Total system memory is less than the memory configured to the VMs Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 2/34

Memory Overcommitment II Techniques supporting Overcommitment Demand Paging: The hypervisor swaps pages in and out of the memory Virtual Machine Migration: Migration of VM to address the memory need Content Based Page Sharing Ballooning Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 3/34

Ballooning Ballooning - Memory reclamation mechanism Balloon driver: module inside the Guest OS VMM inflates and deflates to reclaim some pages from a VM Balloon Controller : Logic to operate the balloon driver (monitors memory need for a VM) Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 4/34

Balloon Controller When to balloon out? Amount of memory to balloon out Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 5/34

Motivation and Problem Description Motivation Many Balloon controllers - use different parameters for decision making Given a workload scenario, which controller to use? Problem Statement Evaluate the controllers with different workload settings Goals Controllers and set of workloads The behaviour of a controller with different workload setups Evaluate controller behaviour in a memory overcommit scenario Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 6/34

Outline Algorithms of balloon controllers Experiment Setup and Workloads Results in non-overcommit scenario Experiments with overcommit setup Conclusion Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 7/34

Balloon Controller Algorithms TWS based ballooning [Chiang] Estimates the True Working Set (TWS) size of the VM using measures like swapin, refault and Committed AS When Guest s physical memory allocation is greater than its WSS, the swap in and refault events are close to zero. FSM for tws controller Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 8/34

Modifications to tws controller - CAS based controllers mod-tws controller When in FAST or SLOW state, the reduction is by some % of current allocation When there is a change in CAS, increase or decrease the allocation by the difference in the amount of CAS Gives importance to swapin and refault new-mod-tws controller Decrease in CAS indicates decrease in memory allocation When there is swapin or refault, consider decrease in CAS as well Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 9/34

bbox controller Estimation of WSS [vmware-esx-server] Tracks VM s memory pages Uses sampling Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 10/34

bbox controller Algorithm 1 bbox controller logic 1: x numb of pages tracked 2: wss numb of pages accessed in sampled interval 3: rss numb of resident pages 4: if wss > 70%of x then 5: increment curr alloc by 20 % of curr alloc 6: else 7: next alloc = rss wss x 8: end if Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 11/34

RSS based controllers I rss-swap and new-rss-swap are the controllers which are based on RSS. Make use of swap and refault rss-swap controller Initial memory allocation is set to RSS If swapin or refault occurs, increase allocation by swapin and refault Else set allocation to RSS new-rss-swap controller Set memory allocation to sum of averages of swapin, refault and RSS observed in the decision interval period Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 12/34

Managing overcommit scenario I Controllers take per VM decisions Need for a global policy or controller which manages memory across the VMs Global controller policy Take reclaimable memory Host free memory is subtracted from total demand Take deficit in the ratio of the memory allocations Distribute the obtained memory in the ratio of their demands ratio = curr minthresh max alloc minthresh (1) Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 13/34

Example overcommit memory management Total memory at host is 1100 MB, min threshold is 100 MB Table : Example of an overcommit scenario VM1 VM2 VM3 VM4 curr alloc 250 250 300 300 max alloc 500 500 500 500 demand (controller) 100 200 200 100 max mem (reclaimed) 150 150 200 200 ratio 0.375 0.375 0.5 0.5 normalized ratio 0.21 0.21 0.29 0.29 reclaim 128.57 128.57 171.43 171.43 ratio of demands 0.17 0.33 0.33 0.17 give 100 200 200 100 final alloc 221.43 321.43 328.57 228.57 Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 14/34

Workloads used Workloads Kernel Compile: CPU intensive, disk io - Dacapo: Set of workloads, memory need fluctuates over time Memcached: in memory key-value store Sysbench OLTP: Database benchmark, read-only, read-write Metrics for Comparison Performance Comparison Memory Saved Index (msi): msi = max allocation possible curr alloc max allocation possible (2) Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 15/34

Analysis in non overcommit scenarios Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 16/34

Experiment Setup IBM blade server with 8 CPU cores with 24GM RAM with 3.10.52 Linux kernel Steps for the experiments Cleanup inside the VM Invoke resource utilization capture scripts inside the VM Start the server which sends the values of metrics as seen inside the VM to the client at the host Start the workload Stop the resource capture scripts inside the VM and save them once the workload finishes Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 17/34

Controllers performance vs memory savings Desirable portion is top right area Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 18/34

Per workload results Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 19/34

Kernel Compile Workload - Performance and Memory Savings Observations tws saves slightly more memory than mod-tws and new-mod-tws. CAS is less (around 250MB) and keeps fluctuating so tws ignores refault and swapin Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 20/34

Dacapo Workload - Performance and Memory Savings Observations CAS value varies between 500 and 700 MB and thus not much msi when compared to kernel compile tws saves more when compared to mod-tws, new-mod-tws as it follows CAS and ignores swap and refault Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 21/34

Sysbench read-only workload - Performance and Memory Savings Observations tws saving memory at the cost of performance degradation Minimal memory savings for mod-tws, new-mod-tws and rss-swap as this is io intensive and refaults result in high Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 22/34

Msi vs performance metrics (di=12 sec) tws m-tws nm-tws rss-sw n-rss-sw bbox wl p m p m p m p m p m p m kc x x dac x x x mc(1:9) mc(7:3) x x x sb ro x x x x x x sb rw x x x x x x Workloads and controllers - relative comparison indicates a performance which is atleast 75% of baseline indicates a memory savings of atleast 35% Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 23/34

Conclusions based on the results bbox: Does not perform well for any of the workload settings except memcached with a set to get ratio of 1:9 new-rss-swap: Shows results similar to those of the bbox. Points out the importance of refault tws : Performs well in all cases except sysbench readonly, sysbench read-write and mecached with a set to get ratio of 7:3.Might be an indication that tws controller is not suitable for workloads which exhibit swap or refault. The mod-tws, new-mod-tws and rss-swap: Close in terms of performance given or the memory savings for all the workload settings Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 24/34

Analysis in overcommitted scenario Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 25/34

Experiment Setup Workloads : kernel compile, dacapo, memcached, sysbench, dacapo2 Each VM is set to have a maximum memory allocation of 1GB Sysbench workload is set to have a read-only transaction load For memcached, the set to get ratio is set to 1:1 The experiment involves cleanup inside the VMs, starting all the workloads, moving the logs once they finish The overcommit ratio was varied from 1 to 3 to 6 with decision interval fixed at 12 seconds Overcommit ratio is defined as : sum of memory configured to V Ms total physical memory at host (3) Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 26/34

Table : Performance Values of workloads with ocr=3 relative to ocr=1 kc time dac time dac2 memc tput sysb tws 0.96 1.05 0.90 0.99 1.20 mod-tws 1.27 1.40 1.50 0.35 0.70 new-mod-tws 1.25 1.26 1.20 0.37 0.76 rss-swap 1.16 1.33 1.68 0.45 0.61 Table : Performance Values of workloads with ocr=6 relative to ocr=1 kc time dac time dac2 memc tput sysb tws 1.56 3.16 0.77 0.41 0.10 mod-tws 2.55 4.00 3.43 0.18 0.00 new-mod-tws 2.22 3.08 2.31 0.15 0.00 rss-swap 2.25 4.50 3.78 0.21 0.01 As the overcommit ratio increased, the performance of workloads deteriorated. Drastic performance change when ocr increased from 3 to 6 Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 27/34

Conclusions and Future Work Conclusions Evaluation of controllers in non-overcommit scenarios Overcommit scenario - more work to be done Extensions Analysis in overcommit scenario space - combinations of workloads, decision interval Make the black box controller swap aware Incorporate sharing and evaluate overcommit levels when ballooning and sharing co-exist Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 28/34

Thank You I would like to thank my advisor, Prof. Purushottam Kulkarni for his constant guidance and support throughout the project. I also would like to thank Debadatta Mishra for his support in the form of useful discussions related to work. Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 29/34

Bibliography I [1] Carl A.Waldspurger Memory Resource Management in VMware ESX Server, In proc. Fifth Symposium on Operating Systems Design and Implementation,OSDI 02 [2] Shashank Rachamalla, Debadatta Mishra, Purushottam Kulkarni Share-o-meter: An empirical analysis of KSM based memory sharing in virtualized systems, 20th Annual International Conference on High Performance Computing, HiPC 2013, Bengaluru (Bangalore), Karnataka, India, December 18-21, 2013 [3] Jui-Hao Chiang, Han-Lin Li, Tzi-cker Chiueh Working Set-based Physical Memory Ballooning, 10th International Conference on Autonomic Computing, ICAC 13 [4] Weiming Zhao, Zhenlin Wang, Yingwei Luo Dynamic memory balancing for virtual machines,. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), ACM Press, pages 21-30, March 11-13, 2009, Washington, DC, USA Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 30/34

Bibliography II [5] L. Chin-Hung Evaluating the effectiveness of memory overcommit techniques on kvm-based hosting, 2012 [6] Sailee Jain An empirical analysis of ballooning based memory management techniques, Technical report, Department of Computer Science, Indian Institute of Technology, Bombay, 2014 Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 31/34

Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 32/34

Poor performance from tws controller in case of sysbench read-only(di=12) Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 33/34

Good performance from mod-tws controller in case of sysbench read-only(di=12) Ravali, CSE, IIT Bombay M.Tech. Project Stage 2 34/34