Maximizing Virtual Machine Perfrmance An intrductin t perfrmance tuning Written by Mattias Sundling Quest Sftware, Inc. December 6, 2010 - Dcument versin 1.2- Whitepaper
Cntents Maximizing Virtual Machine Perfrmance... 3 Intrductin... 3 Requirements... 3 Virtual hardware and guest OS... 3 vcpu... 3 Memry... 5 Disk... 7 Netwrk... 9 Delete unnecessary devices frm yur virtual hardware and guest OS... 9 Acknwledgement... 10 Summary... 10 2
Maximizing Virtual Machine Perfrmance Intrductin VM perfrmance is ultimately determined by the underlying physical hardware and the hypervisr that serves as the fundatin fr yur virtual infrastructure. The cnstructin f this fundatin has becme simpler ver the years, but there are still several areas that shuld be fine-tuned in rder t maximize the VM perfrmance in yur envirnment. While sme f the cntent f this writing will be generic tward any hypervisr, this dcument fcuses n VMware ESX(i) 4.1. This is an intrductin t perfrmance tuning and nt intended t cver everything in detail. Mst tpics have links t sites that cntains deep-dive infrmatin if yu wish t learn mre. Requirements VMware ESX(i) 4.1 - If yu are running an lder versin, make sure t upgrade. Perfrmance and scalability have increased significantly since ESX(i) 3.x. ESX(i) 4.1 ffers sme imprvements ver ESX(i) 4.0 as well. Virtual machine hardware versin 7 this hardware versin intrduces features t increase perfrmance. If yu are nt running Virtual Hardware versin 7 make sure t upgrade VMware Tls first, then shutdwn the VM s guest OS. In the VI Client, right-click the VM and select Upgrade Virtual Hardware. Warning, nce yu upgrade Virtual Hardware versin t 7 yu will lse backward cmpatibility t ESX(i) 3.x, s if yu have a mixed envirnment make sure t upgrade all ESX(i) hsts first. Virtual hardware and guest OS The sectins belw make recmmendatins n hw t cnfigure the varius hardware cmpnents fr best perfrmance as well as what ptimizatins can be dne inside the guest OS. vcpu Start with 1 vcpu - mst applicatins wrks well with that. After sme time yu can evaluate CPU utilizatin and applicatin perfrmance. If the applicatin respnse is pr yu can add additinal vcpus as needed. If yu start with multiple vcpus and determine that yu have ver prvisined, it can be cumbersme t revert, depending f yur OS (see HAL). vfglight with Exchange cartridge lks beynd the hypervisr int the applicatin layer 3
Make sure yu select the crrect Hardware Abstractin Layer (HAL) in the guest Operating System. The HAL is the driver in the Operating System fr the CPU; chices are Uni-Prcessr (UP) single prcessr r Symmetric Multiprcessing (SMP) multiple prcessrs. Windws 2008 uses the same HAL fr bth UP and SMP, which makes it easy t dwngrade the number f CPUs. Windws 2003 and earlier have different HAL drivers fr UP versus SMP. Windws autmatically changes the HAL driver when ging frm UP t SMP. It can be very cmplicated t g frm SMP t UP, depending n the OS and versin. If yu have a VM running Windws 2003 SP2 r later which yu have dwngraded frm 2 vcpu t 1 vcpu yu will still have the multiprcessr HAL in the OS. This will result in slwer perfrmance than a system with crrect HAL. The HAL driver can be manually updated, hwever Windws versins prir t Windws 2003 SP2 cannt be easily crrected. I have persnally experienced systems with an incrrect HAL driver t cnsume mre CPU, which can ften peak t unnecessary high CPU utilizatin percentages nce the system gets stressed. Make sure yur multi-prcessr VMs have an OS and applicatin that supprt multi-threading and can take advantage f it. If nt, yu are wasting resurces. This example shws a VM with almst same CPU utilizatin acrss all vcpus. That means OS and applicatin are multi-threaded. CPU scheduling ESX 2 used strict c-scheduling which required a 2 vcpu VM t have 2 pcpu available at the same time. At this time physical CPUs had single r dual cre, which lead t slw perfrmance when hsting t many VMs. ESX(i) 3 intrduced relaxed cscheduling which allws a 2 vcpu VM t be scheduled even thugh that there are nt 2 pcpu available at the same time. ESX(i) 4 refines the relaxed c-scheduler even further, increasing perfrmance and scalability. 4
CPU % Ready The best indicatin that a VM is suffering frm CPU cngestin n an ESX(i) hst is when CPU % Ready reaches 5-10% ver time, in this range further analysis might be needed. Values higher than 10% is definitely shwing a critical cntentin. This means that the VM has t wait fr the ESX(i) hst t schedule its CPU requests due t CPU resurce cntentin with ther VMs. This perfrmance metric is ne f the mst imprtant nes t mnitr in rder t understand the verall perfrmance in a virtual envirnment. This can nly be seen in the hypervisr and as a result CPU utilizatin inside the guest OS might be very high. CPU % Ready is an imprtant metric t understand VM perfrmance Memry When yu create a VM yu allcate a certain amunt f memry t it. There is a feature in the virtual machine settings knwn as Memry Limit (which ften hurts mre than it helps). The functin f this setting is designed t limit the hypervisr memry allcatin t a value ther than what is actually assigned. This means that the guest OS will still see the full amunt f memry allcatin hwever the hypervisr will nly allw use f physical memry up t the amunt f the Memry Limit. The nly use case I have fund fr this is an applicatin that requires 16 GB memry (as an example) t install r start, but it nly uses 4 GB in peratin. Yu can create a Memry Limit at a much lwer value than the actual memry allcatin. The guest OS and applicatin will see the full 16 GB memry but the ESX(i) hst limit the physical memry t 4 GB. But in reality, the memry limit ften gets set n VMs that yu had n intentin t limit. This can happen when yu mve VMs acrss different resurce pls r perfrm a P2V f a physical system. This may als happen as the result f a knwn bug in vcenter which will randmly set a memry limit n a virtual machine, r wrst f all, in the templates previusly cnfigured which results in all deplyed VMs inheriting this setting. As further explanatin: if yu allcate 2 GB memry t a VM, and there is a limit at 512 MB, the guest OS will see 2 GB memry but the ESX(i) hst will nly allw 512 MB physical memry. If guest OS require mre than 512 MB memry, the memry balln driver will start t inflate t let guest OS decide what pages are actively being used. If balln can t reclaim any mre memry, guest OS will start t swap. If the balln can t deflate r if memry usage is t high n ESX(i) server it will start t use memry cmpressin and then VMkernel swapping as last resrt. Ballning is a first warning signal and guest OS / ESX(i) hst swapping will definitely negatively impact the VM perfrmance as well the ESX(i) hst and the strage subsystem that have t serve as virtual memry. Fr further explanatin see: http://www.vmguru.cm/index.php/articles-mainmenu- 62/mgmt-and-mnitring-mainmenu-68/96-memry-behavir-when-vm-limits-are-set vfglight: Detect, Diagnse and Reslve VM prblems. Memry limits will be detected by a rule, diagnse telling yu what is wrng and ptinally wrkflws t autmate the reslutin. 5
Memry sizing When cnfiguring the amunt f VM memry, cnsider the fllwing: T much memry will increase the VM memry verhead - yur VM density (number f VMs per hst) will nt be as high as it culd be. T little memry can result in guest OS swapping -yur perfrmance will be affected negatively. T determine the crrect amunt f memry yu need t mnitr active memry utilizatin ver at least 30-90 days t be able t see patterns. Sme systems might nly be used during a certain perid f the quarter, but used very heavily during that perid. Memry definitins: Granted: Physical memry being granted t VM by ESX(i) hst Active: Physical memry actively being used by VM. Ballned: Memry being used by the VMware Memry Cntrl Driver t allw VM OS t selectively swap memry. Swapped: Memry being swapped t disk. Fr a cmplete list f metrics and descriptins see: http://cmmunities.vmware.cm/dcs/doc-5600 Memry utilizatin (Active Memry) in this example is very lw ver time, which makes it safe t decrease memry setting withut affecting VM and applicatin perfrmance. Memry reclamatin It is a best practice t right-size memry allcatin in rder t avid placing extra lad n ESX(i) hsts due t memry reclamatin. There are several techniques that ESX(i) hst uses t reclaim VM memry. After all, yu want t run as many VMs as pssible and will prbably ver-cmmit memry (allcate mre than yu have). Ballning*: Reclaiming memry by increasing memry pressure inside the VM this requires VMware Tls. D nt disable ballning, as it will negatively impact perfrmance. If yu experience a lt f ballning, try t vmtin the VM t anther hst, as it will allcate all memry back t VM. Als, make sure yu dn t have a fixed memry limit cnfigured n the VM. Swapping*: Reclaiming memry by having ESX(i) hst swap ut VM memry t disk. Memry cmpressin*: Reclaiming memry by cmpressing pages befre they are swapped ut t disk. Transparent Page Sharing: Reclaiming memry by remving redundant pages with same cntent (in the same VM r acrss VMs). 10% f VM memry allcatin can be used as cmpressin cache. * Only active when ESX(i) hst is experiencing memry cntentin. Fr mre details: http://www.vmware.cm/files/pdf/techpaper/vsp_41_perf_memry_mgmt.pdf http://frankdenneman.nl/2010/06/memry-reclaimatin-when-and-hw/ 6
Disk Nw, lets mve n t the mst cmplex building blck f the fundatin, the disk cnfiguratin: ParaVirtualized SCSI (PVSCSI) cntrller PVSCSI prvides better thrughput and lwer CPU utilizatin. Studies have shwn, that with PVSCSI implemented, a 12 % increase in thrughput and 18 % decrease f CPU utilizatin in cmparisn t the LSI Lgic based cntrller. http://blgs.vmware.cm/perfrmance/2009/05/350000-i-peratins-per-secndne-vsphere-hst-with-30-efds.html VMware benchmarking PVSCSI versus LSI Lgic: http://www.vmware.cm/pdf/vsp_4_pvscsi_perf.pdf ESX(i) 4.1 has imprved PVSCSI t be able t handle lw disk I/O where lder ESX (i) versins had prblem with queuing that culd result in latency. http://vpivt.cm/2010/02/04/pvscsi-and-lw-i-wrklads Fr mre infrmatin n hw t cnfigure PVSCSI: http://kb.vmware.cm/selfservice/micrsites/search.d?language=en_us&cmd=displaykc&externalid=1010398 Separate OS, Swap and Data disks int separate VMDK files. This will help perfrmance and Data Prtectin (primarily by excluding the swap data) Cnsider creating a separate virtual disk cntrller fr each disk. This will allw higher disk I/O than single cntrller. Pwer ff VM and change SCSI ID 1:0, 2:0 and s n and yu will get additinal cntrllers. LUN sizing and VM placement Create the datastres with crrect size (500-1000GB). LUNs that are t big will result in t many VMs, SCSI reservatin cnflicts, and ptentially lwer disk I/O caused by metadata lcking (i.e.vmtin, VM pwer n, snapsht). vstrage API fr Array Integratin (VAAI) is a new API in ESX(i) 4.1 that takes sme f the heavy lifting frm the hypervisr and mves it t the strage hardware. If yur hardware supprts it, yu will be able t run bigger datastres withut perfrmance prblems. It als helps lwer metadata lcking mentined abve. Fr mre details: http://www.yellw-bricks.cm/2010/11/23/vstrage-apis-fr-array-integratin-aka-vaai/ Use a 8MB blck size when creating the datastres as it has n negative impact n perfrmance and it can hld larger VMDK files. It is required that yu have the same blck size n all datastres if yu want t leverage VAAI. Mnitr the LUN perfrmance and identify any latency as quickly as pssible t ensure that disk I/O is streamlined acrss all LUNs. 7
vfglight Strage: Mnitring paths, thrughput and latency is imprtant VMFS and guest OS Alignment If yu create new VMFS vlumes frm the vcenter client, the vlumes will be aligned crrectly fr yu. If yu created the VMFS vlumes during ESX(i) installatin yur vlumes will be unaligned. The nly way t fix this is t Strage vmtin all VMs in the affected datastre t a new datastre and then recreate it frm the vcenter client. Windws 2008, 7 and Vista aligns NTFS vlumes by default. All prir Windws server OS misalign the disks. Yu can nly align the disk when yu create it. Mst Linux distributins have this misalignment tendency as well. On average, prperly aligning the disks can increase perfrmance by 12% and decrease latency by 10%. Fr mre infrmatin see: http://www.vmware.cm/pdf/esx3_partitin_align.pdf Quest voptimizer Pr can detect and reslve alignment prblems n existing disks fr Windws and Linux. T learn mre: http://www.quest.cm/vptimizer%2dpr/ Strage I/O Cntrl (SIOC) Frm ESX(i) 4.1, SIOC can be enabled n a per-datastre basis. This can be helpful if yu are cncerned that sme missin critical VMs are nt getting the required disk I/O during times f disk cngestin. Yu can cnfigure disk shares per VM. If there is disk cngestin, the VMs with higher disk shares (shares will be used nly when there s cntentin) have the pririty fr mre disk I/O this wrks the same way as memry shares. 8
Netwrk Physical netwrk Make sure yu have multiple redundant physical NICs at 1 Gbit/s r 10 Gbit/s speeds cnnected t VM virtual netwrk switches. VMXNET3 The netwrk driver in the guest OS can be updated frm default E1000 t VMXNET3 (paravirtualized netwrk driver, same enhancements as paravirtualized strage described abve and it can leverage 10Gbit/s netwrk speeds). Cautin! IP address will reset t DHCP and a new MAC address will be generated. Make sure t capture yur ld settings. Ipcnfig /all >c:\ip.txt (will capture yur settings int ip.txt fr Windws). Optinally, the ptin t enable jumb frames n the netwrk can help maximize the packets that traverse the envirnment. Set MTU t 9000 in guest OS driver, vswitch, and physical netwrk prts (end t end). Yur netwrk infrastructure must als supprt jumb frames. ESX(i) 4.1 supprt Fault Tlerance n VMXNET3 guest OS. Fr mre detailed infrmatin and perfrmance test see: http://www.vmware.cm/pdf/vsp_4_vmxnet3_perf.pdf, http://kb.vmware.cm/selfservice/micrsites/search.d?language=en_us&cmd=displaykc&externalid=1001805 Netwrk I/O Cntrl (NetIOC) NetIOC allws yu t cntrl the netwrk bandwidth utilized by vmtin, NFS, iscsi, Fault Tlerence, VMs and Management. This can be dne by cnfiguring share r limits and allws yu t cntrl Quality f Service making sure critical cmpnents always get the netwrk bandwidth required. Delete unnecessary devices frm yur virtual hardware and guest OS Unnecessary devices in the virtual hardware and inside the guest OS will take unnecessary CPU and memry resurces t emulate. If yu dn t use them make sure t delete them. Cleaning up inside the guest OS will nt gain s much in perfrmance; it s mre f a husekeeping thing. Flppy, CD, USB, Serial Prt, Cm Prt, Sund Less devices means less verhead n yur VM Cleanup f deleted hardware in the OS as well Fr Windws At the cmd prmpt, type: Set devmgr_shw_nnpresent_devices=1 Start Device Manager (devmgmt.msc) In Device Manager Shw hidden devices Delete all nn present devices 9
Acknwledgement Thanks t my clleagues at Quest Sftware: Tmmy Pattersn, Chris Walker, Paul Martin, Thmas Bryant and Sctt Herld fr reviewing and giving valuable feedback. A special thanks t VMware Trainer and Blgger: Eric Slf at ntpr.nl fr additinal review and finding sme errrs. Summary Tune the fundatin and yu will better utilize the infrastructure. Once the building blcks (CPU, Memry, Disk and Netwrk) are ptimized then the true perfrmance is ultimately determined. 10