Increasing XenServer s VM density

Transcription

1 Increasing XenServer s VM density Jonathan Davies, XenServer System Performance Lead XenServer Engineering, Citrix Cambridge, UK 24 Oct 2013 Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

2 Outline 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

3 Outline Scalability expectations 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

4 Scalability expectations Users expect VM density to scale with hardware Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

5 Scalability expectations Users expect VM density to scale with hardware Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

6 Scalability expectations Users get upset when... Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

7 Scalability expectations XenServer s VM density scalability hard density limit :-( hardware's theoretical capacity XS 6.1 (and earlier) Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

8 Scalability expectations XenServer s VM density scalability hard density limit :-( hardware's theoretical capacity XS 6.1 (and earlier) XS 6.2 practical density limit (depending on nature of VMs) hard density limit :-) Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

9 Outline Hard limits 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

10 Hard limits Enumerated causes of limitations Hard limit 1: dom0 event channels Cause of limitation XenServer uses a 32-bit dom0 This means 1,024 dom0 event channels #define MAX_EVTCHNS(d) \ (BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d)) Various VM functions use a dom0 event channel VM density hard limit Mitigation for XS 6.2 Mitigation for future 225 VMs per host (PV with 1 vcpu, 1 VIF, 1 VBD) 150 VMs per host (HVM with 1 vcpu, 1 VIF, 3 VBDs) Hack for dom0 to enjoy 4,096 event channels 800 VMs per host (PV with 1 vcpu, 1 VIF, 1 VBD) 570 VMs per host (HVM with 1 vcpu, 1 VIF, 3 VBDs) Change the ABI to provide unlimited event channels this would remove the limit Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

11 Hard limits Enumerated causes of limitations Hard limit 2: blktap2 device minor numbers Cause of limitation blktap2 only supports up to 1,024 minor numbers (despite the kernel allowing up to 1,048,576) #define MAX_BLKTAP_DEVICE 1024 Each virtual block device requires one device VM density hard limit 341 VMs per host (with 3 disks per VM) Mitigation for XS 6.2 Double this constant to 2, VMs per host (with 3 disks per VM) Mitigation for future Move away from blktap2 altogether? this would remove the limit Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

12 Hard limits Enumerated causes of limitations Hard limit 3: number of aio requests Cause of limitation VM density hard limit Each blktap2 instance creates an asynchronous I/O context for receiving 402 events. Default system-wide number of aio requests was 444,416 in XS VMs per host (with 3 disks per VM) Mitigation for XS 6.2 Set fs.aio-max-nr to 1,048, VMs per host (with 3 disks per VM) Mitigation for future Increase fs.aio-max-nr further or use storage driver domains this would remove the limit Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

13 Hard limits Enumerated causes of limitations Hard limit 4: dom0 grant references Cause of limitation VM density hard limit Mitigation for XS 6.2 Windows VMs use receive-side copy (RSC) by default in XS 6.1. netback allocates (at least) 22 grant-table entries per virtual interface for RSC. dom0 had a total of 8,192 grant-table entries in XS VMs per host (with 1 interface per VM) Don t use RSC in Windows VMs anyway this removes the limit Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

14 Hard limits Enumerated causes of limitations Hard limit 5: connections to xenstored Cause of limitation xenstored uses select(2), which can only listen on 1,024 file descriptors. #define FD_SETSIZE 1024 qemu opens 3 file descriptors to xenstored. VM density hard limit Mitigation for XS 6.2 Mitigation for future 333 VMs per host (HVM) Make two qemu watches share a connection 500 VMs per host (HVM) Upstream qemu doesn t connect to xenstored this will remove the limit Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

15 Hard limits Enumerated causes of limitations Hard limit 6: connections to consoled Cause of limitation VM density hard limit Mitigation for XS 6.2 Similarly, consoled uses select(2) Each PV domain opens 3 fds to consoled 341 VMs per host (PV) Use poll(2) rather than select(2) in consoled this removes the limit Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

16 Hard limits Hard limit 7: dom0 low memory Enumerated causes of limitations Cause of limitation VM density hard limit Mitigation for future Each running VM eats about 1MB of dom0 lowmem around 650 VMs per host Use a 64-bit dom0 this will remove the limit Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

17 Summary of hard limits Hard limits Summary of hard limits Limits on number of HVM guests with 1 vcpu, 1 VBD, 1 VIF (with PV drivers) Limitation XS 6.1 XS 6.2 Future dom0 event channels no limit blktap minor numbers no limit aio requests no limit dom0 grant references 372 no limit no limit xenstored connections no limit consoled connections no limit no limit no limit dom0 low memory no limit Overall limit very high Limited by event channels xenstored something else! Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

18 Summary of hard limits Hard limits Summary of hard limits Limits on number of HVM guests with 1 vcpu, 3 VBDs, 1 VIF (with PV drivers) Limitation XS 6.1 XS 6.2 Future dom0 event channels no limit blktap minor numbers no limit aio requests no limit dom0 grant references 372 no limit no limit xenstored connections no limit consoled connections no limit no limit no limit dom0 low memory no limit Overall limit very high Limited by event channels xenstored something else! Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

19 Summary of hard limits Hard limits Summary of hard limits Limits on number of PV guests with 1 vcpu, 1 VBD, 1 VIF Limitation XS 6.1 XS 6.2 Future dom0 event channels no limit blktap minor numbers no limit aio requests no limit dom0 grant references no limit no limit no limit xenstored connections no limit no limit no limit consoled connections 341 no limit no limit dom0 low memory no limit Overall limit very high Limited by event channels dom0 lowmem something else! Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

20 Hard limits 500 Windows VMs on a host Summary of hard limits Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

21 Outline Soft limits 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

22 Soft limits xenstored High dom0 CPU utilisation by xenstored top - 16:29:33 up 36 min, 1 user, load average: 0.80, 0.56, 0.47 Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie Cpu(s): 40.1%us, 40.0%sy, 0.0%ni, 17.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: k total, k used, k free, 23696k buffers Swap: k total, 0k used, k free, k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7339 root S :10.22 xenstored 6665 root S :04.03 fe 7225 root S :03.35 xcp-rrdd 7269 root S :03.47 xcp-rrdd-iostat 7413 root m 21m 8932 S :10.28 xapi 7283 root S :08.65 xcp-rrdd-xenpm root S :00.40 v6d root R :02.31 top 1 root S :00.83 init 2 root S :00.00 kthreadd 3 root RT S :00.01 migration/0 Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

23 Soft limits xenstored High dom0 CPU utilisation by xenstored dom0 vcpus domu vcpus Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

24 Soft limits xenstored High dom0 CPU utilisation by xenstored xenstored's dom0 vcpu other dom0 vcpus domu vcpus Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

25 Soft limits xenstored High dom0 CPU utilisation by xenstored Cause of limitation Mitigation for XS 6.2 xenstored CPU utilisation bottleneck Reduce xenstore use by XenServer s toolstack: remove some spurious writes replace polling with watching Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

26 Soft limits High dom0 CPU utilisation due to qemu qemu top - 16:40:27 up 2:07, 1 user, load average: 89.62, 87.22, Tasks: 1015 total, 65 running, 950 sleeping, 0 stopped, 0 zombie Cpu(s): 23.4%us, 55.5%sy, 0.0%ni, 4.8%id, 0.0%wa, 0.0%hi, 15.4%si, 0.5%st Mem: k total, k used, k free, 3804k buffers Swap: k total, 0k used, k free, k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7143 root R :21.63 rpciod/ root R :14.34 ovs-vswitchd tcpdump R :35.12 tcpdump root R :00.11 top S :56.30 qemu-dm S :27.34 qemu-dm R :28.65 qemu-dm S :26.97 qemu-dm S :53.19 qemu-dm S :28.86 qemu-dm R :51.71 qemu-dm R :31.18 qemu-dm S :27.97 qemu-dm root S :12.92 nfsiod S :58.46 qemu-dm S :31.33 qemu-dm S :56.82 qemu-dm Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

27 Soft limits qemu burning dom0 CPU qemu 200 idle Windows guests, each qemu utilising 3% of a CPU means 6 dom0 vcpus wasted! Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

28 Soft limits What is qemu busy doing? qemu Emulated device qemu events per VM per second USB 221 CD-ROM 38 Buffered I/O & RTC timer 13 Parallel port 1 Serial port 1 VNC 1 qemu monitor 1 Mitigation for XS 6.2 Use an event-channel for buffered I/O notifications Provide options to disable all emulated devices Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

29 Outline Benchmarks 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

30 Booting 90 Win7 VMs Benchmarks Bootstorm 2500 Tampa (XS 6.1) Clearwater (XS 6.2) Time to fully boot 90 VMs (25 at a time) 2000 Elapsed time (s) XS 6.2 is 60% faster VM index Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 2.1GHz) Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

31 Booting 120 Win7 VMs Benchmarks Bootstorm 2500 Tampa (XS 6.1) Clearwater (XS 6.2) Time to fully boot 120 VMs (25 at a time) 2000 Elapsed time (s) XS 6.2 is 75% faster VM index Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 2.1GHz) Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

32 Booting 200 Win7 VMs Benchmarks Bootstorm Tampa (XS 6.1) Clearwater (XS 6.2) Time to fully boot 200 VMs (25 at a time) XS 6.1 can't even get 200 VMs running! Elapsed time (s) It took XS 6.2 just 13 minutes to boot 200 VMs (on this hardware) VM index Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 2.1GHz) Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

33 Benchmarks LoginVSI LoginVSI: number of usable Windows VMs number of VMs performing acceptably XS 6.1 XS 6.2 number of VMs running Machine used: Quad 10-core Intel 2.27GHz Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34

34 Questions Questions? Jonathan Davies (Citrix) Increasing XenServer s VM density 24 Oct / 34