Scaling in a Hypervisor Environment Richard McDougall Chief Performance Architect VMware
VMware ESX Hypervisor Architecture Guest Monitor Guest TCP/IP Monitor (BT, HW, PV) File System CPU is controlled by scheduler and virtualized by monitor Monitor supports: BT (Binary Translation) HW (Hardware assist) PV (Paravirtualization) VMkernel Scheduler Memory Allocator Virtual NIC Virtual Switch NIC Drivers Virtual SCSI File System I/O Drivers Memory is allocated by the VMkernel and virtualized by the monitor Physical Hardware Network and I/O devices are emulated and proxied though native device drivers
Evolution of Performance for Large Apps on ESX 00% Mission Critical Apps ESX.x VI 3.0 VI 3.5 vsphere 4.0 General Population Of Apps Overhead:-5% Overhead:30-60% Overhead:0-40% Overhead:0-30% VCPUs:8 VCPUs: VCPUs: VCPUs:4 VM RAM:55GB VM RAM:3.6 GB VM RAM:6 GB VM RAM:64GB Phys RAM: TB Phys RAM:64GB Phys RAM:64GB Phys RAM:56GB PCPUs:64 core PCPUs:6 core PCPUs:6 core PCPUs:64 core IOPS:350,000 IOPS:<0,000 IOPS:0,000 IOPS:00,000 N/W:0 Gb/s N/W:380 Mb/s N/W:800 Mb/s N/W:9 Gb/s 64-bit OS Support Monitor Type: Gen- 64-bit OS Support 30 VMs per host Binary Translation HW Virtualization Gen- HW 5 vcpus per host Monitor Type: Virtualization Monitor Type: EPT VT / SVM Monitor Type: NPT Ability to satisfy Performance Demands
Databases: Why Use VMs Rather than DB Virtualization? Virtualization at hypervisor level provides the best abstraction Each DBA has their own hardened, isolated, managed sandbox Strong Isolation Security Performance/Resources Configuration Fault Isolation Scalable Performance Low-overhead virtual Database performance Efficiently Stack Databases per-host
Large Oracle Transaction Workload: SwingBench Database: # of Users = 5,0,87 # of Products = 5,0,87 Db size = 5.GB Test Test duration = 0 mins # of Users per run = 30 Think time = 0 New customer - % Browse products - 8% Order products - 8% Process orders - 5% Browse orders - 8% Oracle g (..0) RHEL4 x86_64 SGA: 3GB PGA: GB RHEL5 U 64 bit.6.8-53..3.el5 Dell PowerEdge 950 Mem: 8GB Two dual core) Intel(R) Xeon(R) CPU 560 @ 3.00GHz processor 4GB cache Storage: CX 3-40 (30 disks) ESX version: 3.5 build# 607 Number of VMs: vcpu: 4 Mem: 6GB vdisk: 6GB vnic:
Measuring the Performance of DB Virtualization Throughput Delivered Minimal Overheads
Transaction Rate (Ratio to -way VM) Single VM Performance: Well-Known Database OLTP Workload Next Generation Intel Xeon based 8-pCPU server RHEL 5. Oracle gr In-house ESX Server < 5% overhead for 8 vcpu VM 8,900 total DB transactions per second Near-perfect scalability from to 8 vcpus 60,000 I/O operations/second A fair-use implementation of the TPC-C workload; results are not TPC-C compliant
Single VM Handles Most Demanding Applications ;) = Sun Fire 5k (ca. 00)
The world changed @ 3Ghz: Multi-Core Focus Scaling by clock frequency no-longer possible Memory Latency Dominates Performance now achieved by implementing multiple CPU cores
Moore s Law: Direct Correlation to Number of Cores Billion transistors and growing 8080 8008 4004 86 8086 Pentium 4 Pentium III Pentium II Pentium 486 DX 386 4 Million 970 980 990 000 00,000,000,000 00,000,000 0,000,000,000,000 00,000 0,000,000
Estimated Logical CPU Counts: -Socket Server (Assuming hardware threading is a given) 000 00 0 Sun UltraSPARC T Sun UltraSPARC T AMD Opteron DualCore x86 Threading Nehalem Intel Clovertown AMD Barcelona 990 995 000 005 00 03 x86 RISC
Number of Cores used VMware vsphere enables you to use all those cores 000 Oracle VMWare ESX Scaling: Keeping up with core counts 00 SQL Server Exchange Virtualization provides a means to exploit the hardware s increasing parallelism 0 Avg. Cust, Application Avg. Four Socket Web Servers 990 995 000 005 00 05 Year Most applications don t scale beyond 4/8 way
Large Database Consolidation Study 45000 40000 Aggregate TPM vs. Number of VMs 00 90 Scaling to 6 Cores, 56GB RAM! 35000 80 30000 70 5000 60 Aggregate TPM 50 CPU Utilization 0000 40 5000 30 0000 0 5000 0 0 0 3 4 5 6 7 # of VMs
Aggregate Response Time (secs) Oracle Performance (Response time) Average response time vs. Number of VMs 0.0 0.8 0.6 00 90 80 Average response time CPU Utilization 0.4 70 0. 60 0.0 50 0.08 40 0.06 30 0.04 0 0.0 0 0.00 3 4 5 6 7 0 # of VMs
Scaling of Virtual Web vs Physical
MS Exchange VM Scaling Single Host Physical Throughput ~8000 Mailboxes (50% of Virtual)
POWER SUPPLY POWER CAP A D FANS 3G 4B 5E PROC POWER SUPPLY 6H 7C 8F POWER SUPPLY POWER CAP A D FANS 3G 4B PROC 9i ONLINE SPARE 9i MIRROR 8F 5E 7C OVER TEMP POWER SUPPLY 6H INTER LOCK 6H 5E 7C 8F DIMMS 4B 3G 9i ONLINE SPARE D PROC 9i MIRROR A 3 4 5 6 8F 7C OVER TEMP INTER LOCK 6H 5E DIMMS 4B 3G D PROC A 3 4 5 6 3 4 3 4 5 6 7 8 5 6 7 8 PLAYER PLAYER HP ProLiant DL380G6 HP ProLiant DL380G6 POWER SUPPLY POWER CAP A D FANS 3G 4B POWER SUPPLY POWER CAP A 5E PROC D FANS POWER SUPPLY 6H 7C 3G 4B 8F 5E PROC 9i ONLINE SPARE POWER SUPPLY 6H 9i MIRROR 7C 8F 8F 7C 9i ONLINE SPARE OVER TEMP 9i MIRROR INTER LOCK 6H 5E 8F DIMMS 4B 7C 3G PROC 3 4 5 6 OVER TEMP INTER LOCK 6H D 5E A DIMMS 4B 3G D PROC A 3 4 5 6 3 4 3 4 5 6 7 8 5 6 7 8 PLAYER PLAYER HP ProLiant DL380G6 HP ProLiant DL380G6 Example migration scenario 4_4_0_0 with DRS vcenter Imbalanced Balanced Cluster Heavy Load Lighter Load