Monitoring a Virtualized Database Server. Thomas LaRock Senior DBA, Confio Software

Transcription

1 Monitoring a Virtualized Database Server Thomas LaRock Senior DBA, Confio Software 1

2 Who Am

3 I Wrote A Book!

4 4 What Is Everyone Talking About?

7 7 Enter virtualization!

10 Agenda We will focus on VMWare Why? Would you optimize for Yahoo! Search? Or focus on Google instead? Database monitoring for VM Resource bottlenecks Scenarios Wrap-up 10

11 Takeaways DB monitoring 11

12 Takeaways DB monitoring FAIL 12

13 Takeaways DB monitoring vsphere FAIL 13

14 Takeaways DB monitoring vsphere FAIL FAIL 14

15 Takeaways DB monitoring vsphere How to save time FAIL FAIL 15

16 Takeaways DB monitoring vsphere How to save time FAIL FAIL WIN 16

17 17 VMWare Host

18 VMWare Common Terms ESX/ESXi the hypervisor and foundation for VMware products Physical Host underlying hardware where ESX is installed 18

19 VMWare Common Terms ESX/ESXi the hypervisor and foundation for VMware products Physical Host underlying hardware where ESX is installed Virtual Machine (VM) container inside host that looks like a physical machine 19

20 VMWare Common Terms ESX/ESXi the hypervisor and foundation for VMware products Physical Host underlying hardware where ESX is installed Virtual Machine (VM) container inside host that looks like a physical machine vcenter Server centralized management 20

21 VMWare Common Terms ESX/ESXi the hypervisor and foundation for VMware products Physical Host underlying hardware where ESX is installed Virtual Machine (VM) container inside host that looks like a physical machine vcenter Server centralized management vsphere Client Admin and Monitoring 21

22 VMWare Administration 22

23 Database Monitoring Challenges Standard tools include: Perfmon SQL Server DMVs 23

24 Database Monitoring Challenges Standard tools include: Perfmon SQL Server DMVs Waits and queues are vital 24

25 Database Monitoring Challenges Standard tools include: Perfmon SQL Server DMVs Waits and queues are vital Need to examine hosts vsphere, vcenter 25

26 Database Monitoring Challenges Standard tools include: Perfmon SQL Server DMVs Waits and queues are vital Need to examine hosts vsphere, vcenter Dirty, filthy, liar named Task manager 26

27 27 Why should I use vsphere?

28 vsphere Challenges TMI 100s of counters no indication of importance 28

29 vsphere Challenges TMI 100s of counters no indication of importance Not enough detailed data 29

30 vsphere Challenges TMI 100s of counters no indication of importance Not enough detailed data Keeps details only for a day by default rolls to hourly 30

31 vsphere Challenges TMI 100s of counters no indication of importance Not enough detailed data Keeps details only for a day by default rolls to hourly GUI performance can be slow at times 31

32 vsphere Challenges TMI 100s of counters no indication of importance Not enough detailed data Keeps details only for a day by default rolls to hourly GUI performance can be slow at times Graphs are isolated; can only see one type of chart at a time 32

33 vsphere Challenges TMI 100s of counters no indication of importance Not enough detailed data Keeps details only for a day by default rolls to hourly GUI performance can be slow at times Graphs are isolated; can only see one type of chart at a time Hard to mix Memory, CPU, Storage, etc 33

34 Stop Wasting Time! I hate wasting time! 34

35 Stop Wasting Time! I hate wasting time! Need to quickly eliminate possible root cause CPU, memory, disk I/O, network 35

36 Stop Wasting Time! I hate wasting time! Need to quickly eliminate possible root cause CPU, memory, disk I/O, network I want to investigate VM layer for issues 36

37 Stop Wasting Time! I hate wasting time! Need to quickly eliminate possible root cause CPU, memory, disk I/O, network I want to investigate VM layer for issues I want to eliminate host/guest as culprit BEFORE trying to tune a query! 37

38 CPU Configuration Concepts vcpu Reservation - amount of CPU guaranteed Limit limits the amount of CPU Shares sets priority for this VM 38

39 CPU Configuration Concepts vcpu Reservation - amount of CPU guaranteed Limit limits the amount of CPU Shares sets priority for this VM Know how many vcpus you need? Probably not, so start with 1 and go up as needed Remember vcpu * GHz formula 39

40 CPU Monitoring Databases are not typically CPU bound 40

41 CPU Monitoring Databases are not typically CPU bound BMFF metric VM Ready Time Similar to runnable queue Amount of time VM was ready to run, but needed to wait for CPU resource 41

42 CPU Monitoring Databases are not typically CPU bound BMFF metric VM Ready Time Similar to runnable queue Amount of time VM was ready to run, but needed to wait for CPU resource Other metrics include: VM CPU utilization Host CPU utilization 42

43 Is There a (CPU) Problem? Where does it hurt? DB performance is slow, for no apparent reason Host server has high CPU utilization 43

44 Is There a (CPU) Problem? Where does it hurt? DB performance is slow, for no apparent reason Host server has high CPU utilization Possible causes CPU over committed Additional VM guests (planned or unplanned) 44

45 Is There a (CPU) Problem? Where does it hurt? DB performance is slow, for no apparent reason Host server has high CPU utilization Possible causes CPU over committed Additional VM guests (planned or unplanned) Potential remedies Move VM guests back to where they belong Reconfigure vcpu settings Purchase more CPUs 45

46 CPU Scenario Measure host CPU usage AVG > 75% -or- Peak > 90% 46

47 CPU Scenario Measure host CPU usage AVG > 75% -or- Peak > 90% Check guest VM ready time If any vcpu > 2000ms (or 10%) 47

48 CPU Scenario Measure host CPU usage AVG > 75% -or- Peak > 90% Check guest VM ready time If any vcpu > 2000ms (or 10%) Host CPU saturation exists Balance guest resources Get more CPUs for the host 48

49 Memory Configuration Concepts RAM assigned to VM guest Reservation Amount of RAM guaranteed Limits Limits the amount of RAM for the VM Shares Sets the priority for the VM Ballooning unused memory that was given back for use on other VMs Swapping memory (could be active) given back forcibly for use on other VMs Shared memory identical memory pages are shared among VMs 49

50 Memory Configuration Concepts RAM assigned to VM guest Reservation Amount of RAM guaranteed Limits Limits the amount of RAM for the VM Shares Sets the priority for the VM Ballooning unused memory that was given back for use on other VMs Swapping memory (could be active) given back forcibly for use on other VMs Shared memory identical memory pages are shared among VMs Set Memory Reservation >= DB Memory 50

51 Memory Monitoring Databases are often memory bound Often due to bad configurations 51

52 Memory Monitoring Databases are often memory bound Often due to bad configurations BMFF Swapping, Ballooning If you got the memory, you want to keep it 52

53 Memory Monitoring Databases are often memory bound Often due to bad configurations BMFF Swapping, Ballooning If you got the memory, you want to keep it Other Metrics include VM (and Host) Memory Utilization VM Memory Reservation VM Memory Limit 53

54 Is There a (Memory) Problem? Where does it hurt? Low memory counters (buffer/proc cache, PLE) Host ballooning > 0 54

55 Is There a (Memory) Problem? Where does it hurt? Low memory counters (buffer/proc cache, PLE) Host ballooning > 0 Possible causes Host/VM/SQL memory misconfigured 55

56 Is There a (Memory) Problem? Where does it hurt? Low memory counters (buffer/proc cache, PLE) Host ballooning > 0 Possible causes Host/VM/SQL memory misconfigured Potential remedies Set VM memory reservation = memory provisioned Do not allow over commit of CPU Verify vcpus are not idle 56

57 Memory Scenario Measure Available Mbytes Is it < 100, or < 10% (which do you prefer?) 57

58 Memory Scenario Measure Available Mbytes Is it < 100, or < 10% (which do you prefer?) Check VM Ballooning Is it > 0? Doesn t matter, set min/max to non-default values 58

59 Memory Scenario Measure Available Mbytes Is it < 100, or < 10% (which do you prefer?) Check VM Ballooning Is it > 0? Doesn t matter, set min/max to non-default values Assume min/max is set, check for lock pages in memory granted to service account If not, then DO IT! 59

60 Disk Configuration Concepts You are sharing your storage! VMFS VMWare File System Datastore access point to storage Storage issues are usually related to configuration and not capabilities of ESX Follow best practices from storage vendor Create dedicated datastores for databases Isolate data and log files 60

61 Disk Monitoring BMFF Latency Host maxtotallatency Host Device Latency (by device) VM Command Latency (for all VMs) 61

62 Disk Monitoring BMFF Latency Host maxtotallatency Host Device Latency (by device) VM Command Latency (for all VMs) Other metrics include Disk Read Rate Disk Write Rate 62

63 Is There a (Disk) Problem? Where does it hurt? High latencies 63

64 Is There a (Disk) Problem? Where does it hurt? High latencies Possible causes Overloaded storage Overloaded network I/O intensive queries Memory/CPU issues 64

65 Is There a (Disk) Problem? Where does it hurt? High latencies Possible causes Overloaded storage Overloaded network I/O intensive queries Memory/CPU issues Potential remedies Reconfigure storage and/or network Rewrite queries 65

66 Disk Scenario Monitor disk latency Is Host maxtotallatency > 30ms? 66

67 Disk Scenario Monitor disk latency Is Host maxtotallatency > 30ms? Is VM Command Latency >= 30ms for your VM? Look for PAGEIOLATCH_XX waits Tune Disk I/O intensive processes on database Are Memory / CPU issues causing I/O problems 67

68 Disk Scenario Monitor disk latency Is Host maxtotallatency > 30ms? Is VM Command Latency >= 30ms for your VM? Look for PAGEIOLATCH_XX waits Tune Disk I/O intensive processes on database Are Memory / CPU issues causing I/O problems Review device latency Review Disk Read/Write rates for that device 68

69 Network Concepts How big is that pipe? vswitch software switch inside of VMKernel vswitch can be tied to 1 or more NICs VMWare claims to handle over 30GB/sec Most db s don t need that much 69

70 Network Concepts How big is that pipe? vswitch software switch inside of VMKernel vswitch can be tied to 1 or more NICs VMWare claims to handle over 30GB/sec Most db s don t need that much VMXNET paravirtualized network adapter Installed into guest O/S capable of 1Gbps Minimizes overhead between VM and Host Requires VMware Tools 70

71 Network Monitoring BMFF Dropped Packets (Received, Transmitted) 71

72 Network Monitoring BMFF Dropped Packets (Received, Transmitted) Other Metrics Include Network Rate 72

73 Network Monitoring BMFF Dropped Packets (Received, Transmitted) Other Metrics Include Network Rate Look for dropped packets Any errors on host NICs? One NIC getting more traffic than another? 73

74 Network Monitoring BMFF Dropped Packets (Received, Transmitted) Other Metrics Include Network Rate Look for dropped packets Any errors on host NICs? One NIC getting more traffic than another? Is Network Rate is getting close to max? Find which VM is causing the load 74

75 Is There a (Network) Problem? Where does it hurt? Network performance is slow or intermittent 75

76 Is There a (Network) Problem? Where does it hurt? Network performance is slow or intermittent Possible causes Overloaded host CPU Single NIC Wrong network adapter drivers 76

77 Is There a (Network) Problem? Where does it hurt? Network performance is slow or intermittent Possible causes Overloaded host CPU Single NIC Wrong network adapter drivers Potential remedies Reduce CPU pressure on host NIC teaming More info at 77

78 Network Scenario Monitor droppedtx, droppedrx for all vmnic objects Are they > 0? 78

79 Network Scenario Monitor droppedtx, droppedrx for all vmnic objects Are they > 0? Check for overloaded host CPU Add additional vcpu Add additional vnic 79

80 Network Scenario Monitor droppedtx, droppedrx for all vmnic objects Are they > 0? Check for overloaded host CPU Add additional vcpu Add additional vnic Check VM driver configuration for network devices Bad config options lead to bad network perf 80

81 Quick BMFF Sheet Resource Metric Host / VM Description CPU Ready VM CPU time spent in ready state Usage Both CPU usage as a percentage during a defined interval Memory Swapin, Swapout Both Memory the host swaps in/out from/to disk (per VM, or cumulative over host) Vmmemctl Both Amount of memory reclaimed from resource pool by way of ballooning Disk maxtotallatency Host Highest latency value across all disks used by the host. devicelatency Host Average time to complete a command from the physical device. totallatency Host Average latency in all guests. Network droppedtx, dropped Rx Both Drop packets per second 81 usage Both Sum of data transmitted and received

82 82 Questions?

83 For More Information