Virtualization management tools Course materials may not be reproduced in whole or in part without the prior written permission of IBM. 5.4
Unit objectives After completing this unit, you should be able to: Describe standard AIX/virtualization monitoring tools Identify freeware monitoring tools Describe virtualization management and monitoring tools such as IBM Systems Director IBM Tivoli Monitoring
Utilization data management (1 of 3) Displays events related to resource utilization or state changes Managed system must be configured for data to be sampled Change the setting for utilization data collection Use chlparutil to change the sampling rate for the managed system at the command line lslparutil can be used at the command line to view/gather data
Utilization data management (2 of 3) Sampling event example (logical partition utilization) LPARs utilization percentage Sampling event example (physical processor pool utilization) Physical pool utilization percentage
Utilization data management (3 of 3) Shared processor pool Shared Pool 01 utilization percentage Shared memory pool utilization Shared memory pool
Normal monitoring tools (1 of 5) iostat, vmstat, sar, topas, and so on Use Performance Utilization Resource Register (PURR) Use the Scaled PURR (SPURR) if the hardware supports it Support POWER6 features topas cross partition monitor output topas -C (topas -cecdisp on vio server) topas D topas -@ lparstat Operational modes: Monitoring, Hypervisor summary, Hypervisor calls, operating system configuration, Monitor multiple shared processor pools, dedicated partition in donating mode. mpstat Logical CPU statistics Voluntary and involuntary logical context switching (ilcs and vlcs) Multiple shared processor pools, dedicated partition in donating mode
Normal monitoring tools (2 of 5) topas recording and reporting: topasrec can be used to record local system data and cross LPAR data in binary format Start recording: # topasrec L o <output file> (local data) # topasrec -C o <output file> (cross partitions data) Use smitty Stop_Recording to stop topas recording. Listing running recording: # topasrec l Pid User Fmt Start_time Path Status 245850 root bin 12:58:23,Oct06,2009 /usr/lpp/perfagent/ Running Summary report: # topasout -R summary <output file> Detailed report: # topasout -R detailed /etc/perf/topas_cec.080816
Normal monitoring tools (3 of 5) SMIT panels introduced to operate on topas recording The following options are provided To start/stop persistent recording (24x7) To start/stop WLE data collection To choose the type of recording Binary/nmon style local recording CEC recording (cross partitions data) List available and completed recordings Generate reports on the completed recording
Normal monitoring tools (4 of 5) Topas CEC reports Summary Detailed
Normal monitoring tools (5 of 5) nmon Installed by default in AIX Can be started by running command nmon or topas_nmon Can be started by pressing ~ from topas screen topas_nmon can also record local data in nmon format
nmon analyzer and nmon consolidator nmon analyzer Post processing tool to analyze performance. Analyze files produced by topasout in nmon format Cross partitions statistics cannot be analyzed http://www.ibm.com/developerworks/wikis/display/wikiptype/nmonanalyser nmon consolidator Need to use nmon consolidator if you want to get a report for the entire machine.
VIO Server 2.1 monitoring commands Virtual I/O Server version 2.1 monitoring commands $ topas * $ topas -cecdisp $ fcstat * $ svmon * $ wklmgr $ wklagent $ wkldout $ seastat $ vmstat * $ viostat $ sysstat * Denotes standard AIX commands
VIO monitoring using topas (1 of 4) From the VIO, run topas cecdisp and press v to show the VIOs monitoring panel.
VIO monitoring using topas (2 of 4) From topas VIOS panel, move the cursor to a specific VIOS and press d to get the detailed monitoring
VIO monitoring using topas (3 of 4) From topas panel in the Virtual I/O Server, press E to display the shared Ethernet adapter configuration and statistics Topas monitor for host: sys14_vios3 Interval: 2 Wed Sep 23 17:37:07 2009 =================================================================================== Network KBPS I-Pack O-Pack KB-In KB-Out ent3 (SEA PRIM) 0.2 0.9 0.9 0.1 0.2 \--ent0 (PHYS) 0.2 0.9 0.5 0.1 0.1 \--ent4 (VETH CTRL) 0.1 0.0 2.8 0.0 0.1 \--ent2 (VETH) 0.1 0.0 0.5 0.0 0.0 lo0 0.0 0.0 0.0 0.0 0.0 The number of kilobytes received per second over the monitoring interval Device Tree reflected in the ASCII-art tree display The number of data packets sent per second over the monitoring interval The number of data packets received per second over the monitoring interval Total throughput=kb-in+kb-out
VIO monitoring using topas (4 of 4) From topas panel in the Virtual I/O Server, press D to display the Disks panel then d to display the adapters statistics Topas Adapter View : sys044_vios1interval: 2 Fri Oct 16 11:23:05 2009 =============================================================================== Adapter KBPS TPS KB-R KB-W fcs0 204.3K 2.4K 204.3K 0.0 vhost1 0.0 0.0 0.0 0.0 Adapters Throughput vhost0 204.3K 4.8K 2.4K 2.4K =============================================================================== Vtargets/Disks Busy% KBPS TPS KB-R ART MRT KB-W AWT MWT AQW AQD hdisk2 95.7 204.3K 2.4K 204.3K 0.4 1.8 0.0 0.0 0.0 0.6 0.9 hdisk1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 hdisk0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Physical hdisk Busy% and statistics
AIX MPIO paths monitoring using topas From topas Disk panel in an AIX client partition, press 'd' to toggle on/off Adapter panel Topas Adapter View : sys044_lpar1interval: 2 Fri Oct 16 12:43:52 2009 =============================================================================== Adapter KBPS TPS KB-R KB-W vscsi0 152.0K 2.4K 2.4K 0.0 vscsi1 0.0 0.0 0.0 0.0 =============================================================================== Vtargets/Disks Busy% KBPS TPS KB-R ART MRT KB-W AWT MWT AQW AQD hdisk0 100.0 150.0K 2.3K 150.0K 1.2 72.2 0.0 0.0 113.80.0 0.0 Press 'm' to toggle on/off Path panel Topas MPIO Monitor for host:sys044_lpar1 Interval 2 Fri Oct 16 11:10:51 2009 =============================================================================== Disk Busy% KBPS TPS KB-R ART MRT KB-W AWT MWT AQW AQD hdisk0 100.0 214.0K 2.5K 214.0K 1.1 3.6 0.0 0.0 0.0 0.0 0.0 =============================================================================== Path Busy% KBPS TPS KB-R KB-W Path1 0.0 0.0 0.0 0.0 0.0 Paths Throughput Path0 100.0 214.0K 2.5K 214.0K 0.0
VIO monitoring using Workload Manager Maintenance commands available since Virtual I/O Server 1.3 (for long term monitoring) > wkldmgr (workload manager) > wkldagent (provide recording capabilities) > wkldout (process workload manager-related recordings) Tool used for monitoring is a subset of AIX WLM (Workload Manager) The WLM is started in passive mode Does not regulate the CPU, memory, or disk utilization but monitor them. wkldout [-report reporttype] [-interval MM] [-beg HHMM] [-end HHMM][-fmt[-mode modetype]] [-graph] filename <xmwlm_recording_file> $wkldout report summary filename /home/ios/perf/wlm/xmwlm Report: System Summary --- hostname: sys23_vio Start:12/06/07 11:36:18 Stop:12/06/07 12:16:18 Int: 5 Min Mem: 2.0GB Shared SMT: ON Logical CPUs: 4 Psize: 3 time InU Us Sy Wa Id PhysB Ent %Entc RunQ WtQ -------------------------------------------------- 11:41:17 2.0 0 1 0 99 0.21 0.25 1.4 1.0 0.0 11:46:17 2.0 0 1 0 99 0.30 0.25 1.8 1.1 0.0 11:51:17 2.0 0 1 0 99 0.27 0.25 1.7 0.8 0.0 11:56:17 2.0 0 1 0 40 56.2 88.2 28.2 1.0 0.0 Wkldout summary report Nmon analyzer output
Freeware monitoring tools (1 of 3) LPARMon Simple graphical tool that shows what is going on in the managed system.
Freeware monitoring tools (2 of 3) Ganglia is an Open Source cluster performance monitoring tool Extended to include POWER5/6 features like shared processor LPARs, entitlement, physical CPU usage and so on. Monitors LPARs (including VIO Server) on a single or multiple managed systems. http://www.ibm.com/developerworks/wikis/display/wikiptype/ganglia
Freeware monitoring tools (3 of 3) lpar2rrd LPAR CPU statistics and documentation tool for IBM POWER servers Produces historical CPU utilization graphs of LPARs and shared CPU usage. Collects complete physical (HW) and logical configuration of all managed systems and their LPARs including changes in their state and configuration http://www.ibm.com/developerworks/wikis/display/virtualization/lpar2rrd+tool
Performance toolbox POWER systems and virtualized environments can be monitored by Performance toolbox (PTX)
Management tools: IBM Systems Director Integrated suite of systems management tools, providing: Monitoring and management of systems across heterogeneous IT environments Simplify management of physical and virtual platform resources Monitors and reports system health LPAR LPAR LPAR LPAR LPAR LPAR LPAR VIO VIO Discovers physical and shared I/O
IBM Systems Director: Welcome
IBM Systems Director: Power Systems Management
IBM Systems Director VMControl features VMControl Express Edition Lifecycle management and relocation of virtual machines of all IBM servers platform Manage Power Systems virtual server resources Create, modify and delete virtual machines Relocate virtual machine VMControl Standard Edition Lifecycle management and deployment of virtual images on Power Systems (AIX) and System z (Linux) Optional upgrade from Express to Standard edition via purchased license key activation Create and manage virtual machines Virtual machine relocation Capture, import, create virtual images Deploy virtual images Maintain virtual images in a centralized library Resource pool management features (future) VMControl Express Edition VMControl Standard Edition VMControl Enterprise Edition
IBM Systems Director VMControl
IBM Tivoli Monitoring Consolidated monitoring of physical and virtual resources Improves mean-time-to-recovery by visualizing the virtual world to solve virtual performance problems Real-time and historical data assists in separating intermittent problems from reoccurring problems within peak workloads Out-of-the-box reporting allows clients to quickly provide executive level reports and identify resource bottlenecks
IBM Tivoli Monitoring: AIX/System p architecture TEPS TEP client Console server Console database ITM server Management server Topology availability performance HMC OS HMC/IVM TEMS VIOS availability health performance Warehouse AIX availability health performance System P Server HMC agent AIX CEC agent AIX VIOS premium agent VIOs AIX premium agent AIX AIX premium agent AIX AIX premium agent AIX
IBM Tivoli Monitoring: Hypervisor view Global CPU and memory allocation Total CPU and memory allocated to LPARs
IBM Tivoli Monitoring: AIX LPAR view CPU, memory, disk, network info per LPAR
IBM Tivoli Monitoring: Network mapping VIOS view of network information Shows how network interfaces are mapped to LPARS
IBM Tivoli Monitoring: Virtual storage mapping VIOS view of disk mapping and utilization information Shows virtual storage mapping for VIO Server Shows virtual storage mapping detail for VIO Server
IBM Tivoli Monitoring: Device status
Checkpoint 1. Which command can be used to check and monitor a shared Ethernet adapter failover on a Virtual I/O Server? 2. Which command can be used to start recording with topas? 3. What IBM tool is an integrated suite of systems management tools, designed for monitoring and managing heterogeneous IT environments? 4. True or False? IBM Tivoli Monitoring monitors and manages system and network applications on a variety of operating systems, tracks the availability and performance of your enterprise system, and provides reports to track trends and troubleshoot problems.
Checkpoint solutions 1. Which command can be used to check and monitor a shared Ethernet adapter failover on a Virtual I/O Server? Run topas then press E. 2. Which command can be used to start recording with topas? topasrec 3. What IBM tool is an integrated suite of systems management tools, designed for monitoring and managing heterogeneous IT environments? IBM Systems Director 4. True or False? IBM Tivoli Monitoring monitors and manages system and network applications on a variety of operating systems, tracks the availability and performance of your enterprise system, and provides reports to track trends and troubleshoot problems.
Unit summary Having completed this unit, you should be able to: Describe standard AIX/virtualization monitoring tools Identify freeware monitoring tools Describe virtualization management and monitoring tools such as IBM Systems Director IBM Tivoli Monitoring