IBM Tivoli Storage Productivity Center V4.2.1 Performance Management Best Practices

Transcription

1 IBM Tivoli Storage Productivity Center V4.2.1 Performance Management Best Practices Version 2.5 (March 7, 2011) Sergio Bonilla Xin Wang IBM Tivoli Storage Productivity Center development, San Jose, CA

2 Second Edition (March 2011) This edition applies to Version 4, Release 2, of IBM Tivoli Storage Productivity Center Copyright International Business Machines Corporation All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. 2

3 Table of Contents 1 Notices Legal Notice Trademarks Acknowledgement Other IBM Tivoli Storage Productivity Center Publications IBM Tivoli Storage Productivity Center Performance Management Overview Disk Performance and Fabric Performance Performance Metrics Setup and Configuration Performance Data collection Adding a Device Create Threshold Alerts Create Performance Monitor Check Performance Monitor Status Retention for performance data Common Issues General issues ESS and DS Related Issues DS4000/DS5000 Related Issues HDS Related Issues Top Reports and Graphs a Storage Administrator May Want to Run Tabular Reports Drill up and Drill down Historic Charts Batch Reports Constraint Violation Reports Top Hit Reports SAN Planner and Storage Optimizer Summary Reference Appendix A Available Metrics Appendix B Available Thresholds Appendix C DS3000, DS4000 and DS5000 Metrics

4 1 Notices 1.1 Legal Notice This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non- IBM products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, 4

5 companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces. 1.2 Trademarks The following terms are trademarks or registered trademarks of the International Business Machines Corporation in the United States or other countries or both: AIX Passport Advantage Tivoli Storage DB2 pseries WebSphere DS4000, DS6000, DS8000 Redbooks (logo) XIV Enterprise Storage Server Redbooks zseries server iseries Storwize Tivoli The following terms are trademarks or registered trademarks of other companies: Microsoft, Windows, Windows XP and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of the Open Group in the United States and other countries. Java, Solaris, and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Intel is a registered trademark of the Intel Corporation or its subsidiaries in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. CLARiiON and Symmetrix are registered trademarks of the EMC Corporation. 5

6 HiCommand is a registered trademark of Hitachi Data Systems Corporation. Brocade and the Brocade logo are trademarks or registered trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. McDATA and Intrepid are registered trademarks of McDATA Corporation. Cisco is a registered trademark of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries. Engenio and the Engenio logo are trademarks or registered trademarks of LSI Logic Corporation. Other company, product, or service names may be trademarks or service marks of others. 1.3 Acknowledgement The materials in this document have been collected from explicit work in the IBM Tivoli Storage Productivity Center development lab, other labs within IBM, from experiences in the field at customer locations, and contributions offered by people that have discovered valuable tips and have documented the solution. Many people have helped with the materials that are included in this document, too many to properly acknowledge here, but special thanks goes to Xin Wang who compiled the original version of this document. It is a source of information for advanced configuration help and basic best practices for users wanting to get started quickly with Tivoli Storage Productivity Center. 1.4 Other IBM Tivoli Storage Productivity Center Publications IBM Tivoli Storage Productivity Center Performance Management Best Practices is a supplement to the Tivoli Storage Productivity Center publications that are available. It is intended to be a supplement to the Tivoli Storage Productivity Center publications, providing additional information to help implementers of Tivoli Storage Productivity Center with configuration questions and to provide guidance in the planning and implementation of Tivoli Storage Productivity Center. It is expected that an experienced Tivoli Storage Productivity Center installer will use this document as a supplement for installation and configuration, and use the official Tivoli Storage Productivity Center publications for overall knowledge of the installation process, configuration, and usage of the Tivoli Storage Productivity Center components. This document is not intended to replace the official Tivoli Storage Productivity Center publications, nor is it a self-standing guide to installation and configuration. You can find the entire set of Tivoli Storage Productivity Center publications at These documents are essential to a successful implementation of Tivoli Storage Productivity Center, and should be used to make sure that you do all the required steps to install and configure Tivoli Storage Productivity Center. You should have the official publications available in either softcopy of printed form, read them and be familiar with their content. 6

7 2 IBM Tivoli Storage Productivity Center Performance Management 2.1 Overview There are three main functions for IBM Tivoli Storage Productivity Center performance management: performance data collection, performance thresholds/alerts, and performance reports. The product can collect performance data for devices - storage subsystems and fibre channel switches - and store the data in the database up to a user-defined period. The product may collect performance data from either IBM devices using native-agent APIs, or IBM and non-ibm devices that are managed by CIM agents that are at least SMI-S 1.1 compliant. The product can set thresholds for important performance metrics, and when any boundary condition is crossed, can notify the user via , SNMP, or other alerting mechanisms. And lastly, the product can generate reports, historic trend charts, and help analyze the bottleneck of a performance congestion by drilling down to threshold violated components and the affected hosts. The combination of those functions can be used to monitor a complicated storage network environment, to predict warning signs of system fallout, and to do capacity planning as overall workload grows. The collected performance data may also be utilized by both the Storage Optimizer and SAN Planner functions. The IBM Tivoli Storage Productivity Center Standard Edition (5608-WC0) includes performance management for both subsystems and switches, while IBM Tivoli Storage Productivity Center for Disk (5608-WC4) is only for storage subsystems. IBM Tivoli Storage Productivity Center Basic Edition (5608-WB1) and IBM Tivoli Storage Productivity Center for Data (5608-WC3) do not include performance management function. 2.2 Disk Performance and Fabric Performance Performance management for subsystems is done via the disk manager. Data collection for subsystems can be scheduled under Disk Manager -> Monitoring -> Performance Monitors, and subsystem performance reports are under Disk Manager -> Reporting -> Storage Performance. Performance management for fibre channel switches is done via the fabric manager. Data collection for switches can be scheduled under Fabric Manager -> Monitoring -> Switch Performance Monitors, and switch performance reports are under Fabric Manager -> Reporting -> Switch Performance. Some disk performance and all fabric performance require a CTP certified CIMOM that is at least SMI-S 1.1 compliant. Devices that do not require a CIM agent for disk performance collection include some DS8000 subsystems, and all SVC, Storwize V7000, and XIV Storage subsystems. These devices use native-agent APIs introduced in IBM Tivoli Storage Productivity Center V4.2. 7

8 2.3 Performance Metrics IBM Tivoli Storage Productivity Center can report on various performance metrics, which indicate the particular performance characteristics of the monitored devices. Two very important metrics for storage subsystems are the throughput in I/Os per sec and the response time in milliseconds per I/O. Throughput is measured and reported in several different ways. There is throughput of an entire box (subsystem), or of each cluster (ESS) or controller (DS6000, DS8000), or of each I/O Group (SVC, Storwize V7000). There are throughputs measured for each volume (or LUN), throughputs measured at the Fibre Channel interfaces (ports) on some of the storage boxes and on fibre channel switches, and throughputs measured at the RAID array after cache hits have been filtered out. For storage subsystems, it is common to separate the available performance statistics into two separate domains, the front-end and the back-end of the subsystem. Front-end I/O metrics are a measure of the traffic between the servers and the storage subsystem, and are characterized by relatively fast hits in the cache, as well as occasional cache misses that go all the way to the RAID arrays on the back end. Back-end I/O metrics are a measure of all traffic between the subsystem cache and the disks in the RAID arrays in the backend of the subsystem. Most storage subsystems give metrics for both kinds of I/O operations, front- and back-end. We need to always be clear whether we are looking at throughput and response time at the front-end (very close to system level response time as measured from a server), or the throughput and response time at the back-end (just between cache and disk). The main front-end throughput metrics are: Total I/O (overall) Read I/O (overall) Write I/O (overall) The corresponding front-end response time metrics are: Overall Response Time Read Response Time Write Response Time The main back-end throughput metrics are: Total Backend I/O (overall) Backend Read I/O (overall) Backend Write I/O (overall) The corresponding back-end response time metrics are: Overall Backend Response Time Backend Read Response Time Backend Write Response Time It is important to remember that the response times taken in isolation of throughput rates are not terribly useful, because it is common for components which have negligible throughput rates to exhibit large (bad) response times. But in essence those bad response times are not significant to the overall operation of the storage environment if they occurred for only a handful of I/O operations. It is therefore necessary to have an understanding of which throughput and response time combinations are significant and which can be ignored. To help in this determination, IBM Tivoli Storage Productivity Center V4.1.1 introduced a metric called Volume Utilization Percentage. This metric is based on both I/O and Response Time of a storage volume and 8

9 is an approximate measure of the amount of time the volume was busy reading and writing data. It is therefore safe to ignore bad average response time values for volumes with very low utilization percentages, and conversely, those volumes with the highest utilization percentages are the most important for the smooth operation of the storage environment and are most important to exhibit good response times. When implementing storage tiering using 10K, 15K, or even SSD drives, the most highly utilized volumes should be considered for being placed on the best performing underlying media. Furthermore, it will be advantageous to track any growth or change in the throughput rates and response times. It frequently happens that I/O rates grow over time, and that response times increase as the I/O rates increase. This relationship is what capacity planning is all about. As I/O rates and response times increase, you can use these trends to project when additional storage performance (as well as capacity) will be required. Depending on the particular storage environment, it may be that throughput or response times change drastically from hour to hour or day to day. There may be periods when the values fall outside the expected range of values. In that case, other performance metrics can be used to understand what is happening. Here are some additional metrics that can be used to make sense of throughput and response times. Total Cache Hit percentage Read Cache Hit Percentage NVS Full Percentage Read Transfer Size (KB/Op) Write Transfer Size (KB/Op) Low cache hit percentages can drive up response times, since a cache miss requires access to the backend storage. Low hit percentages will also tend to increase the utilization percentage of the backend storage, which may adversely affect the back-end throughput and response times. High NVS Full Percentage (also known as Write-cache Delay Percentage) can drive up the write response times. High transfer sizes usually indicate more of a batch workload, in which case the overall data rates are more important than the I/O rates and the response times. In addition to the front-end and back-end metrics, many storage subsystems provide additional metrics to measure the traffic between the subsystem and host computers, and to measure the traffic between the subsystem and other subsystems when linked in remote-copy relationships. Such fibre channel port-based metrics, primarily I/O rates, data rates, and response times are available for ESS, DS6000, DS8000, SVC, and Storwize V7000 subsystems. ESS, DS6000, and DS8000 subsystems provide additional break-down between FCP, FICON, and PPRC operations at each port. SVC and Storwize V7000 subsystems provide additional breakdown between communications with host computers, backend managed disks, and other nodes within the local cluster, as well as remote clusters at each subsystem port. XIV subsystems do not provide portbased metrics as of IBM Tivoli Storage Productivity Center V Similar to the Volume Utilization Percentage mentioned earlier, IBM Tivoli Storage Productivity Center V4.1.1 also introduced the Port Utilization Percentage metric (available for ESS, DS6000, and DS8000 storage subsystems). The Port Utilization Percentage is an approximate measure of the amount of time a port was busy, and can be used to identify over-utilized and under-utilized ports on the subsystem for potential port balancing. For subsystems where port utilizations are not available, the simpler Port Bandwidth Percentage metrics provide a measure of the approximate bandwidth utilization of a port, based on the port s negotiated speed, and can be used in a similar fashion. However, beware that the Port Bandwidth Percentages can potentially provide misleading indicators of port under-utilization when ports are not under-utilized if there is a performance bottleneck elsewhere in the fabric or at the port s communication partner. For fibre-channel switches, the important metrics are Total Port Packet and Total Port Data, which provide the traffic pattern over a particular switch port, as well as the Port 9

10 Bandwidth Percentage metrics providing indicators of bandwidth usage based on port speeds. When there are lost frames from the host to the switch port, or from the switch port to a storage device, the dumped frame rate on the port can be monitored. All these metrics can be monitored via reports or graphs in IBM Tivoli Storage Productivity Center. Also there are several metrics for which you can define thresholds and receive alerts when measured values do not fall within acceptable boundaries. Some examples of supported thresholds are: Total I/O and Total Data Thresholds Total Backend I/O and Total Backend Data Thresholds Read Backend Response Time and Write Backend Response Time Thresholds Total Port I/O (Packet ) and Data Thresholds Overall Port Response Time Threshold Port Send Utilization Percentage and Port Receive Utilization Percentage Thresholds Port Send Bandwidth Percentage and Port Receive Bandwidth Percentage Thresholds Please see Appendix A for a complete list of performance metrics that IBM Tivoli Storage Productivity Center supports and Appendix B for a complete list of thresholds supported. The important thing is to monitor the throughput and response time patterns over time for your environment, to develop an understanding of normal and expected behaviors. Then you can set threshold boundaries to alert you when anomalies to the expected behavior are detected. And you can use the performance reports and graphs to investigate any deviations from normal patterns or to generate the trends of workload changes. 3 Setup and Configuration 3.1 Performance Data collection Performance data may be collected either directly from select device types using native-agent API or from devices managed by a CIM agent (CIMOM) using SMI-S interfaces. Devices that do not require a managing CIM agent include DS8000, SVC, Storwize V7000, and XIV subsystems. Devices that require a CIM agent include non-ibm subsystems and switches, ESS subsystems, DS4000 and DS5000 subsystems, and DS6000 subsystems. XIV subsystems require version 10.1 or higher to collect performance data. For devices that require the use of a CIM agent, you need to make sure all of the following prerequisites are met before adding the CIM agent to TPC: The version of the CIMOM and the firmware for the device is supported. A CIMOM is installed in the environment, either as a proxy on another server or embedded on the device it manages. For subsystems or switches on a private network, be sure to have the CIMOM installed on a gateway machine so the IBM Tivoli Storage Productivity Center server on a different network can communicate with it. The CIMOM is configured to manage the intended device. See IBM TotalStorage Productivity Center V3.2 Hints and Tips on how to install and configure a CIMOM: 10

11 &dc=db500&uid=swg &loc=en_us&cs=utf-8&lang=en The following steps cover setup and configuration for both subsystems and switches Adding a Device You may add a device to collect performance data from to TPC through either: 1. Launching the IBM Tivoli Storage Productivity Center -> Configure Devices panel. 2. Clicking the Add Storage button on the Disk Manager -> Storage s panel. 3. Clicking the Add Fabric button on the Fabric Manager -> Fabrics panel. 4. Clicking the Add button on the Administrative Services -> Data Sources -> Storage s panel. 5. Clicking the Add CIMOM button on the Administrative Services -> Data Sources -> CIMOM Agents panel. The first four options will launch a Configure Devices wizard, at different stages of the configuration settings. Adding a device using the first option allows the user to configure either a storage subsystem or fabric/switch to be used for future performance monitors. The second and fourth options automatically choose the Storage option and advances to a panel that allows you to choose the subsystem type, while the third option automatically chooses the Fabric/Switches option. Going through the first panel also allows the user to choose to add and configure a new device or to configure a previously discovered device. Adding a storage subsystem allows you to choose between configuring an IBM DS8000, IBM XIV, IBM SAN Volume Controller/IBM Storwize V7000, or other storage subsystems managed by a CIM agent. Adding fabric/switch allows you to choose between configuring a Brocade, McData, Cisco, QLogic, or Brocade/McData device. The Configure Devices panel updates depending on the device chosen. Fields with labels in bold are required in order to connect to the chosen device type. If you are configuring other devices, read the CIMOM documentation to get all the information required to connect to the CIMOM. In order to collect performance data for fabrics, you must choose to configure a CIMOM agent for the device, instead of configuring an out of band fabric agent. Configuring a device using these wizards will perform the necessary discovery and initial probe of the device required in order to run a performance data collection against the device. 11

12 3.1.2 Create Threshold Alerts A performance threshold is a mechanism by which you can specify one or more boundary values for a performance metric, and can specify to be notified if the measured performance data for this metric violates these boundaries. Thresholds are applied during the processing of performance data collection, so a performance monitor for a device must be actively running for a threshold to be evaluated and a violation to be recognized The Tivoli Storage Productivity Center ships with several default thresholds enabled (see Appendix B for a full list of thresholds supported) that do not change much with the environment, but metrics such as throughput and response time can vary a lot depending on the type of workload, model of hardware, amount of cache memory etc. so that there are no recommended values to set. Boundary values for these thresholds have to be determined in each particular environment by establishing a base-line of the normal and expected performance behavior for the devices in the environment. After the base-line is determined, thresholds can then be defined to trigger if the measured performance behavior falls outside the normally expected range. Thresholds are device type and component type specific, meaning that each threshold may apply to only a subset of all supported device types and to only a subset of supported component types for each device type. Every threshold is associated with a particular metric; checking that metric s value at each collection interval determines whether the threshold is violated or not. To create an alert for subsystem thresholds, go to Disk Manager -> Alerting -> Storage Alerts, right-click to select create storage subsystem alert (see Figure 1): Alert tab In the triggering condition area, select from the drop down list a triggering condition (threshold alerts have names ending with Threshold ), ensure that the threshold is enabled via the checkbox at the top of the panel, and then enter the threshold boundary values for the desired boundary conditions. Tivoli Storage Productivity Center V4.1.1 and above allows decimal values in the threshold boundary values, prior versions only allow integer values. Alert tab Some thresholds are associated with an optional filter condition, which is displayed in the triggering condition area. If displayed, you can enable or disable the filter, and if enabled, can set the filter boundary condition. If the filter condition is triggered, any violation of this threshold will be ignored when the filter is enabled. Alert tab In the alert suppression area, select whether to trigger alerts for both critical and warning conditions or only critical conditions or not to trigger any alerts. The suppressed alerts will not alert log entries or cause any action to be taken as defined in the triggered action area, but they will still be visible in the constraint violation reports. Alert tab In the alert suppression area, select whether to suppress repeating alerts. You may either suppress alerts until the triggering condition has been violated continuously for a specified length of time or to suppress subsequent violations for a length of time after the initial violation. Alerts suppressed will still be visible in the constraint violation reports. Alert tab In the triggered action area, select one of the following actions: SNMP trap, TEC/OMNIbus event, login notification, Window s event log, run script, or . Storage subsystem tab move the subsystem(s) you want to monitor into the righthand panel (Selected subsystems). Make sure these are the subsystems for which you will define performance monitors Save the alert with a name 12

13 Figure 1. Threshold alert creation panel for storage subsystems. To create an alert for switch thresholds, go to Fabric Manager -> Alerting -> Switch Alerts, right-click to select create switch alert, and follow the same steps as for subsystems described above. There are a few points that need to be addressed in order to understand threshold settings: 1. There are two types of boundaries for each threshold, the upper boundary (stress) and lower boundary (idle). When a metric s value exceeds the upper boundary or is below the lower boundary, it will trigger an alert. 2. There are two levels of alerts, warning and critical. The combination of boundary type and level type generates four different threshold settings: critical stress, warning stress, warning idle and critical idle. Most threshold values are in descending order (critical stress has the highest value that indicates high stress on the device, and critical idle has the lowest value) while Cache Holding Time is the only threshold in ascending order. 3. If the user is only interested to receive alerts for certain boundaries, the other boundaries should be left blank. The performance manager will only check boundary conditions with input values, therefore no alerts will be sent for the condition that is left blank. 4. The storage subsystem alerts will be displayed under IBM Tivoli Storage Productivity Center -> Alerting -> Alert logs -> All, as well as under Storage. Another important way to look at the exception data is to look at constraint violation reports. This is described in section

14 3.1.3 Create Performance Monitor A performance data collection is defined via a mechanism called a monitor job, and that can be run manually (immediately), can be scheduled for one-time execution, or can be scheduled for repeated execution, as desired. Only after the device has been probed successfully, can a monitor job be run successfully. To create a performance monitor on a subsystem, go to Disk Manager -> Monitoring -> Performance Monitors, right-click to select create performance monitor. Storage tab - move the subsystem(s) you want to monitor into the right-hand panel (Selected subsystems) Sampling and scheduling tab enter how frequently the data should be collected and saved (the smaller the interval length, the more granular the performance data), when the monitor will be run and the duration for the collection Save the monitor with a name To create a performance monitor for a switch, go to Fabric Manager -> Monitoring -> Switch Performance Monitors, follow the same steps as above, substituting storage subsystem with switch. The monitor will start at the scheduled time, but the performance sample data will be collected a few minutes later. For example, if your monitor is scheduled to start at 9 am to collect with an interval length of 5 minutes, the first performance data might be inserted into the database minutes later, and the second performance data will be inserted after 5 more minutes. Only after the first sample data is inserted into the database, in this case, around 9:10 or 9:15 am, you will be able to view the performance reports. Because of this, there are some best practice information related how to set up the schedule and duration for a performance monitor: 1. Monitor duration if a monitor is intended to run for a long time, choose to run it indefinitely. The performance monitor is optimized such that running indefinitely will be more efficient than running, stopping, and starting again. 2. You should only have one performance monitor defined per storage device. 3. Prior to v4.1.1, if you want to run the same monitor at different workload periods, set the duration to be 1 hour less than the difference between the two starting points. This gives the collection engine one hour to finish up the first collection and shutdown properly. For example, if you want to start a monitor at 12 am and 12 pm on the same day, the duration for the 12 am collection has to be 11 hours or less, so the monitor can start again at 12 pm successfully. The same is true for a repeated run. If you want to run the same monitor daily, be sure the duration of the monitor will be 23 hours or less. If you want to run the same monitor weekly, the duration of the monitor will need to be 7x24-1 = 167 hours or less. 4. Later versions of IBM Tivoli Storage Productivity Center no longer have the limitation of running back-to-back performance monitors for the same device with a 1 hour reduction of the duration required. To avoid overlapping performance monitors for back-to-back monitors, v4.1.1 and higher will automatically reduce the duration of the preceding run by one interval period to allow sufficient time for the monitor to end. There is no data loss, as the subsequent monitor will retain the previous interval s performance data for the next delta. During a performance sample collection, the hourly and daily summary for each performance metric are computed based on the sample data. The summary data reflects the performance characteristics of the component over certain time periods while the sample data shows the performance right at that moment. 14

15 One more thing to notice for a performance monitor and the sample data: the clock on the server might be different from that on the device. The performance monitor always uses device time on the sample it collects, then converts it into the time zone (if it s different) of the IBM Tivoli Storage Productivity Center server Check Performance Monitor Status When the performance monitor job starts to run, you begin to collect performance data for the device. You should check the status of the monitor job, make sure it runs and continues running. Expand on Performance Monitors, right click on the monitor, select Job History, and check the status of the job you want to view. Alternatively, you may navigate to IBM Tivoli Storage Productivity Center -> Job Management and find the performance job in the list of jobs. If the status is blue, the monitor is still running without issues. If the status is yellow, you can check out the warning messages. The monitor will continue to run with warning messages. For example, if there are missing a sample data warning messages, the monitor will continue to run, and only if the monitor misses all the data it should collect, the status will turn red, and the monitor failed. If the status is green, the monitor completed successfully. To view the job log, select the performance job from the list of scheduled jobs. The runs of the particular job will be listed in the bottom panel. Expand the run you are interested in, select the job, and click the View Log File(s) button. Normally the job log will have error messages logged for a failed collection. There are a few common issues that may lead to failed data collection. See section 3.3 for details. 3.2 Retention for performance data After the monitor is created, the user should configure the retention of performance data in the database. Expand Administrative Services > Configuration > Resource History Retention, under Performance Monitors there are three options: retention for collected sample data (labeled per performance monitoring task ), retention for aggregated hourly data, and retention for daily data. Sample data is the data that is collected at the specified interval length of the monitor, for example data collected every 5 minutes. The default retention period for sample data is 14 days. This means that by default, IBM Tivoli Storage Productivity Center keeps the individual 5 minute samples for 14 days before they are purged. Individual samples are summarized into hourly and daily data, for example the sum of 12 of the 5 minute samples are saved as an hourly performance data record, and the sum of 288 such samples are saved as a daily performance data record. The default retention periods for hourly and daily data are 30 days and 90 days, respectively. You can change all those values based on your need to retain historical performance information, but please be aware of the implication to the size of IBM Tivoli Storage Productivity Center database if performance data is kept longer, especially the sample data. Here are a few formulas the user can use to estimate the size of performance data: For subsystems, the biggest component is volume, and the biggest performance sample data will be that of volumes. For switches, the performance data is proportional to the number of ports in a switch. Assuming: NumSS = number of subsystems NumSW = number of switches 15

16 NumV = average number of volumes in a subsystem NumPt = average number of ports in a switch CR = number of sample data collected per hour (for a sample interval of 5 minutes, this should be 60/5 = 12 samples ) Rs = retention for sample data in days Rh = retention for hourly summarized data in days Rd = retention for daily summarized data in days The estimated space required may be calculated using the following formulas: Storage subsystem performance sample data size = NumSS * NumV * CR * 24 * Rs * 200 byte Storage subsystem performance aggregated data size = NumSS * NumV * (24 * Rh + Rd) * 200 byte Switch performance sample size= NumSw * NumPt * CR * 24 * Rs * 150 Switch performance aggregated data size = NumSw * NumPt * (24 * Rh + Rd) * Common Issues There are a few known issues that may lead to failed performance data collection. Most of them are related to the configuration of devices or the environment. Here are a few hints and tips on those known issues: General issues Invalid data returned by either the firmware or CIMOM of managed devices may cause a data spike during a polling interval. These spikes will skew the results of averaged and aggregated data, resulting in unreliable performance data that are also capable of reducing the effectiveness of the Storage Optimizer. To alleviate this, code was introduced in to detect when a data spike occurred and the performance data for that polling interval wouldn t be inserted into the database. This will result in a message indicating that the polling interval was skipped in the performance monitor s job log when a spike is detected ESS and DS Related Issues Any firewalls between the ESS CIMOM host server and the ESS subsystem should be configured to allow LIST PERFSTATS traffic through. If this is not possible, then both the ESS CIMOM host server and the ESS subsystem must be on the same side of any existing firewalls. In addition to this, all IP ports on the CIMOM server above 1023 opened to receive performance data from the ESS. The port bandwidth usage percentage for DS6000 subsystems may be displayed as N/A in the reports. This is due to the port speeds not being available from the device. The DS6000 CIMOM may be upgraded to version to reduce the likelihood. The storage pools of DS8000 subsystems containing space efficient volumes will have incomplete performance data collected. The performance manager is unable to determine if the space efficient volumes are fully allocated, making it impossible to manage the performance for the ranks, the arrays associated with those ranks, and the device adapters associated with those arrays, since it cannot determine the performance impact of those volumes. Rather than present 16

17 the user with inaccurate and misleading information, the performance manager will not aggregate the volumes metrics to the higher level components DS4000/DS5000 Related Issues Both clusters need to be defined to the Engenio CIMOM. If only one cluster of the DS4000 or DS5000 is defined to the Engenio CIMOM, performance data will be collected only for the one cluster, while volume and other components information are still collected for both clusters HDS Related Issues Tivoli Storage Productivity Center is capable of collecting some performance statistics from HDS subsystems with HDvM 6.2, but there are currently known limitations to the performance metrics being returned. As such, Tivoli Storage Productivity Center does not claim support for monitoring HDS subsystems with HDvM 6.2. For more information regarding this limitation, please see: 4. Top Reports and Graphs a Storage Administrator May Want to Run After performance data is collected for the subsystem and the switch, there are a few ways to use the data and interpret key metrics via reports, via charts, drill down to problematic components, review of constraint violation reports, or export of the data for customized reports. The following table describes the performance reports supported for each device type in IBM Tivoli Storage Productivity Center Note that not all SMI-S BSP subsystems support each report type. For example, only certain versions of DS3000, DS4000, and DS5000 return performance data that can be displayed in the By Controller and By Port reports. Please see Appendix C for a list of metrics and reports supported by DS3000, DS4000, and DS5000. Device Type ESS, DS6000, DS8000 SVC, Storwize V7000 Performance Report Type By By Controller By Array By Volume By Port By By Volume By I/O Group By Module/Node By Managed Disk Group 17

18 XIV SMI-S BSP (eg. DS4000, DS5000) Switch By Managed Disk By Port By By Volume By Module/Node By By Controller By Volume By Port By Switch By Port Tabular Reports The most straight forward way to view performance data is to go through the corresponding component manager s reporting function to view the most recent data. For subsystem performance reports, go to Disk Manager -> Reporting -> Storage Performance, then select one of the options to view the data. The options on report types as shown in Figure 2: By for box level aggregate/averages for ESS, DS, SVC, Storwize V7000, XIV, and SMI-S BSP By Controller for ESS clusters, and DS and select SMI-S BSP controllers By Array for ESS and DS arrays By Volume for ESS, DS, XIV, and SMI-S BSP volumes/luns, and SVC and Storwize V7000 vdisks By Port ESS, DS, SVC, Storwize V7000, and SMI-S BSP FC ports onto storage box By I/O Group for SVC and Storwize V7000 I/O groups By Node for SVC and Storwize V7000 nodes By Managed Disk Group for SVC and Storwize V7000 managed disk groups By Managed Disk for SVC and Storwize V7000 managed disks 18

19 Figure 2. IBM Tivoli Storage Productivity Center Performance Reports Options for Disk Manager. On the right-hand panel, the available metrics for the particular type of device are in the included columns, and the user can pick which metric not to include in the performance report. The user also could use the selection button to pick components specific to the device type, and use the filter button to define criteria of their choosing. Select the display latest performance data option to generate a report on the most recent data. Historic reports can be created by choosing either the date/time range or by defining how many days in the past to include in the report. You may display either the latest sample, hourly, or daily data for either the latest or historic reports. If the selection is saved, this customized report will show up under IBM Tivoli Storage Productivity Center -> Reporting -> My Reports > [Admin] s Reports. [Admin] here is the login name used to define the report. See more information regarding this topic in the IBM Tivoli Storage Productivity Center User s Guide. For switch performance reports, go to Fabric Manager -> Reporting -> Switch Performance. A report may be created in a similar fashion as a subsystem report. The supported report types are: By Switch By Port 19

20 Drill up and Drill down Based on the latest sample data, drill up and drill down can be done between different components for ESS, DS8000, DS6000, SVC, Storwize V7000, and XIV. For ESS/DS8000/DS6000, user can drill down to reports in this path by clicking on the magnifying glass icon by the row: By Controller -> By Array -> By Volume reports. And drill up works on reverse direction. For example, by looking at the performance of a volume, you can drill up to the performance of the underlying array to see if there is more information; and while you are at the performance report for an array, you can drill down to all the corresponding volumes to see which volume is imposing significant load on the array. For SVC and Storwize V7000, a user can drill down to reports in this path: by Mdisk Group -> by Mdisk. Historic Charts A historic chart can be created by clicking the chart icon at the top of the tabular report and choosing history chart on the panel that is displayed. The history chart may be created for all the rows from the previous tabular report or for a selected subset chosen prior to clicking the chart icon. One or more metrics may be chosen to display in the report, as well as whether to sort the chart by the metric(s) chosen or by component. Once the historic chart is generated, you may modify the date and time ranges included in the chart and click generate chart again. You may view the trend of the data by clicking the show trends button. The historic data in tabular form may not always be available in older versions of IBM Tivoli Storage Productivity Center; however, in those cases the report can be exported into different formats for analysis in other tools once the by sample history chart is displayed. Click on the File option, there are various print and export options. You can print the graph to a format such as html, pdf etc. and export the data into CSV file for archiving or input to a spreadsheet. It s desirable to track growth in the I/O rate and response time for a particular subsystem or switch using the historic chart. Also, retention of the performance data in the IBM Tivoli Storage Productivity Center database is limited, and eventually it will be purged out of the database. It is important to develop a set of graphs and reports to summarize and visualize the data, and keep periodic snapshots of performance. The key is to monitor normal operations with key metrics, develop an understanding of expected behaviors, and then track the behavior for either performance anomalies or simple growth in the workload. See Figure 3 for an example of the throughput chart (Total I/O ) for a DS8000. This data is an hourly summary of I/O rate for this DS8000 for past day. This data can also easily be exported into other format for analysis. 20

21 Figure 3. Historic chart on Total I/O rate for a DS8000. Batch Reports Another way to backup the performance data is to use batch reports. This saves a report into a file on a regular scheduled basis. You can create a batch report by going to IBM Tivoli Storage Productivity Center -> Reporting -> My Reports, right-click on the Batch Reports node, and select Create Batch Report. In order to create batch reports, a data agent must be installed locally or remotely. Starting with IBM Tivoli Storage Productivity Center V4.1, fix pack 1, the data agent will only be available with the Data and Standard Edition. Prior to installing a data agent, an agent manager must be installed, and the device and data servers must have been registered with the agent manager. For additional information regarding installing the data agent or agent manager see: 0_t_installing_agents.html When creating a batch report, the Report tab allows you to choose the type of performance report for the batch job (see Figure 4). Select either a Storage Performance report or Switch Performance report under their respective nodes in the Available list and click the >> button. Only one performance report may be chosen per batch report job. Once a performance report type has been chosen, the Selection tab will be populated with the available metric columns for that report type. The panel is similar to the tabular report panel in section 4.1 (see Figure 2) and features the same options. On the Options tab, select which agent to run the report on (this will determine the location of the output file), and choose the type of output file to generate (see Figure 5), such as a CSV file that may be imported into a spreadsheet or an HTML file. Then choose when and how often you want this job to run on the When to Run tab. Then save the batch report. When the batch report is run, the file location is described in the batch job s log. 21

22 For additional information regarding batch reports, see the IBM Tivoli Storage Productivity Center V4.1.1 info center: 0_c_batch_reports.html Figure 4. Choose a performance report for the batch reporting job. 22

23 Figure 5. Choose the agent to run the report on and the type of output to generate. Constraint Violation Reports Another important way to view performance data is through constraint violation reports. For each device type, there are only certain metrics you can set thresholds. See Figure 5 for constraint violation options. Go to Disk Manager -> Reporting > Storage Performance -> Constraint Violations, all subsystems with thresholds violated will show up in the first general report. Similar constraint violation reports exist for switches. Go to Fabric Manager -> Reporting > Switch Performance -> Constraint Violations, you will get switch constraint violation reports. It is very important you set meaningful threshold values that the constraint report can be used to diagnose problems in your environment. The first report shows the number of violations for each subsystem during the last 24 hours. If the normal behavior pattern is studied and the threshold values truly reflect an abnormal condition, the number of violations will indicate the severity of the problem on the subsystem. This can be used daily to monitor all the devices and to analyze the trend in your environment. You can also drill down from the subsystem level to get details on the violations on each component from this general report. In the detailed report panel, you can click on the Affected Volumes tab to generate a report showing details on the affected volumes. Under Volumes, select whether you want the report to show all volumes or only the most active volumes associated with the subsystem and component. Under Performance data, select whether you want the report to display performance data for the volumes. The user can also click on the Affected Hosts tab to generate a report showing details on the affected hosts for ESS/DS6000/DS8000. The volume report here will show the user which host is affected by this threshold violation. In the meantime, a historic graph can be generated based on constraint violation by clicking on the chart icon. All the options described in section 4.3 exist here too. 23