Default Thresholds. Performance Advisor. Adaikkappan Arumugam, Nagendra Krishnappa

Transcription

1 Default Thresholds Performance Advisor Adaikkappan Arumugam, Nagendra Krishnappa Monitoring performance of storage subsystem and getting alerted at the right time before a complete performance breakdown is a challenging task for any admin. To trigger an alert at the right time we need to understand what the right threshold levels are. This document is an attempt to get you started setting up thresholds to monitor Storage subsystem performance using Performance Advisor. The values suggested in this document should be used as a reference point for getting started. It is strongly recommended to modify them as needed to meet the requirements of your environment and workload.

2 Table of Contents PURPOSE & SCOPE... 3 PRE-REQUISITES... 3 INTRODUCTION... 3 COUNTERS THAT MATTER... 4 BASIC PERFORMANCE MANAGEMENT... 4 For Protocols:... 5 For Exchange Server:... 6 For SQL Server:... 7 For Oracle:... 8 ALERT AT VARIOUS LEVELS... 9 PERFORMANCE BASE LINE... 9 ENABLE DIAG COUNTERS EXTERNAL REFERENCES... 15

3 Purpose & Scope Performance Advisor monitors Storage system performance and status by collecting performance counters on Storage systems. But, not many users understand how to use Performance Advisor to monitor and generate alerts when there is performance degradation. This document identifies key counters that need to be monitored and thresholds to set as a starting point. The user can then tune these values to suit his needs once he gets familiar with Performance Advisor and its capabilities in trouble shooting performance issues. An even better approach to find out the right threshold values for your workload is to use the base-lining feature in Performance Advisor. A methodology for doing this will be discussed later in this document. Note: All suggested threshold values are meant to serve as guidance and needs to be tuned according to your workload and environment. Pre-requisites Knowledge and familiarity with the following will help you understand this document better. Performance Advisor A component of Operations Manager Data ONTAP and Performance Counters Read more on Performance Advisor in the DataFabric Manager documentation available on NOW site - Introduction One of the consistent requests from Performance Advisor customers is to provide guidance on which of the basic performance counters should be monitored and what best fits a given workload or environment. Once these counters are identified, we need to determine the correct threshold values to be set so that alarms can be set on threshold breaches. Some of the key workloads where NetApp is being deployed are: Windows Applications Exchange, SQL, SharePoint Virtualization VMware, Hyper-V Home Directories CIFS or NFS Databases DB2, Oracle, Sybase SAP Performance of the servers hosting the applications and the application performance itself can be determined using tools available for server and application performance management. When there is a performance issue with any application, a server admin tries to troubleshoot using the tools he has, and then passes on the issue to the storage admin to troubleshoot. This is reactive in nature when storage is concerned. Until an issue occurs there is no way to determine whether there are any issues on the

4 storage side. Also, it would be of greater advantage if the issue could be identified or diagnosed early when it is building up. Performance Advisor is the right management tool for identifying or troubleshooting performance issues on NetApp storage systems. Performance Advisor can be used to setup multi level thresholds and alerts that can warn you of any probable performance issue that is being built up or has already occurred. Counters that Matter There is a basic set of counters that a user can track in order to do regular performance monitoring of NetApp storage systems. When monitoring a storage system, an administrator would like to be alerted whenever I/O latency goes high for various protocols such as NFS, CIFS, iscsi, FCP or HTTP. CPU and disk utilization remain high for a prolonged period of time and From an application point of view, the above two points boil down to I/O response times how quickly the I/O requests from the application are being served. Note: Certain counters discussed below may be DIAG counters which are usually hidden in Performance Advisor. Also, some counters may be available in newer versions of Data ONTAP only. Basic Performance Management Once storage systems are deployed, they need to be constantly monitored for basic performance issues. The basic counters to start with are: Counter Threshold Comments cpu_busy 40 70% Gives the System CPU resource utilization in percent. If you are trying to manage only a single controller then start with 70%. If it s a HA pair, to take failover into account start with 50%. To determine if there is high cpu utilization due to lot of I/O, analyze system level counters such as total_ops (DIAG), read_ops, write_ops, net_data_recv & net_data_sent sys_avg_latency 20ms Gives the average latency for all operations in the system in milliseconds. This is a DIAG counter, needs DIAG counters to be enabled in PA disk_busy 90% Gives the percentage of time there was at least one outstanding request to the disk. Too busy disks may indicate possibilities of increased latency. For detailed analysis consider using disk level counters such as user_read_latency & user_write_latency. Using this counter may have performance impact as the number of

5 disks is considerably high in any environment. max_disk_busy 90% Gives disk utilization of the busiest disk in the system. Use this counter when disk_busy becomes a performance impact. Note: All values suggested are meant to be a starting point and the user needs to tune it according to his workload and environment. As a starting point, you can set the threshold interval to be around 300 seconds for all the counters that are discussed in this document. The next question is - how to troubleshoot once the above threshold values are breached? What other counters to look for next? For Protocols: To begin with, let s consider a scenario where sys_avg_latency remains higher than 20ms for more than 300s. In this situation, we should first look at the protocol latency counters to see where the possible issues could be. The table below suggests those protocol latency counters and their thresholds. The latency values mentioned are from the storage system perspective only, and it does not include latency in the network. Counter Threshold Value Comments cifs_latency 20ms Gives the average latency for CIFS operations in milliseconds. You can dig deeper to see if it s a read or write latency by monitoring cifs_read_latency or cifs_write_latency counters. nfsv3_avg_op_latency 20ms Gives the average latency of the nfs V3 ops. Use the nfsv3_read_latency, nfsv3_write_latency and nfsv3_ops for further troubleshooting. Similarly for NFSv4 counters. nfsv4_ops_latency 20ms Gives average latency for NFS v4 operations in milliseconds. For further analysis use counters such as nfsv4_ops, nfsv4_read_ops and nfsv4_write_ops. iscsi_latency 20ms Gives average latency for ISCSI operations in milliseconds. Drill down further for detailed trouble shooting using counters such as iscsi_ops, iscsi_read_latency & iscsi_write_latency (latter two being DIAG counters) fcp_latency 20ms Gives average latency for FCP operations in milliseconds. Analyze further if needed using fcp_ops, fcp_read_latency & fcp_write_latency (latte two being DIAG counters) Note: All values suggested are meant to be a starting point and the user needs to tune it according to his workload and environment.

6 Now, let s consider from an application perspective. If you are running home directories using either CIFS or NFS, then you can use the thresholds for CIFS latency and NFS latency, as mentioned in the above table, coupled with few volume level counters. Counter Threshold Comments volume:cifs_latency 20ms volume:nfs_latency 20ms Gives the average time for CIFS operations on the volume. For further analysis to determine the volumes on which the latencies are high, consider using volume level counters such as cifs_ops, cifs_read_latency & cifs_write_latency Gives average time for NFS operations on the volume. For further analysis to determine the volumes on which the latencies are high, consider using few volume level counters such as nfs_ops, nfs_read_latency & nfs_write_latency. Note: All values suggested are meant to be a starting point and the user needs to tune it according to his workload and environment. Thresholds considered in above two tables consider FC/SAS type of disks. SATA disks can have thresholds up to 40ms. Microsoft applications such as Exchange and SQL Server have their own recommendations on what the disk performance should be. The table below contains mappings between their recommendations and NetApp storage performance counters. The counters for Exchange Server were chosen from the TechNet Library for Exchange Server For Exchange Server: Exchange Counter Counter Threshold Comments LogicalDisk(*)\Avg. Disk sec/transfer Data Disk: LogicalDisk(*)\Avg. Disk sec/read PhysicalDisk(*)\Avg. Disk sec/read Data Disk: LogicalDisk(*)\Avg. Disk sec/write PhysicalDisk(*)\Avg. Disk lun:avg_latency 20ms For healthy disks, this counter shows approximately 20 ms. Counter values larger than 20 ms, or with large spikes, indicate a possible disk issue (for example, failure or slow speed). lun:avg_read_latency 20ms Shows the average time, in milli seconds, of a read of data from the disk. Should be below 20 milliseconds (ms) at all times on average. lun:avg_write_latency 100ms Shows the average time, in milli seconds, of a write of data to the disk. Should be below 100 ms at all times on average.

7 sec/write Log Disk: LogicalDisk(*)\Avg. Disk sec/read Log Disk: LogicalDisk(*)\Avg. Disk sec/write TEMP/Page file disks: LogicalDisk(*)\Avg. Disk sec/read lun:avg_read_latency 20ms lun:avg_write_latency 10ms lun:avg_read_latency 10ms If disk writes are high, it is possible that read latencies are also affected as a direct correlation with high write times. Shows the average time, in milli seconds, of a read of data from the disk. Should be below 20 ms on average. Shows the average time, in milli seconds, of a write of data to the disk. Should be below 10 ms on average. Shows the average time, in milli seconds, of a read of data from the disk. Should be below 10 ms on average. TEMP/Page file disks: LogicalDisk(*)\Avg. Disk sec/write lun:avg_write_latency 10ms Shows the average time, in milli seconds, of a write of data to the disk. Should be below 10 ms on average. Note: All values suggested are meant to be a starting point and the user needs to tune it according to his workload and environment. For SQL Server: SQL Counter Counter Threshold Comments Avg. Disk Sec/Read lun:avg_read_latency 20ms Gives the average time, in seconds, of a read of data from the disk. Avg. Disk Sec/Write lun:avg_write_latency 20ms Gives the average time, in seconds, of a write of data to the disk. Physical Disk: %Disk Time No disk time counter for LUNs 50% Gives the percentage of elapsed time that the selected disk drive was busy servicing read or write requests. Avg. Disk Reads/Sec lun:read_ops Gives the rate of read operations on the disk. Ensure that this number is less than 85 percent of the disk capacity. The disk access time increases exponentially beyond 85 percent capacity. Avg. Disk Writes/Sec lun:write_ops Gives the rate of write operations on the disk. Ensure that this number is less than 85 percent of the disk capacity.

8 The disk access time increases exponentially beyond 85 percent capacity. Avg. Disk Queue lun: queue_depth_lun 2 Gives the average number of physical read and write requests that were queued on the selected physical disk during the sampling period. If your I/O system is overloaded, more read/write operations will be waiting. If your disk queue length frequently exceeds a value of 2 during peak usage of SQL Server, you might have an I/O bottleneck. Note: All values suggested are meant to be a starting point and the user needs to tune it according to his workload and environment. For Oracle: Now, let s consider another huge application Oracle. For Oracle we can try to define the below counters and set thresholds starting with the below values and then tune it based on your requirement. Counter Threshold Comments lun:avg_read_latency 20ms Average read latency in milliseconds for all operations on the LUN lun:avg_write_latency 17ms Average write latency in milliseconds for all operations on the LUN Average latency of read operations observed over all LUNs in the system accessed over FCP in milliseconds. fcp:fcp_read_latency 20ms Use this counter if you are hosting a database on FCP LUNs Average latency of write operations observed over all LUNs in the system accessed over FCP in milliseconds. fcp:fcp_write_latency 17ms Use this counter if you are hosting a database on FCP LUNs Average latency of read operations observed over all LUNs in the system accessed over iscsi in milliseconds. iscsi:iscsi_read_latency 20ms Use this counter if you are hosting a database on iscsi LUNs. Average latency of write operations observed over all LUNs in the system accessed over iscsi in milliseconds. iscsi:iscsi_write_latency 17ms Use this counter if you are hosting a database on iscsi LUNs. Average latency for NFSv3 read operations in nfsv3:nfsv3_read_latency 20ms milliseconds. Use this counter if you are hosting a database over an NFSv3 export. nfsv3:nfsv3_write_latency 17ms Average latency for NFSv3 write operations in milliseconds. Use this counter if you are hosting a

9 database over an NFSv3 export. Average time for reads to the volume. Can be used for volume:read_latency 20ms detailed troubleshooting to find out which volume is causing latencies seen by the application. Average time for writes to the volume. Can be used for volume:write_latency 17ms detailed troubleshooting to find out which volume is causing latencies seen by the application. Average time for NFS writes to the volume. Can be volume:nfs_write_latency 17ms used for detailed troubleshooting to find out which volume is causing latencies seen by the application. Average time for NFS reads to the volume. Can be volume:nfs_read_latency 20ms used for detailed troubleshooting to find out which volume is causing latencies seen by the application. Average time for block protocol reads from the volume. volume:san_read_latency 20ms Use this counter for detailed analysis when hosting a database over SAN infrastructure Average time for block protocol writes to the volume. volume:san_write_latency 17ms Use this counter for detailed analysis when hosting a database over SAN infrastructure Note: All values suggested are meant to be a starting point and the user needs to tune it according to his workload and environment. Alert at various levels Setting alerts to go off only when one specific threshold is breached might be too late for any proactive steps that you could possibly take. Hence, you may want to be alerted at the initial stages when there are some hints towards performance degradation. To understand this better, let s consider a CIFS home directory service based on a SLA the latency for home directories should not exceed 20ms for more than 5 mins. This kind of SLA may need you to be alerted when latency hits 15ms for more than 10mins, so that if there are any issues with the underlying storage subsystem, you can take corrective actions and not violate the SLA. For situations like these, you can setup multi-level alerts by setting up 3 alerts at various threshold values, as shown in the table below. Alert Level Warning Error Critical Threshold cifs_latency > 15ms for 60s cifs_latency > 20ms for 120s cifs_latency > 20ms for 180s Performance Base Line One of the key features that were delivered in Performance Advisor 4.0 is the base-lining feature. This feature helps in base-lining an environment and suggests threshold values based on 4 metrics mean, min, max and percentile. Use the suggested value to set your thresholds with a few clicks. Let s see how you can achieve this.

10 Install Operations Manager 4.0 in your environment and allow Performance Advisor (PA) to collect all performance data from all storage systems deployed. PA begins to collect all performance counters from storage systems and stores it in its database. Allow this to happen for a sufficient period of time within which you have reached your high, low and normal workloads. Consider an example to baseline CPU utilization. Using PA, open the CPU Utilization chart as shown below. Click on the View Actions button on top and choose Metrics.

11 Once you click on Metrics in the above shown menu, you ll now reach the below screen. Click on Metrics Settings to choose the available metrics. Once you have selected the metrics, click OK.

12 Click on Filter Settings to define the period during which the workload was optimum and you would want thresholds to be set based on this workload duration. Now choose the cpu_busy counter and the metric of interest, in this case percentile. PA will now plot the 90% horizontal line on the graph, indicating the baseline.

13 Now to set a threshold on this base-line, Right-click and choose Add threshold and the counter cpu_busy. This brings up the Add Threshold Wizard. Click on Suggest Value to get values based on various time periods from last 1Hr to 1 Year. You can also set custom time periods to obtain the suggested value based on the base-lined environment as shown in below.

14 Choose OK then Finish to finish the wizard and complete the process of adding a threshold. Enable DIAG counters To enable DIAG counters on Performance Advisor, run below CLI on the DataFabric Manager server. dfm option set perfadvisorshowdiagcounters=yes To enable data collection for a particular DIAG counter if they are not getting collected by default, use the below CLI, dfm perf data enable [-p] <host-name-or-id> <perf-object-type> { all <counter-name>...}

15 External References Troubleshooting Performance Problems in SQL Server Exchange - Monitoring Mailbox Servers NetApp provides no representations or warranties regarding the accuracy, reliability or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customer s responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document NetApp, Inc. All rights reserved. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, DataFabric, Data ONTAP, MultiStore, NearStore, NOW, SnapMirror, Snapshot, SnapVault, and vfiler are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. Intel and Pentium are registered trademarks and Xeon is a trademark of Intel Corporation. Linux is a registered trademark of Linus Torvalds. VMware is a registered trademark of VMware, Inc. Microsoft and Windows are registered trademarks of Microsoft Corporation. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.