Taking the Guesswork out of Networked Storage Performance and Capacity Management

Enterprise Storage Taking the Guesswork out of Networked Storage Performance and Capacity Management Hitachi HiCommand Tuning Manager Software An Application Brief By Steve Smith January 2005

Executive Summary To support company competitiveness and customer satisfaction, a chief goal of every IT organization is to ensure top performance by detecting and resolving potential problems before they occur. Historically, IT organizations have tried to manage performance and capacity with data-gathering processes involving a combination of command-line scripts, custom applications, and/or vendor-supplied tools to capture and enter performance and capacity data into spreadsheets. The typical spreadsheet process, while helpful, cannot easily produce historical performance or capacity trends, or forecast future trends for each application s resources. This goal requires a tool that allows you to define and continuously monitor and store critical performance and capacity thresholds, and send a warning to the proper persons to alert them of a potential problem before it occurs. Hitachi HiCommand Tuning Manager software is an advanced performance and capacity management solution that was developed to solve these challenges. This application brief examines a sampling of the basic reporting, analyzing, forecasting, and troubleshooting capabilities of Tuning Manager software. This solutions ability to query a historical database for performance and capacity trend analysis on each component of your SAN can allow you to easily correlate the current changes in performance with recent changes to the physical configuration, software, workload, or other environmental changes that may be causing changes in an application s performance. Performance management and capacity planning are essential to corporate competitiveness, agility, and revenue generation. When you can manage your networked storage infrastructure to the highest levels of capacity utilization and performance, you may be able to defer hardware and software upgrades, which, in turn, can free up money for other investments.

Contents The Challenge... 1 Hitachi HiCommand Tuning Manager Software... 2 Understand Your Current Environment... 3 Example Reports... 6 Capacity Reports...7 Performance Reports...7 Problem Avoidance with Tuning Manager Alerts...11 Implementing an Alert...13 Testing Alerts...15 Solving Performance Problems with Tuning Manager Software...16 Performance Problem Scenario...17 Summary...22

Taking the Guesswork out of Networked Storage Performance and Capacity Management Hitachi HiCommand Tuning Manager Software An Application Brief By Steve Smith The Challenge Today s complex IT environments often include dozens or even hundreds of application and database servers and various operating systems connected to terabytes of data, which resides on several classes or tiers of storage systems. Often the servers and storage are located on one or more storage area networks (SANs) connected by SAN switches and/or storage directors with hundreds of interconnected switch ports. Just knowing what assets are installed, where or how much of these resources are being utilized and by whom, and when more processing capacity will be required, is a goal that few organizations have fully achieved without countless hours of manual data collection and analysis. And even then, the analysis is often based on incomplete or obsolete information. With competitive pressures driving shorter application development cycles and frequent organizational changes including mergers and acquisitions, IT departments cannot afford to guess when upgrades will be required to accommodate sudden changes in IT service requirements. Conversely, over-budgeting and over-configuring to compensate for performance and capacity planning guesswork is wasteful and unacceptable. These challenges have driven organizations to physically and/or logically consolidate systems, storage, and even data centers, in an effort to reduce complexity and costs, improve asset utilization and service levels, and reduce the risk of application downtime. Although these strategies may help, there are still fundamental challenges that can only be solved with the right software tools and sound IT management processes by an adequately trained staff. There is an old expression that states, If you can t measure it, you can t manage it. The task of efficiently managing IT infrastructure complexity is further exacerbated because each server, operating system, database, SAN switch, and storage system has its own monitoring and reporting software tool, each with a different user interface, a different data repository, and a new set of skills to learn. Historically, those organizations that have tried to manage performance and capacity have used a data-gathering process involving a combination of command-line scripts, custom applications, and/or vendor-supplied tools to capture and enter performance and capacity data into spreadsheets. The data is manually manipulated and analyzed to produce reports that may not be a complete picture of the environment or may not be current by the time the report is produced. And, although a manual data gathering process may help to view performance or capacity at a point in time, it will not help resolve performance problems in real time when they occur or automatically notify you when critical thresholds are near. 1

A real-time software monitor can view the current state of the host, file system, database, SAN, and storage resources that are being used by an application and allow you to contrast that information with a historical view of the normal behavior or baseline performance of those resources during past periods of acceptable performance that were previously stored in the monitor software s database. The ability to query a historical database for performance and capacity trend analysis on each component of the SAN can allow you to easily correlate the current changes in performance with recent changes to the physical configuration, software, workload, or other environmental changes that may be causing changes in an application s performance. The goal of every IT organization is to detect and resolve potential problems before they occur. Here again, the spreadsheet process cannot easily produce historical performance or capacity trends, or forecast future trends for each application s resources. This goal requires a tool that allows you to define and continuously monitor and store critical performance and capacity thresholds, and send a warning to the proper persons to alert them of a potential problem before it occurs. Hitachi HiCommand Tuning Manager software is an advanced performance and capacity management solution that was developed to address these challenges. Hitachi HiCommand Tuning Manager Software Tuning Manager software is a solution that monitors, reports, and forecasts storage performance and capacity in the context of the entire storage network, including hosts, file systems, database applications, SAN switches, and the internal disk storage components of all tiers of Hitachi storage systems. Tuning Manager software periodically discovers the capacity of storage devices, file systems, hosts, databases and network attached storage (NAS) devices. It then consolidates, analyzes, and reports on current performance, capacity, utilization, historical and future usage trends, and notifies you of any predetermined threshold violations. Tuning Manager software provides the information you need to make informed decisions and more efficiently operate your Hitachi storage environment while taking the guesswork out of planning and budgeting for future growth. Tuning Manager reports, charts, and other metrics help you to: : : Replace risky guesswork with fact-based decisions : : Identify all Hitachi storage systems on the network and their current performance, capacity, and utilization : : Determine how many and what kind of servers exist on your whole network and its subnetworks, and the storage they are consuming : : Determine which storage systems are either under- or over-utilized or under- or over-allocated : : Quantify file systems total capacity, amount used, and amount remaining : : Detect and prevent capacity shortages with proactive alerts when critical thresholds are violated : : Detect and prevent potential performance bottlenecks with alerts when predetermined thresholds are violated : : Isolate the root cause of application response time problems by monitoring and reporting the performance of all components in the path from the host and database application to the disk parity group 2

: : Determine when to acquire additional storage capacity : : Understand the relationship between the host s file systems, logical devices, and corresponding physical storage components : : Understand the RAID configuration of the storage system associated with a host s file systems Understand Your Current Environment One of the early steps to proactively managing your storage capacity and performance with Tuning Manager software is to decide which metrics and thresholds to monitor in order to fully understand your particular environment. Tuning Manager software collects, stores, and analyzes hundreds of metrics that are available to you in the standard reports, as well as for developing customized reports and analysis. The metrics fall within the following broad categories and most are available for real-time viewing, historical trending, forecasting, and troubleshooting. The list also includes some examples of the metrics available in each category. : : Storage Systems. Average/minimum/maximum Read Input Output Operations per Second (IOPS); Write IOPS; Read MB/sec and Write MB/sec transferred for total storage system, for each storage port, each disk parity group, or each logical device (LDEV); and the cache capacity and cache percentage utilized : : Servers. Storage capacity allocated and percentage capacity used/percentage capacity free, I/O queuing time; I/O service times, IOs per second, paging operations, swapping, memory used, percent kernel mode, percent user mode, wait time : : File Systems. Capacity MB/used MB, capacity used percentage/capacity free percentage, growth rate, Read IOPS, Write IOPS, Read MB/sec, and Write MB/sec : : Device Files. Average service time, average wait time, queue lengths, Read IOPS percentage/write IOPS percentage, Read MB/sec transferred, Write MBs/sec transferred : : SAN Switches. Link failures, loss of signal count, CRC errors, buffer credits, bytes/sec received/transmitted, frames/sec received/transmitted : : Oracle Database Instances. : : SQL Server Database Instances. Tuning Manager software lets you view and analyze capacity and performance of all storage systems and their ports, logical devices, and disk storage groups as viewed from the perspective of their associated servers, databases, and file systems. However, a complete picture also requires the ability to view all SAN components from the reverse direction. Tuning Manager software also lets you view each server, database, and file system from the perspective of their associated storage system, port, logical device, or disk storage group. The illustrations below provide some examples of the various views available to you. The left side of the screen in Figure 1 shows the Tuning Manager Resource Tree, a hierarchical display with successive levels of detail about your network of servers, its subnetworks, hosts, storage, and applications. As you select each level in the tree, the metrics and charts about the resource and its subresources are displayed in the Information Frame in the right side of the screen. In the Information Frame you can select either Capacity or Performance Information. 3

Figure 1. HiCommand Tuning Manager Resource Tree Tuning Manager software provides a hierarchical display of information regarding your network of servers, its subnetworks, hosts, storage, and applications. In Figure 2, after displaying a storage system from the Storage level of the Resource Tree with the Performance option, Tuning Manager software displays a performance summary in the information frame at the top of the screen and also offers the option to drill down into the detailed performance information of the Storage System Ports, the Disk Array Groups, or the Logical Disks for the selected storage system. To select the subresource type you want to see, just click on the Ports, Array Groups, or Logical Disks link in the subresources area and a performance report is displayed for the selected subresource. 4

Figure 2. Storage System Performance Summary When viewing Tuning Manager software s performance summary, you gain the options drill down into the detailed performance information of the Storage System Ports, Disk Array Groups, or Logical Disks for the selected storage system. Using the HiCommand Tuning Manager Resource Tree and the many hyperlinks that are embedded in the reports allows you to navigate through all of the resources in the network in either direction, up or down, or view the capacity or performance of each resource. As an example, you can display a Disk Array Group in a particular storage system and then find and display the performance and capacity of all of the hosts, file systems, or Oracle instances associated with that particular Disk Array Group. Conversely, you can select a host, file system, or an Oracle instance, and navigate down to display all of the storage, ports, logical devices, or disk array groups associated with that particular host, file system, or Oracle instance. This bi-directional view of interdependencies is crucial for problem diagnosis and resolution. It s also invaluable when reconfiguring or consolidating your environment, or for any other activity that requires a thorough understanding of the component interdependencies within your storage infrastructure. As an example, consider diagnosing an Oracle application performance problem with Tuning Manager software, where the Oracle application s server resources and the SAN switch fabric have already been eliminated as the root cause of the bottleneck. With Tuning Manager software you then discover a disk parity group that has an unusually high amount of I/O activity as measured in IOPS or MB/sec. A simple click on the List Connected Servers button on the Tuning Manager screen as shown in Figure 3, will display a detailed summary of the I/O consumption for each server and device file that is accessing the disk parity group in question. By correlating the time period that the IOPS increased on one server, with the decline in performance on the Oracle application server, you can conclude that the server access contention for this disk parity group is causing the performance problem. This bi-directional view is essential when trying to determine the root cause of an application performance problem. A more detailed illustration of a similar troubleshooting scenario is illustrated later in this paper. 5

Figure 3. List Connected Servers The Connected Servers Performance view accessed from Advanced Information section of the Tuning Manager software screen displays a detailed summary of the I/O consumption for each server and device file that is accessing the disk parity group in question. Example Reports This section provides some examples of the standard Performance and Capacity reports that are provided as part of the Tuning Manager solution sets that accompany the base product. All of the standard reports can be easily modified. Additionally, the Tuning Manager report generator provides a simple, menu-driven method to develop your own custom reports and analysis by accessing the hundreds of real-time and historical metrics in the Tuning Manager data stores. The resource levels that can be viewed and analyzed are listed below. The reports can be requested for a current real-time snapshot, any past point in time, or as a historical trend analysis over a specified time period. The historical data that is stored in the Tuning Manager database repository can also be used as input for the Tuning Manager reports that forecast future performance or capacity trends. All of these reports can be customized and printed or exported for use with other software tools. Below is a partial list of the standard reports that are provided with Tuning Manager software. More than 150 different reports are provided with the Tuning Manager base product. 6

Capacity Reports : : All servers on the network all servers displayed in a tree structure, including subnetworks, servers, and file systems : : All servers on a subnetwork(s) : : Selected individual servers : : File systems : : Top 10 file systems with the least amount of capacity remaining : : Device files : : Oracle total : : Oracle instance : : Oracle tablespaces : : SQL Server database Performance Reports : : All servers on the network all servers displayed in a tree structure, including subnetworks, servers, and file systems : : All servers on a subnetwork(s) : : Selected individual servers : : File systems : : Device files : : Top 10 devices with the longest average service times : : All storage on the network : : Selected storage systems, including logical disks, array groups, and ports : : Storage ports : : Top 10 ports with the highest IOPS : : Top 10 ports with highest MB/sec transferred : : Disk array groups : : Storage logical devices : : SAN fabric : : SAN switch : : Switch ports : : Oracle applications 7

: : Oracle instances : : Oracle tablespace : : SQL Server database Figure 4 is an example of the File System Capacity report for all of the servers on the network and Figure 5 is an example of the File System Capacity report for all of the file systems on the Mercury server. The columns in each report can be sorted in descending or ascending sequence by simply clicking on the column heading so that you can quickly identify the file systems with the most or the least capacity available or utilized. In Figure 5, the two graphs at the bottom of the screen in the Favorite Charts window contain pre-selected capacity forecast graphs that will be displayed whenever the File System Capacity report is displayed. All of this data can be exported to a spreadsheet for further manipulation by other software applications. Figure 4. File System Capacity Whole Network Tuning Manager software enables a view of capacity for all of the servers on the network. Figure 5. File System Capacity Subresources The subresource view provided by Tuning Manager software allows you to observe capacity for all the file systems on a specific server. 8

Tuning Manager software provides the option to view reports in several different formats, including lists, graphs, and tables. Figure 6 illustrates an example of a bar chart report that shows the capacity and usage of three different Oracle Database Instances at a particular point in time. The example in Figure 7 illustrates a line chart that tracks the capacity and utilization history of the same three Oracle Instances over a period of several hours. The time period reported in this example is short (four hours) for illustration purposes only. In normal practice you might want to run this report to measure a much longer period for historical trend analysis, perhaps several weeks or a month. Figure 6. Oracle Instances Point-in-Time Capacity This sample Tuning Manager bar chart report shows the capacity and usage of three different Oracle Database Instances at a particular point in time. 9

Figure 7. Oracle Instances Capacity in Hours Timeframe The example in Figure 7 illustrates a line chart that tracks the capacity and utilization history of the same three Oracle Instances over a period of several hours. Figure 8 shows an example of the Storage Performance History report. When troubleshooting performance problems it is necessary to identify exactly when the changes in the performance behavior of an application started to occur. Many performance problems can be traced back to a change in the software or hardware infrastructure or a change in the application s transaction workload. Using Tuning Manager software s historical repository of performance and capacity information you can easily correlate any changes to the application s infrastructure or transaction workload with the changes in the application s performance behavior. You can use Tuning Manager software to capture a baseline performance profile of an application during normal, acceptable levels of performance. These baseline measurements can then be used later as part of a troubleshooting or trend analysis process to help identify which resource(s) are performing outside of their normal range in the baseline measurements. The example in Figure 8 also illustrates the ability to view the reports in a spreadsheet format, in a graph format, or both. The top of Figure 8 shows a line chart of the Cache Usage, and there are also buttons on the top of the screen to select Storage Port or Disk performance graphs. The previous was just a sampling of more than 150 standard reports that are provided with Tuning Manager software. The reports can be customized, or you can build your own reports with the report generator that is provided. Once you understand Tuning Manager software s reporting capabilities, you can begin to develop a plan for the ongoing monitoring of your key capacity and performance management metrics. 10

Figure 8. Storage Performance History Tuning Manger software offers the choice of graph or spreadsheet (or both) views of application performance profiles. These baseline measurements can then be used later as part of a troubleshooting or trend analysis process to help identify which resource(s) are performing outside of their normal range in the baseline measurements. Problem Avoidance with Tuning Manager Alerts With Tuning Manager software, critical thresholds can be automatically monitored so that the proper personnel are notified when these thresholds are breached. Additionally, you can provide pre-defined actions that can be automatically executed well in advance of potential problems, thus avoiding application interruptions and dissatisfied end users or customers. The Tuning Manager software agents continuously monitor hundreds of metrics about your entire storage environment, including 11

performance and capacity of servers, Oracle and SQL Server databases, file systems, SAN switches, and the internal components of Hitachi storage systems. Figure 9 provides an actual example of the Current Alerts List, a report that displays a list of current outstanding alerts. In addition to being automatically notified for each Alert, the system administrator can view this report at any time to see the current status of all alerts. The information displayed in the Current Alerts List is shown in Table 1. Table 1. Current Alerts List Information Display Column Date Resource Data Category Threshold Status Message Description Date and time the alert condition occurred The resource associated with the alert The resource type responsible for generating the alert : : Server : : File system : : Device file : : Storage system : : Storage port : : Array group : : Logical disk : : Oracle Instance : : TableSpace : : Datafile : : Switch : : Switch port The threshold condition that triggered the alert, according to the value in the Status field The status of the resource at the alert time : : Red: critical : : Yellow: warning : : Green: OK The value of the metric that triggered the alert 12

Figure 9. Current Alerts List The Tuning Manager agents continuously monitor hundreds of metrics about your entire storage environment and the software produces a Current Alerts List, a report that displays a list of current outstanding alerts, such as the critical status of resources shown in this sample. Implementing an Alert In order to avoid unwanted performance threshold alerts during predictable bursts of activity, or to filter out periodic spikes in activity of very short duration, Tuning Manager software offers several ways of controlling when and how an alert condition will be evaluated. In the Alert Setting 1 window shown in Figure 10, you are presented with the option to set specific times to monitor an alert, perhaps only during critical processing periods, or to monitor the alert at all times. In order to avoid triggering an alert for a very infrequent or brief spike, the Damping function allows you to specify the number of times a threshold must be reached within a specific time interval, before the alert is triggered. The fields in Alert Setting 1 display are defined as follows: : : Alert Name the unique name of this alert : : User Name the user account responsible for creating this alert : : Modification Date the last time this alert was edited : : Period of Watching : : Always Evaluated when checked, the alert condition will be monitored constantly : : From restricts alert monitoring to a finite starting time : : To restricts alert monitoring to a finite ending time : : Damping : : Damping Enabled when checked, an alert is triggered only when multiple instances occur within a finite time period : : Occurrence(s) the number of times an alert condition must occur within a specified time interval before the alert is triggered 13

: : Intervals the number of successive samples to be collected. The alert is triggered only when the number specified in Occurrences is reached within the specified Intervals. For example: You specify that there must be at least six Occurrences of the specified alert condition within 10 successive samples or Damping Intervals. : : Data Category the resource type to be monitored Figure 10. Alert Setting 1 Tuning Manager software offers several ways of controlling when and how an alert condition will be evaluated. The Alert Setting 2 screen in Figure 11 is where you specify the actual thresholds that will trigger a Warning Alert and a Critical Alert. You can set a warning threshold when a critical resource is approaching an unacceptable level, at which point you may want to monitor it closely or take some preemptive action. An example might be the available free space in a critical file system or database that is below an acceptable level, or when the IOPS of a disk parity group has reached a level that will impact the performance of a business-critical application. Figure 11. Alert Setting 2 The actual thresholds that will trigger a Warning Alert and a Critical Alert can be specified on Tuning Manager software s Alert Setting 2 screen. 14

After you have defined the conditions that trigger an alert, the next step is to define the action to be taken when the alert condition is triggered. Tuning Manager alert actions can be any or all of the following: : : E-Mail. You can send a message to one or more people, notifying them of the alert condition. Tuning Manager software can display the metrics as part of the e-mail message by embedding one or more variables in the message. For example, the following message includes the actual metric variables in the message text. Alert: %%Hostname only has %%Free free out of a total capacity of %%Capacity : : SNMP. You can create an SNMP Trap so that an alert can be sent to another management application or console. : : Command. You can execute commands or predefined automation scripts that may take some predefined action to resolve or mitigate a problem. : : Event Log/Syslog. You can specify a message to be inserted in the server s system log or, in the case of Microsoft Windows, in the Event Log. An alert can also be sent to another management application or console with log monitoring capability. The final step in defining the Tuning Manager alerts is the use of the Bind function. Binding allows you to define an alert once for example, setting a threshold alert when a file system has less than 10 percent of its space available and then easily bind that alert definition to many other file systems. For example, you may want to apply that rule to all servers on a particular subnetwork because they all belong to a critical application. Similarly, you may want to set a threshold on the performance in IOPS or MB/sec transferred for all storage ports or disk array groups in one or more storage systems. Testing Alerts Tuning Manager software also provides the ability to easily verify your alert definition by testing the alert in advance. As shown in Figure 12, by clicking on Test in the rightmost column of the Alert that you want to test, you are presented with the alert activation window that is shown in Figure 13. Figure 12. Click on Test to Select the Alert that You Want to Test Binding allows you to define an alert once and then easily bind that alert definition to many other file systems. Tuning Manager software also provides the ability to easily verify your alert definition by testing the alert in advance. 15

In Figure 13, by filling in the value 12 in the box labeled Alert Test Condition Value, you can simulate the triggering of this alert by clicking on the test button. Since this Alert definition is supposed to trigger a Warning message when the Mercury server capacity exceeds 10GB, by simulating a 12MB capacity you can test whether the alert definition will be triggered properly. Figure 13. Alert Activation Test Window Tuning Manager software allows you to test your alert setup by entering a value that should prompt a Warning message. The result of the test is displayed in a popup status window in Figure 14. Figure 14. Alert Test Status A popup window displays the results of your alert setup test. Solving Performance Problems with Tuning Manager Software This application brief has examined a few of Tuning Manager software s powerful reporting, analyzing, and forecasting capabilities. This section provides practical examples, which depict Tuning Manager software s diagnostic and troubleshooting capabilities, in addition to its resource management capabilities. As shown in the previous section of this paper, the ability to set threshold alerts and actions will help you avoid as many of the performance problems as you are able to anticipate. However, in a complex networked storage environment it is difficult to anticipate and set alerts for every potential performance problem. An additional requirement for maintaining optimum service levels is the ability to resolve 16

performance problems quickly when they do occur. This requires a software tool that can quickly retrieve and display a snapshot view of the current problem environment, as well as a historical view of the same environment before the problem occurred. With the ability to rapidly view and compare the key performance metrics before and after a problem has occurred, you are able to quickly isolate the root cause. Consider the following example of a common type of performance problem and how Tuning Manager software can help isolate the root cause. Performance Problem Scenario High Performance Inc. (HPI), like many enterprises that are trying to reduce their storage management complexity and cost, has consolidated several of their Hitachi Thunder 9500 V Series modular and Lightning 9900 V Series enterprise storage systems into one Hitachi TagmaStore Universal Storage Platform. The Hitachi Storage Area Management Suite of software, including Tuning Manager software, has dramatically simplified HPI s storage management tasks by providing a single console to configure, monitor, and manage multiple tiers of Hitachi storage systems. The ability to directly attach the Thunder family and Lightning family systems to the Universal Storage Platform has enabled HPI to perform their consolidation in a phased approach, while protecting their investment. Like many other large businesses, the last few days of the month are usually the heaviest IT processing periods for HPI. This month however, during the morning of the heaviest processing period, the CEO of HPI asked the CFO to produce a series of financial reports to be delivered to the boardroom by the next morning. The reports required the use of a decision support software application that would mine the corporate database for the required information. Within 15 minutes after the data mining application was started, the help desk began to get phone calls from customers who were experiencing very poor response time and timeouts from the HPI customer service Web portal. The help desk was not aware of any reason for the problem and immediately notified the IT system administrator. The system administrator logged in to the Tuning Manager console to investigate the problem. The Tuning Manager software agents continually gather hundreds of performance metrics and store them for instant recall if needed. Since the problem symptom was a slow application, the first step was to investigate the server named Mars that was hosting the application in question. Figure 15 shows the Tuning Manager server performance summary report when the Mars server is selected from the Resource Tree. By clicking the Performance Reporter button at the bottom of Figure 15, the Performance Reporter function is launched and the server performance reports are displayed, as shown in Figure 16. 17

Figure 15. Application Server Performance Summary Select Application Server Mars in Navigation Frame to display performance summary for that server. Click on Port for Port Performance Report. Performance Reporter: Platform Use Tuning Manager software s Performance Reporter for detailed Analysis of Application Server Mars. Tuning Manager software enables you to choose and display a performance summary for a specific server and then drill down for a detailed analysis. The Performance Reporter function allows you to display either the standard Tuning Manager reports or customized reports with a simple mouse click. The Performance Reporter can analyze and report realtime or very recently collected data that is still resident in the local agent s data store, or it can report any past time period by querying the Tuning Manager database repository for historical information. Figure 16 shows a few of the server performance metrics that can be reported and analyzed. The reports clearly show that the server CPU Utilization is normal, the Available Memory report does not look unusual, and the memory page swap report does not show anything unusual. However, the Device File I/O Report shows that the I/O activity has slowed down considerably during the period when the application response time was reported to be very slow. This indicates that there is some correlation between a drop in I/O activity and the slow response time, and that the problem root cause is probably in the storage or I/O path. At this point you may want to view the SAN switch reports to rule out any problem in the SAN that may be causing the I/O slowdown. For brevity, this example will assume the problem is not in the SAN; however Tuning Manager software has a very extensive reporting capability that includes detailed performance and error reporting for SAN switches and for each individual port on each switch. Moving down the I/O path from the server and SAN with Tuning Manager software to find the root cause, the next step would be to examine the storage ports for any bottlenecks. In order to view the Storage Port Performance report go back to Figure 15 Server Application Performance Summary and click on the column heading titled Ports. The resulting display in Figure 17 is the Storage Port Performance Analysis. 18

Figure 16. Performance Reporter Server Performance CPU % Utilization report Available Memory report Memory page swap report Device File I/O report Device File I/O activity appears to have slowed down; problem may be in storage or I/O path. Tuning Manager software allows for the analysis of various server metrics, offering easily accessible reports on memory usage, device file I/O, and CPU usage. Figure 17. Storage Port Performance Analysis 1. View of ports related to a specific server, from server view. Storage View Server View 2. Select Port History to View recent activity. The Port I/O workload was higher for one of the storage ports. It may be cause of performance degradation of the Mars server. 3. The I/O activity on Port CL0-A increased while the Mars server throughput decreased. Tuning Manager reports allow you to compare activity and response times during an exact time period, as a first step toward diagnosing performance issues. In Figure 17, the reports indicate that during the exact time period that the Storage Port CL0-A activity increased sharply, the customer service application s transaction response time decreased sharply. This may indicate that the storage port s I/O activity is originating from another server that is sharing the storage port with the customer service application. 19

Tuning Manager software has a very easy way to investigate this very common but difficult to diagnose problem. As shown in Figure 18, on the Storage Performance report, you would select the Storage Port CL0-A from the Resource Tree and click the List Connected Servers button. The resulting display is a list of all of the servers and device files that are sharing Storage Port CL0-A. By clicking on a column heading you can sort each column ascending or descending sequence and quickly spot the servers that are generating the highest I/O activity during the period in question. Figure 18. List Connected Servers 1: Select port CL0-A and click List Connected Servers. 2: Sort the information based upon the individual server column to observe which server increased the port I/O load. Tuning Manager software can list all of the servers and device files sharing a specific storage port and sort them to provide a view of those generating the highest I/O activity. Apparently, in this example, MyServer3, which is sharing the same storage port as the customer service application server, is generating a significantly large amount of I/O activity. You can validate the assumption that MyServer3 is causing server bandwidth contention at the storage port, by going back to the Server Performance Report for MyServer3 and looking at the I/O Performance Report. As shown in Figure 19, the I/O Performance Report for MyServer3 clearly shows a sharp increase in I/O activity for MyServer3 at the same point in time customer service application was experiencing a sharp increase in application transaction time. It is now pretty certain that the root cause of the problem is that the two servers are experiencing access contention for the same storage port. There are several ways to solve such a problem. Before discussing a solution, it should be pointed out here that with Tuning Manager software you can correlate approximately at what IOPS or MB/sec threshold that the application response time started to slowed down; then you can use those metrics to set new alerts with Tuning Manager software to help prevent a reoccurrence. 20

Figure 19. Storage Port Analysis 1: Analyze the MyServer3 performance status with the Performance Reporter. 2: The I/O Performance Report of MyServer3 shows sharp increase. This bandwidth contention is probably causing the performance problem on the Mars server. Performance Reporter: Platform Tuning Manager software provides an I/O Performance Report, which offers and at a glance look the selected server s performance levels over a period of time. One solution to this problem is to reconfigure the servers so that they no longer share the same storage port. Another way is to use Hitachi Server Priority Manager software (formerly Priority Access) to set a lower access priority to MyServer3. Also, an extension to this problem scenario is a situation in which the above investigation did not show any significant server access contention for any storage port. Tuning Manager has the ability to probe deeper into the Hitachi storage systems, down to the disk array group performance metrics. The next logical step in this scenario would be to display the Disk Array Performance reports and examine them for the same type of access contention that was examined for the storage ports. This example has presented just one of many ways that Tuning Manager software can help you resolve difficult performance problems quickly. Additionally, you can avoid repeating the same problems because you can set threshold alerts more precisely by analyzing at what performance threshold the problems originated. 21

Summary Performance management and capacity planning are essential to corporate competitiveness, agility, and revenue generation. When you can manage your networked storage infrastructure to the highest levels of capacity utilization and performance, you may be able to defer hardware and software upgrades, which, in turn, can free up money for other investments. In today s real-time, 24/7 economies, customer satisfaction and your company s competitiveness often depend on application response time, and if it is too slow too often, the result may be lost customers. IT managers and storage administrators need to track performance and utilization trends and predict future requirements to ensure that upgrades are budgeted before IT service levels are impacted. When problems occur they must be resolved quickly and prevented from repeating. All of these critical business needs can be addressed by HiCommand Tuning Manager software. Only a small portion of Tuning Manager software s basic reporting, analyzing, forecasting, and troubleshooting capabilities have been examined in this application brief. For more information regarding Tuning Manager software and other related Hitachi software solutions, you are encouraged to visit the Hitachi Data Systems corporate Web site www.hds.com or to call your local Hitachi Data Systems representative. 22

Hitachi Data Systems Corporation Corporate Headquarters 750 Central Expressway Santa Clara, California 95050-2627 U.S.A. Phone: 1 408 970 1000 www.hds.com info@hds.com Asia Pacific and Americas 750 Central Expressway Santa Clara, California 95050-2627 U.S.A. Phone: 1 408 970 1000 info@hds.com Europe Headquarters Sefton Park Stoke Poges Buckinghamshire SL2 4HD United Kingdom Phone: + 44 (0)1753 618000 info.eu@hds.com Hitachi Data Systems is registered with the U.S. Patent and Trademark Office as a trademark and service mark of Hitachi, Ltd. The Hitachi Data Systems logotype is a trademark and service mark of Hitachi, Ltd. HiCommand is a registered trademark of Hitachi, Ltd. TagmasStore, Thunder, and Lightning are trademarks of Hitachi Data Systems Corporation. All other product and company names are, or may be, trademarks or service marks of their respective owners. Notice: This document is for informational purposes only, and does not set forth any warranty, express or implied, concerning any equipment or service offered or to be offered by Hitachi Data Systems. This document describes some capabilities that are conditioned on a maintenance contract with Hitachi Data Systems being in effect, and that may be configuration dependent, and features that may not be currently available. Contact your local Hitachi Data Systems sales office for information on feature and product availability. Hitachi Data Systems sells and licenses its products subject to certain terms and conditions, including limited warranties. To see a copy of these terms and conditions prior to purchase or license, please go to http://www.hds.com/products_services/ support/warranty.html or call your local sales representative to obtain a printed copy. If you purchase or license the product, you are deemed to have accepted these terms and conditions. 2005, Hitachi Data Systems Corporation. All Rights Reserved. WHP-170-01 January 2005