E N T E R P R I S E M A N A G E M E N T Windows NT Performance Monitor A Practical Approach By Paul Del Vecchio The Dell Application Solution Center uses several workload and benchmarking tools to stress, monitor, analyze, and tune solutions during customer and independent software vendor engagements. This article describes how to get the most out of one of their more popular choices, Windows NT Performance Monitor (Perfmon). Windows NT Performance Monitor (Perfmon) may be the single most powerful yet most underutilized feature of the Windows NT operating system. Perfmon is widely used by the performance benchmarking community, which relies on the tool daily to identify system and applicationlevel performance issues. However, most end users either ignore the tool completely or fail to make full use of its tremendous potential. IT professionals often fail to fully realize the value Perfmon can add by enabling them to perform advanced performance analysis on their Windows NT infrastructures. Note: This article describes the operation of Perfmon in a Windows NT context. According to Microsoft, Perfmon.exe in Windows 2000 looks and operates like Performance Monitor in Windows NT. Getting Started with Perfmon To launch Perfmon, simply type start perfmon at the command prompt. You can also invoke Perfmon by navigating to Start Menu > Programs > Administrative Tools > Performance Monitor or Start Menu > Run > Perfmon. After you have launched Perfmon, you must select a target system, or computer, to monitor. This system can be the localhost from which you launched Perfmon or another Windows NT system on the local network. Because of the overhead associated with running Perfmon, it is recommended that Perfmon run on a remote computer while scrutinizing production servers for performance issues across a network. To select a system, click on the plus sign icon in the Perfmon toolbar. This action invokes a network browser showing localhost as the default monitoring target. After you have selected the system you wish to monitor, choose an object, or subsystem, to monitor. These include such components as system, memory, network interfaces, or disk I/O subsystems. Next, choose the counters you wish to monitor. A counter is an indicator of a subsystem that monitors performance that can be quantitatively measured, such as the percent CPU time, pages in and out per second for www.dell.com/powersolutions PowerSolutions 95
Figure 1. Perfmon Host, Object, and Counter Selection Dialog Box virtual memory, or packets sent and received per second for the network interface. Figure 1 shows the Perfmon dialog box to select hosts, objects, and counters. After selecting the target system, its objects, and counters to monitor, you can run Perfmon in any of three modes: Chart mode graphically displays information on the counters of a selected object and allows you to monitor the system in real time Log mode collects all counter data from selected objects, allowing you to analyze the logged data later Report mode raw data from counters that can be easily imported to spreadsheets like Microsoft Excel for the creation of graphs and charts Getting the Most Out of Perfmon A common mistake among Perfmon users is failing to fully utilize the tool. Of the three modes, most users run Perfmon in the default mode (chart) and never explore what the other two modes do. Charting in real-time mode poses two significant limitations: first, data is not captured, but merely viewed for a short time before it simply falls off the display; and second, detailed analysis of on-screen data in real time is difficult, if not impossible. The Log and Report modes overcome these limitations by allowing the user to capture the data rather than simply viewing it for a limited time interval. Chart Mode Provides a Quick Glance In chart mode, Perfmon displays information on a selected object s counters in either a horizontal graph (default) or vertical histogram. The default graph view is usually adequate; however, the histogram may be useful for comparing the values of many instances, such as the thread state of many threads belonging to a single process. The horizontal graph fails to present this type of comparison in a way that is easily digestible and understandable to the user. In chart mode with the vertical histogram view, you can adjust the sampling rate of Perfmon, by either increasing or decreasing the rate at which the tool samples and displays data. The default rate is one second; that is to say, samples are taken and displayed at one-second intervals. The vertical histogram offers only real-time information from the last sample taken. Once the counter changes value, the previous data is gone and cannot be retrieved: Perfmon does record an average, but over a limited period of time. In chart mode with the horizontal graph view, you can monitor the levels of specific counters in real time and also receive history information. The amount of history available is extremely limited, however, because the display size is relatively small. In either case, chart mode is not intended to capture data. Its main purpose is to give the user a quick glance at a few metrics over a short period; it is not intended to provide in-depth monitoring capability over time. Log Mode Captures Information In chart mode, you can monitor the system in real time. Even if you decrease the sample rate from the default of one second, you are not capturing information. To properly analyze system performance, you must log the data from the counters for analysis later. To log counter data, follow these steps: 1. Bring up the log dialog box by selecting the disk icon on the Perfmon toolbar. 2. Select the plus sign icon on the toolbar to again display the browser, which allows you to select both the host and the objects you wish to capture. 3. From the Options menu in the Perfmon menu bar, select Log. 4. In the log dialog box, name your log file and specify the path you wish to store it in. Specify a sample rate, then save the log file. 5. From the plus sign icon on the toolbar, select the objects you wish to monitor. All of the counters associated with a particular object will be captured and available for analysis later. 6. Select Start Log in the log dialog box to start logging system performance. Once the log as been started, you will notice the status of the log change from closed to collecting. This means Perfmon has begun collecting object and counter information and is saving the data to your log file. You can return to the chart mode while your log is collecting information, to view some counters of interest to you in real time. The disk icon appears in the lower right-hand corner of the chart along with its current size. This is to remind the user that a logging session is currently underway and will continue to grow in size until the user stops the collection process. 96 PowerSolutions
7. Select the log dialog box and click Stop Log to halt the collection process. Your log file is now ready for immediate analysis in the chart mode. To view its contents, simply select Chart from the drop-down menu on the toolbar and select Data From. The choices are current activity (real time) or log file, in which case a file browser dialog will appear enabling the user to browse the system for their newly created Perfmon log. To obtain the greatest benefit, you should log several objects on a particular system, depending on what application is running and what subsystems are involved in moving and processing data. For example, Web server applications tend to stress processor and network resources, but not memory or disk resources. Database applications tend to stress processor, memory, and disk resources, but rarely network resources. Messaging applications tend to stress all of the above. You should begin in log mode by monitoring more objects rather than less, at least until you have deemed that particular subsystems are not at risk for potential performance issues. After capturing the right data by logging the appropriate objects on a particular system, you can move to the analysis stage. To help narrow your area of focus, you should analyze the counters of a single object. Do not bring up counters across multiple objects for analysis until you have properly analyzed each object separately and documented all your findings (data). Because Perfmon was written with hardware and software subsystems in mind, it is logically organized to lend itself to views and presentations sorted by subsystem type. For example, the System object enables you to monitor counters such as % Processor Time, % Privileged Time, System Calls/sec, Interrupts/sec, and so on. These counters are all related to hardware and software system-level resources. The Memory object allows you to scrutinize counters such as the paging file reads or writes per second, memory utilization, and cache faults per second. When logging, choosing the right objects to capture is not your sole consideration. It is important to choose a sample rate that best suits your needs. The log file size will also depend upon the sample interval, as well as the number ADDING HARDWARE WAS NOT ENOUGH: A PERSONAL EXPERIENCE WITH PERFORMANCE MONITORING One particular software vendor engaged the ASC in an effort to optimize its messaging application written for Windows NT 4.0. The application was a messaging server, compliant with Simple Mail Transfer Protocol (SMTP), Post Office Protocol 3 (POP3), and Internet Message Access Protocol 4 (IMAP4), designed to send and receive e-mail within a corporate IT infrastructure and over the Internet. While testing the application, we ran a messaging benchmark against the server to generate stresses typical of messaging server workloads. We used Mailstone 2.0, one of the premier messaging benchmarks at that time. After collecting Perfmon data via logging mode, we scrutinized the logs with the chart mode. After reviewing the logs, we noted a potential disk I/O performance problem: specifically, the transfer latencies and disk queue length counters (sec/transfer and average disk queue length) under the logical disk object. By doing this, we determined the disk subsystem was saturated. As requests from the application made their way through the OS to the disk I/O subsystem (SCSI device driver), the time to process a request by the subsystem steadily increased past acceptable limitations. After determining that a disk I/O bottleneck existed, our first instinct was to add more spindles (physical disk drives) to the RAID (Redundant Array of Independent Disks) array that contained the messaging stores. We believed this strategy would help to reduce the growing transfer latencies during our tests. The RAID array was already properly tuned to deal with the small, random nature of the I/O requests sent by the messaging application. Regardless of the robustness of the I/O subsystem we put in place, the system continued to suffer from severe disk transfer latency issues. We decided to look further into the I/O subsystem of the application architecture rather than simply adding more hardware. We used a file system tool called NT Filemon, a tool developed by a group known then as NT Internals, now SysInternals Group. Filemon is a lower level tool (provides more detailed information) than Perfmon and enables the user to measure which processes interact with the Windows NT file system (NTFS), at what rate, and for what duration. We decided to measure the file system hits that resulted from sending a single mail message. This seemed like a good first test. Instead of capturing the expected four to five hits on the file system per mail message, to our surprise we found that our mail message interacted with NTFS more than 30 times. The developers worked quickly to correct the issue in their application s I/O architecture. They modified the code to utilize main memory for work that had previously utilized NTFS to temporarily store message header, data, and error control information in the form of caches. The real solution lied in modifying software, not hardware, and enabled us to break through the disk I/O bottleneck. www.dell.com/powersolutions PowerSolutions 97
of objects you monitor. If you are monitoring a production server for performance issues over a normal workday (8 to 10 hours), your sample interval should be somewhat longer than if you were running a 10-minute benchmarking test. A fairly frequent sample interval, such as 5, 10, or 15 seconds, captures specific events and produces very granular and detailed information. It also generates a lot of data, and therefore can lead to massive log files when you sample systems over long periods of time. Since disk space is sometimes a very sought-after resource, you might want to take this into consideration. If you log with sample intervals that are too long on the other hand, you may miss crucial system events or transitions. For benchmarking scenarios, in which tests rarely last longer than 30 to 60 minutes, a log file sample interval between 5 and 15 seconds is ideal. For longer tests, or for monitoring production servers over several hours, you should bump the interval to a number between 15 and 30 seconds, and in some cases to one minute or greater. There is no substitute for experience, so experiment often to determine which interval is best for your particular needs. To end the logging process, return to the log dialog box and select Stop. The log file then closes and is ready for TESTING AND TUNING AT THE DELL APPLICATION SOLUTION CENTER The Dell Application Solution Center (ASC) is a laboratory environment designed to help customers and software vendors test and tune their applications and solutions before they go live. Dell ASC engineers are specifically trained in the performance-tuning disciplines and adhere to a strict process methodology. The heart of the Dell ASC methodology is a process known as the top-down approach, which focuses on scrutinizing the entire solution stack from the top down. The approach starts at the system level (CPU, memory, I/O), works down through the application level (application code and logic), and concludes at the micro-architecture level by testing for Intel processor instruction-level performance issues (Level 1 and Level 2 cache; branch prediction; singleinstruction multiple-data, stream processing; and so on). A Dell ASC lab engagement involves the installation and configuration of a customer solution or vendor application on Dell server and storage hardware, followed by stress testing to evaluate performance. Stress testing involves one of several workload or benchmarking tools to generate stresses representative of real-world client/server activity. For more information about the Dell ASC, contact a Dell representative or visit www.dell.com. Figure 2. Perfmon Log Dialog Box for Selecting Sample Rate, Log Path, and Filename analysis in chart mode. Figure 2 shows the log dialog box for selecting sample rate, log path, and filename. Reporting Mode The reporting mode simply shows raw data as taken from Perfmon. This enables the user to import the data to a spreadsheet program for analysis, creation of charts or graphs, and so forth. Its purpose is primarily to enable the formatting of data taken and not the collection of it. When using the reporting function, the raw data from object counters is displayed in a table-like format. The reporting mode lends itself to documentation better than charting but makes visual comparison difficult and relationships harder to recognize. Combine all Perfmon Modes for Greater Benefit You should use all three modes together to attain the full benefit of Perfmon. The log mode is the most useful for capturing data for detailed analysis in the chart mode. The report mode is most useful for viewing and formatting raw data. The input of the chart can be toggled between the current system activity and data from a log file captured earlier. Proceeding to Analysis After properly logging with Perfmon, you can begin analysis in the chart mode. The best method is to select each object individually, analyze it, write down all the displayed data, and determine if a performance problem exists. Figure 3 shows some typical Perfmon objects and associated counters that Dell ASC engineers often monitor in the Dell Application Solution Center. You can begin the analysis stage by pulling up Perfmon in the chart mode and choosing the Data From option. This action selects your log file as the input, or source. Next, begin analysis by adding counters from a single object; you 98 PowerSolutions
POPULAR PERFMON OBJECTS AND COUNTERS may want to start with the System object. If you choose System, begin adding the associated counters mentioned in Figure 3, such as % CPU Time, % Privileged Time, and so on. Figure 4 shows 12 popular counters and their recommended ranges for normal system operation. Resolve Performance Issues The performance monitor logs may show that one or more of your target s subsystems appear to be excessively utilized or that some component has reached its saturation point. Your first instinct could be to provide additional hardware resources at each bottleneck; but this strategy is not always the correct approach. If you have now identified a resource contention issue on your system, you should properly isolate the root cause of that contention. First, you must understand what the hardware needs to do and why. The operating system simply facilitates and carries out requests from applications. Applications can request a system (and its hardware) to perform work in a variety of ways. There are good and bad ways to perform work or move data on any operating system. Solid, source-controlled network operating systems rarely exhibit true scalability problems. Whether they do or do not is immaterial, because they are always static, beyond your control, and finite for that particular release or service release. True, combative scalability issues exist within the software applications written for a particular NOS. Therefore, you should be at least as familiar with the applications you are running, their behaviors and architectures, as the operating system and hardware you are running them on. Using Perfmon as a First Defense Although hardware subsystems often require adjustment, tuning, or modification to defeat resource contention issues, you must examine all the factors carefully before tackling a particular performance foe. In addition, you should store all of your performance monitor logs for future reference, as both hardware and software technology change at a rapid pace. This will help you to identify problems in the future by comparing the performance information of a properly tuned system with one you believe to be performance impaired. The key to a healthy system is to use Windows NT Perfmon often to measure and monitor its performance for potential resource contention issues. Let Perfmon be your first line of defense. Use lower level tools (such as NT Filemon) when necessary, to increase the granularity and detail of your data, as well as your understanding of the root causes of these problems, and also for data correlation purposes. Never allow one piece of data from a single source to determine you have found a potential performance problem or resolution! Objects System Memory Logical Disk Network Interface Web Service Active Server Pages SQL Server General Stats SQL Server Locks Counters % CPU Time % Privileged Time Context Switches/sec File Control Operations/sec File Data Operations/sec System Calls/sec Interrupts/sec Pages/sec Page Faults/sec Demand Zero Faults/sec Transition Faults/sec Cache Faults/sec Pages In/sec Pages Out/sec Average Disk Queue Length Average Disk sec/transfer Disk Bytes/sec Disk Reads/sec Disk Writes/sec Disk Transfers/sec Average Disk Bytes/Transfer Bytes Total/sec Output Queue Length Packets/sec Packets Sent/sec Packets Received/sec Current Anonymous Users Get Requests/sec CGI Requests/sec Requests/sec Requests Rejected Request Execution Time Request Wait Time Request Queue Length Cache Hit Ratio I/O Lazy Writes/sec I/O Transactions/sec Max Tempdb Space Used RA (read ahead) Pages Found in Cache Page Locks Exclusive Lock Waits/sec Max Users Blocked Total Blocking Locks Lock Escalations/sec Lock Wait Time Note: This list is not exhaustive and does not define individual counters. For a complete list of all counters and their technical definitions, click on the Explain button in the counter selection dialog box within Perfmon. Figure 3. Popular Perfmon Objects and Counters www.dell.com/powersolutions PowerSolutions 99
COMMON COUNTERS AND RECOMMENDED RANGES Counters (Parent Object) Recommended Range % CPU Time (System) 0-90% (> 90% indicates potential processor bottleneck; may also indicate thread contention problem; investigate Context Switches/sec and System Calls/sec for potential thread issues) % Privileged Time (System) 0-40% (> 40% indicates excessive system activity; correlate with System Calls/sec) Context Switches/sec (System) File Control Operations/sec (System) File Data Operations/sec (System) System Calls/sec (System) Interrupts/sec (System) Pages/sec (Memory) Average Disk Queue Length (Logical Disk) Average Disk sec/transfer (Logical Disk) Bytes Total/sec (Network Interface) Packets/sec (Network Interface) 0-10,000 (> 10,000 may indicate too many threads contending for resources; correlate with System Calls/sec and threads counter in Windows NT Task Manager to identify process responsible) Ratio dependent (The combined rate of file system operations that are neither reads nor writes [file control/manipulation only, non-data related]. Inverse of File Data Operations/sec) Ratio dependent (Combined rate of all read/write operations for all logical drives. Inverse of File Control Operations/sec) 0-20,000 (> 20,000 indicates potentially excessive Windows NT system activity; correlate with Context Switches/sec and threads counter in Windows NT Task Manager to identify process responsible) 0-5000 (> 5000 indicates possible excessive hardware interrupts; justification is dependent on device activity) 0-200 (> 200 warrants investigation into memory subsystem; define reads (pages in) versus writes (pages out); check for proper paging file and resident disk configuration; May indicate application memory allocation problems, heap management issues) 0-2 (> 2 indicates potential disk I/O bottleneck due to I/O subsystem request queue growing; correlate with Average Disk sec/transfer) 0-.020 (>.020 seconds indicates excessive request transfer latency and potential disk I/O bottleneck; define reads/sec versus writes/sec; correlate with Average Disk Queue Length) Depends upon interface type (10baseT, 100baseT) A potential network I/O bottleneck exists when throughput approaches theoretical maximum for interface type. (For example, 10baseT theoretical maximum = 10 1,000,000 bits = 100 Mbits/sec divided by 8 = 12.5 Mbytes/sec) Depends upon interface type (10baseT, 100baseT) Note: These general guidelines are meant only to help detect a potential performance issue. Further investigation will confirm or deny whether a performance issue really exists. Figure 4. Common Counters and Recommended Ranges You also should consider the Microsoft Software Development Kits (SDK) and Windows NT and Windows 2000 Resource Kits for a veritable wealth of low-level tools to help you identify hardware and software performance issues. Additional Resources The following sources provide additional information about performance monitoring: Zen and the Art of Performance Monitoring: Windows NT 3.51 Resource Kit, Volume 4: http://msdn.microsoft.com/library/default.asp?url=/ library/winresource/dnwinnt/s7883.htm Microsoft Platform Software Development Kit (SDK) home page: http://msdn.microsoft.com/library/ default.asp?url=/library/psdk/portals/mainport.htm Windows NT 4.0 and Windows 2000 Resource Kits for MSDN subscribers SysInternals: http://www.sysinternals.com Winternals: http://www.winternals.com Paul Del Vecchio (paul.j.delvecchio@intel.com) is a senior performance analyst for the System Software Engineering Lab at Intel Corporation. He has spent the past six years helping customers, ISVs, and OEMs achieve optimal performance of their solutions that run on Intel architecture. 100 PowerSolutions