Destiny performance monitoring white paper

Transcription

1 Destiny performance monitoring white paper Monitoring server indicators Overview This paper provides an introduction to monitoring server indicators relevant to Destiny. It serves as a starting point for performance monitoring rather than delving deeply into troubleshooting. If you suspect performance issues, contact Follett Software Technical Support before making any changes to your Destiny or SQL Server configuration. See the Destiny Technical Support section later in this document. This document is organized into monitoring the Destiny application server and monitoring the database server. Comments are equally applicable whether your application and database are on the same physical server or two separate servers. Copyright Follett Software Company Page 1

2 Monitoring Tools Monitoring tools are available to view, record, and script local and remote server administration. While this document is focused on system counters to monitor, these main tools are mentioned here for further follow up if you desire. WMI: Windows Management Instrumentation Windows Management Instrumentation (WMI) is a method of remotely administering Microsoft software. It is Microsoft s implementation of the WBEM and CIM object model standards for representing server components. WMI classes (objects) provide the information that allows a Windows server to be remotely administered. For example, WMI can be used to run defragmentation reports and to defragment hard disks remotely and via scripts. For more information in WMI, see Windows PowerShell Windows PowerShell is an interactive shell and scripting language for administering Windows servers. PowerShell can be used to script the monitoring of WMI classes on remote Windows servers. The Get- WMIObject cmdlet provides access to the WMI classes. For more information on PowerShell, see SNMP Simple Network Management Protocol (SNMP) is a set of standards for monitoring devices in a network. SNMP software can be used to monitor a heterogeneous network environment, including Windows-based servers. SNMP services on Windows servers can be enabled to provide accessibility to SNMP clients (monitoring applications). In addition, WMI can be configured as an SNMP client to access SNMPcompliant devices and make them available via WMI. For more information on SNMP and WMI, see PerfMon Windows Performance Monitor (PerfMon) is an application for viewing and recording performance indicators in a Windows system. PerfMon can monitor various OS and SQL Server counters. Relevant PerfMon counters will be discussed in detail later in this document. The Destiny Application There are several areas that can be checked when monitoring the Destiny application, including verifying certain aspects of the Destiny configuration. About Destiny / Show Details When logged into Destiny as the Destiny administrator, the "About Destiny" window offers a "Show Details" option. You can reach the "About Destiny" window after logging in by clicking the Follett Software Company link at the bottom of any page. Clicking Show Details provides a snapshot of Destiny content statistics and component version information. About Destiny / Server Connectivity Test In addition, found in the "About Destiny" window is a Server Connectivity Test that can be used to test network connection and download speeds from any client computer to the Destiny application server. This feature is available to anyone that is currently logged in to Destiny, although only certain privileged users have the option to run the test continuously. Note: To run these tests, JavaScript must be enabled on the client computer. The browser and JavaScript requirements for this test are the same as for Destiny Quest. Copyright Follett Software Company Page 2

3 From the Server Connectivity Test page, click Start Test to perform a quick sweep across three different file size levels to get a quick snapshot of how the network timings look from that client machine. The tabulated results show the following pieces of information: File size - The size of the file that was downloaded for this test Sever time - The current date and time on the server the test took place. Connect time - The amount of time it took to submit the file download request to the server Download time - The amount of time it took for the client to download the file from the server Total time - Complete round-trip time (connect time + download time) The tabulated results are also color-coded to highlight acceptable download times. Green rows indicate that the network connection is satisfactory; yellow or red rows could indicate a network problem between the client and server that may need further investigation. Note: Because of browser differences and limitations, there may be inaccuracies or overhead associated with these timings, so they are best used to compare two similarly set up client machines using the same web browser. The server connectivity test can also be used similar to the Alarm utility to continuously test network connection over a period of time if the logged-in user has certain rights (that is, normal student patrons should not have access to this feature). At the server connectivity test page, select the "Enable interval testing" check box. Optionally, specify the interval in minutes and the size of the file to download, or use the defaults of one minute and 50KB file size. Upon clicking Start Test, the test will run at the interval specified until the browser is closed or the Stop Test button is clicked. The last 10 tests will be shown in the tabulated results along with the minimum, maximum, and average total times accumulated over the entire test run. Server connectivity test results are also logged to the Destiny application server logs, so that Technical Support can retrieve the data automatically from the logs so the end user does not need to manually collect and send this information. Destiny Server Health Reports Destiny v8.5 and higher includes server health reports to help you monitor your Destiny server s performance. These reports are the quickest and simplest way to get a picture of overall Destiny transaction performance. They present information graphically and numerically, including transaction volume and response times, bandwidth, server uptime, and the types of browsers used to access Destiny. To view System Health Reports, log in to the district's Welcome page as the Destiny Administrator, click the Setup link in the upper corner of the window, and open the Reports tab. If you're part of a Destiny consortium, log in to the consortium's Welcome page as the Super Administrator, click the Setup link in the upper corner of the window, and open the Reports tab. Four types of reports are available: The Browser Report displays information about the browsers through which the users access Destiny. The Bandwidth Report displays the amount of data the server is sending and receiving. The Request/Response Report displays the number of requests your server receives and how quickly it responds. The Server Uptime Report displays the amount of your server's uptime and downtime. To quickly see server performance View the Request/Response Report to see how quickly the Destiny application is processing transactions. This can be compared on the same report to the number of incoming requests at any point of time to see how the server handles different load levels. Note that these response times record the time to process and send transactions on the Destiny server, and depending on your network topology, do not reflect round-trip network latency between the end user and the Destiny server. Copyright Follett Software Company Page 3

4 Destiny server logs The Destiny application writes to a text log that rolls over to a new file each day. A record of each transaction is written as well as various internal system data including a running count of free memory inside Destiny s Java Virtual Machine (JVM). In addition, any exceptions that occur are written to the log. You can view the server logs by navigating to \FSC-Destiny\jboss\server\destiny\log. The current log is always named serverlog.txt. It can be viewed with Notepad. Exception messages In the log file, some exceptions are normal and can safely be ignored. Note any excessive exceptions that could affect performance or indicate performance issues, such as persistent transaction timeouts. Startup time One indicator to note in serverlog.txt is the Destiny service startup time. This is only recorded in the serverlog.txt file on the day(s) the service is started. Search for the text "started in" to find the line that looks like the following: INFO Server () JBoss (MX MicroKernel) <..> Started in 0m:59s:394ms A startup time that significantly exceeds one minute could indicate server-to-server connectivity issues between Destiny and SQL Server, or an overloaded server if both are on one physical server. Backups It is very important to monitor that backups occur on schedule and complete successfully. Backups should be verified for the following: The FSC-Destiny folder The destiny and master SQL databases The destiny SQL log file, if using Full recovery model Verify successful backups through your backup management tool and ensure that failure notification is in place. Additionally, verify backups periodically through Destiny. Log in as the Destiny administrator and go to "Setup". You will see the notification "Your last SQL backup was " at top of screen. If you observe that backup times are becoming excessively long, make sure you are using an appropriate backup strategy, including the recovery model (simple or full) and full versus differential backups. If you are using the Full recovery model, verify that transaction log backups run at least hourly. Also, verify the performance of the communications path between your backup agent and SQL Server, and of the backup hardware (i.e., tape drive or remote server). JVM configuration Destiny configuration settings for the JVM have a vital effect on maximizing performance. These settings are contained in \FSC-Destiny\fsc\bin\config.properties. These settings determine internal memory allocation, cache sizes, and the dual-vm option for servers with more than 4GB RAM. Normally these configuration settings are set once when Destiny is installed. They may benefit from updating if your server hardware has changed (e.g. amount of memory). If you suspect a configuration problem affecting performance, Follett Technical Support can help you verify appropriate settings. Note: Dual-VM setup does not apply to Destiny running on a 64-bit OS. Virus scanning We recommend excluding the \FSC-Destiny folder and below from virus scanning. This is especially important if "live" or "real-time" scanning is enabled, which scans files when they are read from disk into memory. This can significantly affect Destiny performance when it needs to load its software components. Copyright Follett Software Company Page 4

5 The Server and Windows There are several areas to assess when monitoring the server and Windows itself. Windows Event Logs Windows event logs should be examined periodically. These logs will show the following: Application log: Info, Warn, and Error conditions including Destiny and SQL Server start/stop, etc. System log: System startup/shutdown, hardware and controller failures, and other system events. Security log: Windows session logon/logoff events and any server security alerts. The event log viewer can be accessed in Windows Control Panel / Administrative Tools. Task Manager Windows Task Manager is the simplest way to get a quick view of current server resource usage. Task Manager shows processor usage by CPU, in total, and by process. The general industry guideline for Total CPU usage is that continuous usage over 80% may indicate a CPU bottleneck. The Processes tab displays CPU % by process. The Destiny process appears as "java.exe". You may need to click "Show processes from all users" to see it. Task Manager also shows total physical memory and the amount currently in use. Task Manager shows percent usage for each network connection. If you have a two-server Destiny configuration, note the usage between the app and database servers during load. Memory On servers with a 32-bit version of Windows, certain boot parameters must be explicitly enabled if you wish to use more than 4GB of physical memory. If you have more than 4GB physical memory and suspect this is not currently enabled, contact Follett Technical Support for configuration details. Network If your end users are experiencing longer round-trip transaction times than the Destiny server health reports or serverlog.txt have recorded, they may be experiencing network latency. These types of delays can be introduced by network bandwidth saturation or various network hardware failures or configurations such as firewalls and proxy servers. Diagnosing network latency is beyond the scope of this document. A good starting point is to measure round trip end user response time and compare it to internal Destiny transaction process time as recorded in serverlog.txt. Physical disk drives Disk drives need to be monitored regularly for available space, fragmentation, and file system errors. Available drive space can be checked manually by opening Properties for the drive. Available space can also be checked via any of the monitoring and scripting tools such as WMI and PerfMon referred to in this document. Physical disk drives defragmenting Physical hard drives should be defragmented daily, or minimally weekly. Fragmentation occurs over time as files become written to clusters spread over a wide area of a disk drive. High fragmentation levels can have a significant effect on file read/write performance. Fragmentation can be checked and corrected via Windows Defragment utility or various third-party tools. Defragmentation can also be scheduled or scripted by using Task Scheduler, WMI, or the Defragment utility, depending on your Windows system. From the Tools tab of drive Properties, select "Defragment Now " The Check Disk tool can be used to report and fix file system errors. To view the Check Disk UI, navigate to the Tools tab of the drive s Properties dialog and click the Check Now button. File system errors can also be checked and fixed from a command prompt using the "chkdsk" command. Copyright Follett Software Company Page 5

6 App server performance monitoring with PerfMon Windows Performance Monitor (PerfMon) is an application for monitoring performance-related key indicators in your Windows system. This document will highlight some of the most important counters relative to Destiny application and database servers. PerfMon can be used to observe values in real-time or to sample and record values over a period of time to review later. PerfMon can also be used to locally or remotely monitor an instance of SQL Server. Thresholds are given below for most counters, as general guidelines. Compare your particular installation to the guidelines if you are experiencing performance issues. Observe the values over time and identify any negative trends. Investigate any that are consistently out of range. A good starting point for understanding counter detail is SQL Server Books Online. Further information can be found in the "Additional Resources" section later in this document. To start PerfMon, type perfmon at the commandline. For information on how to use PerfMon, including how to set up specific counters, see PerfMon s Help menu. Counter definitions below are quoted from PerfMon and/or SQL Server 2005 Books Online. Threshold guidelines have been obtained from the resources listed in "For more info" section. The three most important performance areas to monitor on the Destiny app server are processor, memory, and disk. SQL Server-specific counters are detailed later in "Database server performance monitoring with PerfMon". Processor and system counters Issues in these counters could indicate a CPU bottleneck. Processor and system counters Processor: % Processor Time "The amount of time the CPU spends executing a thread that is not idle." Continuous usage over 80% may indicate a CPU bottleneck. Processor: % Privileged Time "The percentage of time the processor spends on execution of Microsoft Windows kernel commands, such as processing of SQL Server I/O requests." Should be <30% of % Processor Time. Consistently high values when Physical Disk counters are high could indicate a physical disk bottleneck. Processor: % User Time "The percentage of time that the processor spends on executing user processes such as SQL Server." System:Context Switches/sec "The combined rate at which all processors on the computer are switched from one thread to another." Should be <5000 * # processors If >5000, could be due to other applications on the server or hyper-threading. Copyright Follett Software Company Page 6

7 Memory counters Issues in these counters could indicate memory pressure. Memory counters Memory: Available Mbytes "The amount of physical memory, in Megabytes, immediately available for allocation to a process or for system use." Should be > 100MB. Memory: Pages Input/sec "The rate at which pages are read from disk to resolve hard page faults." Should be < 10/sec. Memory: Pages/sec "The rate at which pages are read from or written to disk to resolve hard page faults." Ideally <100 on a slow disk subsystem or <600 on a fast disk subsystem. Paging File: %Usage "The percentage of the paging file that was in use." Average value should be well under 70%. A high value indicates that excessive disk swapping is occurring Ensure adequate RAM is available Generally best to allow Windows to adjust the page file size The page file can also be moved to an unused volume for better performance Paging File: %Usage Peak "The greatest percentage of the paging file that was in use during the sample interval." Should be < 70%. Copyright Follett Software Company Page 7

8 Physical disk and network counters Issues in these counters could indicate I/O-related bottlenecks. Physical disk and network counters PhysicalDisk: Avg. Disk sec/read "The average time, in seconds, of a read of data from the disk." Note: 1 Msec = seconds. Optimal: <8 Msec. Poor: >20 Msec. PhysicalDisk: % Disk Time "The percentage of elapsed time that the selected disk drive was busy servicing read or write requests." If this counter is high (>90%), see Current Disk Queue Length. PhysicalDisk: Avg. Disk sec/write "The average time, in seconds, of a write of data to the disk." Non-write-cached drives: Optimal: <8 Msec. Poor: >20 Msec. Write-cached drives: Optimal: <1 Msec. Poor: >4 Msec. Continuous high values indicates a disk bottleneck. PhysicalDisk:Avg. Disk Queue Length and Current Disk Queue Length "The average [or current] number of both read and write requests that were queued for the selected disk during the sample interval." Read-aheads can cause this value to be fairly high without adversely affecting performance are not unusual. Per SQL Server 2005 Books Online, the number of waiting I/O requests should be sustained at no more than 1.5 to 2 times the number of spindles that make up the physical disk. Network Interface: Bytes Total/ sec "The rate at which bytes are sent and received over each network adapter, including framing characters." Use this counter to monitor data sent through this NIC. Alternatively, Bytes Received/sec and Bytes Sent/sec will give further detail. Network Interface: Packets/sec "The rate at which packets are sent and received on the network interface." "For TCP/IP frames actual bandwidth is computed as packets/sec * 1500 * 8 / Mbps." Network Interface: Current bandwidth "An estimate of the current bandwidth of the network interface in bits per second (BPS)." Copyright Follett Software Company Page 8

9 Monitoring the database server This section highlights the main areas to monitor relative to Destiny s database. For further information, see the resources listed in the "Additional Resources" section. Enterprise Manager / Management Studio Depending on your edition of SQL Server, you can use the Enterprise Manager, Management Studio, or Management Studio Express software to view and manage aspects of your SQL database. Management Studio Express is available from as a separate download. SQL Server logs SQL Server writes important messages to the SQL Server logs. These can be found in Management Studio Management SQL Server Logs, and should be checked periodically. Server properties Management Studio displays information about the configuration of the overall SQL Server software. Right-click the server icon and select Properties to access this data. Here you will see these types of information relevant to SQL Server performance monitoring (among others): General server information Operating system, processors, physical memory, etc. Memory Configured and Running values for allocated memory Processors Processor affinity and SQL Server priority SQL Server minimum and maximum memory Minimum and Maximum server memory allocated to SQL Server can be viewed and set in Management Studio as mentioned above. Once set, these values normally do not need to be changed. Note: These values can be set by Destiny during startup if configured in the Destiny config.properties file. If you have questions about your SQL Server memory allocation, please contact Follett Software Technical Support. See the Destiny Technical Support section later in this document. Database properties Management Studio also gives a snapshot of current database information. Right-click the destiny database and select Properties to quickly see last database and log file backup, current database size and space available. The Database Properties dialog also shows the location of the data and log files and their autogrowth settings. SQL Server capacity A vital area to monitor is the size of your database, log files, and available space. These can be seen in Management Studio as mentioned above. The database file(s), which have file extensions of.mdf and.ndf, have an allocated external file size. This file size can be configured to grow automatically as needed. It is vital to ensure you always have adequate drive space for these files to grow. The "Space Available" value in the database properties dialog shows how much of the space inside the allocated file size is available to store additional data before the file must grow. If you are using the Full recovery model, then the contents of your SQL log file (extension.ldf) will grow between backups. It is vital to monitor the size of the log file and available drive space to ensure the log file can grow as needed. Frequent backups of the transaction log are absolutely essential to ensure that the transaction log does not continue to grow endlessly. If the transaction log grows so large that it fills available disk space, you will not be able to perform transactions on your Destiny database. Additionally, SQL Server may mark the database as suspect. If you encounter this situation, see the following Microsoft Knowledgebase article, "A transaction log grows unexpectedly or becomes full on a computer that is running SQL Server" at Copyright Follett Software Company Page 9

10 SQL Server Profiler Another tool that can be useful for monitoring SQL Server is the Profiler tool. This tool allows you to record a trace of SQL Server activity. Many types of internal activity can be selectively traced. This powerful tool is used when troubleshooting suspected issues. Like PerfMon, Profiler offers a current live view and a trace/review option. SQL Server 2005 creates a trace that runs by default without incurring significant overhead. This can be reviewed periodically in Profiler. See SQL Server 2005 Books Online at This document will focus on the PerfMon tool, which may be more appropriate for basic performance monitoring. For more information about the Profiler tool, see SQL Server 2005 Books Online and the "Additional Resources" section of this document. SQL Server 2005 Dynamic Management Views (DMV) SQL Server offers Dynamic Management Views (DMV), which effectively expose internal SQL Server performance-related statistics as relational tables so they can be used in SQL queries. Consequently SQL-based reports can be defined that incorporate these internal values. Much of the data available through DMV is also available through PerfMon (see below). For more information about DMV, see the "Additional Resources" section later in this document. Database server performance monitoring with PerfMon PerfMon is an excellent tool for basic performance monitoring of SQL Server. There are many SQLspecific objects (counters) available in PerfMon. This section highlights basic PerfMon SQL counters including those most relevant to Destiny performance. Counter definitions below are quoted from PerfMon and/or SQL Server 2005 Books Online. Threshold guidelines have been obtained from the resources listed in "For more info" section. If your database is running on a separate server from the Destiny application, monitor the same server counters as listed for the Destiny App server plus the following SQL Server-specific counters. SQL Server PerfMon counters Process: % Processor Time (sqlservr) "% Processor Time is the percentage of elapsed time that all of process threads used the processor to execution instructions." Continuous usage over 80% may indicate a CPU bottleneck. System: Processor Queue Length "The number of threads in the processor queue." Should be < 5* #CPU. Exceeding this may be due to other applications on the server. SQLServer:Memory Manager: Memory Grants Pending "Total number of processes waiting for a workspace memory grant." Should be < 1. Higher or rising could be due to insufficient memory. SQLServer:Buffer Manager: Page Life Expectancy "Number of seconds a page will stay in the buffer pool without references." In other words, how long a page stays in memory before it is flushed. Should be >= 300 (5 minutes). Consistently low could indicate memory pressure. A sudden drop could indicate a memory bottleneck. SQLServer: AccessMethods: Forwarded Records/sec "Number of records per second fetched through forwarded record pointers." Should be: <10 percent of SQL Statistics: Batch Requests/sec. Copyright Follett Software Company Page 10

11 SQLServer: AccessMethods: Page Splits/sec "Number of page splits per second that occur as the result of overflowing index pages." Should be: <20 percent of SQL Statistics: Batch Requests/sec. SQLServer:BufferManager: Buffer Cache Hit Ratio "Percentage of pages found in the buffer cache without having to read from disk." Ideally should be >95%. A sudden drop or consistently low (<90%) could indicate memory pressure. SQLServer:BufferManager: Free List Stalls/sec "Number of requests per second that had to wait for a free page." Should be <2. "Total number of pages on all free lists." Should be >640 SQLServer:BufferManager: Free Pages SQLServer:BufferManager: Lazy Writes/sec "Number of buffers written per second by the buffer manager's lazy writer." Should be <20. Monitor the trend. Could indicate memory pressure. SQLServer:BufferManager: Page Lookups/sec "Number of requests per second to find a page in the buffer pool." Should be: Page Lookups/sec / SQL Statistics: Batch Requests/sec <100 SQLServer:BufferManager: Page Reads/sec, Page Writes/sec Number of physical database page reads, writes issued per second. Should be <90. SQLServer:SQL Statistics: Batch Requests/sec and SQLServer Statistics:SQL Compilations/sec (to calculate plan re-use) Calculate plan re-use as (BatchRequests/sec SQL Compilations/sec) / BatchRequests/sec Ideally, plan re-use should be >90%. Could cause CPU bottleneck. Additional resources WMI Accessing SNMP devices with WMI PowerShell scripting tool Microsoft TechNet / SQL Server TechCenter Microsoft SQL Server Solution Center SQL Server Books Online "Monitoring Resource Usage (System Monitor)" SQL Server Database Engine Administering the Database Engine Monitoring and Tuning for Performance Monitoring Resource Usage (System Monitor) SQL Server Books Online "Using SQL Server Objects" Copyright Follett Software Company Page 11

12 "SQL Server Performance The Crib Sheet" by Grant Fitchey Good intro to PerfMon and Profiler. Some developer-focused info. Screencast series: "PerfMon counters for the SQL Server DBA" Introduction to SQL Server performance monitoring counters "Troubleshooting Performance Problems in SQL Server 2005", Microsoft "Top SQL Server 2005 Performance Issues for OLTP Applications" Goes into more counters including DMV to help pinpoint performance area. Published on sqlcat.com, the Microsoft Customer Advisory Team website. "SQL Server Performance Monitoring with Perfmon" by Brent Ozar SQL Server Books Online "Default trace enabled option" "Monitor system information with SQL Server 2005 s default trace" Microsoft TechNet LogParser CodePlex PAL Performance Analysis of Logs tool In addition, there are many Microsoft and third-party tools, books, training classes, and other resources for Windows and SQL Server performance management. For more information Destiny Technical Support The Destiny Support Team at Follett Software Company is ready to assist or consult with you regarding any questions or concerns. SUPPORT articles are available anytime, anywhere. TELEPHONE SUPPORT is available Monday through Friday, 7:00 AM to 6:00 PM Central time. You can speak with a Technical Support Analyst by calling (800) SUPPORT is available at techsupport@fsc.follett.com or For additional assistance, please contact your Follett Software sales executive or Follett Technical Support at (800) Copyright Follett Software Company Page 12