PATROL From a Database Administrator s Perspective September 28, 2001 Author: Cindy Bean Senior Software Consultant BMC Software, Inc.
3/4/02 2
Table of Contents Introduction 5 Database Administrator Tasks 5 PATROL for Database Management 5 Enterprise Database Monitoring 5 Heterogeneous Database Monitoring 5 Historical Data Collection 6 Event Data 9 Monitoring Blackout 9 SQL Statement Capture 9 User Monitoring 10 Reports 11 Extensibility 11 Customize View 11 Corrective Actions 12 Notification 12 Log Filter Customization 12 Summary 12 Frequently Asked Questions 13 3/4/02 3
3/4/02 4
Introduction Database administrators (DBAs) are responsible for keeping relational databases available at peak performance. Database availability is very critical in today s business world most businesses operate 24 hours a day, 7 days a week, 365 days a year through telephone operators and the Internet. If your application s database is down, your customer may go elsewhere. This lost revenue is difficult to calculate, but must be avoided. DBAs perform tasks to verify that databases are available. They typically work in a reactive, fire-fighting mode, attempting to stay ahead of the tidal wave. Database Administrator Tasks DBAs perform routine tasks to verify that database servers are running and the maintenance tasks are completing successfully. Some of the common tasks are: Viewing server log files for errors Verifying that database servers are running Checking backup jobs Calculating database space These tasks are executed daily if not several times per day. The tasks can be done manually by logging into the database or by viewing operating system files. Most DBAs use scripts. Scripts usually send an email with the results of whether the task failed or succeeded. If successful jobs do not send an email, then you run the risk of thinking that the job completed when it may not have executed due to system problems. Reviewing the email results can be very time consuming. Failed jobs usually have subject lines or rules set up to be easily identified but must be reviewed and repaired before the job can be executed successfully. DBAs can manage a few database servers with manual methods and scripts, but increasing the number of database servers increases the need for more DBAs or better methods. PATROL for Database Management PATROL for database management s wide range of automated, proactive monitoring capabilities continuously manages your database environments, preventing costly outages and protecting your bottom line. By automating critical but routine administrative functions and spotlighting the information necessary to make faster and better decisions, PATROL enhances the contribution and value of your DBA group. PATROL offers the broadest range of database support available today, improving the availability of all popular database environments. For the DBA, this means a solution to keep databases available and performing to ensure that end users have access to the information they need, when they need it. PATROL is a 24x7x365 administrator that watches operating systems, databases and applications to ensure availability. PATROL gives DBAs the ability to automate and simplify complex, routine, and error-prone tasks so they can concentrate on more analytical tasks. Enterprise Database Monitoring PATROL allows the DBA to view the entire system not just the database server. Databases are frequently blamed for performance issues, but often the operating system, the network, or the application are the cause of performance problems. PATROL gives the DBA a view of the other system components so database performance can be ruled out as the cause. Heterogeneous Database Monitoring PATROL allows the DBA to check the health of all the database servers in one view. The databases can exist on several different machines, be of different database types, and run on different operating systems. PATROL supports all of the popular database environments including Oracle, Microsoft SQL Server, DB2 Universal Database, Sybase, and Informix, and runs on all of the major operating system platforms, including several flavors of Unix and Microsoft Windows. PATROL also monitors the operating system and popular database applications such as SAP/R3, PeopleSoft, Oracle Applications, and others. The DBA can determine if potential problems exist by looking 3/4/02 5
at one console and save time by not having to look several different places to check their systems. Each database vendor provides a tool to manage their database, but only PATROL can view and provide consistency across all of the popular relational databases from one console. PATROL monitors the critical components for each database type, watching for conditions that will affect the availability of the database. PATROL identifies conditions such as tablespaces or databases running out of space, long running transactions, increased response time, increased CPU utilization, and many others. The DBA can be notified when resources are running low and prevent a total depletion of resources. An example of this is a large power company that is replacing their scripts with PATROL to monitor their critical applications on Oracle, DB2 UDB, and Microsoft SQL Server. Their environment was becoming too large to maintain scripts and manually process problems. PATROL was programmed to automatically open trouble tickets in Remedy, replacing another manual process by the operators. Historical Data Collection PATROL has the ability to save historical data for 365 days. How often is the DBA actually present when a problem occurs? PATROL saves the data so the DBA can go back in history and view events and determine if the problem shows a trend or is just a single occurrence. Parameters can collect textual data and store them in the parameter history as annotated data points. SQL statements can be captured by the SQL Snapshot feature discussed later. The figures below show two examples of parameters storing text data. The Sybase ASE long running transaction graph below shows a data point storing a report with details on the process that was long running. The Database Administrator can view the details of the long running transaction, including the SQL statement that was causing the condition, at a later time. 3/4/02 6
The Oracle parameter indicates that a tablespace cannot extend its database file. The annotated data point contains the details on the tablespace and why it cannot extend. Historical information can assist the DBA in analyzing problems when he is not able to view the problem directly or when problems occur outside of business hours. PATROL can be used to correlate the data from different parameters in a multi-line graph. You can view related data from the same server or view the same parameter from different database servers. The figures below show an example of two parameters from the same Oracle instance and an example with the same parameter from four Sybase instances. 3/4/02 7
This graph shows an Oracle instance with two response time parameters. The blue area shows the total response time of the client connection. The green area shows the time of the SQL query. The total response time should be slightly higher than the query time. A larger difference could indicate network or operating system problems but show that the database response time is within acceptable service levels. This graph shows the amount of memory usage for multiple Sybase ASE servers. This allows the DBA to compare how the ASE servers are consuming memory. 3/4/02 8
Event Data PATROL also generates and maintains a history of events that occur on the Agent. An event is defined as a change in the environment. Events are used to trigger warnings and alerts or just to provide information. Event data is viewed through the PATROL Event Manager Browser. The Event Manager Browser allows you to filter events based on things such as type, status, host and origin. Once an event is highlighted, you can view the details, add comments, view expert advice and bring up a web page. The DBA can then use the Event Manager Browser to see what events occurred on a database instance during a suspect time period. The figure below shows the events for all the agents being monitored from one PATROL Console. A DBA returns from lunch with voice mail from a user who could not log into the database server. Typically, the DBA could not do anything. PATROL allows the DBA to look at a list of events and see that there were no database errors, but there were network connection failures at 12:51pm. The network errors were preventing the user from connecting to the database. Monitoring Blackout PATROL has the ability to set up alarm blackout periods during times of hardware or software maintenance. This prevents the database administrator from being contacted on erroneous alarms. These blackouts can also be used by the DBA to set up blackouts during backups or special batch jobs. SQL Statement Capture DBAs are tasked with diagnosing application problems caused by poorly performing SQL statements. PATROL provides the ability to view the executing SQL statements through the SQL Snapshot and SQL Viewer components. SQL Snapshot allows you to view the last SQL statement sent by a user or process to the database server. If the process is executing a trigger or stored procedure, the name and current executing line displays. The SQL Viewer utility captures SQL statements submitted to the database server and records more detail about the statement. The DBA can perform problem analysis even if the problem has ceased. An example of SQL Snapshot was shown earlier with the long running transaction parameter under Historical Data Collection. SQL Snapshot is programmed as a corrective action for several parameters to retrieve the SQL 3/4/02 9
statement and store the statement with the parameter history. In PATROL for Oracle, SQL Viewer data can be collected and then analyzed by SQL-Explorer for Oracle. SQL-Explorer uses advanced technology to deliver expert advice on how to tune SQL and use Oracle Hints. It exploits superior intelligence capabilities to decide which indexing schemes to use for tables, enabling you to perform expert-level tuning functions even if you are not experienced in SQL. SQL-Explorer for Oracle displays the queries captured by SQL Viewer in PATROL for Oracle. SQL-Explorer allows you to tune the captured SQL statements. PATROL offers SQL Snapshot or SQL Viewer for all the supported database types and offers both for some databases. User Monitoring PATROL for database management comes with the ability to monitor users. Monitoring for problem user situations is enabled by default. These situations include users that run long transactions, hold locks too long, blocks other users or users that consume too many database resources. These conditions cause other users on the database to question the availability of the database even though it is running. PATROL for Oracle monitors for problem users and alerts if a user blocks another process, uses too much CPU, is idle, or has a runaway query. In addition to monitoring for problem users, PATROL can monitor specific user processes that the DBA chooses. Different statistics are monitored for the selected users based on the type of database. An example of this is a large bank that uses PATROL to track users and collects historical information on their system and user CPU time to be used later for capacity planning. 3/4/02 10
Reports PATROL can generate reports on various elements of the monitored database environment. Reports vary by type of database, but typically include information on instance configuration, space management, users, logs or jobs. Below is an example of some of the reports available for Oracle. Agent Query creates reports by constructing SQL statements to query PATROL on the status of objects in the PATROL environment. Reports can be generated based on monitored computers, applications, instances or even specific parameters. PATROL provides queries, called Quick Query, to generate simple reports on selectable criteria. An example would be a report on all instances in alarm state. PATROL data can be extracted and loaded into a database for customized reports, can be loaded into Microsoft Office products, or can be collected into a database for Service Reporting. Service Reporting is an integrated reporting tool that combines data collection, data storage, and data retrieval with a powerful report creation, customization, and viewing facility. Service Reporting provides a visual presentation of systems management data being collected by PATROL. An example is a large insurance company that feeds the data from PATROL into Excel for daily outage reports for the DBA s manager. This lets the manager know how well the group is meeting their service level requirements. Extensibility PATROL contains parameters and reports to monitor the most common features of a database server. PATROL can be extended to add a DBA s favorite reports and scripts through menu commands and parameters. This provides a centralized repository for all tasks and their results run on the database. One utility company modified a parameter in PATROL for Oracle and created a menu command for the DBA to run when users complain about the performance of the Oracle database. The menu command generates a report of the top ten users and their amount of logical reads, physicals reads, and CPU consumption. All it took was a simple modification to a provided parameter. Customize View The PATROL Console is your personal view of the environment you wish to monitor. Custom folders or containers can be created to organize the agents or database instances in a logical view that makes sense to your business. The PATROL Console allows you to load only the components that you want to monitor although the PATROL Agent is monitoring additional components monitored by other consoles. 3/4/02 11
Corrective Actions PATROL can be customized to correct a problem without intervention. DBAs can program their expertise into PATROL to correct a condition before it causes a problem and can be notified that the problem was avoided instead of receiving the middle of the night phone call. Recovery actions only include reports when PATROL is first installed since each company has a different environment. The default reports gather important information to help the DBA solve the problem in a timely manner. An example of this is a government agency that was very impressed by the report generated by PATROL for Oracle by the LockConflicts parameter and the fact that PATROL generated the report automatically. Notification PATROL can be customized to provide notification of warnings and alarms. Notification is set up through recovery actions or state change actions. Recovery actions are initiated by the PATROL Agent and state change actions are initiated by the PATROL Console. Which one to use is determined by the software needed for notification. Recovery actions that send email or pages must have the appropriate software or modem or have access from each PATROL Agent. State change actions need the appropriate software or modem on a monitoring console. PATROL does not require a console to be attached for the Agents to monitor but state change actions do require a console. Additional notification rules can be established by installing PATROL Operations Manager, another product included with the PATROL base product. Log Filter Customization PATROL provides monitoring of the database alert log or error log. When first installed, PATROL searches for very generic strings such as error or warning. This may cause PATROL to alarm on messages that may not be critical in your environment. PATROL allows the DBA to modify the strings that are monitored so that PATROL only alarms on very specific messages. Strings can even be excluded from monitoring. Summary PATROL for database management can provide DBAs with a better method of managing database servers. PATROL assists the DBA by monitoring and managing enterprise, heterogeneous environments from one console, tracking historical data and events and capturing SQL statements. The DBA can be assured that his environment is being proactively monitored while he attends to his other administrator duties. 3/4/02 12
Frequently Asked Questions 1. I manage a small number of database servers. Why do I need something like PATROL? PATROL can help you monitor the everyday tasks like checking the database status and the logs making your time more available to work on important tasks like SQL tuning and analysis. Although you only have a small number of database servers to monitor now, your company is growing and the database environment will grow too. Setting up tools when the environment is small allows the growth to be more manageable. 2. I have my own scripts to monitor my database. Why I should I replace my scripts with PATROL? Most DBAs do not have the time to write and maintain scripts. PATROL duplicates the monitoring of many DBAs scripts so you can let PATROL do the work for you. PATROL maintains its code when new versions of databases and operating systems are released. PATROL runs on several Unix and Windows platforms compared to scripts that may need editing when moved to a different operating system. PATROL also stores needed username and passwords to run the tasks. The passwords are encrypted in PATROL. Many scripts hardcode the password or access the password stored in a file, hidden as much as possible but not secure. If you have custom scripts specific to your application, you can add them to PATROL and run all of your scripts from one place. 3. The database servers come with a native monitoring tool such as Oracle s Enterprise Manager. Why do I need PATROL? PATROL provides a centralized, common user interface that can monitor multiple database types. BMC supports the major database platforms: Oracle, Sybase, Informix, Microsoft SQL Server, and DB2 Universal Database, and has monitoring for CA-Ingres, Progress, and RDB. The native monitoring tools typically monitor only their own databases. DBAs can learn one tool to monitor multiple database types instead of learning a tool for each database that is running in their enterprise. 4. I ve heard PATROL is difficult to use. Once you have an understanding of the PATROL architecture and spend an hour navigating the PATROL Console, you will see that PATROL is not difficult to use. The monitoring components require a few configuration steps to begin monitoring an application. Default settings are provided for parameter thresholds, but may not be appropriate for your environment. The PATROL Console provides a graphical user interface to change these settings. PATROL can be installed and be monitoring your systems within a few days compared to weeks or even months with other systems management products. PATROL also allows the development of custom components that requires more training. PATROL provides value to your environment because it can be customized to fit your environment. For more information, visit BMC Software on the Web at http://www.bmc.com. BMC Software, the BMC Software logos, and all other BMC Software product or service names are registered trademarks or trademarks of BMC Software, Inc. IBM and DB2 are registered trademarks of International Business Machines Corp.; Oracle is a registered trademark, and the Oracle product names are registered trademarks or trademarks of Oracle Corp. All other registered trademarks or trademarks belong to their respective companies. Copyright 2002 BMC Software, Inc. All rights reserved. 3/4/02 13