TUTORIAL WHITE PAPER Application Performance Management Investigating Oracle Wait Events With VERITAS Instance Watch
TABLE OF CONTENTS INTRODUCTION...3 WAIT EVENT VIRTUAL TABLES AND VERITAS INSTANCE WATCH...4 ARE ALL WAIT EVENTS IMPORTANT?...5 WAIT EVENTS AND SYSTEM PERFORMANCE PROBLEMS...5 SPECIFIC WAIT EVENTS AND PERFORMANCE ISSUES...6 BUFFER BUSY WAITS...7 DB FILE SCATTERED READ...8 THE SESSION WAIT LIST WAIT EVENT INFO COLUMN IDENTIFIES SPECIFIC DATA FILES ASSOCIATED WITH SPECIFIC WAIT EVENTS....8 DB FILE SEQUENTIAL READ...8 ENQUEUE...9 FREE BUFFER WAITS...9 LATCH FREE...10 LIBRARY CACHE PIN...10 LOG BUFFER SPACE...10 LOG FILE SWITCH (COMPLETION; ARCHIVING NEEDED; CHECKPOINT INCOMPLETE; CLEARING LOG FILE)...11 LOG FILE SYNC...11 SQL*NET MESSAGE FROM DBLINK...11 WRITE COMPLETE WAITS...11 CONCLUSION...11 2
INTRODUCTION Many database performance problems are not as obvious as lock contention or a poorly tuned SQL statement. It s often useful to determine where, and by how much, user processes are spending time. This is often referred to as wait-based or bottleneck analysis. Oracle includes a wait event interface that provides invaluable information for diagnosing and solving many complex performance issues. The Session Wait List in VERITAS Instance Watch takes these wait events and adds a graphical, intuitive interface to help expedite the isolation and identification of the root cause of database performance problems based on excessive wait events. Wait event timing information is only available when the Oracle initialization parameter TIMED_STATISTICS is set to TRUE. The diagnostic value of this timing information far outweighs the negligible overhead associated with collecting it, so all databases beyond release 7.2 should have this setting. (NOTE: Except for release 8.1.5 where some version-specific Oracle bugs can cause performance problems when TIMED_STATISTICS=TRUE) Wait Time vs. Resource Contention 1000 900 Wait Time (msec.) 800 700 600 500 400 300 200 100 0 Wait times rise with increasing resource contention Resource Contention 3
WAIT EVENT VIRTUAL TABLES AND VERITAS INSTANCE WATCH There are three main virtual performance tables through which Oracle reports wait event information. These are: 1. V$SESSION_WAIT 2. V$SYSTEM_EVENT 3. V$SESSION_EVENT. V$SESSION_WAIT offers a current snapshot of user waiting in the database. It contains a row for each session that displays the event for which that session is currently waiting or the last event it waited for (in case the session is currently using the CPU). Several additional parameters in the list provide event-specific information and can be used to do further investigation. For example, the latch number for the latch free wait event can precisely identify which latch is being waited for. Session Wait List Button VERITAS Instance Watch s Session Wait List button on the Instance Overview screen is a window into V$SESSION_WAIT. In addition to presenting the basic information from V$SESSION_WAIT, the Session Wait List decodes the additional parameters for selected events to provide additional diagnostic detail. In Instance Watch Replay mode, where a methodical step-by-step evaluation of events leading to a performance problem can be seen, the Session Wait List provides valuable insight into recent performance problems. For convenience, the VERITAS Instance Watch Session Wait List can also be accessed from the Session Detail screen. In this case, it provides a list of wait event totals and timing information for a specific 4
session in the V$SESSION_EVENT virtual table. When this list is refreshed in DELTA mode, it shows the distribution and timing of events over the refresh interval for the session. This feature can be quite helpful when analyzing performance problems specific to certain points in an application by refreshing the list before and after the session has performed the performance critical operations. ARE ALL WAIT EVENTS IMPORTANT? Several wait events recorded by Oracle are sometimes called "idle" waits. These are points in the code where waiting is expected and normal, so for purposes of diagnosing performance issues they can be ignored. The following wait events fall into this category: client message SQL*Net message from client SQL*Net more data from client rdbms ipc message pipe get Null event pmon timer smon timer parallel query dequeue PX Idle Wait dispatcher timer lock manager wait for remote message SQL*Net message to client PL/SQL lock timer virtual circuit status wakeup time manager WAIT EVENTS AND SYSTEM PERFORMANCE PROBLEMS An Oracle database has well over 100 identifiable wait events. Some of these are specific to various specialized subsystems, such as Oracle Parallel Server or Parallel Query, that may not be in use and thus will never be incurred by user processes. Also, some wait events are of such short duration that it would be rare to "catch" them in any given snapshot of the system, even though they may be occurring. Documentation for wait events, as well as some discussion of what they mean, can be found starting with Appendix A of the Oracle8 Reference Manual. Although wait events existed prior to Oracle 8, they were an undocumented feature. Sessions are typically involved in wait events as a normal part of Oracle processing. Significant accumulation of wait time for events that should be transient and short-lived may be indicative of performance problems. To get a sense for the overall wait event profile in the database, the following query on V$SYSTEM_EVENT can be issued after the database has been running for some time under normal load: SELECT event,total_waits,time_waited,(time_waited/total_waittime)*100 pct_total FROM v$system_event SE,(SELECT SUM(time_waited) total_waittime FROM v$system_event WHERE event NOT IN ( client message, SQL*Net message from client, SQL*Net more data from client, rdbms ipc message 5
, pipe get, Null event, pmon timer, smon timer, parallel query dequeue, PX Idle Wait, dispatcher timer, lock manager wait for remote message, SQL*Net message to client, PL/SQL lock timer, virtual circuit status, wakeup time manager ) ) TOT WHERE total_waits > 0 AND time_waited > 1000 AND event NOT IN ( client message, SQL*Net message from client, SQL*Net more data from client, rdbms ipc message, pipe get, Null event, pmon timer, smon timer, parallel query dequeue, PX Idle Wait, dispatcher timer, lock manager wait for remote message, SQL*Net message to client, PL/SQL lock timer, virtual circuit status, wakeup time manager ) ORDER BY pct_total DESC; This query will show the list of potentially significant wait events in the system and the percentage of the important wait time that has been spent on each. It provides a nice snapshot of where Oracle wait time is accumulating. SPECIFIC WAIT EVENTS AND PERFORMANCE ISSUES The following sections describe specific wait events that may be related to known performance issues and tuning opportunities. No action may be necessary based on the occasional occurrence of any of these wait events. Again, Oracle sessions are always involved in wait events in the normal course of processing. Significant numbers of sessions waiting for the same event(s) or a few sessions waiting a very long time for certain events may indicate a problem, or at least the need for further investigation. 6
BUFFER BUSY WAITS The session is waiting to access a block in the buffer cache. Depending upon its frequency, this may indicate a buffer contention problem. See the VERITAS Instance Watch Data Base Buffer Busy Wait Details to help identify the types of blocks that may be causing contention as tuning solutions are specific to the block types involved. Data Base Buffer Busy Wait Details helps identify the types of blocks that may be causing contention 7
DB FILE SCATTERED READ The session is waiting on a multi-block read associated with a full table scan. The Session Wait List -- Wait Event Info column will identify the specific data file associated with the full scan, as well as the block# within the data file. From here, users can hot-link to the Session Detail facility to discover the SQL statement and table involved in the full scan. The Session Wait List Wait Event Info column identifies specific data files associated with specific wait events. DB FILE SEQUENTIAL READ The session is waiting on a single block read of either a table or index block. The Wait Event Info column will identify the specific data file associated with the read, as well as the block# within the data file. If many sessions are waiting for reads from the same data file there may be load balancing problems in the I/O subsystem. Use the VERITAS Instance Watch Data File I/O List to investigate how well I/O is distributed among Oracle data files. 8
The Instance Watch Data File I/O List shows how well I/O is distributed among Oracle data files ENQUEUE The session is waiting for a lock request to be satisfied. Normally, this will be for access to a database table, although there are many different types of enqueues. The Wait Event Info column shows additional details about the object for which the lock is being requested. When this wait event appears, the Lock Contention alert should appear red. Double-click on the alert to display the Contention Lock Details list, which shows the object and users involved in the contention. The System Performance area of the Instance Overview display showing a red alert for lock contention. FREE BUFFER WAITS The session is waiting for a free buffer in the buffer cache. The DBWR process may not be keeping up with the volume of dirty buffers in the cache. Consider adding more DBWR processes using the Oracle DBWR_PROCESSES initialization parameter or the DBWR_IO_SLAVES parameter. 9
LATCH FREE The session is waiting for a latch. Latches are normally transient serialization mechanisms, so frequent occurrence of this wait event usually indicates serious contention problems that are compromising performance. The Session Wait List Wait Event Info column will identify the specific latch requested. Use the VERITAS Instance Watch Latch Get Miss Details list to get more information about latch contention. The help section problem descriptions and solutions related to specific latches. Detailed help is a click away The Latch Get Miss Details list provides detailed information and help for latch contention. LIBRARY CACHE PIN The session is waiting to pin a library cache object but another session has the object pinned in an incompatible mode. This is normally only seen if there is heavy contention for a particular object or statement in the shared pool. If this event appears with any frequency, it may indicate a serious problem with Oracle and/or the application and should be pursued with Oracle Worldwide Support. LOG BUFFER SPACE The session is waiting to write information to the redo log buffer but cannot because LGWR is not keeping up with the volume. Increase the Oracle LOG_BUFFER initialization parameter and/or move the redo log files to faster disks or ones with less contention. It may also help to add multiple LGWR processes by increasing the Oracle LGWR_IO_SLAVES initialization parameter. 10
LOG FILE SWITCH (COMPLETION; ARCHIVING NEEDED; CHECKPOINT INCOMPLETE; CLEARING LOG FILE) The session is waiting for a log switch. If multiple sessions are waiting for these events, then perhaps the archiver has run out of file system space and become stuck. It may also help to add multiple archive processes through the Oracle LOG_ARCHIVE_MAX_PROCESSES initialization parameter. LOG FILE SYNC The session is waiting for redo information to be flushed to disk and confirmed. Frequent waits for this event may be relieved by reducing the commit frequency or speeding up redo writing (faster disks or perhaps multiple LGWR_IO_SLAVES). SQL*NET MESSAGE FROM DBLINK The session is waiting to get data from a remote database over a database link connection. When many sessions wait a long time on this event, there may be network performance issues. Fetching large amounts of data across database links is slow so, if this is a significant drain on performance, a review of the distributed architecture may be in order. Consider localizing remote data using replication. WRITE COMPLETE WAITS The session is waiting for a buffer block to finish writing to disk. When significant numbers of sessions have this event, it may mean that the DBWR process is not keeping up with the volume of dirty buffers in the cache. Consider adding multiple DBWR processes using the Oracle DBWR_PROCESSES initialization parameter or the DBWR_IO_SLAVES parameter. CONCLUSION Oracle wait events are crucial to identify and solve many difficult database performance issues. VERITAS Instance Watch s Session Wait List expedites this effort. It exposes Oracle wait events at the session level by producing a snapshot on the Instance Overview page. An overall summary is also shown on the Session Detail page. Speed is of the essence when solving any performance problem that can negatively impact business processes. VERITAS Instance Watch helps significantly reduce the amount of time and effort to isolate, identify, and solve performance problems in a single Oracle instance or across the enterprise. VERITAS Software Corporation Corporate Headquarters 350 Ellis Street Mountain View, CA 94043 650-527-8000 or 866-837-4827 For additional information about VERITAS Software, its products, or the location of an office near you, please call our corporate headquarters or visit our Web site at www.veritas.com. 11