Lock Out Your Locking Problems Part 2. Lennart Henäng Svenska Handelsbanken AB

Transcription

1 1 F11 Lock Out Your Locking Problems Part 2 Lennart Henäng Svenska Handelsbanken AB October 15, :45 15:45 Platform: DB2 for z/os In this second part of the two-part presentation we will dive into the information that DB2 for z/os can provide you with in order to get hold of locking conflicts. Various system parameters controlling the behaviour of DB2 with regard to locking are described. Messages and trace records showing deadlocks, timeouts, and long running transactions will be looked at in detail. Our ideas on a home grown poor man's locking conflict monitor will be discussed. We will start with a reference to part 1. We will then continue with a description of the messages and trace records that DB2 can provide for deadlocks, timeouts, and long running transactions respectively. We look at related system parameters that control the locking conflict resolution behaviour as well as the way DB2 produces messages and trace records. The next section discusses a proposal for a poor man's locking conflict monitor. The proposed monitor retrieves trace records by using the Instrumentation Facility Interface and saves the information immediately to a set of DB2 tables. The proposed monitor accepts subscriptions and based on these subscriptions, it can mail relevant locking conflict information to responsible DBAs. The subscription part is built as a set of stored procedures and triggers. The presentation will hilite any differences between DB2 V8 and DB2 9.

2 2 Thanks Peter Backlund Haakon Roberts Claes Arrenius Olle Nyman 2 I want to thank Peter Backlund of Peter Backlund DB2-Konsult AB who is one of the most dedicated DB2er I know. It is always a fun learning experience to work with Peter. I also want to thank Haakon Roberts of DB2 for z/os Development that has been very helpful in answering my questions about locking. I also want to thank two colleagues at Handelsbanken. Claes Arrenius, the author of an application that scans DB2 messages and put locking conflict information into a DB2 table. Ola Nyman, that has been helpful in testing deadlocks and timeouts with DL/I batch. Finally I want to thank my other colleagues including my manager - at Handelsbanken that has been patient while I ve been working on this presentation.

3 3 Svenska Handelsbanken Group Universal bank established 1871 The biggest bank in Sweden and the third biggest bank in the Nordic area* Lending to general public: > SEK billion Operating profit: SEK 19.4 billion (2007, incl. SPP) Total staff: branches 461 in Sweden 171 in Great Britain and in the Nordic Countries outside Sweden * Refers to lending to the public 3 Svenska Handelsbanken Group has 660 branches in total. The largest coverage is in Sweden, Great Britain and the other nordic countries. The subsidiaries of Svenska Handelsbanken Group are: Handelsbanken Finans, Stadshypotek, Handelsbanken Liv, and Handelsbanken Fonder

4 4 What was in part 1? Locking from an application point of view Lock size Lock mode Lock duration row, page, table S (share) or X (exclusive) commit or momentarily Now we will take a look from a technical point of view 4 As a reminder of what we were talking about in part 1, we recap the main DB2 locking terminology from an application standpoint. It had to do with the size of the locks, or rather how much data that is locked by a lock, the mode of a lock, basically read or update, and the duration av a lock for how long ít will be held. In this part, we will look at locking from a more technical point of view to see what locking conflicts we can run into and how DB2 tries to solve such conflicts and how DB2 tell you about their existence.

5 5 Agenda Defining locking conflicts DB2s behaviour and how to adjust it DB2 Messages and Instrumentation Timeout Deadlock Lock escalation Long runners How to manage locking conflicts 5 The presentation will take us thru examples, messages, and trace records for timeouts, deadlocks, lock escalations as well as long runners. We will also see what kind of controls we have to modify DB2s behaviour in the event of a timeout, deadlock, lock escalation, or long runner. We start off by defining the most common locking conflicts, timeout and deadlock, and see how the application will be informed about them. Second, we delve into the behaviour of DB2 and how to adjust it to our will. Third, we look at how DB2 communicates information about locking conflict events to its administrators by messages and instrumentation (trace records). Finally, we present an idea on how to manage locking conflicts in order to make applications run as smooth and problem free as possible.

6 6 B-1 Timeout (1/2) TRANSACTION A X RESOURCE t 6 First of all, we have the timeout conflict that occurs if a transaction tries to access a resource that is already locked by another transaction. In this example, TRANSACTION A, at a certain time, has got an exclusive lock on a resource (which can be a table, page, or row). IBMs definition: Abnormal termination of either the DB2 subsystem or of an application because of the unavailability of resources. Installation specifications are set to determine both the amount of time DB2 is to wait for IRLM services after starting, and the amount of time IRLM is to wait if a resource that an application requests is unavailable. If either of these time specifications is exceeded, a timeout is declared.

7 7 B-1 Timeout (2/2) TRANSACTION A X RESOURCE t S or X timeout TRANSACTION B When the wait-time has been exceeded we get a timeout SQLCode -911, SQLState Reasoncode 00C9008E 7 While TRANSACTION A, holds on to its exclusive lock, TRANSACTION B comes along and requests a lock on the very same resource. Since any lock request from TRANSACTION B is incompatible with TRANSACTION A s exclusive lock, TRANSACTION B has to wait (be suspended in DB2 talk) to be granted the required lock. That is a basic behaviour of lock management. However, if TRANSACTION B has to wait for too long, DB2 decides to stop waiting and tell the application that it won t get the requested lock. TRANSACTION B will experience a TIMEOUT where DB2 has rolled back the complete transaction. This has to be done no matter if TRANSACTION B has made numerous updates before the timeout or not. TRANSACTION B will be rolled back and the application will receive an SQLCODE - 911, SQLSTATE together with a reasoncode 00C9008E. So, what is too long to wait for a lock? We ll talk about that a bit later in the presentation, but for now, it is enough to know that it s controlled by the installation. Note! If the ROLLBACK fails, and the application does not abend, the application receives SQLCODE -913, SQLSTATE together with a reasoncode 00C9008E. In this case the application is in control of the transaction and can choose to commit or rollback.

8 8 B-2 Timeout (1/2) RESOURCE t S TRANSACTION C 8 This is another example where TRANSACTION C has been granted an S-lock on the resource.

9 9 B-2 Timeout (2/2) TRANSACTION D X timeout RESOURCE t S TRANSACTION C When the wait-time has been exceeded we get a timeout SQLCode -911, SQLState Reasoncode 00C9008E 9 TRANSACTION D comes along requesting an X-lock, which is incompatible with the S-lock of TRANSACTION C. Again, TRANSACTION D will be suspended for the installation defined wait-time. Since TRANSACTION C does not release its lock within the wait-time, DB2 decides to stop waiting and tell the application that it won t get the requested lock. TRANSACTION D will experience a TIMEOUT where DB2 has rolled back the complete transaction. Again, the application will receive an SQLCODE -911, SQLSTATE together with a reasoncode 00C9008E.

10 10 B-3 Deadlock (1/4) TRANSACTION E X RESOURCE 1 t 10 IBMs definition of a deadlock: Unresolvable contention for the use of a resource, such as a table or an index. This example starts off with TRANSACTION E acquiring an X-lock on RESOURCE 1.

11 11 B-3 Deadlock (2/4) TRANSACTION E X RESOURCE 1 RESOURCE 2 t X TRANSACTION F 11 TRANSACTION F comes along and acquires an X-lock on another resource, RESOURCE 2. So far, everything is in order; each transaction has control over its own resource.

12 12 B-3 Deadlock (3/4) TRANSACTION E X S or X RESOURCE 1 RESOURCE 2 t X TRANSACTION F 12 TRANSACTION E is moving along and asks for an S- or X-lock on RESOURCE 2. TRANSACTION E will be suspended, waiting for TRANSACTION F s X-lock to be released. There is still no real problem here apart from that TRANSACTION E could be timed out due to TRANSACTION F holding on to its X-lock for too long (as we ve learnt earlier in the presentation).

13 13 B-3 Deadlock (4/4) TRANSACTION E X S or X RESOURCE 1 RESOURCE 2 t X S or X => deadlock TRANSACTION F This is a classical example of a deadlock SQLCode -911, SQLState Reasoncode 00C But, TRANSACTION F moves along and asks for an S- or X-lock on RESOURCE 1. And suddenly we have a classical deadlock situation where TRANSACTION E is waiting for TRANSACTION F and TRANSACTION F is waiting for TRANSACTION E. When just waiting for a lock, the wait-time could be set to a higher value to make the waiting transaction wait longer and eventually get the lock it asked for when the blocker releases its lock. But, a deadlock will never be solved by waiting. It s a definitive problem that has to be solved by DB2, as early as possible. The installation can control how often DB2 (or rather IRLM) will check for deadlock situations. When detected, DB2 will choose a deadlock victim that will have its transaction rolled back and then notified by SQLCODE -911 and reasoncode 00C The other transaction is then granted the lock it asked for and can continue its processing. Which transaction will DB2 choose as deadlock victim? Basically it will be the transaction that has written the fewest log records. As we will se later, DB2 assign worth values on the transactions and the one with the lowest worth value will be rolled back. In a situation where one of the transactions has updated a NOT LOGGED tablespace and the other has not, DB2 will always chose the transaction that did NOT update a NOT LOGGED tablespace as the victim! That is regardless of how many log records that have been written by the two transactions respectively. If both has updated a NOT LOGGED tablespace, the one that wrote most log records is the winner.

14 14 B-4 Deadlock single resource TRANSACTION E S X RESOURCE t S X => deadlock TRANSACTION F Result: Deadlock 14 A deadlock can also occur on a single resource. This slide shows such an example. TRANSACTION E acquires an S-lock on RESOURCE. Short thereafter, TRANSACTION F also acquires an S-lock on RESOURCE. This is perfectly OK, as we learned in the first part of this presentation. Next, TRANSACTION E asks for an X-lock on RESOURCE and has to wait because the X-lock is not compatible with TRANSACTION F s S-lock. Then, TRANSACTION F asks for an X-lock on RESOURCE and has to wait because the X-lock is not compatible with TRANSACTION E s S-lock. So, the two transactions will wait on each other for ever we have a deadlock situation. Again DB2 has to choose a deadlock victim and rollback its transaction to resolve the situation making it possible for the other transaction to continue processing.

15 15 DB2s behaviour and how to adjust it 15 This section will describe how to influence the behaviour of DB2 in locking conflicts.

16 B-5 Deadlock & Timeout parms The most important parameters are IRLMRWT , default 60 Resource wait time in seconds a multiple of DEADLOK Defined in DSNZPARM Cannot be updated online, DB2 has to be restarted for a change DEADLOK 1 5, , default 1 Time (in seconds or milliseconds) between deadlock-checks Defined in startup JCL for IRLM Can be changed dynamically by MODIFY IRLMPROC command Use milliseconds, e.g. 500 (especially in Data Sharing) 16 IRLMRWT: The value that is specified for this option must be a multiple of the DEADLOCK TIME on installation panel DSNTIPJ because IRLM uses its deadlock timer to initiate time-out detection and deadlock detection. This value is rarely the actual time. For data sharing, the actual timeout period is longer than the time-out value. The IRLMRWT parameter CANNOT be changed dynamically by using the SET PARM= command. DB2 calculates the timeout period as follows: 1. Divide RESOURCE TIMEOUT by DEADLOCK TIME 2. Round to the next largest integer 3. Multiply that integer by DEADLOCK TIME In non-data-sharing systems, the actual time that a transaction waits on a lock before timing out varies between the timeout period and the timeout period plus one DEADLOCK TIME interval. MIN LOCAL TIMEOUT = timeout period MAX LOCAL TIMEOUT = timeout period + DEADLOCK TIME value AVERAGE LOCAL TIMEOUT = timeout period + DEADLOCK TIME value/2 For example, if the timeout period for a given transaction is 60 seconds and the DEADLOCK TIME value is 5 seconds, the transaction waits between 60 and 65 seconds before timing out, with the average wait time of 62.5 seconds. This is because timeout is driven by the deadlock detection process, which is activated on a timer interval basis. In a data sharing environment, because the deadlock detection process sends inter-system XCF messages, a given transaction typically waits somewhat longer before timing out than in a non-data-sharing environment. MIN GLOBAL TIMEOUT = timeout period + DEADLOCK TIME value MAX GLOBAL TIMEOUT = timeout period + 4 * DEADLOCK TIME value AVERAGE GLOBAL TIMEOUT = timeout period + 2 * DEADLOCK TIME value For example, if the timeout period for a given transaction is 60 seconds and the DEADLOCK TIME value is 5 seconds, the transaction waits between 65 and 80 seconds before timing out, with the average wait time of 70 seconds. This is because timeout is driven by the deadlock detection process, which is activated on a timer interval basis. The DEADLOK parameter for IRLM can be changed dynamically by a MODIFY IRLMPROC command. MODIFY irlmproc,set DEADLOCK=nnnn Specifies the number, in milliseconds, indicating how often the local deadlock processing is scheduled. nnnn must be a number from 100 through 5000 milliseconds. If a member of a sysplex group and all IRLMs are not enabled for subsecond deadlock processing, message DXR106E is issued. 16

17 17 B-6 Actual timeout time For non-data-sharing systems MIN LOCAL TIMEOUT = IRLMRWT MAX LOCAL TIMEOUT = IRLMRWT + DEADLOK AVERAGE LOCAL TIMEOUT = IRLMRWT + DEADLOK/2 For data-sharing systems MIN GLOBAL TIMEOUT = IRLMRWT + DEADLOK MAX GLOBAL TIMEOUT = IRLMRWT + 4 * DEADLOK AVERAGE GLOBAL TIMEOUT = IRLMRWT + 2 * DEADLOK 17 DB2 calculates the timeout period as follows: 1. Divide RESOURCE TIMEOUT by DEADLOCK TIME 2. Round to the next largest integer 3. Multiply that integer by DEADLOCK TIME In non-data-sharing systems, the actual time that a transaction waits on a lock before timing out varies between the timeout period and the timeout period plus one DEADLOCK TIME interval. MIN LOCAL TIMEOUT = timeout period MAX LOCAL TIMEOUT = timeout period + DEADLOCK TIME value AVERAGE LOCAL TIMEOUT = timeout period + DEADLOCK TIME value/2 In a data sharing environment, because the deadlock detection process sends inter-system XCF messages, a given transaction typically waits somewhat longer before timing out than in a non-data-sharing environment. MIN GLOBAL TIMEOUT = timeout period + DEADLOCK TIME value MAX GLOBAL TIMEOUT = timeout period + 4 * DEADLOCK TIME value AVERAGE GLOBAL TIMEOUT = timeout period + 2 * DEADLOCK TIME value

18 18 B-7 Example of timeout times IRLMRWT = 5 and DEADLOK = 500 For non-data-sharing systems Min 5 seconds Max 5.5 seconds Avg 5.25 seconds For data-sharing systems Min 5.5 seconds Max 7 seconds Avg 6 seconds Note: Max and average values can be larger, depending on # of waiters or heavy load on IRLM 18 Let s calculate the theoretical timeout times that we could expect using the formulas on the previous slide. First, for a non-data-sharing system For example, if the timeout period for a given transaction is 5 seconds and the DEADLOCK TIME value is 1 second, the transaction waits between 5 and 5.5 seconds before timing out, with the average wait time of 5.25 seconds. This is because timeout is driven by the deadlock detection process, which is activated on a timer interval basis. And then, for a data-sharing system For example, if the timeout period for a given transaction is 5 seconds and the DEADLOCK TIME value is 500 milliseconds, the transaction waits between 5.5 and 7 seconds before timing out, with the average wait time of 6 seconds. This is because timeout is driven by the deadlock detection process, which is activated on a timer interval basis. However, the maximum or average values can become larger, depending on the number of waiters in the system or if a heavy IRLM workload exists.

19 19 B-8 Timeout multiplier by type Type IMS BMPs IMS DL/I batch IMS Fast Path Nonmessage processing BIND subcommand processing STOP DATABASE command processing Utilities Retained locks for all types Application accessing not logged tablespaces All other types Multiplier >= 3 1 Modifiable yes yes no no no yes yes no no Parameter NOTE! All modifiable timeout multipliers can be changed online BMPTOUT (1-254) DLITOUT (1-254) UTIMOUT (1-254) RETLWAIT (0-254) A timeout hits different load in different ways. An IMS transaction that is timed out while waiting for a lock, automatically gets rescheduled which means that the user probably doesn t even notice the timeout, apart from a longer response time. A batch job that has done an amount of updates to the database will take longer to rollback, and it has to execute its restart logic to continue its work. Or, if it s a DL/I Batch it will abend and terminate when hit by the timeout and has to be physically restarted. DB2 provides controls to set multipliers for the timeout period based on what type of processing is going on. The table on this page shows the different types, the default multiplier, whether it is possible to modify it, and the name of the parameter, if any. Retained lock wait (RETLWAIT) works a bit different as compared to the other timeout multipliers. RETLWAIT indicate how long a transaction should wait for a lock on a resource if another DB2 in a data sharing group has failed and is holding an incompatible lock on that resource. If you use the default, 0, applications do not wait for incompatible retained locks, but instead the lock request is immediately rejected, and the application receives a resource unavailable SQLCODE. The value that you use is a multiplier that is applied to the connection's normal time-out value. For example, if the retained lock multiplier is 2, the timeout period for a call attachment connection that is waiting for a retained lock is 1 * 2 (1 for the normal CAF timeout period, 2 for the additional time that is specified for retained locks). In other words, it is a multiplier to be multiplied with one of the other multipliers. In DB2 9 you can turn off logging on a tablespace. When an application is accessing such a tablespace, DB2 will guarantee that it will have a multiplier that is at least three. For example, an IMS transaction will have three times the IRLMRWT while a DL/I batch will have six times the IRLMRWT. All modifiable timeout multipliers can be changed online via the SET PARM command.

20 20 B-9 Longrunners Three important values in DSNZPARM URCHKTH 0 255, default 0 Warning if application has been active during x checkpoints without committing URLGWTH , default 0 Warning if application has written x thousand log-records without committing LRDRTHLD , default 0 Warning if a read claim has been held more than x minutes These warnings are written to the SYSLOG and as trace records when Statistics Trace Class 3 is on 20 We want to avoid all types of locking conflicts. If we can make programs behave properly, they won t create problems for other programs accessing the same tables. We have to avoid creating timeouts which primarily can be done by frequent commits and we have to avoid deadlocks which is more a data access pattern issue. Long-running transactions might also result in a lengthy DB2 restart or a lengthy recovery situation for critical tables. DB2 provides ways to detect programs that doesn t commit as frequent as they should. This function is controlled by three parameters; URCHKTH, URLGWTH, and LRDRTHLD. URCHKTH specifies the number of checkpoint cycles that are to complete before DB2 issues a warning message to the console and instrumentation for an uncommitted unit of recovery (UR). Specify a value that is based on how often a checkpoint occurs in your system and how much time you can allow for a restart or shutdown. For example, if your site s checkpoint interval is 5 minutes and the standard limit for issuing commits with units of recovery is 20 minutes, divide 20 by 5 to determine the best value for your system. URLGWTH specifies the number of log records that are to be written by an uncommitted unit of recovery (UR) before DB2 issues a warning message to the console and instrumentation. Specify the value in 1-K (1000 log records) increments. LRDRTHLD specifies the number of minutes that a read claim can be held by an agent before DB2 before DB2 issues a warning message to the console and instrumentation to report it as a long-running reader. All the above ZPARMs can be changed online with the SET PARM command.

21 21 Messages and Instrumentation 21 This section will describe the two ways that DB2 communicates with its administrators about locking conflict events.

22 22 B-10 Timeout - application select deptname from dsn8910.dept DEPTNAME DSNE610I NUMBER OF ROWS DISPLAYED IS 0 DSNT408I SQLCODE = -911, ERROR: THE CURRENT UOW HAS BEEN ROLLED BACK DUE TO DEADLOCK OR TIMEOUT. REASON 00C9008E, TYPE OF RESOURCE , AND RESOURCE NAME DSN8D91A.DSN8S91D.X'000002' DSNT418I SQLSTATE = SQLSTATE RETURN CODE DSNT415I SQLERRP = DSNXRRC SQL PROCEDURE DETECTING ERROR DSNT416I SQLERRD = SQL DIAGNOSTIC INFORMATION DSNT416I SQLERRD = X'FFFFFF42' X'FFFFFF9C' X'00C9008E' X'00D2C6E3' X'C3C81005' X' ' SQL DIAGNOSTIC INFORMATION 22 In this example we have an ordinary timeout situation on a tablespace page in the DSN8910.DEPT table. The SQLCA contains the SQLCODE, SQLSTATE, reasoncode, resource type, as well as the resource name. In this example, we are using SPUFI that in turn uses DSNTIAR to format the contents of the SQLCA. Please note that SQLCA does not contain any information on other processes involved in the timeout. SQLCode -911 SQLState ReasonCode 00C9008E Type of Resource = Table Space Page (resource types are documented in Appendix A of the DB2 9 Messages manual) Resource Name DSN8D91A.DSN8S91D.X i.e. page 2 in the DSN8S91D tablespace in database DSN8D91A

23 23 B-11 Timeout - SYSLOG DSNT376I -GT8G PLAN=DSNESPCS WITH CORRELATION-ID=GOLD106 CONNECTION-ID=TSO LUW-ID=ADCD.GT9GLU1.C =315 THREAD-INFO=GOLD106:*:*:* IS TIMED OUT. ONE HOLDER OF THE RESOURCE IS PLAN=DSNESPCS WITH CORRELATION-ID=GOLD105 CONNECTION-ID=TSO LUW-ID=ADCD.GT9GLU1.C232096C408D=313 THREAD-INFO=GOLD105:*:*:* ON MEMBER GT9G DSNT501I -GT8G DSNILMCL RESOURCE UNAVAILABLE CORRELATION-ID=GOLD106 CONNECTION-ID=TSO LUW-ID=ADCD.GT9GLU1.C =0 REASON 00C9008E TYPE NAME DSN8D91A.DSN8S91D.X'000002' 23 DB2 will document the timeout on the MVS console. Here we get some more information since DB2 is giving us the identity of ONE HOLDER OF THE RESOURCE in the DSNT376I message. DSNT501I gives us the reason code, resource type (resource types are documented in Appendix A of the DB2 9 Messages manual) and the resource name.

24 B-12 Display locks (DB2 V8) -dis db(dsnd881a) locks DSNT360I -GT8G *********************************** DSNT361I -GT8G * DISPLAY DATABASE SUMMARY * GLOBAL LOCKS DSNT360I -GT8G *********************************** DSNT362I -GT8G DATABASE = DSN8D81A STATUS = RW DBD LENGTH = DSNT397I -GT8G NAME TYPE PART STATUS CONNID CORRID LOCKINFO DSN8S81D TS RW TSO GOLD106 H-IS,S,C DSN8S81D TS RW TSO GOLD105 H-IX,S,C DSN8S81E TS -THRU 0004 DSN8S81P TS 0001 RW RW 24 DB2 gives you the ability to get a snapshot on what locks are held on a database by the command DISPLAY DATABASE() LOCKS. If there is a longrunning batch job that has acquired a lock on the resource and does not COMMIT frequent enough, you might spot the lock holder(s) by using this command. The example on the slide does not show the locking conflict situation mentioned in the previous slides. It just want to show an example of what information is available in DB2. As implied above, there are very rare situations where you can spot a locking conflict by using the command. Furthermore, as soon as you have a timeout or deadlock reported, the locking conflict is already resolved and there is no information to be shown by DB2. In this example of command output from DISPLAY DATABASE(DSND881A) LOCKS on a DB2 V8 subsystem, we can see that there are two lock holders on the DSN8S91D tablespace in database DSN881A, namely GOLD106 and GOLD105. The LOCKINFO consists of a lock status, followed by a dash -, a lock state, a lock type and a lock duration delimited by commas. For this example, we can see that GOLD106 has a status of Hold, a lock state/mode of IS, a table space type of lock (S), with a Commit duration on the DSN8S81D tablespace. We can also see that GOLD105 has a status of Hold, a lock state/mode of IX, a table space type of lock (S), with a Commit duration on the very same tablespace. As we learned in the first part of this presentation, a lock in IS mode is compatible with a lock in IX mode, so everything is in order. The problem with this display output is to identify the thread that each row refers 24

25 25 B-13 Display locks (DB2 9) -dis db(dsn8d91a) locks DSNT360I -GT9G *********************************** DSNT361I -GT9G * DISPLAY DATABASE SUMMARY * GLOBAL LOCKS DSNT360I -GT9G *********************************** DSNT362I -GT9G DATABASE = DSN8D91A STATUS = RW DBD LENGTH = DSNT397I -GT9G NAME TYPE PART STATUS CONNID CORRID LOCKINFO DSN8S91D TS RW TSO GOLD106 H-IS,S,C - AGENT TOKEN 278 DSN8S91D TS RW TSO GOLD105 H-IX,S,C - AGENT TOKEN TB TSO GOLD106 H-IS,T,C - AGENT TOKEN TB TSO GOLD105 H-IX,T,C - AGENT TOKEN In DB2 9, there is an enhancement in the output from the DISPLAY DATABASE() LOCKS command. There is now an additional line in the output identifying the lock holders thread by its agent token. By some reason, this is only true for local lock holders. In this example, there are also additional information on table locks held by the two agents. Table names does not fit in the tablespace name column so they are represented with a number, an OBID within the database. The content of the different fields is documented under message DSNT397I in the DB2 9 Messages manual. Compatibility of different lock modes is documented in the section Compatibility of lock modes in the DB2 9 Performance Monitoring and Tuning Guide.

26 26 B-14 Timeout - trace PRIMAUTH CONNECT ORIGAUTH END_USER WS_NAME ORIGAUTH CORRNAME CONNTYPE PLANNAME DESCRIPTION GOLD106 TSO GOLD106 'BLANK' 'BLANK' GOLD106 GOLD106 TSO DSNESPCS TIMEOUT DATA TIMEOUT HEADER NUMBER OF HOLDERS/WAITERS: 1 LOCK HASH VALUE: X' ' LOCK RES TYPE: DATA PAGE LOCK DBID: 266 OBID: 2 REQUESTED FUNCTION: LOCK REQUESTED STATE: SHARED REQUESTED DURATION: MANUAL WAITERS CACHED STMT ID: X' D... H O L D E R PRIMAUTH : GOLD105 PLAN NAME: DSNESPCS CORR ID: GOLD105 CONN: TSO LOCK STATE: EXCLUSIVE LOCK DURATION: COMMIT MEMBER: GT8G TRANSACT : 'BLANK' WS_NAME: 'BLANK' END_USER: 'BLANK' STMT ID : X' ' 26 As mentioned earlier, DB2 will also write a trace record for timeouts if STATISTICS CLASS(3) is started (or if explicitly asked for). This slide shows the output from OMEGAMON for DB2 batch report for IFCID 196. The trace record has a standard header that identifies the thread that experienced the timeout and got rolled back. In this case it is the GOLD106 user. The timeout header shows information about what kind of lock the thread requested and on what resource. In this example, GOLD106 requested a shared lock on tablespace DSN8S91D (DBID 266, OBID 2 see previous list from the catalog) with duration manual. The holder part of the trace record shows that the holder of the incompatible lock is GOLD105 that has an exclusive lock with duration COMMIT. We need to know more about the two threads involved in this locking conflict and DB2 provides us with a hint on the SQL statements that are involved. In the timeout header you can find the waiters cached stmt id, in this case it s X D. And for the holder the stmt id is X These numbers refer to the SQL statement cache, which is used by dynamic SQL. Now, we only have to look into the cache to find the SQL statements

27 27 B-15 Timeout - SQL WAITERS CACHED STMT ID: X' D' = 109 Explain stmtcache all or Explain stmtcache stmtid 109 Select from Dsn_Statement_Cache_Table where stmtid = GOLD106 select deptname from dsn8810.dept H O L D E R STMT ID : X' ' = GOLD105 update dsn8810.dept set deptname ='Planning' where deptno = 'B01' 27 The waiters cached stmt id was X D which translates to 109 in decimal notation. The holders stmt id was X which translates to 68 in decimal notation. One way to read the statement cache is to use the EXPLAIN STMTCACHE statement which will returns rows for cached statement to the DSN_STATEMENT_CACHE_TABLE. These rows contain identifying information about the statements in the cache, as well as statistics that reflect the execution of the statements by all processes that have executed the statement. Running a SELECT statement against the DSN_STATEMENT_CACHE_TABLE will display the SQL statement in question.

28 28 B-16 Deadlock - application UPDATE DSN8810.PROJ SET PROJNAME ='Planning' WHERE projno='op1010'; DSNE615I NUMBER OF ROWS AFFECTED IS 1 DSNE616I STATEMENT EXECUTION WAS SUCCESSFUL, SQLCODE IS SELECT deptname FROM DSN8810.DEPT; DEPTNAME DSNE610I NUMBER OF ROWS DISPLAYED IS 0 DSNT408I SQLCODE = -911, ERROR: THE CURRENT UOW HAS BEEN ROLLED BACK DUE TO DEADLOCK OR TIMEOUT. REASON 00C90088, TYPE OF RESOURCE , AND RESOURCE NAME DSN8D81A.DSN8S81D.X'000002' DSNT418I SQLSTATE = SQLSTATE RETURN CODE DSNT415I SQLERRP = DSNXRRC SQL PROCEDURE DETECTING ERROR DSNT416I SQLERRD = SQL DIAGNOSTIC INFORMATION DSNT416I SQLERRD = X'FFFFFF42' X'FFFFFF9C' X'00C90088' X'00D2C6E3' X'C3C81005' X' ' SQL DIAGNOSTIC INFORMATION 28 This transaction updates the PROJ table and then tries to read from the DEPT table. Another transaction, that this transaction doesn t know anything about, has probably been updating the DEPT table and then tried to read from the PROJ table. DB2 detects a deadlock and finds out who is going to be the victim. The deadlock victim is then rolled back and notified by returning control after the SELECT against the DEPT table with SQLCODE -911 and reason code 00C The example is using SPUFI which in turn uses DSNTIAR to format the DSN408I message. The message also contains the resource type and resource name. In this case it is page 2 in tablespace DSN8S81D in database DSN8D81A. No information about the other transaction is provided in the SQLCA.

29 29 B-17 Deadlock - SYSLOG DSNT375I -GT8G PLAN=DSNESPCS WITH CORRELATION-ID=GOLD106 CONNECTION-ID=TSO LUW-ID=ADCD.GT8GLU1.C2320F682B81=338 THREAD-INFO=GOLD106:*:*:* IS DEADLOCKED WITH PLAN=DSNESPCS WITH CORRELATION-ID=GOLD105 CONNECTION-ID=TSO LUW-ID=ADCD.GT8GLU1.C2320F58CDCA=337 THREAD-INFO=GOLD105:*:*:* ON MEMBER GT8G DSNT501I -GT8G DSNILMCL RESOURCE UNAVAILABLE CORRELATION-ID=GOLD106 CONNECTION-ID=TSO LUW-ID=ADCD.GT8GLU1.C2320F682B81=0 REASON 00C90088 TYPE NAME DSN8D81A.DSN8S81D.X'000002' 29 DB2 will document the deadlock on the MVS console. Here we get some more information since DB2 is giving us the identity of the other threads that were involved in the deadlock in the DSNT375I message. In the explanation of DSNT375 you can read Plan plan-id2 identifies one of the members of the deadlock. DB2 does not attempt to identify all survivors of a deadlock or all participants in a deadlock in the DSNT375I message. Plan plan-id2 in message DSNT375I might be just one of several plans holding locks on the desired resource. DSNT501I gives us the reason code, resource type (resource types are documented in Appendix A of the DB2 9 Messages manual) and the resource name.

30 30 B-18 Deadlock trace (1/2) DEADLOCK HEADER INTERVAL COUNT: WAITERS INVOLVED: 2 TIME DETECTED: 08/08/08 18:57:24.42 R E S O U R C E LOCK RES TYPE: ROW LOCK DBID: 266 OBID: 25 RESOURCE ID: X' E' B L O C K E R PRIMAUTH : GOLD106 PLAN NAME : DSNESPCS CORR ID : GOLD106 CONN ID : TSO MEMBER : N/A DURATION : COMMIT STATE : EXCLUSIVE ACE : 2 TRANSACTION : 'BLANK' WS_NAME : 'BLANK' END_USER: 'BLANK PROGRAM NAME: DSNESM68 LOCATION : 'BLANK' PCKG/COLL ID: DSNESPCS CONS TOKEN : X'149EEA901A79FE48' STMT ID : X' E' STATUS : HOLD QW0172HF: X'12' W A I T E R PRIMAUTH : GOLD105 PLAN NAME : DSNESPCS CORR ID : GOLD105 CONN ID : TSO MEMBER : N/A DURATION : MANUAL STATE : SHARED ACE : 3 TRANSACTION : 'BLANK' WS_NAME : 'BLANK' END_USER: 'BLANK PROGRAM NAME: DSNESM68 LOCATION : 'BLANK' PCKG/COLL ID: DSNESPCS CONS TOKEN : X'149EEA901A79FE48' STMT ID : X' ' WORTH : X'12' QW0172WG: X'30' 30 As mentioned earlier, DB2 will also write a trace record for deadlocks if STATISTICS CLASS(3) is started (or if explicitly asked for). This slide shows the output from OMEGAMON for DB2 batch report for IFCID 172. The trace record has a standard header that identifies the thread that was chosen as the deadlock victim and got rolled back. It is not included in the printout, but in this case it is the GOLD106 user. This slide shows the first resource involved in the deadlock. It is a row lock on table DSN8810.PROJ in database DSN8D81A (DBID 266, OBID 25). The blocker is GOLD106 which holds an X-lock with duration COMMIT on row x E. We also have information down to package version to identify the running program. We also have the cached statement id available. The waiter for this lock is GOLD105 that is requesting an S-lock on the very same row for manual duration. Please note that the waiter has got a worth value associated with it. The worth value is X 12 (18 in decimal notation).

31 31 B-19 Deadlock trace (2/2) R E S O U R C E LOCK RES TYPE: DATA PAGE LOCK DBID: 266 OBID: 2 RESOURCE ID: X' ' B L O C K E R PRIMAUTH : GOLD105 PLAN NAME : DSNESPCS CORR ID : GOLD105 CONN ID : TSO MEMBER : N/A DURATION : COMMIT STATE : EXCLUSIVE ACE : 3 TRANSACTION : 'BLANK' WS_NAME : 'BLANK' END_USER: 'BLANK PROGRAM NAME: DSNESM68 LOCATION : 'BLANK' PCKG/COLL ID: DSNESPCS CONS TOKEN : X'149EEA901A79FE48' STMT ID : X' ' STATUS : HOLD QW0172HF: X'12' W A I T E R PRIMAUTH : GOLD106 PLAN NAME : DSNESPCS CORR ID : GOLD106 CONN ID : TSO MEMBER : N/A DURATION : MANUAL STATE : SHARED ACE : 2 TRANSACTION : 'BLANK' WS_NAME : 'BLANK' END_USER: 'BLANK PROGRAM NAME: DSNESM68 LOCATION : 'BLANK' PCKG/COLL ID: DSNESPCS CONS TOKEN : X'149EEA901A79FE48' STMT ID : X' F' WORTH : X'11' QW0172WG: X'30' 31 This is the second resource involved in the deadlock. It is a page lock on a page in tablespace DSN8S91D in database DSN8D91A (DBID 266, OBID 2). The blocker is GOLD105 which holds an X-lock with duration COMMIT on page x We have information down to package version to identify the running program. We also have the cached statement id available. The waiter for this lock i GOLD106 that is requesting an S-lock on the very same page for manual duration. Please note that the waiter for this resource has got a worth value of X 11 (17 in decimal notation). According to the message in a previous slide, GOLD106 is the one chosen as deadlock victim and being rolled back. It s due to the fact that GOLD106 has the lowest worth value (x 11 as compared to x 12 ).

32 32 B-20 Deadlock - SQL R E S O U R C E LOCK RES TYPE: ROW LOCK DBID: 266 OBID: 25 RESOURCE ID: X' E' B L O C K E R STMT ID : X' E' = GOLD106 UPDATE DSN8810.PROJ SET PROJNAME ='Planning' WHERE projno='op1010' W A I T E R STMT ID : X' ' = GOLD105 select projname from dsn8810.proj R E S O U R C E LOCK RES TYPE: DATA PAGE LOCK DBID: 266 OBID: 2 RESOURCE ID: X' ' B L O C K E R STMT ID : X' ' = GOLD105 update dsn8810.dept set deptname ='Planning' where deptno = 'B01' W A I T E R STMT ID : X' F' = GOLD106 SELECT deptname FROM DSN8810.dept 32 When combining the deadlock trace from OMEGAMON for DB2 and the contents of the statement cache, you can get a clear picture of what has happened and the source of the locking conflict. In this slide you can see how GOLD106 updates the PROJ table and that GOLD105 updates the DEPT table and that GOLD106 waits to select from the DEPT table at the same time as GOLD105 waits to select from the PROJ table. A classical deadlock situation!

33 33 B-21 Lockmax - application Create tablespace nomax in gold106d lockmax 0 locksize row; Create table tm (c1 integer not null) in gold106d.nomax; Insert into tm with t(n) as (select 1 from sysibm.sysdummy1 union all select n+1 from t where n<20000) select n from t; 57011(-904) Unsuccessful execution caused by an unavailable resource. Reason code: "00C90096", type of resource: " ", and resource name: "GOLD106D.NOMAX.X'000029' '.X'38'". 00C90096 NUMLKUS exceeded 39 (x 27 ) pages * 255 rows + 56 (x 38) rows = rows (locks) 33 In this example, we will force a situation where NUMLKUS will be exceeded. We do it by telling DB2 to shut off lock escalation (which, by the way, is a mechanism to avoid excessive number of locks) by setting LOCKMAX 0 on the tablespace, requesting row locking by setting LOCKSIZE ROW and then try to insert rows into the table tm. LOCKMAX specifies the maximum number of page, row, or LOB locks an application process can hold simultaneously in the table space. If a program requests more than that number, locks are escalated. Specifying zero (0), indicates that the number of locks on the table or table space are not counted and escalation does not occur. Resource 304 = Tablespace RID Since MAXROWS is not defined on the tablespace, the default number of rows on a page is 255. According to the message, NUMLKUS is reached on page x 29 and row number x 38. Since page x 0 is the header page and page x 1 is the space map page, we can calculate the NUMLKUS for this DB2 system. We have x 27 pages of 255 rows and trieds to insert row number x 38 when we encounter the x 27 is 39 in decimal notation and x 38 is 56 in decimal notation. The number of rows added = 39* = which tells us that NUMLKUS is 10000, the default value! Note that this situation is only a problem for the application in itself, so DB2 will not put any information about this on the MVS console or in trace records called by STATISTICS CLASS(3).

34 34 B-22 Lock escalation - SYSLOG Alter tablespace gold106d.nomax lockmax 700; Insert into tm with t(n) as (select 1 from sysibm.sysdummy1 union all select n+1 from t where n<20000) select n from t; DSNI031I -GT9G DSNILKES - LOCK ESCALATION HAS OCCURRED FOR RESOURCE NAME = GOLD106.LOCKMAX LOCK STATE = X PLAN NAME : PACKAGE NAME = DISTSERV : SYSSH200 COLLECTION-ID = NULLID STATEMENT NUMBER = CORRELATION-ID = aqt.exe CONNECTION-ID = SERVER LUW-ID = C0A800A7.ODCB THREAD-INFO = GOLD106 : BACKLUND-X60 : gold106 : aqt.exe 34 We now tell DB2 to allow a maximum of 700 locks on our tablespace and then insert 2000 rows into the table tm. It works like a charm, the application will get SQLCODE 0 after the completion of the insert. However, under the covers, DB2 has escalated the row locks to a tablespace lock. This can severely impact other users of this tablespace, so DB2 will put information about the event on the MVS console and in trace records. Note that LOCKMAX can also be set to SYSTEM, which means that DB2, for this tablespace, will use the subsystem wide default lockmax aka NUMLKTS which has a default value of Also note the detailed information about what statement is hit by the lock escalation. This is great for a program using static SQL, but for dynamic it doesn t help much.

35 35 B-23 Trace records to care for IFCID 172 Deadlocks IFCID 196 Timeouts IFCID 313 Long running URs Based on # of checkpoints (URCHKTH) Based on # of logrecords (URLGWTH) Based on minutes for a read claim (LRDRTHLD) IFCID 337 Lock escalations 35 These are the four important trace records when it comes to locking conflicts. As we recall from an earlier slide in this presentation, for DB2 to catch long running URs there are three zparms that can be used; URCHKTH, URLGWTH, and LRDRTHLD. The IFCID 313 and 337 are rather straight forward. They describe the culprit and when its bad behavior was detected by DB2. The IFCID 172 and 196 is somewhat more complex in structure. We will look into that on the next slide.

36 36 B-24 IFCID Deadlock OP Header LL Headers QW0172 (resource) Hash Value Locking Flag QW0172HE Resource Name 1st QW0172 Holders ID Member Lock State QW0172HE Interval Counter Lock Duration # of resources STCK Waiters ID Lock Function Requested State Lock Duration Worth Member Holders ID Cached Stmt ID Package Name Collection Name Consistency Token Waiters ID Cached Stmt ID Package Name Collection Name Consistency Token As many QW0172 as there are resources Standard Header IFCID # of areas SSID STCK IFCID Seq # DEST Seq # Trace Mask Trace ID Commit Count Correlation Header AUTHID CORRID CONNID PLAN OPERID Data Sharing Header Member Name Group Name 36 This is a pictorial way to illustrate the contents of the IFCID 172 trace record. It is NOT complete nor fully accurate but is meant to serve as a tool to get to understand how these records are put together. The IFCID 172 is written based on the thread that was choosen as a deadlock victim. The correlation header tells you which thread was rolled back. QW0172HE is the deadlock header that tells you when the deadlock was detected and how many resources that was involved in the deadlock. Holders ID and Waiters ID are not fields in the trace record, but denotes a number of fields including planname, correlation id, connection id, LUW id, thread token as well as client information. QW0172 is the resource record that tells you about the holder and the waiters for the resource. There is one QW0172 for each resource. The other headers are standard DB2 instrumentation headers. The Distributed Header is not included on the slide.

37 37 B-25 IFCID Timeout OP Header LL Headers QW0196HE 1st QW0196 QW0196HE (resource) # of holders Hash Value Resource Name Lock Function Lock State Lock Duration Req Owning WU Timeout Interval Timeout Counter Waiter Stmt ID QW0196 Holder or prio waiter Holders ID Owning WU Member Lock State Lock Duration Holders Stmt ID As many QW0196 as there are holders Standard Header IFCID # of areas SSID STCK IFCID Seq # DEST Seq # Trace Mask Trace ID Commit Count Correlation Header AUTHID CORRID CONNID PLAN OPERID Data Sharing Header Member Name Group Name 37 This is a pictorial way to illustrate the contents of the IFCID 196 trace record. It is NOT complete nor fully accurate but is meant to serve as a tool to get to understand how these records are put together. The IFCID 196 is written based on the thread that was timed out. The correlation header tells you which thread was rolled back. QW0196HE is the timeout header that tells you about the resource and the lock requested by the waiter as well as its cached statement id. It also shows how many holders that were involved in the timeout. QW0196 is the record that identify a holder or priority waiter, its held or requested lock and its cached statement id. A priority waiter is another transaction that is also waiting for a lock on the same resource, but is before the timed out transaction in the queue. There is one QW0196 for each holder/priority waiter. Holders ID is not a field in the trace record, but denotes a number of fields including planname, correlation id, connection id, LUW id, thread token as well as client information. The other headers are standard DB2 instrumentation headers. The Distributed Header is not included on the slide.

38 38 B-26 How to identify the culprit By messages * New message in DB2 9 DSNT375I Deadlock PLAN YES COLLECTION - PACKAGE - CONSISTE NCY TOKEN - STATEMENT NUMBER - CACHED STATEMENT ID - DSNT376I Timeout YES R035I, J031I Longrunner YES N/A N/A N/A N/A N/A DSNI031I Lock escalation YES YES YES - YES - DSNU120I* Deadlock YES DSNU121I* Timeout YES The MVS console messages don t provide all the information we need, they have information that relates them back to a unique transaction. But, we also want to be able to track down the individual SQL statements that actually created the locking conflict. Longrunning transactions are different to the other three events since DB2 is only reporting a potential for a transaction to create a locking conflict. Most likely there is no individual SQL statement that creates the potential for locking conflicts, so there is no need to point out the statement. The only message that references an SQL statement is DSNI031I that provides a statement number that references an SQL statement in a package. In DB2 9 for z/os there are two new messages, DSNU120I and DSNU121I, related to utilities that encounter a deadlock or a timeout. Please see the Reference Materials for the structure of these messages. These messages are similar to DSNT375I and DSNT376I respectively, but contains additional information normally found only in trace records. Still, they don t contain information to pinpoint the SQL-statement involved in the locking conflict.

39 39 B-27 How to identify the culprit By trace records PLAN COLLECTION PACKAGE CONSIST ENCY TOKEN STATEMENT NUMBER CACHED STATEMENT ID 172 Deadlock YES YES(a) YES(a) YES - YES 196 Timeout YES YES 313 Longrunner YES N/A N/A N/A N/A N/A 337 Lock escalation YES YES YES - YES YES (a) Does not support 128 bytes long unicode names 39 The trace records have information that relates them back to a unique transaction. But, we also want to be able to track down the individual SQL statements that actually created the locking conflict. Longrunning transactions are different to the other three events since DB2 is only reporting a potential for a transaction to create a locking conflict. Most likely there is no individual SQL statement that creates the potential for locking conflicts, so there is no need to point out the statement. In IFCID 172, the collection name has a maximum of 18 unicode characters and the package name has a maximum of 8 unicode characters. IFCID 337 is the only one that references static SQL statements in package. IFCID 172, IFCID 196, and IFCID 337 all reference dynamic SQL statements in the dynamic statement cache. We suggest that DB2 development take measures to make IFCID 172, IFCID 196 complete based on what is already done in IFCID 337 For IFCID 172 we want statement number for static SQL as well as support for long names of collection and package For IFCID 196 we want statement number for static SQL and to have collection, package, and consistency token added (with long names where it applies) For IFCID 337 we want to have consistency token added

40 40 An additional example 40 Here is an additional example that shows what happens if we run into a deadlock situation with three resources and three threads involved.

41 41 B-28 Three-way Deadlock TRANSACTION G TRANSACTION H TRANSACTION I S RESOURCE 1 X => deadlock t S RESOURCE 2 X S RESOURCE 3 X 41 This slide has a different way to show a deadlock compared to the slides in the beginning of this presentation. However, it shows a situation where we have three transactions running against three resources and they end up in a deadlock. TRANSACTION I is granted an S-lock on RESOURCE 1, TRANSACTION G is granted an S-lock on RESOURCE 2, and TRANSACTION H is granted an S-lock on RESOURCE 3. Then, TRANSACTION I requests an X-lock on RESOURCE 2, TRANSACTION G requests an X-lock on RESOURCE 3. So far, so good. At this stage, we have two potential timeouts coming along. Then lastly, TRANSACTION H requests an X-lock on RESOURCE 1. And suddenly, we are in a deadlock situation where all three transactions are waiting for each other in a ring. I is waiting for G, G is waiting for H, and H is waiting for I. The three transactions end up in a deadlock. How will DB2 handle this situation? Let s see on the next slide!

42 42 B-29 The resolution TRANSACTION I TRANSACTION H TRANSACTION G DB2 choses to ROLLBACK TRANSACTION G Its S-lock on R 2 is released Its request for X-lock on R 3 is cancelled TRANSACTION I gets its X-lock on R 2 TRANSACTION H has to wait for its X-lock on R 1 42 OK, so we now have three transactions waiting in a ring. TRANSACTION I is waiting for TRANSACTION G, TRANSACTION G is waiting for TRANSACTION H, and TRANSACTION H is waiting for TRANSACTION I. DB2 has to break the deadlock in some way, and in this it choses to rollback TRANSACTION G. This means that its s-lock on RESOURCE 2 is released and its request for X-lock on RESOURCE 3 is cancelled. This in turn means that TRANSACTION I gets its X-lock on RESOURCE 2 and that TRANSACTION H has to wait for its X-lock on RESOURCE 1 until TRANSACTION I releases it. So, we might run into a timeout situation here if TRANSACTION I doesn t release its lock in time.