Oracle 10g Performance Case Studies

Transcription

1 Oracle 10g Performance Case Studies Martin Frauendorfer Technical Support Consultant, SAP AG 1

2 Table of Contents. Overview Enqueue Analysis Runtime Analysis I/O Analysis Comparison Analysis Summary SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 2 2

3 1. Overview Intension Oracle 10g provides plenty of new features and possibilities in the area of performance analysis, e.g.: Active Session History Automatic Workload Repository Advisors Improvements in the wait event area Improvements in the SQL optimization area This presentation intends to show some real-life examples of how Oracle 10g features can effectively support the analysis and resolution of performance problems. With Oracle <= 9i DBAs often had to say: Sorry, this problem happened before but not any longer there s no way I can analyze it now. With Oracle 10g DBAs can often determine the root cause even if the error situation no longer exists. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 3 3

4 1. Overview The following case studies are part of this presentation: Enqueue Analysis: Shows how to determine details for a past enqueue wait situation Runtime Analysis: Shows how to determine details for a past long running transaction I/O Analysis: Shows how to determine detect bottleneck situations in the I/O area Comparison Analysis: Shows how to compare different time frames in terms of Oracle performance SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 4 4

5 2. Enqueue Analysis (1) A check for the distribution of relevant Oracle server time revealed: SELECT EVENT, TOTAL_WAITS, TIME_WAITED, AVG_MS, ROUND(RATIO_TO_REPORT(TIME_WAITED) OVER () * 100) PERCENT FROM ( SELECT SUBSTR(EVENT, 1, 30) EVENT, TOTAL_WAITS, TIME_WAITED, ROUND(TIME_WAITED_MICRO / TOTAL_WAITS / 1000, 2) AVG_MS FROM V$SYSTEM_EVENT WHERE WAIT_CLASS!= 'Idle' AND EVENT NOT IN ('db file parallel write', 'log file parallel write', 'log file sequential read', 'control file parallel write', 'control file sequential read', 'Log archive I/O') UNION SELECT 'CPU' EVENT, NULL, VALUE, NULL FROM V$SYSSTAT WHERE STATISTIC# = 12 ORDER BY 3 DESC) WHERE ROWNUM <= 10; 16 % of the time spent waiting for TX enqueues! EVENT TOTAL_WAITS TIME_WAITED AVG_MS PERCENT db file sequential read , , enq: TX - row lock contention , , CPU 0, ,00 9 log file sync , , db file scattered read , , read by other session , , Datapump dump file I/O , , direct path write temp , , direct path read temp , , SQL*Net message to client , , SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 5 This check is a very useful step to get a first impression what happens inside a database in terms of performance. The EVENT NOT IN condition is necessary because some events are additional idle events from an SAP perspective. The new Oracle 10g categorization WAIT_CLASS = Idle was used in order to make sure that uncritical idle wait events are not considered. The results indicate that, since database startup, 16 % of the relevant Oracle server time was caused by TX enqueues. This is much more than we usually want to see and can have significant impact on the database performance for specific transactions or during specific time frames. 5

6 2. Enqueue Analysis (2) In the following we check when there was an enqueue peak time: SELECT TO_CHAR(END_INTERVAL_TIME, 'YYYY-MM-DD HH24:MI:SS') END_INTERVAL_TIME, TOTAL_WAITS, TIME_WAITED_MICRO, ROUND(DECODE(TOTAL_WAITS, 0, 0, TIME_WAITED_MICRO / TOTAL_WAITS / 1000), 2) AVG_WAIT_MS FROM ( SELECT HSS.END_INTERVAL_TIME, HSE.EVENT_NAME, HSE.TOTAL_WAITS - LAG(HSE.TOTAL_WAITS, 1) OVER (ORDER BY HSS.SNAP_ID) TOTAL_WAITS, HSE.TIME_WAITED_MICRO - LAG(HSE.TIME_WAITED_MICRO, 1) OVER (ORDER BY HSS.SNAP_ID) TIME_WAITED_MICRO FROM DBA_HIST_SYSTEM_EVENT HSE, DBA_HIST_SNAPSHOT HSS WHERE HSE.SNAP_ID = HSS.SNAP_ID AND HSE.EVENT_NAME = 'enq: TX - row lock contention ORDER BY HSS.SNAP_ID DESC ) WHERE TOTAL_WAITS >= 0; END_INTERVAL_TIME Peak on 9th of May in the afternoon! TOTAL_WAITS TIME_WAITED_MICRO AVG_WAIT_MS :00: :00: :00: :00: :00: :00: :00: :00: :00: :00: :00: :00: :00: :00: :00: :01: :01: SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 6 DBA_HIST_SYSTEM_EVENT can show you (per default) on an hourly basis which wait events were active, to what extent, and with which average wait time. Between 3 and 5 p.m. on 9th of May there is an obvious peak in total and average wait time for TX enqueues. So we should now focus on this time frame. 6

7 2. Enqueue Analysis (3) Now let s see which enqueue waits happened during the peak time: SELECT TO_CHAR(ASH.SAMPLE_TIME, 'YYYY-MM-DD HH24:MI:SS') SAMPLE_TIME, ASH.SESSION_ID, ASH.BLOCKING_SESSION, O.OBJECT_NAME, S.SQL_TEXT FROM DBA_HIST_ACTIVE_SESS_HISTORY ASH, DBA_HIST_SQLTEXT S, DBA_OBJECTS O WHERE ASH.SQL_ID = S.SQL_ID (+) AND ASH.CURRENT_OBJ# = O.OBJECT_ID (+) AND ASH.EVENT like 'enq: TX - row lock contention' AND ASH.SAMPLE_TIME BETWEEN TO_TIMESTAMP(' :30:00', 'dd.mm.yyyy hh24:mi:ss') AND TO_TIMESTAMP(' :30:00', 'dd.mm.yyyy hh24:mi:ss') AND ASH.SESSION_STATE = 'WAITING' ORDER BY SAMPLE_TIME DESC; SAMPLE_TIME Many waits for REPOSRC, Some waits for DDFTX and one wait for a LOB Blocking sessions are 1110 and 1022 SESSION_ID BLOCKING_SESSION OBJECT_NAME :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 SYS_LOB :56: , ,00 DDFTX~ :56: , ,00 DDFTX~ :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC :56: , ,00 REPOSRC... SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 7 DBA_HIST_ACTIVE_SESS_HISTORY contains information about active sessions on a 10 seconds basis for the last 7 days. It is a history view for V$ACTIVE_SESSION_HISTORY where information about active sessions is stored on a 1 second basis for the last few hours. 7

8 2. Enqueue Analysis (4) What s the big picture of the Oracle enqueue wait situation? Session 1110 Holds lock on DDFTX Session 1022 Waits for DDFTX lock Holds lock on REPOSRC Other blocked sessions Is this a static situation? What is session 1110 doing? Other blocked sessions SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 8 8

9 2. Enqueue Analysis (5) Let s check the lock holders in the relevant time frame: SELECT TO_CHAR(ASW.SAMPLE_TIME, 'YYYY-MM-DD HH24:MI:SS') SAMPLE_TIME, ASW.BLOCKING_SESSION SESSION_ID, COUNT(*) "#WAITERS", ASH.BLOCKING_SESSION, ASH.TIME_WAITED, DECODE(ASH.SESSION_STATE, NULL, 'INACTIVE', 'WAITING', ASH.EVENT, 'CPU') ACTION, TO_CHAR(SUBSTR(HST.SQL_TEXT, 1, 4000)) SQL_TEXT FROM DBA_HIST_ACTIVE_SESS_HISTORY ASH, DBA_HIST_ACTIVE_SESS_HISTORY ASW, DBA_HIST_SQLTEXT HST WHERE ASH.SQL_ID = HST.SQL_ID (+) AND ASH.SAMPLE_TIME (+) = ASW.SAMPLE_TIME AND ASH.SESSION_ID (+) = ASW.BLOCKING_SESSION AND ASW.EVENT = 'enq: TX - row lock contention' AND ASW.SESSION_STATE = 'WAITING' GROUP BY TO_CHAR(ASW.SAMPLE_TIME, 'YYYY-MM-DD HH24:MI:SS'), ASW.BLOCKING_SESSION, ASH.BLOCKING_SESSION, ASH.TIME_WAITED, DECODE(ASH.SESSION_STATE, NULL, 'INACTIVE', 'WAITING', ASH.EVENT, 'CPU'), TO_CHAR(SUBSTR(HST.SQL_TEXT, 1, 4000)) ORDER BY TO_CHAR(ASW.SAMPLE_TIME, 'YYYY-MM-DD HH24:MI:SS') DESC; Lock situation is static, session 1110 is INACTIVE, so it must be active on SAP side SAMPLE_TIME SESSION_ID #WAITERS BLOCKING_SESSION TIME_WAITED ACTION :52: , ,00 0,00 enq: TX - row lock contention :52: ,00 6 0,00 0,00 INACTIVE :51: , , ,00 enq: TX - row lock contention :51: ,00 6 0,00 0,00 INACTIVE :51: , ,00 0,00 enq: TX - row lock contention :51: ,00 6 0,00 0,00 INACTIVE :51: , ,00 0,00 enq: TX - row lock contention :51: ,00 6 0,00 0,00 INACTIVE :51: , , ,00 enq: TX - row lock contention :51: ,00 6 0,00 0,00 INACTIVE :51: , ,00 0,00 enq: TX - row lock contention :51: ,00 6 0,00 0,00 INACTIVE... SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 9 9

10 2. Enqueue Analysis (6) What do we know now? We have a lot of information about the Oracle side lock dependencies. But we don t know what happened on the SAP side at this point, because there is no Active Session History available for the SAP work processes. So we can t say what the lock holding session 1110 was doing all the time. This is a limiting factor for our analysis. In this particular case we re in luck because one SM66 snapshot was taken on the SAP side showing the work process activities (see next page). Via V$SESSION we were able to map Oracle session 1110 to client process (column PROCESS). Unfortunately Oracle doesn t store the PROCESS column in V$ACTIVE_SESSION_HISTORY, so that a mapping work process <-> Oracle session is difficult if work processes were restarted since the problem happened. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 10 10

11 2. Enqueue Analysis (7) SM66 snapshot showing SAP work process activities: Server Name No. Type PID Status Reason Sem Start Error CPU Time User Report Action Table sr3p022_fsp_00 0 DIA running Yes 3102 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 3 DIA running Yes 3102 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 4 DIA running Yes 93 IN3137 CL_FINB_ Sequential read /1ACC/43D_ sr3p022_fsp_00 5 DIA running Yes 3103 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 6 DIA running Yes 3103 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 7 DIA running Yes 3103 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 8 DIA running Yes 3061 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 17 DIA running Yes 3103 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 22 DIA running Yes 3059 SAPSYS <rpc(gen Direct read REPOSRC sr3p022_fsp_00 23 DIA running Yes 1202 X CL_FINB_ Sequential read /1ACC/43D_ sr3p022_fsp_00 24 DIA running Yes 877 SAPSYS <rpc(gen Direct read REPOSRC... sr3p072_fsp_00 25 BTC running Yes 3130 BATCH_CRA SAPLSDIF Insert DDFTX sr3p072_fsp_00 26 BTC stopped Yes 3130 BATCH_CRA /BA1/CL_ sr3p072_fsp_00 27 BTC stopped Yes 3129 BATCH_CRA /BA1/SAP sr3p072_fsp_00 28 BTC stopped SLEEP Yes 6 BATCH_CRA /BA1/R2_ sr3p072_fsp_00 29 BTC stopped Yes 3129 BATCH_CRA GP45PYIK sr3p072_fsp_00 30 BTC running Yes 3129 BATCH_CRA SAPLSDIF Insert DDFTX sr3p072_fsp_00 31 BTC stopped Yes 3128 BATCH_CRA GP45PYIK sr3p072_fsp_00 32 BTC stopped Yes 3128 BATCH_CRA /BA1/CL_ sr3p072_fsp_00 33 BTC stopped Yes 3128 BATCH_CRA /BA1/SAP sr3p072_fsp_00 34 BTC stopped Yes 3128 BATCH_CRA GP45PYIK sr3p072_fsp_00 35 BTC stopped Yes 3127 BATCH_CRA GP45PYIK sr3p072_fsp_00 36 BTC stopped Yes 3127 BATCH_CRA CL_ABAP_ sr3p072_fsp_00 37 BTC stopped Yes 3127 BATCH_CRA /BA1/SAP sr3p072_fsp_00 38 BTC stopped Yes 3126 BATCH_CRA GP45PYIK sr3p072_fsp_00 40 BTC stopped Yes 3125 BATCH_CRA GP45PYIK sr3p072_fsp_00 41 BTC running Yes 3124 BATCH_CRA SAPLSDIF Insert DDFTX sr3p072_fsp_00 42 BTC stopped Yes 3124 BATCH_CRA /BA1/CL_ sr3p072_fsp_00 45 BTC stopped SLEEP Yes 1 ALE_IMP ZIN_SCEN What does the status stopped for process mean? SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 11 11

12 2. Enqueue Analysis (8) Possible Explanation of the Stopped Status With recent kernel releases ABAP programs are not generated in the running LUW. Instead they are executed and committed in another dialog process. Reason: if the generation (which is nothing else then DML operations against tables like REPOSRC or REPOLOAD) happens in the normal LUW, it can potentially take a very long time until the change is committed and so other processes queue up for the enqueues. As long as the generation takes place in the other dialog process, the main process is in status Stopped. Knowing this we can now imagine what the root cause of this problem was: Session 1110 had to wait for its dialog process to finish the generation. The dialog process had to wait for a REPOSRC lock The REPOSRC lock was held by Oracle session 1022 that had to wait for the DDFTX lock So this is a deadlock situation! SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 12 12

13 2. Enqueue Analysis (9) What is the big picture of the overall enqueue wait situation? Session 1110 SAP Batch work process Holds lock on DDFTX Wait for Oracle enqueue on DDFTX Session 1022 Waits for DDFTX lock Holds lock on REPOSRC Other blocked sessions Wait for Oracle enqueue on REPOSRC Wait for SAP generation SAP Dialog work process Doing generation For batch work process Other blocked sessions SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 13 13

14 2. Enqueue Analysis (10) Final Results Although the external generation of ABAP programs should avoid enqueue situations, it has caused critical enqueue waits and even a deadlock situation in this particular case. The reason was bad timing of the generation and the activities of session 1022 that finally lead to a permanent Oracle<->SAP deadlock. Oracle was not able to detect and resolve this deadlock because it can t be aware of the dependency on the SAP side between batch job and its generation dialog process. After having terminated and restarted the job everything ran smoothly because the bad timing didn t repeat. Reason for the generation activity was a previous coding change in the report. SAP note exactly describes this kind of scenario. The described workaround to force a local generation using abap/pxa = gen_local is not a good idea, because other enqueue situations are possible then. We assume that this problem will no longer happen because this kind of bad timing is very unlikely. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 14 14

15 3. Runtime Analysis (1) Problem Situation A customer stated that a certain batch job took more than 15 hours the day before, which is much longer compared to previous runs. Based on transaction SM37 the following details of the job could be determined: Start date: , 12:04:12 End date: , 03:39:34 Work process PID: Based on the Oracle session overview in ST04 (or V$SESSION) the related Oracle sessions were determined: SID Client PID Event Logical Reads Phys. Reads Block Changes SQL*Net message from client SQL*Net message from client Only SID 1134 has performed significant work, so we can focus on this session during the subsequent analysis. This is now a good basis for an analysis based on the Oracle active session history. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 15 The work process obviously opened two different sessions to the same database (865 and 1134). This is quite a normal scenario. The SAP work process has not been restarted since the job ran. Otherwise the mapping of work process to Oracle sessions would no longer be correct. 15

16 3. Runtime Analysis (2) What were the top SQL statements of that session? SELECT COUNT(*) * 10 "ESTIMATED_SECONDS", ASH.SQL_ID, TO_CHAR(S.SQL_TEXT) SQL_TEXT FROM DBA_HIST_ACTIVE_SESS_HISTORY ASH, DBA_HIST_SQLTEXT S WHERE ASH.SQL_ID = S.SQL_ID (+) AND ASH.SESSION_ID = 1134 AND ASH.SAMPLE_TIME BETWEEN TO_TIMESTAMP(' :04:12', 'dd.mm.yyyy hh24:mi:ss') AND TO_TIMESTAMP(' :39:34', 'dd.mm.yyyy hh24:mi:ss') GROUP BY ASH.SQL_ID, TO_CHAR(S.SQL_TEXT) ORDER BY COUNT(*) DESC; ESTIMATED_SECONDS SQL_ID There are only 5 expensive SQL statements with a significant runtime SQL_TEXT bngmgmf5gy9bb SELECT "T00003"."/B20/S_C30POS" AS fuq9r589fwbns SELECT "T00003"."/B20/S_C30POS" AS bg38uqzwyvkwc SELECT "T00003"."/B20/S_C30POS" AS dkut6ufwafv SELECT "T00003"."/B20/S_C30POS" AS d5907v33zfd44 SELECT "T00003"."/B20/S_C30POS" AS kj5mtbnft6p 70 1wa6j3643uba6 50 8w782vt53mfg5 SELECT * FROM "BANK_PROCFL_PROC" WHERE c8kvcz1sgp1qs SELECT "T00003"."/B20/S_C30POS" AS... SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 16 DBA_HIST_ACTIVE_SESS_HISTORY provides a snapshot of active sessions every ten seconds up to 7 days in the past. By explicitly restricting the SELECT to the job time frame and Oracle session 1134 we get exactly the snapshots of our session. The ESTIMATED_SECONDS don t add up to the overall runtime of the job because the job was sometimes active outside the database. In fact some work was parallelized and only coordinated by the master process. Nevertheless we hope that analyzing and optimizing the master process will already provide significant benefit. The output of the selection contains the SQL_ID. The SQL_ID is one of the most useful pieces of information you can get, because it s the key for a lot of detail information for the related SQL statement. 16

17 3. Runtime Analysis (3) Now let s check the wait event and CPU distribution: SELECT COUNT(*) * 10 "ESTIMATED_SECONDS", DECODE(ASH.SESSION_STATE, 'WAITING', ASH.EVENT, 'CPU') ACTION FROM DBA_HIST_ACTIVE_SESS_HISTORY ASH, DBA_HIST_SQLTEXT S WHERE ASH.SQL_ID = S.SQL_ID (+) AND ASH.SESSION_ID = 1134 AND ASH.SAMPLE_TIME BETWEEN TO_TIMESTAMP(' :04:12', 'dd.mm.yyyy hh24:mi:ss') AND TO_TIMESTAMP(' :39:34', 'dd.mm.yyyy hh24:mi:ss') GROUP BY DECODE(ASH.SESSION_STATE, 'WAITING', ASH.EVENT, 'CPU') ORDER BY COUNT(*) DESC; CPU and db file sequential read are the main contributors ESTIMATED_SECONDS ACTION CPU 7410 db file sequential read 110 latch free 80 latch: cache buffers chains 70 log file sync 20 db file scattered read 10 SQL*Net more data to client SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 17 There were no strange wait events responsible for a significant delay. Just CPU and I/O reads, which is normal. The high CPU consumption can be caused, for example, by expensive SQL statements with a high number of buffer gets. 17

18 3. Runtime Analysis (4) How does the top SQL statement look? SELECT SUBSTR(SQL_TEXT, 1 + (i - 1) * 80, 80) SQL_TEXT FROM DBA_HIST_SQLTEXT HST, (SELECT ROWNUM I FROM DBA_HIST_SQLTEXT WHERE ROWNUM <= 60) WHERE HST.SQL_ID = 'bngmgmf5gy9bb'; SELECT "T00003"."/B20/S_C30POS" AS "ZIBI002 F00028","T00001"."/BIC/AO_STATUS" AS "ZIBI002 F00010","T00001"."/BIC/AO_VER_VF" AS "ZIBI002 F00003", "T00001"."/BIC/AO_VER_VT" AS "ZIBI002 F00006","T00001"."/BIC/AO_SYS_VF" AS "ZIBI002 F00004","T00001"."/BIC/AO_SYS_VT" AS "ZIBI002 F00007", "T00003"."/BIC/AO_STATUS" AS "ZIBI002 F00036","T00003"."/BIC/AO_VER_VF" AS "ZIBI002 F00029","T00003"."/BIC/AO_VER_VT" AS "ZIBI002 F00032", "T00003"."/BIC/AO_SYS_VF" AS "ZIBI002 F00030","T00003"."/BIC/AO_SYS_VT" AS "ZIBI002 F00033","T00004"."/BIC/AO_STATUS" AS "ZIBI002 F00047", "T00004"."/BIC/AO_VER_VF" AS "ZIBI002 F00040","T00004"."/BIC/AO_VER_VT" AS "ZIBI002 F00043","T00004"."/BIC/AO_SYS_VF" AS "ZIBI002 F00041", "T00004"."/BIC/AO_SYS_VT" AS "ZIBI002 F00044" FROM "/BIC/AZDDKX00100" "T00001" JOIN "/BIC/AZDBS10300" "T00003" ON "T00001". "/B20/S_C43CLACC" = "T00003". "/B20/S_C43CLACC" JOIN "/BIC/AZDFIX00100" "T00004" ON "T00003". "/B20/S_C41FINST" = "T00004". "/B20/S_C41FINST" WHERE "T00001"."SOURSYSTEM" = '00' AND "T00001"."/BIC/AO_SYS_VF" < ' ' AND "T00001"."/BIC/AO_VAL_VF" <= ' ' AND "T00001"."/BIC/AO_VER_VT" > ' ' AND "T00001"."/BIC/AO_SYS_VT" >= ' ' AND "T00001"."/BIC/AO_VAL_VT" > ' ' AND "T00001"."/B20/S_C55LGENT" = 'L0050' AND NOT ( "T00001"."/BIC/ZMANREL" = '011' OR "T00001"."/BIC/ZMANREL" = '010' OR "T00001"."/BIC/ZMANREL" = '001' OR "T00001"."/BIC/ZMANREL" = '000' ) AND "T00003"."SOURSYSTEM" = '00' AND "T00003"."/BIC/AO_SYS_VF" < ' ' AND "T00003"."/BIC/AO_VAL_VF" <= ' ' AND "T00003"."/BIC/AO_VER_VT" > ' ' AND "T00003"."/BIC/AO_SYS_VT" >= ' ' AND "T00003"."/BIC/AO_VAL_VT" > ' ' AND "T00003"."/BIC/AO_PACKNO" BETWEEN '000000' AND '049999' AND A typical SAP Bank Analyzer Multi-table Join "T00004"."SOURSYSTEM" = '00' AND "T00004"."/BIC/AO_SYS_VF" BETWEEN ' ' AND ' ' AND "T00004"."/BIC/AO_VAL_VF" <= ' ' AND "T00004"."/BIC/AO_VER_VT" > ' ' AND "T00004"."/BIC/AO_SYS_VT" >= ' ' AND "T00004"."/BIC/AO_VAL_VT" > ' ' AND NOT ( "T00004"."/BIC/ZMPRODA" = '0104' OR "T00004"."/BIC/ZMPRODA" = '0107' OR "T00004"."/BIC/ZMPRODA" = '0110' OR "T00004"."/BIC/ZMPRODA" = '0111' OR "T00004"."/BIC/ZMPRODA" = '0114' ) GROUP BY "T00003"."/B20/S_C30POS","T00001"."/BIC/AO_STATUS", "T00001"."/BIC/AO_VER_VF","T00001"."/BIC/AO_VER_VT", "T00001"."/BIC/AO_SYS_VF","T00001"."/BIC/AO_SYS_VT", "T00003"."/BIC/AO_STATUS","T00003"."/BIC/AO_VER_VF", "T00003"."/BIC/AO_VER_VT","T00003"."/BIC/AO_SYS_VF", "T00003"."/BIC/AO_SYS_VT","T00004"."/BIC/AO_STATUS", "T00004"."/BIC/AO_VER_VF","T00004"."/BIC/AO_VER_VT", "T00004"."/BIC/AO_SYS_VF","T00004"."/BIC/AO_SYS_VT"; SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 18 DBA_HIST_SQLTEXT contains the full SQL text of SQL statements in the SQL_TEXT column (of CLOB data type). The SUBSTR approach in the SELECT is necessary if you want to display the full SQL text in report RSORADJV. If you execute the command directly on Oracle side, you can choose a simpler variant. The output had to be manually reformatted in order to get a well readable format. 18

19 3. Runtime Analysis (5) What was the execution plan of this SQL statement? SELECT TO_CHAR(TIMESTAMP, 'YYYY-MM-DD HH24:MI:SS') TIMESTAMP, ID, LPAD(' ', DEPTH) OPERATION ' ' OPTIONS DECODE(OBJECT_NAME, NULL, NULL, ' (' OBJECT_NAME ')') ACTION, COST, CARDINALITY FROM DBA_HIST_SQL_PLAN WHERE SQL_ID = 'bngmgmf5gy9bb' ORDER BY 1 DESC, 2; Nested Loop Joins with Index Skip Scan on Leading Table TIMESTAMP ID ACTION COST CARDINALITY :36:42 0,00 SELECT STATEMENT 6.584,00 0, :36:42 1,00 HASH GROUP BY 6.583,00 74, :36:42 2,00 TABLE ACCESS BY INDEX ROWID (/BIC/AZDDKX00100) 3,00 6, :36:42 3,00 NESTED LOOPS 6.582,00 74, :36:42 4,00 NESTED LOOPS 6.549,00 12, :36:42 5,00 TABLE ACCESS BY INDEX ROWID (/BIC/AZDFIX00100) 4.379,00 2, :36:42 6,00 INDEX SKIP SCAN (/BIC/AZDFIX00100~0) 4.379,00 1, :36:42 7,00 TABLE ACCESS BY INDEX ROWID (/BIC/AZDBS10300) 1.085,00 7, :36:42 8,00 INDEX RANGE SCAN (/BIC/AZDBS10300~02) 1.036,00 479, :36:42 9,00 INDEX RANGE SCAN (/BIC/AZDDKX00100~0) 0,00 82,00 SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 19 DBA_HIST_SQL_PLAN contains historic execution plans of SQL statements. In this case due to the fact that literals rather than bind variables were used in the selection the statement was only executed once. Therefore, there s only one execution plan available. 19

20 3. Runtime Analysis (6) What were the load characteristics of the SQL statement? SELECT TO_CHAR(HSS.END_INTERVAL_TIME, 'YYYY-MM-DD HH24:MI:SS') END_INTERVAL_TIME, HSQ.EXECUTIONS_DELTA EXECUTIONS, HSQ.ROWS_PROCESSED_DELTA RECORDS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.ROWS_PROCESSED_DELTA / HSQ.EXECUTIONS_DELTA), 2) RECORDS_PER_EXEC, HSQ.DISK_READS_DELTA DISK_READS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.DISK_READS_DELTA / HSQ.EXECUTIONS_DELTA), 2) DISK_READS_PER_EXEC, HSQ.BUFFER_GETS_DELTA BUFFER_GETS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.BUFFER_GETS_DELTA / HSQ.EXECUTIONS_DELTA), 2) BUFFER_GETS_PER_EXEC, ROUND(DECODE(HSQ.ROWS_PROCESSED_DELTA, 0, 0, HSQ.BUFFER_GETS_DELTA / HSQ.ROWS_PROCESSED_DELTA), 2) BUFFER_GETS_PER_RECORD, HSQ.ELAPSED_TIME_DELTA / 1000 ELAPSED_TIME_MS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.ELAPSED_TIME_DELTA / HSQ.EXECUTIONS_DELTA / 1000), 3) ELAPSED_TIME_MS_PER_EXEC, HSQ.CPU_TIME_DELTA / 1000 CPU_TIME_MS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.CPU_TIME_DELTA / HSQ.EXECUTIONS_DELTA / 1000), 3) CPU_TIME_MS_PER_EXEC FROM DBA_HIST_SQLSTAT HSQ, DBA_HIST_SNAPSHOT HSS WHERE HSQ.SNAP_ID = HSS.SNAP_ID AND HSQ.SQL_ID = 'bngmgmf5gy9bb' ORDER BY HSS.SNAP_ID DESC; Very high number of buffer gets for 0 records! END_INTERVAL_TIME EXECUTIONS RECORDS RECORDS_PER_EXEC DISK_READS DISK_READS_PER_EXEC BUFFER_GETS BUFFER_GETS_PER_EXEC :00:35 0,00 0,00 0 0, , :00:34 0,00 0,00 0 0, , :00:32 0,00 0,00 0 0, , :00:31 1,00 0,00 0 0, , BUFFER_GETS_PER_RECORD ELAPSED_TIME_MS ELAPSED_TIME_MS_PER_EXEC CPU_TIME_MS CPU_TIME_MS_PER_EXEC SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 20 DBA_HIST_SQLSTAT contains hourly snapshots of the Oracle shared cursor cache. In this case 4 hours are returned because the SQL statement spanned 4 hours. The times per execution values are not really correct in this special case because they only relate to the values in line 4 rather than summing up all lines. Normally this problem is not so significant because the SQL statements don t run so long. The high number of buffer gets is a typical reason for the high CPU consumption we have already seen. 20

21 3. Runtime Analysis (6) Now we just have to optimize that SQL statement... select table_name, column_name, num_distinct, num_buckets, to_char(last_analyzed, 'dd.mm.yyyy hh24:mi:ss') last_analyzed, histogram from dba_tab_columns where table_name in ('/BIC/AZDDKX00100', '/BIC/AZDBS10300', '/BIC/AZDFIX00100'); Histogram statistics, creation date 6 weeks In the past TABLE_NAME COLUMN_NAME NUM_DISTINCT NUM_BUCKETS LAST_ANALYZED HISTOGRAM /BIC/AZDBS10300 SOURSYSTEM 1,00 1, :01:45 FREQUENCY /BIC/AZDBS10300 /B20/S_C30POS 5.896,00 75, :01:45 HEIGHT BALANCED /BIC/AZDBS10300 /BIC/AO_VER_VF 609,00 75, :01:45 HEIGHT BALANCED /BIC/AZDBS10300 /BIC/AO_SYS_VF ,00 1, :01:45 NONE /BIC/AZDBS10300 /BIC/AO_VAL_VF 1,00 1, :01:45 FREQUENCY /BIC/AZDBS10300 /BIC/AO_VER_VT 610,00 75, :01:45 HEIGHT BALANCED /BIC/AZDBS10300 /BIC/AO_SYS_VT ,00 75, :01:45 HEIGHT BALANCED /BIC/AZDBS10300 /BIC/AO_VAL_VT 1,00 1, :01:45 FREQUENCY /BIC/AZDBS10300 /BIC/AO_PACKNO 5.882,00 75, :01:45 HEIGHT BALANCED /BIC/AZDBS10300 /BIC/AO_STATUS 2,00 2, :01:45 FREQUENCY /BIC/AZDBS10300 /B20/S_C43CLACC 5.536,00 75, :01:45 HEIGHT BALANCED /BIC/AZDBS10300 RECORDMODE 1,00 1, :01:45 FREQUENCY /BIC/AZDBS10300 /B20/S_C41FINST 1.992,00 75, :01:45 HEIGHT BALANCED /BIC/AZDDKX00100 /BIC/ZMBUCHAR 3,00 3, :02:49 FREQUENCY /BIC/AZDDKX00100 /BIC/ZMANREL 2,00 2, :02:49 FREQUENCY /BIC/AZDDKX00100 SOURSYSTEM 1,00 1, :02:49 FREQUENCY /BIC/AZDDKX00100 /B20/S_C43CLACC ,00 75, :02:49 HEIGHT BALANCED /BIC/AZDDKX00100 /BIC/AO_VER_VF 2.079,00 75, :02:49 HEIGHT BALANCED /BIC/AZDDKX00100 /BIC/AO_SYS_VF ,00 1, :02:49 NONE... SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 21 21

22 3. Runtime Analysis (7) Final Results Outdated column statistics (both with and without histograms) can be critical in case of date ranges, because existing dates are often a sliding range. Date columns are quite important for the expensive Bank Analyzer SQL statements in question. At the time of the statistics creation (beginning of May) no records with a date of June can have been there, whereas now a significant number exists. As a consequence the cardinality estimation of the CBO can be far away from the reality and wrong join orders are used. As a quick workaround we have created new statistics for the affected tables with BRCONNECT: brconnect u / -c f stats t <table_name> -f collect Afterwards the job runtime improved from 15 hours to 3.5 hours. In order to make sure that statistics for the critical tables are updated regularly, the following approaches are possible: Set ACTIV=U ( unconditioned statistics ) in DBSTATC (note ) Use a smaller STATS_CHANGE_THRESHOLD for statistics creation SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 22 By using a smaller STATS_CHANGE_THRESHOLD new statistics are created when a smaller amount of changes happened to a table. The disadvantage is a possible overhead during statistics creation if for too many tables new statistics are calculated. 22

23 4. I/O Analysis (1) Problem Situation A customer had average db file sequential read times of 8 ms. This is usually an acceptable average value for requesting one block from the operating system or from disk. Nevertheless it doesn t tell us if there are bottlenecks during certain times or for certain files or objects. With Oracle 10g it is now possible to also determine the distribution of wait event runtimes. So we are able to see not only the average value, but also the range of existing values and peak areas. Therefore a more detailed analysis of db file sequential read times was performed for that customer with interesting results. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 23 23

24 4. I/O Analysis (2) What is the distribution of the db file sequential read times? SELECT EVENT, WAIT_TIME_MILLI UPPER_WAIT_TIME_MS, WAIT_COUNT, WAIT_TIME_MILLI * DECODE(WAIT_TIME_MILLI, 1, 0.5, 0.75) * WAIT_COUNT TIME_WAITED, ROUND(RATIO_TO_REPORT(WAIT_TIME_MILLI * DECODE(WAIT_TIME_MILLI, 1, 0.5, 0.75) * WAIT_COUNT) OVER () * 100) PERCENT FROM V$EVENT_HISTOGRAM WHERE EVENT = 'db file sequential read' ORDER BY WAIT_TIME_MILLI; Several thousand reads with a runtime of more than 0.5 seconds! EVENT UPPER_WAIT_TIME_MS WAIT_COUNT TIME_WAITED PERCENT db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read db file sequential read SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 24 V$EVENT_HISTOGRAM contains histogram information for wait events. Two typical peaks can be seen in the output: There are 47 million accesses with a runtime of less than 1 ms. These blocks are typically buffered on lower levels (filesystem cache, I/O sub system cache,...) and no real disk access is necessary. There are more than 100 million accesses with runtimes between 4 and 16 ms. These accesses are normally typical disk accesses. Additionally we see an untypical number of accesses with a runtime of more than 1 second. It is hardly possible to imagine that reading one block from disk takes more than 1 second but here the output tells us that there were already 27 blocks that took longer than 65 seconds to be read from disk. Although the following slides will confirm that the high times are reality you should always take into account the possibility of Oracle statistical bugs if you see such high values. 24

25 4. I/O Analysis (3) When did the long accesses take place? SELECT TO_CHAR(ASH.SAMPLE_TIME, 'YYYY-MM-DD HH24:MI:SS') SAMPLE_TIME, ASH.SESSION_ID, ASH.TIME_WAITED, O.OBJECT_NAME FROM DBA_HIST_ACTIVE_SESS_HISTORY ASH, DBA_HIST_SQLTEXT S, DBA_OBJECTS O WHERE ASH.SQL_ID = S.SQL_ID (+) AND ASH.CURRENT_OBJ# = O.OBJECT_ID (+) AND ASH.EVENT like 'db file sequential read' AND ASH.SESSION_STATE = 'WAITING' and ash.time_waited > ORDER BY SAMPLE_TIME DESC; A lot of high I/O times between 7:55 and 8:00! SAMPLE_TIME SESSION_ID TIME_WAITED OBJECT_NAME (MICRO) :58: /1ACC/43D 1E00B :58: /1ACC/43D 1E00B :58: /BA1/B1_FPTOTALS :58: /BA1/F1_BAS~ :58: FICOT_CONDI~ :57: FICOT_CONDI :57: FICOT_CONDI :57: /1ACC/43D 1E00B :57: FICOT_CONDI :56: /1SGS/41D4F :56: /1SGS/41D4F :56: /BA1/R4_REC_BP :56: /BA1/R4_REC_BP :56: /1ACC/43D 1E00B :56: /1SGS/41D4F :55: /1ACC/43D 1E00B :55: /1SGS/ :55: /1ACC/43D 1E00C~ :55: /BA1/R2_CRE~ :55: FICOT_CONDI :55: /1BA/R4_5541D4F :55: FICOT_POS :55: /1ACC/43D 1E00B SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 25 Just another time the active session history can provide really useful information. In this case we can see that a lot of long running I/O reads happened during a few minutes. In fact the whole I/O was more or less blocked during that time. Some short time frames with similar peaks existed too. The rest of the time the I/O ran smoothly and there were no critical peaks. 25

26 4. I/O Analysis (4) Final Results After knowing the periods when increased I/O read times occurred, we tried to figure out what was special during those periods. Finally we recognized that during these periods a tablespace extension was always performed. During the ADD DATAFILE execution all other I/O was nearly stopped. Knowing this we were able to reproduce the bottleneck situation and address the issue in the areas of the operating system and the hardware partner. There the problem can be analyzed and the bottleneck situation can be optimized. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 26 26

27 5. Comparison Analysis (1) Problem Situation Runtime tests were performed at a customer for a certain SAP functionality. The only difference between two test scenarios was the amount of parallelization on SAP side. It turned out that the summarized database load of the serial test was much lower compared to the database load of the parallelized test. At this point it was necessary to compare the database behavior during two tests with activated and deactivated parallelization. For that purpose we decided to create AWR reports for the two test time frames: Creation of AWR snapshots before and after each test: EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT(); Creation of AWR reports using the awrrpt.sql script (note (3)) SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 27 The test system was only used for the tests, so no significant other load exists that could have an impact on the quality of the analysis results. 27

28 5. Comparison Analysis (2) Let s compare the top SQL statement in both reports: Parallel Test: Elapsed CPU Elap per % Total Time (s) Time (s) Executions Exec (s) DB Time SQL Id yztx06upqp9 Module: CA_OBL_BINREL=================CP INSERT INTO "SRBCSBREL" VALUES( :A0, :A1, :A2, :A3, :A4, :A5, :A6, :A7, :A8, :A9, :A10, :A11, :A12, :A13, :A14 ) Serial Test: Elapsed CPU Elap per % Total Time (s) Time (s) Executions Exec (s) DB Time SQL Id yztx06upqp9 Module: CA_OBL_BINREL=================CP INSERT INTO "SRBCSBREL" VALUES( :A0, :A1, :A2, :A3, :A4, :A5, :A6, :A7, :A8, :A9, :A10, :A11, :A12, :A13, :A14 ) Same number of executions, but 30 % higher elapsed time! SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 28 28

29 5. Comparison Analysis (3) What were the characteristics of this SQL statement? SELECT TO_CHAR(HSS.END_INTERVAL_TIME, 'YYYY-MM-DD HH24:MI:SS') END_INTERVAL_TIME, HSQ.EXECUTIONS_DELTA EXECUTIONS, HSQ.ROWS_PROCESSED_DELTA RECORDS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.ROWS_PROCESSED_DELTA / HSQ.EXECUTIONS_DELTA), 2) RECORDS_PER_EXEC, HSQ.DISK_READS_DELTA DISK_READS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.DISK_READS_DELTA / HSQ.EXECUTIONS_DELTA), 2) DISK_READS_PER_EXEC, HSQ.BUFFER_GETS_DELTA BUFFER_GETS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.BUFFER_GETS_DELTA / HSQ.EXECUTIONS_DELTA), 2) BUFFER_GETS_PER_EXEC, ROUND(DECODE(HSQ.ROWS_PROCESSED_DELTA, 0, 0, HSQ.BUFFER_GETS_DELTA / HSQ.ROWS_PROCESSED_DELTA), 2) BUFFER_GETS_PER_RECORD, HSQ.ELAPSED_TIME_DELTA / 1000 ELAPSED_TIME_MS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.ELAPSED_TIME_DELTA / HSQ.EXECUTIONS_DELTA / 1000), 3) ELAPSED_TIME_MS_PER_EXEC, HSQ.CPU_TIME_DELTA / 1000 CPU_TIME_MS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.CPU_TIME_DELTA / HSQ.EXECUTIONS_DELTA / 1000), 3) CPU_TIME_MS_PER_EXEC, HSQ.IOWAIT_DELTA / 1000 USER_IO_TIME_MS, ROUND(DECODE(HSQ.EXECUTIONS_DELTA, 0, 0, HSQ.IOWAIT_DELTA / HSQ.EXECUTIONS_DELTA / 1000), 3) USER_IO_TIME_MS_PER_EXEC, ROUND(DECODE(HSQ.DISK_READS_DELTA, 0, 0, HSQ.IOWAIT_DELTA / HSQ.DISK_READS_DELTA / 1000), 3) TIME_PER_DISKREAD_MS FROM DBA_HIST_SQLSTAT HSQ, DBA_HIST_SNAPSHOT HSS WHERE HSQ.SNAP_ID = HSS.SNAP_ID AND HSQ.SQL_ID = '94yztx06upqp9' ORDER BY HSS.SNAP_ID DESC; Slower I/O reads during parallel execution! END_INTERVAL_TIME EXECUTIONS RECORDS RECORDS_PER_EXEC DISK_READS DISK_READS_PER_EXEC BUFFER_GETS BUFFER_GETS_PER_EXEC :30:04 107, , , , :00:09 214, , , , :00:00 104, , , , :10:03 425, , , , BUFFER_GETS_PER_RECORD ELAPSED_TIME_MS ELAPSED_TIME_MS_PER_EXEC CPU_TIME_MS CPU_TIME_MS_PER_EXEC USER_IO_TIME_MS TIME_PER_DISKREAD_MS SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 29 The first three lines (of 20th of June) are related to the serial execution. Due to the serialization the total runtime spanned three hours. The fourth line (of 19th of June) is related to the parallel test. The sum of executions in the first three lines is identical to the number of executions in the fourth line (425). Also the records are identical. Surprisingly the number of disk reads was smaller in the slower parallel test (buffering effects?), but the elapsed time was significantly higher. The main difference is the time spent per disk read (serial execution: around 5.7 ms, parallel execution: around 12.6 ms). 29

30 5. Comparison Analysis (4) Final Results The main runtime difference was the different time per disk read during the both tests. This was confirmed also with other SQL statements: 5.7 ms vs ms 7.6 ms vs ms 4.1 ms vs. 6.7 ms Also the overall db file sequential read comparison of the two AWR reports showed similar differences. So we were able to prove that the increased database request time is suffered from decreased read I/O performance. At this point the Oracle analysis is finished and it has to be checked on operating system and hardware level why the I/O read requests suffer significantly from parallelism (bottleneck due to higher parallel load)? SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 30 An AWR Diff report is also a useful approach in this scenario. In this case Oracle maps the related information of the two time frames directly and specifies an increase or decrease (absolutely and in percent). 30

31 5. Summary (1) Summary We have seen some varied real-life examples that were not possible with Oracle 9i and that take significant advantage of Oracle 10g performance monitoring features. In very many situations it is now possible to analyze performance problems even days after they happened. Oracle provides several reports and Advisors to assist during the performance analysis, but sometimes it is necessary to go beyond these features and perform flexible individual selections. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 31 31

32 6. Summary (2) Any questions? SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 32 32

33 Copyright 2007 SAP AG. All Rights Reserved No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, WINDOWS, NT, EXCEL, Word, PowerPoint and SQL Server are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iseries, pseries, xseries, zseries, z/os, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix and Informix Dynamic ServerTM are trademarks of IBM Corporation in USA and/or other countries. ORACLE is a registered trademark of ORACLE Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, the Citrix logo, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, MultiWin and other Citrix product names referenced herein are trademarks of Citrix Systems, Inc. HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MarketSet and Enterprise Buyer are jointly owned trademarks of SAP AG and Commerce One. SAP, SAP Logo, R/2, R/3, mysap, mysap.com and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are trademarks of their respective companies. SAP AG 2007, Oracle 10g Performance Case Studies, Martin Frauendorfer / 33 33