University of Aarhus. Databases IBM Corporation

Transcription

1 University of Aarhus Databases 2009 Kirsten Ann Larsen

2 What is good performance? Elapsed time End-to-end In DB2 Resource consumption CPU I/O Memory Locks Elapsed time = Sync. I/O + CPU + wait time I/O ~ ms CPU ~ µs

3 What determines SQL performance? Access path determined by the DB2 Optimizer At PREPARE for dynamic SQL At BIND for static SQL Optimizer main components: (Query rewrite) Plan enumeration Cost estimation

4 Query rewrite Predicate transitive closure WHERE P1.userid = P2.pid AND P1.userid = 'mis' AND P2.pid = 'mis' Predicate pushdown SELECT name FROM People WHERE office LIKE 'Turing%' AND office IN (SELECT office FROM people WHERE group = 'tap'); AND office LIKE 'Turing%');

5 Query rewrite Subquery to join transformation SELECT pid FROM Participants WHERE meetid IN (SELECT meetid FROM Meetings WHERE owner='mis'); SELECT P.pid FROM Participants P, Meetings M WHERE P.meetid = M.meetid AND M.owner='mis';

6 Plan enumeration Single table access Join algorithm Join sequence No of plans 2 n * 2 n-1 * n! where n = no of tables

7 Single table access SELECT * FROM Person WHERE telno = Table scan Index scan

8 Matching predicates SELECT * FROM Person WHERE zipcode = 8200 CREATE INDEX zip_ix ON Person (zipcode ASC)......

9 Screening predicates SELECT * FROM Person WHERE zipcode = 8200 AND lastname = MOELLER CREATE INDEX adr_ix ON Person (zipcode ASC, street ASC, lastname ASC)......

10 Cluster index CREATE INDEX zip_ix ON Person CLUSTER (zipcode ASC) CREATE INDEX zip_ix ON Person (zipcode ASC)

11 Which h index is best? SELECT * FROM Person WHERE zipcode = 8200 AND age > 30 CREATE INDEX ix1 ON Person CLUSTER (zipcode ASC, age ASC) CREATE INDEX ix2 ON Person (age ASC, zipcode ASC) ix1 best

12 Which h index is best? SELECT * FROM Person WHERE zipcode = 8200 AND age > 30 CREATE INDEX ix1 ON Person CREATE INDEX ix2 ON Person (zipcode ASC CLUSTER, age ASC) (age ASC, zipcode ASC) It depends

13 Multiple l index scan Typically queries on large tables 2 predicates each with its own index Indexes are accessed -> RIDs extracted RIDs combined before table access If OR between predicates RIDs are unioned If AND between predicates RIDs are intersectioned

14 Data access How is data retrieved from disk? Syncronously Asyncronously (prefetch) Where is data found? Buffer pool Disk cache Disk

15 Join method Available indexes Clustering Size of tables and result sets Predicates Local predicates influence size of result set Join predicates influence choice of indexes But also size of result set in inner table

16 Nested loop join Scan outer table For each row satisfying local predicates: Search inner table for rows matching both local and join predicates If row(s) exist concatenate the rows from the two tables Else Discard row for inner join Concatenate NULLs for outer join

17 Nested loop join Well suited for Small outer table or small result set from outer table Index on join predicate on inner table Preferably efficient and highly clustered No join columns (cartesian product) Cannot be used for full outer join

18 Nested loop join SELECT what, pid FROM Meetings M LEFT JOIN Participants P ON M.meetid = P.meetid WHERE M.date = Meetings Participants Result date what meetid ddb ddb Cln Scan outer table maybe using index For each qualifying row... meetid pid Store-Aud mis mis sigurd find all matching rows in inner table using table or index scan what pid ddb Store-Aud ddb mis ddb mis Cln - The result is the concatenation of outer table row and inner table row(s) or NULL

19 Merge scan join Prepare both tables for merge Sort in order of join columns and condense inner table, place in work file Sort and condense outer table and place in work file OR use index on join columns to access outer table Merge Start with first row in each table If match on join columns, concatenate tables to produce new row. Read next row in inner table. If outer row smaller, concatenate it with NULL or discard. Read next row in outer table. When a group of rows in outer table have equal join columns, return to first match in inner table for each row in outer table If inner table smaller, concatenate it with NULL or discard. Read next row in inner table.

20 Merge scan join Well suited for Large tables / many qualifying rows No (efficient) index on join columns Outer join Requires Join columns

21 Merge scan join SELECT what, pid FROM Meetings M, Participants P WHERE M.meetid = P.meetid AND M.date = AND P.status = a Condense and sort outer table - or use index to access in join column order Condense and sort outer table Place in work file Meetings Participants Result date what meetid meetid status pid ddb a Store-Aud ddb a mis Cln a mis Scan outer table a amoeller When join columns smaller a Turing-224 read new row from outer table Scan inner table When join columns smaller or equal read new row from inner table what pid ddb Store-Aud ddb mis ddb mis Result is concatenation of rows with matching join columns Possibly unmatched rows concatenated with NULLS

22 Cost estimation Access path Statistics DB2 Optimizer i Cost

23 Cost estimation Object statistics Table : no of rows, no of pages,... Index : depth of tree, no of pages, clusterratio,... Column : no of distinct values, frequency, highkey, lowkey,... Statistics are gathered using the RUNSTATS utility If no statistics ti ti are present, default values are assumed

24 Filter factor FF = no of qualifying rows / no of rows FF is derived from catalog statistics Independance of predicates is assumed: FF(P 1 AND P 2 ) = FF(P 1 ) * FF(P 2 ) FF(P 1 OR P 2 ) = FF(P 1 ) + FF(P 2 ) SELECT * FROM Meetings WHERE date = AND slot in (10,11) AND owner <> mis COLCARD = 7, FF = 0.14 COLCARD = 6, FF = = 0.32 COLCARD = 3, FF = 1-033=

25 Using filter factors to select index SELECT * FROM Person WHERE zipcode = 8200 AND age = 30 CARD = 2,400, NPAGES = 70,000 COLCARD = 100, FF = 0.01 COLCARD = 35, FF = CREATE INDEX ix1 ON Person NLEVELS = 4 CREATE INDEX ix2 ON Person (zipcode ASC NLEAF = 15,000 (group ASC, name ACS), age ASC) NLEVELS = 4 NLEAF = 1,500 Rows in result set = 001* * ,400, = I/O s on table Index pages to read = Index pages to read = * 15,000 = ,500 = 1,503 Total no of I/Os = Total no of I/Os = = 633 1, = 1,983

26 Using filter factors to choose join order SELECT name FROM Person P, Cars C WHERE C.owner = P.id AND P.zipcode = 8200 AND C.brand = Audi CARDF = 5,600,000 CARDF = 1,900,000 COLCARD = 1406, FF = COLCARD = 78, FF = Qualifying rows from Person: 0, * 5,600,00 = 3,920 Qualifying rows from Cars: 0,013 * 1,900,000 = 24,700

27 Skew SELECT * FROM People WHERE building = TURING AND sex = F ; COLCARD = 10, FF = 0.1 COLCARD = 2, FF = 0.5 Which index is chosen? CREATE INDEX ix1 ON People (building ASC); CREATE INDEX ix2 ON People (sex ASC);

28 Correlation Predicates not always independent SELECT * FROM Meetings WHERE what = ddb COLCARD = 6, FF = 0.16 AND owner = mis COLCARD = 3, FF = 0.33 FF SQL = 0.16 * 0.33 = 0.05 But: no of qualifying rows / no of rows = 2 / 8 = 0.25

29 Missing i statistics ti ti Runstats have not been run Use of parameter markers Defaults are used: CARD = 10,000 NPAGES = CARD / 20 COLCARD = 25 NLEAF = CARD / 300

30 Helping the optimizer i Run frequency stats to reveal skew Run keycard stats t to reveal correlation Use reopt option to re-evaluate evaluate parameter markers at run time Statistical ti ti views

31 Explain Documents the access path chosen by the optimizer Canberun As a command: db2expln From the Control Center: Access plan In connection with bind Populates explain tables

32 Visual explain

33 Visual Explain

34 Visual explain

35 Real life story 1 -who is using all the CPU? SELECT L0.ID,..., L0.SQ_METERS, COALESCE(L2.NO_OF_BEASTS,0) FROM PROD.TRUCK L0, PROD.LOAD L1 LEFT OUTER JOIN (SELECT LOADID, SUM(NO_OF_BEASTS) FROM PROD.LOADLINE GROUP BY LOADID ) L2 ON L1.ID = L2.LOADID WHERE L0.VEHICLE_NO =? AND L1.WEEK =? AND L0.REG_NO = L1.REG_NO AND... ORDER BY WEEKDAY, REG_NO 13 million getpages and 35 cpu minutes in 16 hours

36 Explain output t Cardinality: Cardinality:

37 SQL rewritten SELECT L0.ID,..., L0.KVMETER, SUM(L2.NO_OF_BEASTS) FROM PROD.TRUCK L0, PROD.LOAD L1 LEFT OUTER JOIN PROD.LOADLINE L2 ON L1.ID = L2.LOADID WHERE L0.VEHICLE_NO =? AND L1.WEEK =? AND L0.REG_NO = L1.REG_NO AND... GROUP BY L0.ID,..., L0.SQ_METERS ORDER BY WEEKDAY, REGNO Reduced to1.7 million getpages and 66 cpu seconds in 16 hours

38 Explain output t improved

39 Real life story 2 why does my query not complete? Call from help desk End-users do not get reply from the system Normally 5 10 transactions a minute Response time ~ 1 sec This Saturday morning Appr. 400 transactions running System severely stressed No transactions complete

40 Real life story 2 -why does my query not complete? System maintenance Wrong index was dropped on 280 million row table Constant time query using index now scans entire table! Table scan triggers sequential prefetch Buffer pool is not big enough to hold the table Simultaneous tranactions reading entire table push out each others pages before they have been read Create and rebuild index? Cannot complete because of overheated buffer pool Need to shut system down to restore service to normal

41 Summary DB2 uses a cost-based optimizer to decide the best access path for an SQL query The optimizer uses statistics as basis for calculations l Stats must be current and reflect data Make sure efficient indexes are available Think when you write SQL Optimizer has exponentially many access paths to consider, so rewrite cannot be a major component