Efficient Interval Management in Microsoft SQL Server

Transcription

1 Efficient Interval Management in Microsoft SQL Server Itzik Ben-Gan, SolidQ Last modified: August, 0 Agenda Inefficiencies of classic interval handling Solution: RI-tree by Kriegel, Pötke and Seidl of University of Munich Optimized solution: Static RI-tree by Laurent Martin Potential for integration in SQL Server References

2 Intervals Examples for temporal intervals: sessions, contracts, appointments, shifts Allen s interval algebra: base relations Common relation: X intersects Y Example: return contracts that were active during an input period Classic interval representation: Table: Intervals( id, lower, upper ) Indexes: idx_lower ON Intervals(lower) INCLUDE(upper) idx_upper ON Intervas(upper) INCLUDE(lower) X Y Inefficiencies of Classic Interval Handling Return intervals [lower, upper] that intersect with Classic predicate: WHERE lower AND upper Problem: two range predicates An Index Seek can use only one range predicate as a Seek Predicate Other range predicate is evaluated as a [residual] Predicate Sensitive lower upper logical reads:, CPU time: ms

3 Work used as Foundation for Solution Relational Interval tree (RI-tree) model Paper [RI]: Managing Intervals Efficiently in Object-Relational Databases (Kriegel, Pötke and Seidl of University of Munich) Static RI-tree and SQL Server implementation Article [SRI]: A Static Relational Interval Tree (Laurent Martin) Article [SRI]: Advanced interval queries with the Static Relational Interval Tree (Laurent Martin) Tree.aspx Article [SRI]: Using the Static Relational Interval Tree with time intervals (Laurent Martin) [RIBG]: Interval Queries in SQL Server (Itzik Ben-Gan) * Above sources will be referred to as [RI], [SRI], [SRI], [SRI] and [RIBG] respectively Relational Interval tree (RI-tree) [RI] Virtual backbone binary tree, height = num bits, range: h -, root: h- Fork node: first node within interval when descending the tree with bisection Table: Intervals( id, node, lower, upper ) Indexes: idx_lower ON Intervals(node, lower), idx_upper ON Intervals(node, upper)

4 Fork Node CREATE FUNCTION AS AS INT) RETURNS INT WITH SCHEMABINDING AS BEGIN AS INT = ; -- height = AS INT / ; >= BEGIN ELSE ELSE BREAK; /= ; END; END; Cons: iterative T-SQL; slows down insertions Optimized Fork Node [SRI] Compute fork node from lower and upper: lowest common ancestor Observations: : For any value N, L = leading bits before trailing 0s : For any X starting with L, X s left subtree nodes start with (L-), right start with L : For nonleaf node X and X- ancestors are same, so X can be replaced with X- : Leaf node Z and Z- differ only in last bit; Example: for Z and 0 X for = Z-(00) Hence... Left node: N, = (00), same (0) ancestors L = 0 Example: (00), (000) 000 Right node: (0)

5 Optimized Fork Node Fork node = matching prefix of (lower - ) and upper 0s Let A = (lower - ) ^ upper -- mark different bits 0 ^ 0 = 00 Let B = POWER(, FLOOR(LOG(A, ))) -- first different bit set to like in upper: 000 Let C = upper % B -- keep trailing bits from upper after set bit in B: 0 Let D = upper - C concat and set trailing bits to 0s, voilà fork node: 00 Formula: upper - upper % POWER(, CAST(LOG((lower - ) ^ upper, ) AS INT)) Not clear? Details in [RIBG] Optimized Fork Node Implemented as a computed column CREATE TABLE dbo.intervalsrit ( id INT NOT NULL, node AS upper - upper % POWER(, FLOOR(LOG((lower - ) ^ upper, ))) PERSISTED NOT NULL, lower INT NOT NULL, upper INT NOT NULL, CONSTRAINT PK_IntervalsRIT PRIMARY KEY(id), CONSTRAINT CHK_IntervalsRIT_upper_gteq_lower CHECK(upper >= lower) ); CREATE INDEX idx_lower ON dbo.intervalsrit(node, lower); CREATE INDEX idx_upper ON dbo.intervalsrit(node, upper); * Above are basic index definitions to support intersection queries; additional columns may be needed (filters, included columns) Pros: Much faster inserts Support seamless multi-row modifications Cons: Complex (even more so with date and time requires mapping to integers)

6 Fork node for DATETIME data type [SRI] node AS DATEADD(ns, ((((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 - (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 + ) % POWER(CAST( AS BIGINT), FLOOR(LOG( (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), lower) AS BIGINT) * 0 + DATEPART(mi, lower)) * 0 + DATEPART(s, lower)) * DATEPART(ns, lower) / 0 ) ^ (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 + )) / LOG()))) % ) * 0, Fork node for DATETIME data type [SRI] DATEADD(s, ((((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 - (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 + ) % POWER(CAST( AS BIGINT), FLOOR(LOG( (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), lower) AS BIGINT) * 0 + DATEPART(mi, lower)) * 0 + DATEPART(s, lower)) * DATEPART(ns, lower) / 0 ) ^ (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 + )) / LOG()))) / ) % 00,

7 Fork node for DATETIME data type [SRI] DATEADD(d, (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 - (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 + ) % POWER(CAST( AS BIGINT), FLOOR(LOG( (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), lower) AS BIGINT) * 0 + DATEPART(mi, lower)) * 0 + DATEPART(s, lower)) * DATEPART(ns, lower) / 0 ) ^ Fork node for DATETIME data type [SRI] (((CAST(DATEDIFF(hh, CONVERT(DATETIME, '000', ), upper) AS BIGINT) * 0 + DATEPART(ns, upper) / 0 + )) / LOG())) ) / CAST( AS BIGINT), CONVERT(DATETIME, '000', )))) Could be much shorter if BIGDATEADD and BIGDATEDIFF were implemented:

8 Querying [RI] Left nodes Middle nodes Right nodes All Together leftnodes and rightnodes functions Querying, Left Nodes Left nodes: W = {w on path leading to lower; and w < lower} In below example W = {, } CREATE FUNCTION dbo.leftnodes (@lower AS AS INT) TABLE ( node INT NOT NULL PRIMARY KEY ) AS BEGIN AS INT = ; AS INT / ; -- descend from root node to lower >= BEGIN -- right node -- left node ELSE BEGIN INSERT VALUES(@node); 00 END -- lower ELSE 0 BREAK; /= ; 0 END; RETURN; END; 0

9 Querying, Left Nodes Intervals [lower, upper] that intersect with All intervals registered at w, where upper AS L ON I.node = L.node AND I.upper 000 = lower = upper = w w Querying, Right Nodes Symmetric to Left Nodes (with rightnodes function) Intervals that intersect with input All intervals registered at w, where lower JOIN AS R ON I.node = R.node AND I.lower lower = = w w

10 Querying, Middle Nodes Intervals that intersect with All intervals registered at w, where node SELECT id FROM dbo.intervalsrit WHERE node lower = = w w 0 w Querying, All Together JOIN AS L ON I.node = L.node AND I.upper UNION ALL JOIN AS R ON I.node = R.node AND I.lower UNION ALL SELECT id FROM dbo.intervalsrit WHERE node logical reads: (IntervalsRIT) + (@T - leftnodes) + (@T - rightnodes), CPU time: ms

11 leftnodes and rightnodes Functions Cons: Use iterative T-SQL and table variable In a join, called per row Can be implemented more efficiently Optimized Ancestors Function [RIBG] Ancestors [SRI]: each step clears rightmost set bit and sets bit to the left to Not clear? Details in [RIBG] Example: n = 00 = 00 = 000 = 0000 = 0000 = Identify rightmost set bit in (00): -n: - (0) keep bits from right until first set bit () reverse rest of bits (00 becomes 0) n & -n = n s rightmost set bit (00000) BitMasks (n < num bits - ) n b B Left nodes: ancestors where node SELECT node FROM dbo.ancestors() AS A WHERE node < ; --, Right nodes: ancestors where node SELECT node FROM dbo.ancestors() AS A WHERE node > ; --, CREATE FUNCTION dbo.ancestors(@node AS INT) RETURNS TABLE AS RETURN & b b as node -- compute ancestor FROM dbo.bitmasks WHERE b & -@node; -- b > rightmost set bit

12 Optimized Ancestors Function JOIN AS L ON L.node AND I.node = L.node AND I.upper UNION ALL JOIN dbo.ancestors(@u) AS R ON R.node AND I.node = R.node AND I.lower UNION ALL SELECT id FROM dbo.intervalsrit WHERE node logical reads: (IntervalsRIT) + (BitMasks), CPU time: 0 ms Further Optimization [SRI] JOIN dbo.ancestors(@l) AS L ON L.node AND L.node >= (SELECT MIN(node) FROM dbo.intervalsrit) AND I.node = L.node AND I.upper UNION ALL JOIN dbo.ancestors(@u) AS R ON R.node AND R.node <= (SELECT MAX(node) FROM dbo.intervalsrit) AND I.node = R.node AND I.lower Filter out ancestors outside of range covered by table Eliminates unnecessary index seeks UNION ALL SELECT id FROM dbo.intervalsrit WHERE node logical reads: (IntervalsRIT) + (BitMasks), CPU time: 0 ms

13 Optimized Ancestors Function Pros: More efficient than iterative T-SQL functions Doesn t require a table variable Cons: A bit complex Potential for Integration in SQL Server [RIBG] Model is complex; can add engine support for model and optimizations Indexing: CREATE INDEX myindex ON dbo.intervals[(fcol, fcol,...)] -- leading equality-based filters INTERVAL(lower, upper) -- interval columns [INCLUDE(icol, icol,...)] -- included columns [WITH (INTERSECTS_ONLY = ON)]; -- determines keylist Internally compute fork node and create two B-tree indexes: key([fcol, fcol,,] node, lower[, upper]) [include(icol, icol)] key([fcol, fcol,,] node, upper[, lower]) [include(icol, icol)] Querying: Optimizer support for detecting interval queries with classic predicates Adding declarative SQL with RI-tree engine support More efficient native functions (for advanced users who wish to roll their own): forknode, leftnodes, rightnodes, ancestors Support integer, as well as date and time types

14 References [RI]: Managing Intervals Efficiently in Object-Relational Databases (Kriegel, Pötke and Seidl of University of Munich) [SRI]: A Static Relational Interval Tree (Laurent Martin) [SRI]: Advanced interval queries with the Static Relational Interval Tree (Laurent Martin) Relational-Interval-Tree.aspx [SRI]: Using the Static Relational Interval Tree with time intervals (Laurent Martin) [RIBG]: Interval Queries in SQL Server (Itzik Ben-Gan) Feature Enhancement Request for Microsoft Connect item: