Data Warehousing and OLAP II. Toon Calders

Transcription

1 Data Warehousing and OLAP II Toon Calders

2 What have we seen last time? Datawarehousing Alternative data storage for analysis Geared towards aggregation queries Online Analytical Processing Multidimensional i l view on data: Data Cubes

3 What have we seen last time? Technologies Simple extensions to SQL insufficient Date Country Sales 1st semester Ireland 20 1t 1st semester France 126 1st semester Germany 56 1st semester null 202 2nd semester Ireland 23 2nd semester France 138 2nd semester Germany 48 2nd semester null 209 null Ireland 43 null France 264 null Germany 104 null null 411

4 What have we seen last time? Completely materializing cube is impossible Reason: sparse dimensions Size of cube w.r.t. number of dimensions (500 data points)

5 What have we seen last time? Therefore: partially materialize the cube Discussed the approach of Harinarayan et al. (part, supplier, customer) (part, supplier) (part, customer) (supplier, customer) (part) (supplier) (customer) ()

6 What have we seen last time? Therefore: partially materialize the cube Discussed the approach of Harinarayan et al. (part, supplier, customer) (part, supplier) (part, customer) (supplier, customer) (part) (supplier) (customer) ()

7 Today s Menu Extensions to the View Selection Problem probabilities on queries memory bound II.2 Data storage and indexing star and snowflake schema, multi-cube bitmap, join, and bitmap-join index

8 Extensions (1) Problem The views in a lattice are unlikely to have the same probability of being requested in a query. Solution: We can weight each benefit by its probability.

9 Example Suppose following distribution: a: 0%, b: 7%, c:3%, d:10%, e:5%, f:20%, g: 40%, h:15% a b c d e f g h Expected cost given S = 48 Query Uses? Cost Exp. Cost a a b b c a d b 50 5 e b f f 40 8 g b h f 40 6 Total 48 a 100 b 50 c 75 d 20 e 30 f 40 g 1 h 10

10 Example Solution found by greedy: a: 0%, b: 7%, c:3%, d:10%, e:5%, f:20%, g: 40%, h:15% a b c d e f g h Expected cost given S = 46 Query Uses? Cost Exp. Cost a a b a c a d d 20 2 e e f a g d 20 8 h e Total 46 a 100 b 50 c 75 d 20 e 30 f 40 g 1 h 10

11 Example Optimal Solution: a: 0%, b: 7%, c:3%, d:10%, e:5%, f:20%, g: 40%, h:15% a b c d e f g h Expected cost given S = 39 Query Uses? Cost Exp. Cost a a b a c a d d 20 2 e a f f 40 8 g d 20 8 h f 40 6 Total 39 a 100 b 50 c 75 d 20 e 30 f 40 g 1 h 10

12 Extensions (2) Problem Instead of asking for some fixed number (k) of views to materialize, we might instead allocate a fixed amount of space to views. Solution We can consider the benefit of each view per unit space always select the view with the best trade-off that still fits

13 Example Available space: 85 (a is already materialized) materializing all: 226 b a c d e f g h a 100 b 50 c 75 d 20 e 30 f 40 g 1 query Materialized view b c d e f g h a b c d e f g h Total Avg h 10

14 Example Available space: 84 (g:1) b a c d e f g h query Materialized view b c d e f h a b c d e f g h Total Avg a 100 b 50 c 75 d 20 e 30 f 40 g 1 h 10

15 Example Available space: 74 (g:1, h:10) b a c d e f g h query Materialized view b c d e f a b c d e f g h Total Avg a 100 b 50 c 75 d 20 e 30 f 40 g 1 h 10

16 Example Available space: 54 (g:1, h:10, d:20) b a c d e f g h query Materialized view b e f a b c d e f g h Total Avg a 100 b 50 c 75 d 20 e 30 f 40 g 1 h 10

17 Example Available space: 24 (g:1, h:10, d:20, e:30) b a c d e f g h query Materiali zed view b f a - - b 50 - c - - d - - e - - f - 60 g - - h - - Total Avg a 100 b 50 c 75 d 20 e 30 f 40 g 1 h 10

18 Conclusions View Materialization Materialization of views is an essential query optimization strategy for decision-support applications. Finding optimal solution is NP-hard. Introduction of greedy algorithm There e exists cases which greedy algorithm fails to produce optimal solution Yet, greedy algorithm has a good guarantee Expansion of greedy algorithm.

19 II.2 Data storage and indexing How is the data stored? relational database (ROLAP) Specialized structures (MOLAP) How can we speed up computation? Indexing structures bitmap index join index bitmap-join index

20 How Does it Fit In? We already know what we want to materialize, but HOW are we going to store it? We made the problem smaller but did not solve it Before partial materialization: Answer (supplier) from (part, supplier, customer) After partial materialization: Answer (supplier) from (supplier, customer) Not all queries are of the type SELECT D1,, Dk, sum(m) FROM R GROUP BY D1,, Dk

21 How Does it Fit In? Example of another type of query: SELECT supplier, year, min(price) FROM cube WHERE product = toilet paper and (year = 2009 or year = 2010) GROUP BY supplier, year

22 How Does it Fit In? Two ways for storing the data In a relational database ROLAP Using specialized storage structures MOLAP Two popular OLAP indexing structures and their combination: Bitmap index Join index Bitmap-Join index

23 Implementation Nowadays systems are typically divided into three categories: ROLAP (Relational OLAP) OLAP on top of a relational database MOLAP (Multi-Dimensional OLAP) Use of special multi-dimensional data structures HOLAP: (Hybrid) combination of previous two

24 ROLAP Typical database scheme: star schema fact table is central links to dimensional tables Extensions: snowflake schema dimensions s have hierarchy/extra a information o attached Star constellation multiple l star schemas sharing dimensions i

25 Example of a Star Schema Order Order No Order Date Customer Customer No Customer Name Customer Address City Salesperson SalespersonID SalespersonName City Quota Fact Table OrderNO O SalespersonID CustomerNO ProdNo DateKey CityName Quantity Total Price Product ProductNO ProdName ProdDescr CategoryName CategoryDescr UnitPrice Date DateKey Date Month Year City CityName State Country

26 Example of a Snowflake Schema Order Order No Product ProductNO Category Order Date Customer Fact Table ProdName ProdDescr CategoryName CategoryDescr Customer No Customer Name OrderNO SalespersonID CategoryName UnitPrice Customer Address CustomerNO City Salesperson SalespersonID SalespersonName City Quota ProdNo DateKey CityName Quantity Total Price Date DateKey Date Month City CityName StateName Month Month Year State StateName Country

27 Fact Constellation Multiple fact tables share the same dimensions E.g., (part, customer) shares Customer with (supplier, customer) Customer ID Name Address SC_fact C_ID S_ID amount Supplier ID Name Address SP_fact P_ID S_ID amount Part ID Name Color

28 Summary How is the data stored? Relational database (ROLAP) Specialized structures (MOLAP) How can we speed up computation? Indexing structures bitmap index join index Bitmap join index

29 MOLAP Not on top of relational database most popular design specialized data structures Multicubes vs Hypercubes Not all subcubes b are materialized User identifies set of sparse attributes S, and a set of dense attributes D. Index tree is constructed on sparse dimensions. Each leaf points to a multidimensional array indexed by D.

30 Example product, store are sparse dimensions date and customer-type are dense prod. p 1time ret reg Total prod. p 1/1/ store s1 2/1/ Total prod. p store s2 1time ret reg Total 1/1/ /1/ Total

31 Example product, store are sparse dimensions date and customer-type are dense prod. p 1time ret reg Total prod. p 1/1/ store s1 2/1/ Total E.g., B-tree, R-tree, prod. p store s2 1time ret reg Total 1/1/ /1/ Total

32 Example product, store are sparse dimensions date and customer-type are dense prod. p 1time ret reg Total prod. p 1/1/ store s1 2/1/07 582D array Direct access Total E.g., B-tree, R-tree, prod. p store s2 1time ret reg Total 1/1/ /1/ Total

33 Example product, store are sparse dimensions date and customer-type are dense prod. p E.g., B-tree, R-tree, 1time ret reg Total prod. p 1/1/ store s1 2/1/07 582D array Direct access prod. p store s Total time ret reg Total 1/1/ /1/ Total Linked list

34 Queries Efficiency depends on: does index on sparse dimensions fit into memory? Type of queries: Restrictions ti on all dimensions i Restrictions only on dense Restrictions s only on some sparse se and dense

35 Queries Selection on all attributes: (p,s1,ret,all) prod. p 1time ret reg Total prod. p 1/1/ store s1 2/1/ Total prod. p store s2 1time ret reg Total 1/1/ /1/ Total

36 Queries Only on dense attributes: (all,all,ret, 2/1/07 ) prod. p 1time ret reg Total prod. p 1/1/ store s1 2/1/ Total prod. p store s2 1time ret reg Total 1/1/ /1/ Total

37 Queries Only some sparse and dense attributes: (all,s1,ret, 2/1/07 ) prod. p 1time ret reg Total prod. p 1/1/ store s1 2/1/ Total prod. p store s2 1/1/ prod. p2 store s1 1time ret reg Total 2/1/ Total

38 Queries Only some sparse and dense attributes: (p,all,all, 2/1/07 ) prod. p 1time ret reg Total prod. p 1/1/ store s1 2/1/ Total prod. p store s2 prod. p2 store s1 1time ret reg Total 1/1/ /1/ Total

39 Storing the Cube (summary) Dense combinations of dimensions can be stored in multi-dimensional arrays For every combination of sparse dimensions one sub-cube Sub-cubes b indexed d by sparse dimensions i E.g., B-tree Order of the dimensions plays a role

40 Summary How is the data stored? relational database (ROLAP) Multi-dimensional structure (MOLAP) How can we speed up computation? Indexing structures bitmap index join index Bitmap-join index

41 Bitmap Index: Example Product Country Sales TV Ireland 20 TV France 126 HiFi Germany 56 PC Ireland 23 TV France 138 PC Germany 48 Index for Country: Index for Product Ireland TV France HiFi Germany PC

42 Bitmap Index: Example Index for Country: Ireland France Germany Index for Product TV HiFi PC SELECT sum(sales) FROM PCS WHERE (Country = Ireland or Country = France) and not (Product = TV) Access only tuples corresponding to a 1 in the bitmap: ( ) &! =

43 Bitmap Index Size of bitmaps can be reduced Use, e.g., run-length encoding is encoded as 4x1;3x0;4x1;7x0;4x1;3x0 Or, store a list of 1-positions instead of a full bitmap becomes 1;2;3;7;17 Can reduce the storage space significantly Logical operations can work directly on the encoding

44 Bitmap Index Works poorly for high cardinality domains For every value a bitmap Difficult to maintain Insert tuple = add entry in all bitmaps Therefore often not maintained but completely rebuilt after mass insertions Many commercial products have implemented Many commercial products have implemented bitmap indices

45 Summary How is the data stored? relational database (ROLAP) Specialized structures (MOLAP) How can we speed up computation? Indexing structures bitmap index join index Bitmap-join index

46 Join Index Traditional indexes: value in table rids in same table Join indices: Index on the join of two tables value in one table rids in other table Data warehouse: values of dimensions of star schema rows in fact table. Join indexes can span multiple dimensions

47 Join Index: Example Store(sID, city, country) Product(pID,brand) Inventory(sID, pid, cost) sid pid Cost sid City Country s1 Antwerp B s2 Brussels B s3 Amsterdam NL s1 p1 125 s1 p2 30 s2 p1 150 s2 p2 40 s2 p3 80 pid p1 p2 p3 Brand C C D s3 p2 35 s3 p3 75 Index table Inventory on Country, Brand

48 Join Index: Example Index table Inventory on Country, Brand sid pid Cost r1 s1 p1 125 Country Brand Row ID r2 s1 p2 30 B C r1, r2, r3, r4 r3 s2 p1 150 B D r5 r4 s2 p2 40 NL C r6, r7 r5 s2 p3 80 r6 s3 p2 35 r7 s3 p1 75 Can be used to answer queries involving the same join condition more efficiently

49 Join Index Join index can index tuples in the fact table based on an attribute in one of the dimensions Fact Table OrderNO SalespersonID CustomerNO ProdNo DateKey CityName Quantity Total Price City CityName State Country E.g., Index tuples in the fact table for the attribute Country

50 Summary How is the data stored? relational database (ROLAP) Specialized structures (MOLAP) How can we speed up computation? Indexing structures bitmap index join index Bitmap-join index

51 Bitmap-join index Logical combination of bitmap index and join index Different bitmap-join indices with bitmaps on same table can be combined EXAMPLE Customer Customer No Customer Name Customer Address City Fact Table CustomerNO ProdNo Quantity Total Price Product ProductNO ProdName ProdDescr Category CategoryDescr UnitPrice bitmap-join index on FactTable(Customer.City) bitmap-join index on FactTable(Product.Category) Slice on customers in Antwerp buying VCRs

52 Summary: indices Bitmap index, join-index, bitmap-join index: Can speedup selection queries with arbitrary Boolean combinations of indexed attributes Very interesting for ad-hoc analytical queries Are not easy to update Hence, not very suitable for operational databases with losts of inserts and deletes Typically these indices are completely rebuild after bulk inserts Therefore, very typical for datawarehouses and less suitable for OLTP systems

53 Summary Data warehouse is a specialized database to support analytical queries = OLAP queries Data cube as conceptual model Implementation of Data Cube View selection problem Explosion problem ROLAP vs. MOLAP Indexing structures