OLAP Systems and Multidimensional Expressions I

Transcription

1 OLAP Systems and Multidimensional Expressions I Krzysztof Dembczyński Intelligent Decision Support Systems Laboratory (IDSS) Poznań University of Technology, Poland Software Development Technologies Master studies, first semester Academic year 2014/15 (winter course) 1 / 44

2 Review of the previous lectures Mining of massive datasets Evolution of database systems: operational vs. analytical systems. Dimensional modeling. Operational vs. analytical systems. Extraction, transformation and load of data. 2 / 44

3 Outline 1 Motivation 2 OLAP Servers 3 SQL 4 Summary 3 / 44

5 OLAP systems The next step is to provide solutions for querying and reporting multidimensional analytical data. The goal is to provide efficient solutions for physical representation and processing of these data. 5 / 44

6 Multidimensional reports OLAP servers provide an effective solution for accessing and processing large volumes of high dimensional data. OLAP systems provide tools for multidimensional reporting. 6 / 44

8 Multidimensional cube The proper data model for multidimensional reporting is the multidimensional one. 8 / 44

9 Operators in multidimensional data model Roll up summarize data along a dimension hierarchy. Drill down go from higher level summary to lower level summary or detailed data. Slice and dice corresponds to selection and projection. Pivot reorient cube. Raking, Time functions, etc.. 9 / 44

10 Lattice of cuboids Different degrees of summarizations are presented as a lattice of cuboids. Example for the dimensions: time, product, location, supplier. Using this structure, one can easily show roll up and drill down operations. 10 / 44

11 Total number of cuboids For an n-dimensional data cube, the total number of cuboids that can be generated is: T = (L i + 1), i=1,...,n where L i is the number of levels associated with dimension i (excluding the virtual top level all since generalizing to all is equivalent to the removal of a dimension). For example, if the cube has 10 dimensions and each dimension has 4 levels, the total number of cuboids that can be generated will be: l = 5 10 = 9, / 44

12 Three types of aggregate functions distributive: count(), sum(), max(), min(), algebraic: ave(), std dev(), holistic: median(), mode(), rank(). 12 / 44

13 OLAP servers Relational OLAP (ROLAP), Multidimensional OLAP (MOLAP), Hybrid OLAP (HOLAP). 13 / 44

14 ROLAP ROLAP servers use a relational or post-relational database management system to store and manage warehouse data. Optimization techniques: Denormalization, Materialized views, Partitioning, Joins, Indexes, Query processing. 14 / 44

15 ROLAP Advantages of ROLAP Servers: Scalable with respect to the number of dimensions, Scalable with respect to the size of data, Sparsity is not a problem (fact tables contain only facts), Mature and well-developed technology. Disadvantage of ROLAP Servers: Worse performance than MOLAP, Additional data structures and optimization techniques used to improve the performance. 15 / 44

16 MOLAP MOLAP Servers use array-based multidimensional storage engines. Optimization techniques: Two-level storage representation: dense cubes are identified and stored as array structures, sparse cubes employ compression techniques, Materialized cubes. 16 / 44

17 MOLAP Advantages of MOLAP Servers: Multidimensional views are directly mapped to data cube array structures efficient access to data, Can easily store subaggregates. Disadvantages of MOLAP Servers: Scalability problem in the case of larger number of dimensions, Not tailored for sparse data, Young technology, There are no existing standards. 17 / 44

18 MOLAP Example Logical model consists of four dimensions: customer, product, location, and day In case of customers, products, locations and days, the data cube will contain cells! Huge number of cells is empty: a customer is not able to buy all products in all locations / 44

19 HOLAP HOLAP Servers are a hybrid approach that combines ROLAP and MOLAP technology. HOLAP benefits from the greater scalability of ROLAP and the faster computation of MOLAP. 19 / 44

20 Querying data warehouses There are two main approaches for querying data warehouses: SQL (ROLAP) MDX (MOLAP) 20 / 44

22 Querying the star schema SQL queries 22 / 44

23 SQL queries SQL group by SELECT Name, AVG(Grade) FROM Students grades G, Student S WHERE G.Student = S.ID GROUP BY Name; Name AVG(Grade) Inmon 4.8 Kimball 4.7 Gates 4.0 Todman / 44

24 SQL queries SQL group by SELECT Academic year, Name, AVG(Grade) FROM Students grades G, Academic year A, Professor P WHERE G.Professor = P.ID and G.Academic year = A.ID GROUP BY Academic year, Name; Academic year Name AVG(Grade) 2001/2 Stefanowski /3 Stefanowski /4 Stefanowski /2 S lowiński /3 S lowiński /4 S lowiński /4 Dembczyński / 44

25 SQL queries OLAP extensions in SQL: GROUP BY ROLLUP, GROUP BY CUBE, GROUP BY GROUPING SETS GROUPING and DECODE/CASE OVER Ranking functions 25 / 44

26 SQL queries GROUP BY CUBE SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY CUBE (Time, Product, Location, Supplier); 26 / 44

27 SQL queries GROUP BY CUBE SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY Time, Product, Location, Supplier UNION ALL SELECT Time, Product, Location, *, SUM(Gain) FROM Sales GROUP BY Time, Product, Location UNION ALL SELECT Time, Product, *, Location, SUM(Gain) FROM Sales GROUP BY Time, Product, Location UNION ALL... UNION ALL SELECT *, *, *, *, SUM(Gain) FROM Sales; 27 / 44

28 SQL queries GROUP BY CUBE SELECT Academic year, Name, AVG(Grade) FROM Students grades GROUP BY CUBE(Academic year, Name); Academic year Name AVG(Grade) 2001/2 Stefanowski /2 S lowiński /3 Stefanowski /3 S lowiński /4 Stefanowski /4 S lowiński /4 Dembczyński /2 NULL /3 NULL /4 NULL 3.8 NULL Stefanowski 3.9 NULL S lowiński 3.6 NULL Dembczyński 4.8 NULL NULL / 44

29 SQL queries GROUP BY ROLLUP SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY ROLLUP (Time, Product, Location, Supplier); 29 / 44

30 SQL queries GROUP BY ROLLUP SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY Time, Product, Location, Supplier UNION ALL SELECT Time, Product, Location, *, SUM(Gain) FROM Sales GROUP BY Time, Product, Location UNION ALL SELECT Time, Product, *, *, SUM(Gain) FROM Sales GROUP BY Time, Product UNION ALL SELECT Time, *, *, *, SUM(Gain) FROM Sales GROUP BY Time UNION ALL SELECT *, *, *, *, SUM(Gain) FROM Sales; 30 / 44

31 SQL queries GROUP BY ROLLUP SELECT Academic year, Name, AVG(Grade) FROM Students grades G GROUP BY ROLLUP(Academic year, Name); Academic year Name AVG(Grade) 2001/2 Stefanowski /2 S lowiński /3 Stefanowski /3 S lowiński /4 Stefanowski /4 S lowiński /4 Dembczyński /2 NULL /3 NULL /4 NULL 3.8 NULL NULL / 44

32 GROUP BY GROUPING SETS SQL queries SELECT Time, Product, Location, Supplier, SUM(Gain) FROM Sales GROUP BY GROUPING SETS ((Time), (Product), (Location), (Supplier)); 32 / 44

33 SQL queries GROUP BY GROUPING SETS SELECT Time, *, *, *, SUM(Gain) FROM Sales GROUP BY Time UNION ALL SELECT *, Product, *, *, SUM(Gain) FROM Sales GROUP BY Product UNION ALL SELECT *, *, Location, *, SUM(Gain) FROM Sales GROUP BY Location UNION ALL SELECT *, *, *, Supplier, SUM(Gain) FROM Sales GROUP BY Supplier; 33 / 44

34 SQL queries GROUP BY GROUPING SETS SELECT Academic year, Name, AVG(Grade) FROM Students grades GROUP BY GROUPING SETS ((Academic year), (Name),()); Academic year Name AVG(Grade) 2001/2 NULL /3 NULL /4 NULL 3.8 NULL Stefanowski 3.9 NULL S lowiński 3.6 NULL Dembczyński 4.8 NULL NULL / 44

35 SQL queries GROUPING(<column expression>) Returns a value of 1 if the value of expression in the row is a null representing the set of all values. <column expression> is a column or an expression that contains a column in a GROUP BY clause. GROUPING is used to distinguish the null values that are returned by ROLLUP, CUBE or GROUPING SETS from standard null values. The NULL returned as the result of a ROLLUP, CUBE or GROUPING SETS operation is a special use of NULL. 35 / 44

36 SQL queries GROUPING(<column expression>) SELECT Extra scholarship, AVG(Grade), GROUPING(Extra scholarship) as Grouping FROM Students grades GROUP BY ROLL UP(Extra scholarship); Extra scholarship AVG(Grade) Grouping Yes No NULL NULL / 44

37 SQL queries DECODE(expression, search, result [, search, result]... [, default] ) If the value of expression is equal to search, then result is returned, otherwise default is returned. The functionality is similar to CASE expression, The results of GROUPING() can be passed into a DECODE function or the CASE expression. 36 / 44

38 SQL queries DECODE(expression, search, result [, search, result]... [, default] ) SELECT DECODE(GROUPING(Extra scholarship), 1, "Total Average", Extra scholarship) as Extra scholarship, AVG(Grade) FROM Students grades GROUP BY ROLL UP(Extra scholarship); Extra scholarship AVG(Grade) Yes 4.15 No 3.61 NULL 4.03 Total average / 44

39 SQL queries OVER(): Determines the partitioning and ordering of a rowset before the associated window function is applied. The OVER clause defines a window or user-specified set of rows within a query result set. A window function then computes a value for each row in the window. The OVER clause can be used with functions to compute aggregated values such as moving averages, cumulative aggregates, running totals, or a top N per group results. Syntax: OVER ( [ <PARTITION BY clause> ] [ <ORDER BY clause> ] [ <ROW or RANGE clause> ] ) 37 / 44

40 SQL queries OVER(): PARTITION BY: Divides the query result set into partitions. The window function is applied to each partition separately and computation restarts for each partition. ORDER BY: Defines the logical order of the rows within each partition of the result set, i.e., it specifies the logical order in which the window function calculation is performed. ROW RANGE: Further limits the rows within the partition by specifying start and end points within the partition. This is done by specifying a range of rows with respect to the current row either by logical association or physical association. The ROWS clause limits the rows within a partition by specifying a fixed number of rows preceding or following the current row. The RANGE clause logically limits the rows within a partition by specifying a range of values with respect to the value in the current row. Preceding and following rows are defined based on the ordering in the ORDER BY clause. 38 / 44

41 SQL queries Ranking functions: RANK () OVER: Returns the rank of each row within the partition of a result set. The rank of a row is one plus the number of ranks that come before the row in question. DENSE RANK () OVER: Returns the rank of rows within the partition of a result set, without any gaps in the ranking. The rank of a row is one plus the number of distinct ranks that come before the row in question. NTILE (integer expression) OVER: Distributes the rows in an ordered partition into a specified number of groups. The groups are numbered, starting at one. For each row, NTILE returns the number of the group to which the row belongs. ROW NUMBER () OVER: Returns the sequential number of a row within a partition of a result set, starting at 1 for the first row in each partition. 39 / 44

42 SQL queries Examples: Ranking of the students SELECT Student, Avg(Grade), RANK () OVER (ORDER BY Avg(Grade) DESC) FROM Students grades GROUP BY Student; To sort according to rank, we need to order the resulting relation: SELECT Student, Avg(Grade), RANK () OVER (ORDER BY Avg(Grade) DESC) AS rank of grades FROM Students grades GROUP BY Student ORDER BY rank of grades; 40 / 44

43 SQL queries Examples: Ranking of students partitioned by instructors. SELECT Instructor Name, Student, Avg(Grade), RANK () OVER (PARTITION BY Student ORDER BY Avg(Grade) DESC) AS rank 1 FROM Students grades GROUP BY Student,Instructor Name ORDER BY Instructor Name, rank 1; Moving average of a student: SELECT Student, Academic year, AVG (grades) OVER (PARTITION BY Student ORDER BY Academic year DESC ROWS UNBOUNDED PRECEDING) FROM Students grades ORDER BY Student, Academic year; 41 / 44

45 Summary Three types of OLAP servers: ROLAP, MOLAP, and HOLAP. Several approaches for querying data warehouses. ROLAP servers: SQL and its OLAP extensions. MOLAP servers: / 44

46 Bibliography A. Pelikant, Hurtownie danych. Od przetwarzania analitycznego do raportowania, Helion 2011 SAS OLAP Server MDX Guide Third Edition, 2006 MSDN: SQL Server Developer Center, / 44