Data Warehousing. Wolf-Tilo Balke Silviu Homoceanu Institut für Informationssysteme Technische Universität Braunschweig

Transcription

1 Data Warehousing & Data Mining Wolf-Tilo Balke Silviu Homoceanu Institut für Informationssysteme Technische Universität Braunschweig

2 5. Queries 5. Queries 5.1 OLAP query languages OLAP operators in SQL MDX (MultiDimensional expressions) 5.2 Data modeling Logical modeling - implementation Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 2

3 5.1 How does OLAP work? Presentation Presentation Presentation OLAP Interface HOLAP Server ROLAP Server MDDB MDDB RDBMS RDBMS Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 3

4 5.1 How does OLAP work? OLAP systems Client/server architecture The client displays reports and allows interaction with the end user to perform the OLAP operations and other custom queries The server is responsible for providing the requested data. How? It depends on whether it is MOLAP, ROLAP, HOLAP, etc. Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 4

5 5.1 How does OLAP work? OLAP server High-capacity, multi-user data manipulation engine specifically designed to support and operate on multidimensional data structures It is optimized for Fast, flexible calculation and transformation of raw data based on formulaic relationships Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 5

6 5.1 How does OLAP work? OLAP server may either Physically stage the processed multidimensional information to deliver consistent and rapid response times to end users MOLAP Populate its data structures in real-time from relational or other databases ROLAP Or offer a choice of both HOLAP Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 6

7 5.1 How does OLAP work? We have seen that The best way to represent data at the presentation level is multidimensional Regardless if the storage is multidimensional (MOLAP) or relational (ROLAP) Optimal for analyze purposes: easy to understand by the decision makers, natural representations of the data in businesses, etc Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 7

8 5.1 OLAP query languages Getting from OLAP operations to the data As in the relational model, through queries In OLTP we have SQL as the standard query language However, OLAP operations are hard to express in SQL There is no standard query language for OLAP Choices are: SQL-99 MDX (Multidimensional expressions) Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 8

9 5.1 OLAP query languages SQL-99 Prepare SQL for OLAP queries New SQL commands GROUPING SETS ROLLUP CUBE New aggregate functions Queries of type top k Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 9

10 5.1 SQL-99 Shortcomings of SQL/92 with regard to OLAP queries Hard or impossible to express in SQL Multiple aggregations Comparisons (with aggregation) Reporting features Performance penalty Poor execution of queries with many AND and OR conditions Lack of support for statistical functions Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 10

11 5.1 SQL-99 Multiple aggregations in SQL/92 Create a 2D spreadsheet that shows sum of sales by maker as well as car model Each subtotal requires a separate aggregate query SUV Sedan Sport By make BMW Mercedes By model SUM SELECT model, make, sum(amt) FROM sales GROUP BY model, make union SELECT model, sum(amt) FROM sales GROUP BY model union SELECT make, sum(amt) FROM sales GROUP BY make union SELECT sum(amt) FROM sales Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 11

12 5.1 SQL-99 Comparisons in SQL/92 This year s sales vs. last year s sales for each product Requires a self-join CREATE VIEW v_sales AS SELECT prod_id, year, sum(qty) AS sale_sum FROM sales GROUP BY prod_id, year; SELECT cur.prod_id, cur.year, cur.sale_sum, last.year, last.sale_sum FROM v_sales cur, v_sales last WHERE cur.year = (last.year+1) AND cur.prod_id = last.prod_id; Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 12

13 5.1 SQL-99 Reporting features in SQL/92 Too complex to express RANK (top k) and NTILE ( top X% of all products) Median Running total, moving average, cumulative totals E.g., moving average over a 3 day window of total sales for each product CREATE OR REPLACE VIEW v_sales AS SELECT prod_id, time_id, sum(qty) AS sale_sum FROM sales GROUP BY prod_id, time_id; SELECT end.time, avg(start.sale_sum) FROM v_sales start, v_sales end WHERE end.time >= start.time AND end.time <= start.time + 2 GROUP BY end.time; Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 13

14 5.1 SQL-99 Grouping operators Extensions to the GROUP BY operator GROUPING SET CUBE ROLLUP Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 14

15 5.1 Grouping operators GROUPING SET Used for reporting purposes Replaces the series of UNIONed queries SELECT dept_name, CAST(NULL AS CHAR(10)) AS job_title, COUNT(*) FROM personnel GROUP BY dept_name UNION ALL SELECT CAST(NULL AS CHAR(8)) AS dept_name, job_title, COUNT(*) FROM personnel GROUP BY job_title; Can be re-written as: SELECT dept_name, job_title, COUNT(*) FROM Personnel GROUP BY GROUPING SET (dept_name, job_title); Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 15

16 5.1 Grouping set The issue of NULL values The new grouping functions generate NULL values at the subtotal levels So we have generated NULLs and real NULLs from the data itself How do we tell the difference? Through the GROUPING function return value: GROUPING(job_title) which returns 0 for NULL in the data and 1 for generated NULL Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 16

17 5.1 Grouping operators ROLLUP Produces a result set that contains subtotal rows in addition to regular grouped rows GROUP BY ROLLUP (a, b, c) is equivalent to GROUP BY GROUPING SETS (a, b, c),(a, b), (a), () N elements of the ROLLUP translate to (N+1) grouping sets Order is significant to ROLLUP! GROUP BY ROLLUP (c, b, a) is equivalent with grouping sets of (c, b, a), (c, b), (c), () Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 17

18 5.1 ROLLUP ROLLUP operation, e.g.,: SELECT year, brand, SUM(qty) FROM sales GROUP BY ROLLUP(year, brand); Year Brand SUM(qty) 2008 Mercedes BMW VW Mercedes (year, brand) (year) (year, brand) (year) (ALL) Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 18

19 5.1 Grouping operators CUBE operator Contains all the subtotal rows of a ROLLUP and in addition cross-tabulation rows Can also be thought as a series of GROUPING SETs All permutations of the cubed grouping expressions are computed along with the grand total N elements of a CUBE translate to 2 n grouping sets: GROUP BY CUBE (a, b, c) is equivalent to GROUP BY GROUPING SETS(a, b, c) (a, b) (a, c) (b, c) (a) (b) (c) () Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 19

20 5.1.1 CUBE operator Aggregate Sum SUV SEDAN SPORT Group By (with total) By model Sum SUV SEDAN SPORT By Make Cross Tab BMW MERCBy model Sum By Make & Year By Year By model& Year The Data Cube and The Sub-Space Aggregates By Make SUV SEDAN SPORT By Make & model Sum By model Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 20

21 5.1 CUBE E.g., CUBE operator SELECT year, brand, SUM(qty) FROM sales GROUP BY CUBE (year, brand); Year Brand SUM(qty) 2008 Mercedes BMW VW Mercedes Mercedes 300 BMW 350 VW (year, brand) (year) (year, brand) (year) (brand) (ALL) Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 21

22 5.1 OLAP functions Diagram of OLAP function evaluation... Partitioning Partitioning Sorting Sorting Dynamic Window Dynamic Window Aggregation Aggregation OVER(PARTITION BY ORDER BY ROWS BETWEEN ) RANK(), SUM() Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 22

23 5.1 OLAP functions The window clause Specify that we want to perform an action over a set of rows 3 sub-clauses: Partitioning, ordering and aggregation grouping General format: <aggregate function> OVER ([PARTITION BY <column list>] ORDER BY <sort column list> [<aggregation grouping>]) Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 23

24 5.1 Window clause Moving averages are hard to compute with SQL-92 It involves multiple self joins for the fact table With the window clause we can create dynamical windows: expressed in the <aggregation grouping> SELECT AVG(sales) OVER (PARTITION BY region ORDER BY month ASC ROWS 2 PRECEDING) AS SMA3 Moving average of 3 rows Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 24

25 5.1 Ranking in SQL Ranking operators in SQL Row numbering is the most basic ranking function ROW_NUMBER() returns a column as an expression that contains the row s number within the result set E.g., SELECT SalesOrderID, CustomerID, ROW_NUMBER() OVER (ORDER BY SalesOrderID) as RunningCount FROM Sales WHERE SalesOrderID > ORDER BY SalesOrderID; SalesOrderID CustomerID RunningCount Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 25

26 5.1 Ranking in SQL ROW_NUMBER doesn t consider tied values 2 equal considered values get 2 different returns SalesOrderID RunningCount The behavior is non-deterministic Each tied value could have its number switched!! Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 26

27 5.1 Ranking in SQL RANK and DENSE_RANK functions Allow ranking items in a group The difference between RANK and DENSE_RANK is that DENSE_RANK leaves no gaps in ranking sequence when there are ties Syntax: RANK ( ) OVER ( [query_partition_clause] order_by_clause ) DENSE_RANK ( ) OVER ( [query_partition_clause] order_by_clause ) Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 27

28 5.1 Ranking in SQL SQL99 Ranking e.g., SELECT channel, calendar, TO_CHAR(TRUNC(SUM(amount_sold),-6), '9,999,999') SALES, RANK() OVER (ORDER BY Trunc(SUM(amount_sold),-6) DESC) AS RANK, DENSE_RANK() OVER (ORDER BY TRUNC(SUM(amount_sold),-6) DESC) AS DENSE_RANK FROM sales, products CHANNEL CALENDAR SALES RANK DENSE_RANK Direct sales , Direct sales , Internet , Internet , Partners , Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 28

29 5.1 Ranking in SQL Other flavors of ranking Group ranking RANK function can operate within groups: the rank gets reset whenever the group changes A single query can contain more than one ranking function, each partitioning the data into different groups Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 29

30 5.1 Group Ranking This is accomplished with the PARTITION BY clause E.g., SELECT RANK() OVER (PARTITION BY channel ORDER BY SUM(amount_sold) DESC) AS RANK_BY_CHANNEL CHANNEL CALENDAR SALES RANK _BY_CHANNEL Direct sales ,000 1 Direct sales ,000 2 Internet ,000 1 Internet ,000 1 Partners ,000 1 Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 30

31 5.1 Ranking in SQL The treatment of NULL values: NULLs are treated as normal values A NULL value is equal to another NULL value They are given ranks according to The ASC DESC options provided for measures The NULLS FIRST NULLS LAST clause MONTH SOLD NULL FIRST ASC NULL LAST ASC NULLFIRST DESC NULLLAST DESC Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 31

32 5.1 Ranking in SQL Top k ranking By enclosing the RANK function in a sub-query and then applying a filter condition outside the sub-query SELECT * FROM (SELECT country_id, SUM(amount_sold) SALES, RANK() OVER (ORDER BY SUM(amount_sold) DESC ) AS COUNTRY_RANK FROM sales, products, customers, times, channels WHERE... GROUP BY country_id) WHERE COUNTRY_RANK <= 5; Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 32

33 5.1 NTILE NTILE Not a part of SQL99 standards but adopted by major vendors Splits a set into equal groups It divides an ordered partition into buckets and assigns a bucket number to each row in the partition Buckets are calculated so that each bucket has exactly the same number of rows assigned to it or at most 1 row more than the others Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 33

34 5.1 NTILE SELECT NTILE(3) OVER (ORDER BY sales) NT_3 FROM CHANNEL CALENDAR SALES NT_3 Direct sales ,000 1 Direct sales ,000 1 Internet ,000 2 Internet ,000 2 Partners ,000 3 NTILE(4) quartile NTILE(100) percentage Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 34

35 5.1 MDX MDX (MultiDimensional expressions) Developed by Microsoft Not really brilliant But adopted by major OLAP providers due to Microsoft's market leader position Used in OLE DB for OLAP (ODBO) with API support XML for Analysis (XMLA): specification of web services for OLAP Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 35

36 5.1 MDX Similar to SQL syntax SELECT {Deutschland, Niedersachsen, Bayern, Frankfurt} ON COLUMNS, {Qtr1.CHILDREN, Qtr2, Qtr3} ON ROWS FROM SalesCube WHERE (Measures.Sales, Time.[2008], Products.[All Products]); SELECT axes dimensions, on columns and rows FROM Data source cube specification If joined, data cubes must share dimensions WHERE Slicer - restricts the data area Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 36

37 5.1 MDX Lists Enumeration of elementary nodes from different classification levels E.g. {Deutschland, Niedersachsen, [Frankfurt am Main], USA} Generated elements Methods which lead to new sets of the classification levels Deutschland.CHILDREN generates: {Niedersachsen, Bayern, } Niedersachsen.PARENT generates Deutschland Time.Quarter.MEMBERS generates all the elements of the classification level Functional generation of sets DESCENDENT(USA, Cities): children of the provided classification levels GENERATE ({USA, France}, DESCENDANTS(Geography.CURRENT, Cities)): enumerates all the cities in USA and France Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 37

38 5.1 MDX Sets nesting combines individual coordinates to reduce dimensionality SELECT CROSSJOIN({Deutschland, Sachsen, Hannover, BS}{Ikeea, [H&M-Möbel]}) ON COLUMNS, {Qtr1.CHILDREN, Qtr2} ON ROWS FROM salescube WHERE (Measure.Sales, Time.[2008], Products.[All Products]); Deutschland Sachsen Hannover BS Ikeea H&M- Möbel Ikeea H&M- Möbel Ikeea H&M- Möbel Ikeea H&M- Möbel Jan 08 Feb 08 Mar 08 Qtr2 Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 38

39 5.1 MDX Relative selection Uses the order in the dimensional structures Time.[2008].LastChild : last quarter of 2008 [2008].NextMember : {[2009]} [2008].[Qtr4].Nov.Lead(2) : Jan 2009 [2006]:[2009] represents [2006],.., [2009] Methods for hierarchy information extraction Deutschland.LEVEL : country Time.LEVELS(1) : Year Brackets {}: Sets, e.g. {Hannover, BS, John} []: text interpretation of numbers, empty spaces between words or other symbols E.g. [2008], [Frankfurt am Main], [H&M] (): tuple e.g. WHERE (Measure.Sales, Time.[2008], Products.[All Products]) Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 39

40 5.1 MDX Special functions and filters Special functions TOPCOUNT(), TOPPERCENT(), TOPSUM() E.g. SELECT {Time.CHILDREN} ON COLUMNS, {TOPCOUNT(Deutschland.CHILDREN, 5, Sales.turnover)} ON ROWS FROM salescube WHERE (Measure.Sales, Time.[2008]); Filter function E.g. SELECT FILTER(Deutschland.CHILDREN, ([2008], Turnover) > ([2007], Turnover)) ON COLUMNS, Quarters.MEMBERS ON ROWS FROM salescube WHERE (Measure.Sales, Time.[2008], Products.Electronics); Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 40

41 5.1 MDX Time series Set Values Expressions Choosing time intervals PERIODSTODATE(Quarter, [15-Nov-2008]): returns LASTPERIODS(3, [Sept-2008]): returns [June-2008], [July-2008], [Aug- 2008] Member Value Expressions Pre-periods PARALLELPERIOD(Year, 3, [Sep-2008]): returns [Sep-2005] Numerical functions COVARIANCE, CORRELATION LINEAR REGRESSION Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 41

42 5.1 mdxml XMLA (XML for Analysis) Most recent attempt at a standardized API for OLAP Allows client applications to talk to multi-dimensional data sources In XMLA, mdxml is a MDX wrapper for XML Underlying technologies XML, SOAP, HTTP Service primitives DISCOVER Retrieve information about available data sources, data schemas, server infos EXECUTE Transmission of a query and the corresponding conclusion Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 42

43 5.2 Data Modeling Store dimension Now we know What OLAP looks like From the outside From the inside SQL 99, MDX How data is modeled Conceptual level muml, ME/R Logical level Cubes, dimensions Product dimension Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 43

44 5.2 Data Modeling But how do we implement it On a logical level On a physical level DBMS Independent DBMS Dependent Functional Analysis Application Program Design Requirement Analysis Conceptual Design Logical Design Data requirements Conceptual schema Logical schema Physical Design Transaction Implementation Application Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 44

45 5.2 Implementation Implementation of the multidimensional data model can be: Relational Snowflake-schema Star-schema Multidimensional Array technique Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 45

46 5.2 Implementation Relational Implementation Main goals: As low loss of semantically knowledge as possible e.g., classification hierarchies The translation from multidimensional queries must be efficient The RDBMS should be able to run the translated queries efficiently The maintenance of the present tables should be easy and fast e.g., when loading new data Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 46

47 5.2 Implementation Going from multidimensional to relational Representations for cubes, dimensions, classification hierarchies and attributes Implementation of cubes without the classification hierarchies is easy A table can be seen as a cube A column of a table can be considered as a dimension mapping A tuple in the table represents a cell in the cube If we interpret only a part of the columns as dimensions we can use the rest as measures The resulting table is called a fact table Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 47

48 5.2 Implementation Product 818 Laptops Mobile p. Time Geography Article Store Day Sales Laptops Hannover, Saturn Mobile Phones Hannover Saturn Laptops Braunschweig Saturn Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 48

49 5.2 Snowflake Schema Snowflake-schema Simple idea: use a table for each classification level This table includes the ID of the classification level and other attributes 2 neighbor classification levels are connected by 1:n connections e.g., from n Days to 1 Month The measures of a cube are maintained in a fact table Besides measures, there are also the foreign key IDs for the smallest classification levels Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 49

50 5.2 Snowflake Schema Snowflake? The facts/measures are in the center The dimensions spread out in each direction and branch out with their granularity Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 50

51 5.2 Snowflake Example n 1 Product group Product_group_ID Description Product_categ_ID Product category Product_category_ID Description 1 n Product Product_ID Description Brand Product_gro up_id 1 n n Sales Product_ID Day_ID Store_ID Sales Revenue n n 1 Day Day_ID Description Month_ID Week_ID n 1 1 n Week Week_ID Description Year_ID Month Month_ID Description Quarter_ID n 1 1 Year Year_ID Description Region Region_ID Description Country_ID 1 n n State State_ID Description Region_ID 1 1 n Store Store_ID Description State_ID 1 time Quarter Quarter_ID Description Year_ID n 1 Country Country_ID Description location fact table dimension tables Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 51

52 5.2 Snowflake Schema Snowflake schema Advantages With a snowflake schema the size of the dimension tables will be reduced and queries will run faster If a dimension is very sparse (most measures corresponding to the dimension have no data) And/or a dimension has long list of attributes which may be queried Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 52

53 5.2 Snowflake Schema Snowflake schema Disadvantages Fact tables are responsible for 90% of the storage requirements Thus, normalizing the dimensions usually lead to insignificant improvements Normalization of the dimension tables can reduce the performance of the DW because it leads to a large number of tables E.g., when connecting dimensions with coarse granularity these tables are joined with each other during queries A query which connects Product category with Year and Country is clearly not performant (10 tables need to be connected) Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 53

54 5.2 Snowflake Example n 1 Product group Product_group_ID Description Product_categ_ID Product category Product_category_ID Description 1 n Product Product_ID Description Brand Product_gro up_id 1 n n Sales Product_ID Day_ID Store_ID Sales Revenue n n 1 Day Day_ID Description Month_ID Week_ID n 1 1 n Week Week_ID Description Year_ID Month Month_ID Description Quarter_ID n 1 1 Year Year_ID Description Region Region_ID Description Country_ID 1 n n State State_ID Description Region_ID 1 1 n Store Store_ID Description State_ID 1 Quarter Quarter_ID Description Year_ID n 1 Country Country_ID Description Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 54

55 5.2 Star Schema Star schema Basic idea: use a denormalized schema for all the dimensions A star schema can be obtained from the snowflake schema through the denormalization of the tables belonging to a dimension Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 55

56 5.2 Star Schema - Example Product Product_ID Product group Product category Description 1 n n Sales Product_ID Time_ID Geo_ID Sales Revenue n 1 Time Time_ID Day Week Month Quarter Year 1 Geography Geo_ID Store State Region Country Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 56

57 5.2 Star Schema Advantages Improves query performance for often-used data Less tables and simple structure Efficient query processing with regard to dimensions Disadvantages In some cases, high overhead of redundant data Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 57

58 5.2 Snowflake vs. Star Snowflake vs. Star The structure of the classifications are expressed in table schemas The fact table and dimension tables are normalized The entire classification is expressed in just one table The fact table is normalized while in the dimension table the normalization is broken This leads to redundancy of information in the dimension tables Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 58

59 5.2 Examples Snowflake Star Product_ID Description Brand Prod_group_ID 10 E71 Nokia 4 11 PS-42A Samsung Nokia 4 Bold Berry 4 Prod_group_ID Description Prod_categ_ID 2 TV 11 4 Mobile Pho.. 11 Product_ ID Description Prod. group Prod. categ 10 E71 Mobile Ph.. Electronics 11 PS-42A TV Electronics Mobile Ph.. Electronics 13 Bold Mobile Ph.. Electronics Prod_categ_ID Description 11 Electronics Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 59

60 5.2 Snowflake to Star When should we go from Snowflake to star? Heuristics-based decision When typical queries relate to coarser granularity (like product category) When the volume of data in the dimension tables is relatively low compared to the fact table In this case a star schema leads to negligible overhead through redundancy, but performance is improved When modifications on the classifications are rare compared to insertion of fact data In this case these modifications controlled through the data load process of the ETL reducing the risk of data anomalies Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 60

61 5.2 Do we have a winner? Snowflake or Star? It depends on the necessity Fast query processing or efficient space usage However, most of the time a mixed form is used The Starflake schema: some dimensions stay normalized corresponding to the snowflake schema, while others are denormalized according to the star schema Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 61

62 5.2 Our forces combined The Starflake schema The decision on how to deal with the dimensions is influenced by Frequency of the modifications: if the dimensions change often, normalization leads to better results Amount of dimension elements: the bigger the dimension tables, the more space normalization saves Number of classification levels in a dimension: more classification levels introduce more redundancy in the star schema Materialization of aggregates for the dimension levels: if the aggregates are materialized, a normalization of the dimension can bring better response time Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 62

63 5.2 More Schemas Galaxies In pratice we usually have more measures described by different dimensions Thus, more fact tables Store Store_ID Product Product_ID Sales Product_ID Store_ID Sales Revenue Receipts Product_ID Date_ID Date Date_ID Vendor Vendor_ID Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 63

64 5.2 More Schemas Other schemas Fact constellations Pre-calculated aggregates Factless fact tables Fact tables do not have non-key data Can be used for event tracking or to inventory the set of possible occurrences Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 64

65 5.2 Multidimensional? Multidimensional implementation The representation of the multidimensional data can be implemented relationally with a finite set of transformation steps, however: Multidimensional queries have to be first translated to the relational representation A direct interaction with the relational data model is not fit for the end user Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 65

66 5.2 Multidimensional? Data structures The basic data structure for multidimensional data storage is the array The elementary data structures are the cubes and the dimensions C=((D 1,, D n ), (M 1,, M m )) The storage is intuitive as arrays of arrays, physically linearized More about linearization and related issues will be discussed in the lecture optimization Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 66

67 5.2 Physical Model? Defining the physical structures Setting up the database environment Setting up the appropriate security Preliminary performance tuning strategies Indexing Partitioning Materialization Goal: define the actual storage architecture decide on how the data is to be accessed and how it is arranged Again, this will be discussed in more detail in the lecture on optimization Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 67

68 Next lecture Optimization Indexes Data Warehousing & OLAP Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 68