A Data Warehousing and Business Intelligence Tutorial. Starring Sakila

Transcription

1 A Data Warehousing and Business Intelligence Tutorial Starring Sakila

2 Welcome! Matt Casters Chief Data Integration, Pentaho Roland Bouman Software Engineer, Pentaho Starring Sakila

3 Commercial Open Source Business Intelligence Full BI suite since 2005 Projects: Kettle (DI & ETL), Jfree (Reporting), Mondrian (OLAP), Weka (Data Mining) Community: CDF (Dashboarding), Saiku (OLAP) Recent: Focus on Big Data, esp. Hadoop Pentaho

4 Business Intelligence Data Warehousing Anatomy of a Data Warehouse Physical Implementation Sakila a Star is Born Filling the Data Warehouse Presenting the Data - BI Applications Agenda

5 Part I: Business Intelligence Starring Sakila

6 Skills, technologies, applications and practices to acquire a better understanding of the commercial context of your business Turning data into information useful for business users Management Information Decision Support Business Intelligence

7 months, years: Data mining Should we become an appliance vendor instead of delivering software solutions Executives Strategic OLAP/Analysis weeks, months: In what region should we open a new store? Managers Tactical days, weeks: Who's available for tomorrow's shift Analysts Customers Partners Reporting Employees Operational Business Intelligence Scope

8 Front end Applications: Reports Charts and Graphs OLAP Pivot tables Data Mining Dashboards Back end Infrastructure Data Integration Data Warehouse Data Mart Metadata (ROLAP) Cube Functional Parts of a Business Intelligence Solution

9 Sources Back-end Front-end Reporting Operational Datastore ERP Charts /Graphs Enterprise Data Warehouse CRM OLAP/Analysis Staging Area Meta Data External Data Dashboards Datamarts Data Mining Extract Transform Load Present High Level BI Architecture

10 Part II: Data Warehousing Starring Sakila

11 A database designed to support Business Intelligence Applications Different requirements as compared to Operational Applications Analytic Database Systems (ADBMS) MySQL: Infobright, InfiniDB LucidDB, MonetDB What is a Data Warehouse?

12 Ultimately, it's just a Relational Database Designed for Business Intelligence applications Ease of use Performance Data from various source systems Tables, Columns,... Integration, Standardization, Data cleaning Add and maintain history Corporate memory What is a Data Warehouse?

13 A database designed to support BI applications BI applications (OLAP) differ from Operational applications (OLTP) OLTP: Online Transaction Processing OLAP: Online Analytical Processing Differences: Applications, Data Processing, Data Model What is a Data Warehouse?

14 OLTP OLAP Operational Tactical, Strategic 'Always' on Periodically Available All kinds of users Managers, Directors Many users Few(er) users Directly supports business process Redesign Business Process Keep a Record of Current status Decision support, long-term planning Maintains a history OLTP vs OLAP: Application Characterization

15 OLTP OLAP Transactions Groups Subject Oriented Aspect Oriented Add, Modify, Remove single rows Bulk load, rarely modify, never remove Human data entry Automated ETL jobs Queries for small sets of rows with all their details Scan large sets to return aggregates over arbitrary groups Standard queries Ad-hoc queries OLTP vs OLAP: Data Processing

16 OLTP OLAP Entity-Relationship model Dimensional model Entities, Attributes, Relationships Facts, Dimensions, Hierarchies Foreign key constraints Ref. integrity ensured in loading process Indexes to increase performance Scans on Fact table obliterates indexes Normalized to 3NF or BCNF Denormalized Dimensions (<= 1NF) OLTP vs OLAP: Data Model

17 Sources Back-end Front-end Reporting Operational Datastore ERP Charts /Graphs Enterprise Data Warehouse CRM OLAP/Analysis Staging Area Meta Data External Data Dashboards Datamarts Data Mining Extract Transform Load Present High Level BI Architecture

18 Part III: Dimensional Model Starring Sakila

19 An aspect-oriented logical data model optimized for querying and data presentation Divides data in two kinds: Facts Dimensions What is the Dimensional Model?

20 Facts Measures/Metrics of a Business Process Examples: Cost, Units Sold, Profit Dimensions Context of Business Process Who? What? Where? When? Why? Navigate Facts: Selection, Rollup, Drilldown Provide and maintain history The Dimensional Model

21 Date Dimension Location Dimension All locations America All America North South Europe All Europe East West 2008 Q4 All Months $ 3850 $ 2050 October November December $ 1000 $ 500 $ 1350 $ 750 $ 1500 $ 800 $ 1275 $ 775 $ 1800 $ 300 $ 200 $ 500 $ 500 $ 250 $ 600 $ 475 $ 325 $ 700 $ 800 $ 1000 $ 250 $ 250 $ 250 $ 350 $ 300 $ 400 Dimensional Data Presentation

22 Fact table structure: Several measures Keys to dimension tables Measures: Usually numeric, Additive, Semi-additive Sometimes pre-calculated Rapidly growing! Millions, Billions of rows (Terabytes) The Dimensional Model: Facts

23 Dimension table structure: Surrogate key and descriptive text attributes Relatively few rows Exception: Customer 'Monster' dimension Relatively static Exception: Slowly changing dimensions Used to navigate through fact data Hierarchies The Dimensional Model: Dimensions

24 Selection (Filter) Navigation: Attributes organized in Hierarchies Date dimension examples: Year, Quarter, Month, Day Year, Week, Day Groupings for Aggregation 'Roll up', 'Drill Down' 'Slice and Dice' The Dimensional Model: Navigating data with Dimensions

25 Fact table usually links to a date dimension Dimensions maintain their own history Slowly changing dimensions Type I Overwrite (no history) Type II History kept in rows (versioning) Type III History kept in columns The Dimensional Model: Maintaining History

26 Part V: Physical Implementation Starring Sakila

27 Related metrics stored in a Fact table Fact table references relevant dimensions Each Dimension stored in a Dimension Table Dimension tables shared by multiple fact tables Dimensional Model Implementation: Star Schema

28 Store Date Film Time Rentals Staff Customer Star Schema example: Sakila Rentals

29 dim_film dim_date fact_inventory fact_rental fact_payment dim_staff dim_customer dim_store dim_store Star Schema example: Sakila Rentals

30 Star schema is 'just' an implementation Optimized for simplicity Optimized for performance (?) Heavily denormalized dimensions There is an alternative: Snowflake Normalized dimensions Stars versus Snowflakes

31 Country Year City Quarter Month Store Date Week Language Film Rating Rentals Minute Customer Hour Country Staff City Snow Flake example: Sakila Rentals

32 Part V: A Star is Born Starring Sakila

33 MySQL Sample Database DVD rental business Overly simplified database schema Typical OLTP database Dimensional Model example

34 Category Language Country Actor Film City Store Inventory Address Staff Rental Customer 3NF Source schema: Sakila Rentals

35 Who? Staff Customer When? What? Film Date Fact: Rentals Time Store Where? Target Star Schema

36 Select Business Process Sales, Purchase, Storage,... Define Facts and Key Metrics Facts: Key Event in Business Process Metrics (Fact Attributes): Count or Amount Choose Dimensions and Hierarchies What? When? Where? Who? Why? Dimensional Design

37 Select Business Process Rentals Identify Facts Count (number of rentals) Rental Duration Choose Dimensions What: Films Who: Customer, Staff When: Rental, Return Where: Store Example Business Process: Rentals

38 Inventory Staff Customer Rental A star is born: Rentals 3NF

39 Category Film Category Store Film Inventory Staff Customer Rental A star is born: Rentals 3NF

40 Category Film Category Store Film Inventory Staff Customer Rental A star is born: Denormalize

41 Store Film Category Store Staff Customer Rental A star is born: Denormalize

42 Address Store Film Category Store Staff Rental A star is born Customer

43 Address Store Film Category Store Staff Customer Rental A star is born: Denormalize

44 Language Film Category Address Store Address Store Staff Customer Rental A star is born: Denormalize

45 City City Language Film Category Address Store Address Store Staff Customer Rental A star is born: Denormalize

46 Country Country City City Language Film Category Address Store Address Store Staff Customer Rental A star is born: Rental Snowflake

47 What: Film Where: Store Who: Staff Who: Customer Language Film Country Store Country City Address Staff City Address Category Store Customer Rental A star is born: Rental Star Schema

48 Something is missing... Who? (Customer, Staff) What? (Film) Where? (Store)...? Dimensional Design

49 What: Film Where: Store Who: Staff Who: Customer Rental When: Date When: Time A star is born: Rental Date and Time

50 What: Film Where: Store Who: Staff Who: Customer Rental When: Rental Date When: Rental Time When: Return Date When: Return Time Role Playing: Date/Time for both Rentals and Returns

51

52 Rental Star Schema

53 Part IV: Filling the Data Warehouse Starring Sakila

54 Sources Back-end Front-end BI Applications Data Warehouse ERP Meta Data Staging Area CRM Datamarts External Data Extract Transform Load Reporting Analysis Visualizations Dashboards Data Mining Present High Level Data Warehouse Architecture

55 Physical Design Source to Target Mapping Define how data in the data warehouse is derived from data in the source system(s) Specification for designing the ETL process Column-level mapping Source system, schema, table, column, data type Target dimension/fact, column, defaults Transformation rules, cleansing, lookup, calculation Planning the ETL Process

56 Staging? Changed Data Capture / Extraction Denormalization Derived data / Enrichment Cleansing / Conforming History policy (dimensions) Granularity Dimension Lookup (facts) Designing the ETL Process

57 Flow ETL Engine Transformations Jobs Data flow and processing Workflow of ETL tasks Tools Spoon Kitchen Pan Designing ETL with Kettle

58 Load Dimension Tables Load Fact table Loading a Fact Table

59 Get Customers source data Lookup Address (Denormalize) Update Dimension Loading a Dimension Table

60 Loading a Fact Table

61 Part V: Presenting the Data: BI Applications Starring Sakila

62 months, years: Data mining Should we become an appliance vendor instead of delivering software solutions Executives Strategic OLAP/Analysis weeks, months: In what region should we open a new store? Managers Tactical days, weeks: Who's available for tomorrow's shift Analysts Customers Partners Reporting Employees Operational Business Intelligence Scope

63 Reporting

64 Reporting Mostly Operational Lists and Grouping Typically standardized Typically no or limited interactivity Subreporting

65 months, years: Data mining Should we become an appliance vendor instead of delivering software solutions Executives Strategic OLAP/Analysis weeks, months: In what region should we open a new store? Managers Tactical days, weeks: Who's available for tomorrow's shift Analysts Customers Partners Reporting Employees Operational Scope of Reporting

66 Reporting

67 Analysis

68 Analysis Tactical, Strategic OLAP Online Analytical Processing Pivot tables Typically Interactive Slice and Dice Drilldown Typically Ad-hoc

69 months, years: Data mining Should we become an appliance vendor instead of delivering software solutions Executives Strategic OLAP/Analysis weeks, months: In what region should we open a new store? Managers Analysts Tactical days, weeks: Who's available for tomorrow's shift Customers Partners Reporting Employees Operational Scope of OLAP & Analysis

70 Analysis Interactive Pivot table

71 Data Mining

72 Data Mining Strategic, Tactical Discover hidden patterns in data Machine learning Statistic analysis Typically not interactive, long running Expert matter Not readily consumable by end-users Characteristics of back-end processing

73 months, years: Data mining Should we become an appliance vendor instead of delivering software solutions Executives Strategic OLAP/Analysis weeks, months: In what region should we open a new store? Managers Tactical days, weeks: Who's available for tomorrow's shift Analysts Customers Partners Reporting Employees Operational Scope of Data Mining

74 Data Mining

75 Charts and Graphs

76 Charts and Graphs Operational, Tactical, Strategic Summarize large dataset Not a separate class but a presentation Data Visualization Standardized or ad-hoc Can be interactive Drive a subreport Drive drilldown

77 Dashboarding

78 Dashboarding Operational, Tactical, Strategic Not a separate class but a presentation Bundle: key metrics for a particular role or perspective different views on the same metrics Can contain reports, pivot tables, charts, graphs Typically interactive

79 Dashboard