The data warehousing architecture

Transcription

1 Data Warehousing Erik Perjons, DSV, SU/KTH 1 The data warehousing architecture The back room The front room External sources Operational DBs/ OLTPs/TPSs Extract Transform Load Refresh Serve Analysis/OLAP Productt Time1 Value1 Value11 Product2 Time2 Value2 Value21 Product3 Time3 Value3 Value31 Product4 Time4 Value4 Value41 Query/Reporting Data mining Falö aöldf flaöd aklöd falö alksdf Data sources Legacy systems OLTP/TP systems Back end tools Data staging area The data warehouse Presentation (OLAP) servers Front end tools End user applications Business Intelligence tools 2

2 Why a data warehouse? DB DBMS End user application A DB DB DBMS DBMS End user application B DB DBMS End user application C 3 A data warehousing definition Chaudhiri & Dayal: Data warehousing is a collection of decision support technologies, aimed at enabling the knowledge worker (executive, manager, analyst) to make better and faster decisions. [Chaundhuri&Dayal, 1997] A data warehouse is a decision support system (DSS) according to Kimball [Kimball, 1998] 4

3 Typical DSS/DW queries Which customer groups are most profitable? How much is the total sale by month for each sales office? Are some sales offices failing to sell some otherwise popular products? Is there a correlation between promotion campaigns and sales growth? 5 DW architechture: Source systems Analysis/OLAP Productt Time1 Value1 Value11 External sources Extract Transform Load Serve Product2 Time2 Value2 Value21 Product3 Time3 Value3 Value31 Product4 Time4 Value4 Value41 Query/Reporting Operational DBs/ OLTPs/TPSs Data mining Falö aöldf flaöd aklöd falö alksdf Source systems Data staging area Presentation servers End user applications 6

4 Source systems - characteristics The source data often in OLTP (Online Transaction Processing) systems, also called TPS (Transaction Processing Systems) Operational DBs/ OLTPs/TPSs high level of transaction throughput and are already occupied by the normal operations of the organisation maintain little historical data a OLTP system may be reliable and consistent, but there are often inconsistencies between different OLTP systems, for example they have different types of data format, data structures and semantics in different OLTP systems OLTP vs. DSS (Decision Support Systems). They have different functions. DW is a DSS. 7 DW architechture: Data staging area Analysis/OLAP Productt Time1 Value1 Value11 External sources Extract Transform Load Serve Product2 Time2 Value2 Value21 Product3 Time3 Value3 Value31 Product4 Time4 Value4 Value41 Query/Reporting Operational DBs/ OLTPs/TPSs Data mining Falö aöldf flaöd aklöd falö alksdf Source systems Data staging area Presentation servers End user applications 8

5 The data staging area Often the most complex part in the architecture, and involves... Extract Transform Load Extraction (E) Transformation (T) Load (L) indexing ETL-tools can be used offered by several vendors Scripts for extraction, transformation and load are implemented 9 Data staging area - extraction Extract Transform Load Extraction - copying the data needed for the data warehouse into the staging area for further manipulation, i.e. transformation usually done once every night 10

6 Data staging area - transformations Extract Transform Load Transformation involves data conversion (convert to a common data format by specifying transformation rules) data migration (convert to a common semantics/terminology by specifying transformation rules, replace the string gender by sex ) data cleaning data scrubbing (use domain-specific knowledge (e.g postal adresses) to check the data) data auditing (discover suspicious pattern, discover violation o f stated rules) data aggregation data enrichment (add extern data) 11 DW architechture: Presentation servers Analysis/OLAP Productt Time1 Value1 Value11 External sources Extract Transform Load Serve Product2 Time2 Value2 Value21 Product3 Time3 Value3 Value31 Product4 Time4 Value4 Value41 Query/Reporting Operational DBs/ OLTPs/TPSs Data mining Falö aöldf flaöd aklöd falö alksdf Source systems Data staging area Presentation servers End user applications 12

7 Presentation/OLAP servers OLAP servers What is OLAP? Star-join schemas/dimensional modelling Data Marts ROLAP/MOLAP servers 13 What is OLAP? Acronym for On-line analytical processing A decision support system (DSS) that support ad-hoc querying, i.e. enables managers and analysts to interactively manipulate data. The idea is to allow the users to easy and quickly manipulate and visualise the data through multidimensional views, i.e. different perspectives. Important concepts: ad-hoc querying, interactivity, multidimensional views 14

8 What is OLAP? An OLAP session takes place online in real time, with rapid responses to manager s or analyst s queries, so that analytical or decision-making process is unsturbed [O Brien, 2002] An OLAP system from 1970s: Express, today part of Oracle 9i. Wellknown OLAP system today: Hyperion Essbase, Cognos PowerPlay, IBM DB2 OLAP Server, Microsofts Analysis service, (Note that an OLAP server often replace the Database server in DW) Multidimensional view of data is the foundation for OLAP 15 Spreadsheet output of OLAP tool product product group mounth quarter office region Views/dimensions Column headers (join constraints) Column header (application constraint) Product Group Region First Quarter Group A ABC 1245 Group A XYZ Group B ABC Group B XYZ Answer set representing focal event Views/dimensions Facts/measurses Row headers Drag and drop the views/dimensions in the spreadsheet and read the the facts/measures 16

9 The cube: multidimensional view of the data A popular conceptual model that influenced front end tools (OLAP clients) database and DBMS design (dimensional modelling, ROLAP and MOLAP servers) The data cube: The cube: multidimensional view of the data The data cube: Spreadsheets: office 130 office quarter quarter service service - Numeric measures/facts (e.g. number of, sum, total sales) depends on a set of dimensions 18

10 Multidimensional view of the data promotion campaign office quarter office service office office quarter quarter quarter service product product customer group 19 Multidimensional view of the data Service Quarter Measures/facts Office Customer group Promotion campaign 20

11 Dimensional modelling - Star-join schema Service used - service name - service group 1 0..* Telephone calls - sum ($) - number of calls 0..* Time - date - month - quarter - year 1 0..* 0..* Sales Dimension - seller name - office 1 1 Customer - customer name - address - region - income group 21 Dimensional modelling - Star-join schema Service Dimension Key Service Service group S1 Local call Group A S2 Intern. call Group A S3 SMS Group B S4 WAP Group C 1 0..* Sales Dimension Key Seller Office F11 Anders C Sundsvall F12 Lisa B Sundsvall F13 Janis B Kista Fact table - Transactions Number Sum of calls C210 S1 F :00 3 C210 S3 F :00 1 C212 S2 F :00 1 C213 S1 F :00 1 C214 S4 F : * 0..* 1 Time Dimension Date/ Key Month Quarter Year * Customer Dimension Key Customer Address Region Income group C210 Anna N Stockholm Stockholm B C211 Lars S Malmö Skåne B C212 Erik P Rättvik Dalarna C C213 Danny B Stockholm Stockholm A C214 Åsa S Stockholm Stockholm A 22

12 Dimensional modelling - Star-join schema Service Dimension Key Service Service group S1 Local call Group A S2 Intern. call Group A S3 SMS Group B S4 WAP Group C Sales Dimension Key Seller Office F11 Anders C Sundsvall F12 Lisa B Sundsvall F13 Janis B Kista Fact table - Transactions Number Sum of calls C210 S1 F :00 3 C210 S3 F :00 1 C212 S2 F :00 1 C213 S1 F :00 1 C214 S4 F :00 1 Time Dimension Date/ Key Month Quarter Year S=37:00 Customer Dimension Query: For how much did customers in Sthlm use service Local call in october 1999? Key Customer Address Region Income group C210 Anna N Stockholm Stockholm B C211 Lars S Malmö Skåne B C212 Erik P Rättvik Dalarna C C213 Danny B Stockholm Stockholm A C214 Åsa S Stockholm Stockholm A 23 Snow-flake schema Year Service used - service name Time - date Month Service group Telephone calls - sum ($) - number of calls Region Quarter Sales Dimension - seller name Customer - customer name - address Income group Office 24

13 3 NF modelling vs. Dimensional modelling Key difference is the degree of normalisation 3 NF modelling - a logical design technique to eliminate data redundancy to keep consistency and storage efficiency, and makes transaction simple and deterministic - ER models for operational DBs are usually complex, e.g. they often have hundreds, or even thousands, of entities/tables Dimensional modelling - a logical design technique that present data in a intuitive, i.e. easier to navigate for the user - allow high performance access/queries by eliminating joins (denormalisation) - strategy to handling aggregates and make use of bit vector indexes [Kimball et al, 1998] 25 departemental subsets (e.g. marketing data mart) far smaller data volumes, fewer data sources allows a piecemeal approach to some of the enormous integration problems involved in creating an enterprise wide data warehouse 26

14 Dependent vs. Independent Independent Dependent 27 Service Quarter Calls Service Quarter Office Office Subscription orders Service Quarter Calls Office Subscription orders 28

15 The presentation/olap servers OLAP servers standard relational database management system (RDBMS) extended relational database management system (RDBMS), called Relational OLAP (ROLAP) servers multidimensional database management system (MDBMS), called Multidimensional OLAP (MOLAP) servers hybrid of ROLAP and MOLAP, called Hybrid OLAP (HOLAP) 29 The presentation/olap servers Extended Relational DBMS (ROLAP servers) data stored in RDB star-join schemas support SQL extensions index structures OLAP servers Multidimensional DBMS (MOLAP servers) data stored in arrays (n-dimensional array) direct access to array data structure excellent indexing properties poor storage utilisation, especially when the data is sparse. 30

16 More about presentation servers What is characteristics regarding data warehouse, according to Chaudhiri&Dayal : SQL extensions (operators like Cube, Crossjoin) [see slide number 30] Index structures (bit map indexes, join indexes) [see slide number 30] Materialised views (pre-aggregations) - [see slide number 32] 31 Aggregations Aggregations can be created on-the-fly (see slide number 22-23) or by the process of pre-aggregation, i.e. store the aggregations in separate dimensions and fact tables 32

17 DW architechture: End user applications Monitoring & Administration External sources Operational DBs Extract Transform Load Refresh Metadata repository OLAP servers Serve Analysis Productt Time1 Value1 Value11 Product2 Time2 Value2 Value21 Product3 Time3 Value3 Value31 Product4 Time4 Value4 Value41 Query/Reporting Data mining Falö aöldf flaöd aklöd falö alksdf Source systems Data staging area Presentation servers End user applications 33 End user applications Analysis Productt Time1 Value1 Value11 Product2 Time2 Value2 Value21 Product3 Time3 Value3 Value31 Product4 Time4 Value4 Value41 OLAP tools, BI apps, DSS Query/Reporting tools Data mining Query/Reporting Data mining Falö aöldf flaöd aklöd falö alksdf 34

18 Spreadsheet output of OLAP tool product product group mounth quarter office region Column headers (join constraints) Column header (application constraint) Answer set representing focal event Product Group Region First Quarter Group A ABC 1245 Group A XYZ Group B ABC Group B XYZ Row headers 35 Graphical output of OLAP tool 36

19 Problems of Data Warehousing Complexity of integration Hidden problems with source systems Data homogenisation Underestimation of resources for data loading Required data not captured High maintenance Long duration projects Why not integrating the legacy applications (OLTP systems) instead? 37