Data Warehouse Design

Transcription

1 Data Warehouse Design Modern Principles and Methodologies Matteo Golfarelli Stefano Rizzi Translated by Claudio Pagliarani Mc Grauu Hill New York Chicago San Francisco Lisbon London Madrid Mexico City Milan New Delhi San Juan Seoul Singapore Sydney Toronto

2 Contents Acknowledgments Foreword Preface xiii xv xvii 1 Introduction to Data Warehousing 1.1 Decision Support Systems 1.2 Data Warehousing 1.3 Data Warehouse Architectures Single-Layer Architecture Two-Layer Architecture Three-Layer Architecture An Additional Architecture Classification 1.4 Data Staging and ETL Extraction Cleansing Transformation Loading 1.5 Multidimensional Model Restriction Aggregation 1.6 Meta-data 1.7 Accessing Data Warehouses Reports OLAP Dashboards 1.8 ROLAP, MOLAP, and HOLAP 1.9 Additional Issues Quality Security Evolution Data Warehouse System Lifecycle 2.1 Risk Factors 2.2 Тор-Down vs. Bottom-Up Business Dimensional Lifecycle Rapid Warehousing Methodology 2.3 Data Mart Design Phases Analysis and Reconciliation of Data Sources Requirement Analysis vii

3 Data Warehouse Design: Modern Principles and Methodologies Conceptual Design Workload Refinement and Validation of Conceptual Schemata Logical Design Physical Design Data-Staging Design Methodological Framework Scenario 1: Data-Driven Approach Scenario 2: Requirement-Driven Approach Scenario 3: Mixed Approach Testing Data Marts 58 3 Analysis and Reconciliation of Data Sources Inspecting and Normalizing Schemata The Integration Problem Different Perspectives Equivalent Modeling Constructs Incompatible Specifications Common Concepts Interrelated Concepts Integration Phases Preintegration Schema Comparison Schema Alignment Merging and Restructuring Schemata Defining Mappings 77 4 User Requirement Analysis Interviews Glossary-based Requirement Analysis Facts Preliminary Workload Goal-oriented Requirement Analysis Introduction to Tropos Organizational Modeling Decision-making Modeling Additional Requirements 97 5 Conceptual Modeling The Dimensional Fact Model: Basic Concepts Advanced Modeling Descriptive Attributes Cross-Dimensional Attributes Ill Convergence Shared Hierarchies Multiple Arcs Optional Arcs 115

4 Contents jx Incomplete Hierarchies Recursive Hierarchies Additivity Events and Aggregation Aggregating Additive Measures Aggregating Non-additive Measures Aggregating with Convergence and Cross-dimensional Attributes Aggregating with Optional or Multiple Arcs Empty Fact Schema Aggregation Aggregating with Functional Dependencies among Dimensions Aggregating along Incomplete or Recursive Hierarchies Time Transactional vs. Snapshot Schemata Late Updates Dynamic Hierarchies Overlapping Fact Schemata Formalizing the Dimensional Fact Model Metamodel Intensional Properties Extensional Properties Conceptual Design Entity-Relationship Schema-based Design Defining Facts Building Attribute Trees Pruning and Grafting Attribute Trees One-to-One Relationships Defining Dimensions Time Dimensions Defining Measures Generating Fact Schemata Relational Schema-based Design Defining Facts Building Attribute Trees Other Phases XML Schema-based Design Modeling XML Associations Preliminary Phases Selecting Facts and Building Attribute Trees Mixed-approach Design Mapping Requirements Building Fact Schemata Refining Requirement-driven Approach Design 196

5 X Data Warehouse Design: Modern Principles and Methodologies 7 Workload and Data Volume Workload Dimensional Expressions and Queries on Fact Schemata Drill-Across Queries Composite Queries Nested GPSJ Queries Validating a Workload in a Conceptual Schema Workload and Users Data Volumes Logical Modeling MOLAP and HOLAP Systems The Problem of Sparsity ROLAP Systems Star Schema Snowflake Schema Views Relational Schemata with Aggregate Data Temporal Scenarios Dynamic Hierarchies: Type Dynamic Hierarchies: Type Dynamic Hierarchies: Type Dynamic Hierarchies: Full Data Logging Deleting Tuples Logical Design From Fact Schemata to Star Schemata Descriptive Attributes Cross-dimensional Attributes Shared Hierarchies Multiple Arcs Optional Arcs Incomplete Hierarchies Recursive Hierarchies Degenerate Dimensions Additivity Issues Using Snowflake Schemata View Materialization Using Views to Answer Queries Problem Formalization A Materialization Algorithm View Fragmentation Vertical View Fragmentation Horizontal View Fragmentation 272

6 Contents xi 10 Data-staging Design Populating Reconciled Databases Extracting Data Transforming Data Loading Data Cleansing Data Dictionary-based Techniques Approximate Merging Ad-hoc Techniques Populating Dimension Tables Identifying the Data to Load Replacing Keys Populating Fact Tables Populating Materialized Views Indexes for the Data Warehouse B + -Tree Indexes Bitmap Indexes Bitmap Indexes vs. B + -Trees Advanced Bitmap Indexes Projection Indexes Join and Star Indexes Multi-join Indexes Spatial Indexes Join Algorithms Nested Loop Sort-merge Hash Join Physical Design Optimizers Rule-based Optimizers Cost-based Optimizers Histograms Index Selection Indexing Dimension Tables Indexing Fact Tables Additional Physical Design Elements Splitting a Database Into Tablespaces Allocating Data Files Disk Block Size Data Warehouse Project Documentation Data Warehouse Level Data Warehouse Schemata Deployment Schema 354

7 XU Data Warehouse Design: Modern Principles and Methodologies 13.2 Data Mart Level Bus and Overlapping Matrices Operational Schema Data-Staging Schema Domain Glossary Workload and Users Logical Schema and Physical Schema Testing Documents Fact Level Fact Schemata Attribute and Measure Glossaries Methodological Guidelines A Case Study Application Domain Planning the TranSport Data Warehouse The Sales Data Mart Data Source Analysis and Reconciliation User Requirement Analysis Conceptual Design Logical Design Data-Staging Design Physical Design The Marketing Data Mart Business Intelligence: Beyond the Data Warehouse Introduction to Business Intelligence Data Mining Association Rules Clustering Classifiers and Decision Trees Time Series What-If Analysis Inductive Techniques Deductive Techniques Methodological Notes Business Performance Management 417 Glossary 423 Bibliography 429 Index 445