IMPROVEMENT THE PRACTITIONER'S GUIDE TO DATA QUALITY DAVID LOSHIN

Transcription

1 i I I I THE PRACTITIONER'S GUIDE TO DATA QUALITY IMPROVEMENT DAVID LOSHIN ELSEVIER AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO Morgan Kaufmann Publishers is an Imprint of Elsevier M<

2 CONTENTS Foreword Preface Acknowledgments About the Author xi xiii xxi xxiii Chapter 1 Business Impacts of Poor Data Quality Information Value and Data Quality Improvement Business Expectations and Data Quality Qualifying Impacts Some Examples More on Impact Classification Business Impact Analysis Additional Impact Categories Impact Taxonomies and Iterative Refinement Summary: Translating impact into Performance 16 Chapter 2 The Organizational Data Quality Program The Virtuous Cycle of Data Quality Data Quality Processes Stakeholders and Participants Data Quality Tools Summary 34 Chapter 3 Data Quality Maturity The Data Quality Strategy A Data Quality Framework A Data Quality Capability/Maturity Model Mapping Framework Components to the Maturity Model Summary 49 Chapter 4 Enterprise Initiative Integration Planning Initiatives Framework Initiatives Operational and Application Initiatives 62 v

3 VI CONTENTS 4.4 Scoping Issues Summary 66 Chapter 5 Developing Quality Road Map 67 A Business Case and A Data 5.1 Return on the Data Quality Investment Developing the Business Case Finding the Business Impacts Researching Costs Correlating Impacts and Causes The Impact Matrix Problems, Issues, Causes Mapping Impacts to Data Flaws Estimating the Value Gap Prioritizing Actions The Data Quality Road Map Practical Steps for Developing the Road Map Accountability, Responsibility, and Management The Life Cycle of the Data Quality Program Summary 90 Chapter 6 Metrics and Performance Improvement Performance-Oriented Data Quality Developing Data Quality Metrics Measurement and Key Data Quality Performance Indicators Statistical Process Control Control Charts Kinds of Control Charts Interpreting Control Charts Finding Special Causes Maintaining Control Summary 112 Chapter 7 Data Governance The Enterprise Data Quality Forum The Data Quality Charter 116

4 CONTENTS Vii 7.3 Mission and Guiding Principles Roles and Responsibilities Operational Structure Data Stewardship Data Quality Validation and Certification Issues and Resolution Data Governance and Federated Communities Summary 128 Chapter 8 Dimensions of Data Quality What Are Dimensions of Data Quality? Categorization of Dimensions Describing Data Quality Dimensions Intrinsic Dimensions Contextual Qualitative Dimensions Finding Your Own Dimensions Summary 146 Chapter 9 Data Requirements Analysis Business Uses of Information and Business Analytics Business Drivers and Data Dependencies What Is Data Requirements Analysis? The Data Requirements Analysis Process Defining Data Quality Rules Summary 164 Chapter 10 Metadata and Data Standards Challenges Data Standards Metadata Management Business Metadata Reference Metadata Data Elements Business Metadata A Process for Data Harmonization Summary 189

5 Tangible Viii CONTENTS Chapter 11 Data Quality Assessment Planning Business Process Evaluation Preparation and Data Analysis Data Profiling and Analysis Synthesis of Analysis Results Review with Business Client Summary Rapid Data Assessment - Results 206 Chapter 12 Remediation and Improvement Planning Triage The Information Flow Map Root Cause Analysis Remediation Execution Summary 218 Chapter 13 Data Quality Service Level Agreements Business Drivers and Success Criteria Identifying Data Quality Rules Establishing Data Quality Control The Data Quality Service Level Agreement Inspection and Monitoring Data Quality Metrics and a Data Quality Scorecard Data Quality Incident Reporting and Tracking Automating the Collection of Metrics Reporting the Scorecard Taking Action for Remediation Summary - Managing Using the Data Quality Scorecard Chapter 14 Data Profiling Application Contexts for Data Profiling Data Profiling: Algorithmic Techniques Data Reverse Engineering Analyzing Anomalies Data Quality Rule Discovery Metadata Compliance and Data Model Integrity 254

6 CONTENTS ix 14.7 Coordinating the Participants Selecting a Data Set for Analysis Summary 259 Chapter 15 Parsing and Standardization Data Error Paradigms The Role of Metadata Tokens: Units of Meaning Parsing Standardization Defining Rules and Recommending Transformations The Proactive versus Reactive Paradox Integrating Data Transformations into the Application Framework Summary 277 Chapter 16 Entity Identity Resolution The Lure of Data Correction The Dual Challenge of Unique Identity What Is an Entity? Identifying Attributes Similarity Analysis and the Matching Process Matching Algorithms False Positives, False Negatives, and Thresholding Survivorship Monitoring Linkage and Survivorship Entity Search and Match and Computational Complexity Applications of Identity Resolution Evaluating Business Needs Summary 296 Chapter 17 Inspection, Monitoring, Auditing, and Tracking The Data Quality Service Level Agreement Revisited Instituting Inspection and Monitoring: Technology and Process Data Quality Business Rules Automating Inspection and Monitoring 307

7 X CONTENTS 17.5 Incident Reporting, Notifications, and Issue Management Putting It Together 312 Chapter 18 Data Enhancement The Value of Enhancement Approaches to Data Enhancement Examples of Data Enhancement Enhancement through Standardization Enhancement through Context Enhancement through Data Merging Summary: Qualifying Data Sources for Enhancement 324 Chapter 19 Master Data Management and Data Quality What Is Master Data? What Is Master Data Management? "Golden Record" or "Unified View"? Master Data Management as a Tool MDM: A High-Level Component Approach Master Data Usage Scenarios Master Data Management Architectures Identifying Master Data Master Data Services Summary: Approaching MDM and Data Quality 349 Chapter 20 Bringing It All Together Organization and Management Building the Information Quality Program Techniques and Tools Summary 383 Index 385