A JOURNEY TO TRUSTED DATA RICK ANDREWS NOVEMBER 2013

Transcription

1 TELSTRA TEMPLATE 4X3 BLUE BETA TELPPTV4 A JOURNEY TO TRUSTED DATA RICK ANDREWS NOVEMBER 2013

2 All customer data is collected, used, disclosed and secured in accordance with applicable law Rick Andrews A Journey to Trusted Data Telstra Unrestricted Final ENTERPRISE DATA WAREHOUSE THE INFORMATION FACTORY 1 BUSINESS EVENTS occur & raw data is created 2 3 Subassemblies of INFORMATION may be combined PROCESSES to extract, transform and load data are applied 4 FINISHED INFORMATION PRODUCTS delivered to enable decision making CORPORATE DASHBOARD SELF SERVE ANALYTICS

3 PERCEPTIONS OF DATA QUALITY Rick Andrews A Journey to Trusted Data Telstra Unrestricted Final

4 DATA DIGITAL LIFEBLOOD OF AN ORGANISATION if data is not MANAGED then it can become a RISKY LIABILITY rather than a VALUABLE ASSET

5 AN OLD MANAGEMENT ADAGE THAT STILL APPLIES TO DATA You can t MANAGE what you don t MEASURE

6 THE SOLUTION A DATA QUALITY FIREWALL PURPOSE To automatically and routinely monitor data in the Enterprise Data Warehouse and to raise alerts when data quality violations are detected ANTICIPATED RESULT Improve the COMMUNICATION OF DATA HEALTH to all EDW stakeholders Business view Available Complete Technical view What When Error-free Scale

7 DATA QUALITY FIREWALL FRAMEWORK OVERVIEW Data Sources Enterprise Data Warehouse BI Platform Source System Source System Source System E T L ESA Files E T L Atomic Data Store E T L Data Marts Data Marts D A T A D E L I V E R Y BI Tools Reports / Dashboards Info Delivery DATA QUALITY FIREWALL Testing Metadata Management Data Standards & Technical Integrity Data Governance & Data Stewardship

8 HOW IT WORKS THE SIMPLE VERSION Schedule Outputs Items To Be Tested DQ Rules Engine EDW

9 DIMENSIONS OF DATA QUALITY THE CRITICAL FEW AVAILABLE COMPLETE ERROR-FREE CURRENCY TIMELINESS COMPLETENESS INTEGRITY UNIQUENESS VALIDITY Time elapsed since real world entity was recorded or business event occurred Data is sufficiently up-to-date for the task at hand Data is not missing and is of sufficient breadth and depth for its specified use Correct references to related entities (Referential Integrity, Entity Integrity) Entities exist only once within a data set Conformance to business rules for the entity

10 COMPARING ACTUAL TO EXPECTED Simple Medium Complex Single Table Analysis Multi Table Analysis BR Multi Table Analysis

11 Targeted Builds Selected small scale builds based on user requirements BUILD APPROACH Tier 0 Source to Staging Tier 1 Staging Tier 2 ETL Tier 3 ADS Tier 4 Access Layer Completeness Consistency Currency Custom Integrity Late Landing Completeness Timeliness Validity Volume Mass Builds Systematic large scale builds based on technical authoritative points of truth Rick Andrews A Journey to Trusted Data Telstra Unrestricted Final

12 AN EXAMPLE OLD VOLUME FORECASTING APPROACH Rick Andrews A Journey to Trusted Data Telstra Unrestricted Final

13 AN EXAMPLE OLD VOLUME FORECASTING APPROACH Wide thresholds masking issues

14 AN EXAMPLE NEW VOLUME FORECASTING APPROACH 85% Confidence Limits

15 DRIVE IMPROVEMENTS IN EDW PHYSICAL DATA MODEL EDW PDM Data Quality Firewall EDW Analysis of violations

16 PROVIDE DEVELOPMENT TEAMS WITH INSIGHT INTO DATA QUALITY ISSUES Initiate Discovery Build Test Deploy Production Data Quality Firewall

17 TOP 8 OUTCOMES Detecting and alerting on data quality issues Provide development teams with insight into data quality issues Supporting data quality improvement activities Providing evidence that data quality issues have been resolved Drive improvements in EDW Physical Data Model Assisting in prioritisation of remediation activities Identified improvement opportunities in operational processes Drive compliance to standards

18 THE DATA QUALITY FIREWALL HAS enabled us to develop a WINDOW INTO THE QUALITY OF DATA within our Enterprise Data Warehouse This improved visibility enables us to affect user s TRUST in the data and better MANAGE our data

19