Data Quality Management and Financial Services Loretta O Connor Data Quality Sales Manager Data Quality Divion May 2007 1 PG 961
Content Introduction Defining the Data Quality Problem Solutions for Data Quality Issues Data Quality Reporting Dashboards Data Quality Methodology Successfully Implementing a Data Quality Strategy Customer Examples Demo Q&A PG 962 2
Data Quality: Problem Definition 3 PG 963
Problem statement: Proceedings of the MIT 2007 Information Quality Industry Symposium Poor Data Quality causes numerous business problems TDWI 2006 PG 964 4
Initiatives Driving Data Quality Industry / Business Driver D a t a Q u a l i t y CDI, Master Data Management (All) Radio Frequency Identification (Manufacturing, CPG) Rk Management (Financial) Electronic availability of all services (Government) Regulatory Compliance Basel II Sarbanes Oxley (SOX) Anti-Money Laundering (AML) Internal Drivers Data Warehouse / BI Data Migrations - Mergers and Acquitions Application Consolidation PG 965 5
The Impact Problems Applications crash Angry business people call the operations team Ops track down the problems Problems with the accuracy of the information being reported Fixes being made without audit Impact Applications unavailable Time consuming to trace and fix Unhappy business people Incorrect results Rk concerns Regulatory concerns Root Causes Data didn t arrive Data entry errors Loose rules on source systems Data constency errors File column changes Corrupted Contributory Factors Unclear / fragmented process Problem / ownership - Rk operations - Data providers Multitudes of Log files PG 966 Source: Kevin Allen: Information Quality for Rk Management 6
The Vicious Money Circle Source Systems Branches Subsidary 3 rd Party Agent Network Finance Business Mainframe Load takes too long and th increasing our exposure $ $ $ $ Load Data $ $ $ check $ it $ $ $ Analyze + Rework $ Can t trust the, so we must manually Load Target Applications AML Engine Rk Engine Reports General Ledger CRM SAP Th vicious circle We creates are non-compliant high costs which and cannot we be anticipated know the regulator will see th PG 967 7
Data Quality: The Solution 8 PG 968
Exting fixes IT Operations Unix Scripts Application monitoring Log file analys Manual updates to files to make it work Business Financial Institutions develop entire ecosystems to compensate for poor quality MS Access checks run by business Manual updates to files to make it work Same changes, every week! All Ad Hoc All Manual Expensive to Manage Unreliable And management wonder why the annual IT budget keeps getting bigger? PG 969 Source: Kevin Allen: Information Quality for Rk Management 9
DQM Approach & Methodology ANALYZE ALIGN CLEANSE SUSTAIN Data Quality not a one off exerce! Organizations must not only align and cleanse, but MUST also keep clean over time 1. Identify & Measure Data Quality 1. Content Profiling 2. Scorecarding Analyze Enhance 5. Monitor Data Quality Versus Targets 2. Define Data Quality Rules & Targets 3. Align: e.g. Standardization, removing noe, align product attributes, measures, classification. 4. Implement Quality Improvement Processes 3. Design Quality Improvement Processes 4. Cleanse / Address duplicates 5. Re-Scorecard/Monitor PG 970 10
Data Quality Dimensions Data Exploration Column Column Profiling Profiling Relationship Relationship Redundancy Redundancy the the s s physical physical charactertics charactertics?? Across Across multiple multiple tables? tables? relationships relationships ext ext in in the the set? set? Across Across multiple multiple tables? tables? redundant? redundant? Orphan Orphan Analys Analys Completeness Completeness msing msing or or unusable? unusable? Conformity Conformity stored stored in in a a non-standard non-standard format? format? Data Quality Constency Constency Accuracy Accuracy gives gives conflicting conflicting information? information? incorrect incorrect or or out out of of date? date? Duplication Duplication records records are are duplicated? duplicated? Integrity Integrity msing msing important important relationship relationship linkages? linkages? Range Range scores, scores, values, values, calculations calculations are are outside outside of of range? range? PG 971 11
Sample DQ Issues Duplication: Fuzzy matching Completeness: Conformity: Msing Incorrect Key Values Format Constency: Data Constency: in correct format and complete, Incorrect but breaks Format a business rule Range: Identify outliers Accuracy: Integrity: Using Relationship reference Identification to validate COMPLETENESS CONFORMITY CONSISTENCY DUPLICATION INTEGRITY ACCURACY RANGE PG 972 12
Data Quality Maturity Model Managed Aware Reactive Proactive Attitude DQ seen as a cost DQ for IT DQ driven by business DI and DQ seen as key enabler Tech. Hand coded Silos of DQ Linked projects Fully integrated DQ initiative Benefit Few Few tactical Key tactical gains Strategic PG 973 Drivers depend on where you are and where you want to go 13
Sample Financial Services Business Intelligence Dashboards 14 PG 974
IDQ: Data Accuracy Scorecard PG 975 15
3 rd Party Reporting using IDQ PG 976 16
Methodology 17 PG 977
Scorecarding Back to Source Proceedings of the MIT 2007 Information Quality Industry Symposium 1 2 3 Plan and Scope DQ Approach DQ Plan Plan & Approach Data Acquition Define Data Priorte Key Data Lt Design & Configure Business Rules Quality Criteria 6 5 Data Improvement Data Updates & Data Cleansing New processes Data Validation Communicate Results Assessment Cleansing Guidelines Root Cause Assessment Set Improvement Targets Baseline 4 Business Rules Analyse & Baseline Profile Data Build Scorecard PG 978 18
Customers 19 PG 979
Master Data Management Improve Enable business user to build quality monitoring rules Provide standard platform that could be extended for further quality initiatives Challenge How We Helped Business Value Problems managing trade promotions because of poor quality Data migrations put at rk because of quality sues Ability to monitor and cleanse all types of product, customer and business Flexibility to manage and control different quality problems on one platform Data quality improvement leads to more streamlined supply chain Faster more successful migrations and systems consolidation PG 980 20
ref123 Third Largest Bank in the US KEY BUSINESS IMPERATIVE Informatica In Action IT/BUSINESS INITIATIVE: Regulatory Reporting Regulatory Compliance Compliance with anti-money laundering regulations Provide robust DQ reporting and metrics system for AML Unit DATA QUALITY INITIATIVE: DQ Reporting & Monitoring THE CHALLENGE Enable AML team to build, manage and customize AML business rules Track and monitor quality across key systems INFORMATICA ADVANTAGE Data quality workbench for business users Scorecard aggregating quality metrics from multiple systems RESULTS/BENEFITS Avoided regulatory penalties of up $20m Implemented AML DQ Monitoring ahead of deadline using exting AML team resources Saved estimated $3m+ cost of bespoke of AML solution PG 981 21
Reuters: Global CRM management Key Business Requirements: Fix quality within exting Siebel systems Approach: Provide quality metrics to drive improvement processes Implement one off and ongoing quality processes Challenge Lack of ROI on Siebel due to low quality Poor client management Inaccurate mailing processes Inefficient marketing processes The manual generation of monthly quality reports very inefficient. Solution Informatica Data Quality To implement an automated Data Quality Scorecard per country To implement one off and then ongoing cleansing and standardization Informatica Data Explorer To profile new sources Expected Results Increase in sales force and marketing efficiency Recogned Data Quality metrics process in place PG 982 22