ECCMA Presentation The Golden Record and Survivorship. Bud Walker Director of Data Quality Solutions

Transcription

1 ECCMA Presentation The Golden Record and Survivorship Bud Walker Director of Data Quality Solutions Joseph Vertido MD-MVP Channel Manager / Data Quality Analyst 1

2 Data Quality Problems REPORTING ANALYTICS DASHBOARD DATA WAREHOUSE 2

3 Data Quality Problems Introduction of Inconsistencies Without any form of Data Quality control, allowing data to come in as-is introduces problems. 3

4 Data Quality Problems Standardization Format and Ordering Misspelling Misplacement Casing Validity Accuracy Stale Data 4

5 The Data Quality Cycle GOLDEN RECORD AND SURVIVORSHIP 5

6 Data Quality Problems REPORTING ANALYTICS DASHBOARD DATA WAREHOUSE 6

7 Data Quality Problems Issues with Consolidation Duplicates Fuzzy Duplicates Domain Specific Fuzzy Duplicates DATA WAREHOUSE 7

8 Data Quality Problems Issues with Consolidation Golden Record and Survivorship DATA WAREHOUSE 8

9 The Golden Record / Survivorship What is the Golden Record / Survivorship? The Golden Record or as many refer to as the Survivor Record, is the single accurate representation of the truth Why is it important? The implementation of Survivorship techniques can be long, complicated and tedious. It is important that a business come to an agreement on what Survivorship Criteria to implement. Failure to successfully implement Survivorship Schemas become detrimental to the efficiency and quality of the deduplication process. Ultimately results in loss of information and inaccurate Business Intelligence 9

10 The Golden Record Billing Joseph Vertido 123 Main St, Los Angeles CA Customer Service Joseph Vertido (800) Web Account Joseph Vertido 123 Main St,

11 The Golden Record Billing Joseph Vertido 123 Main St, Los Angeles CA Customer Service Joseph Vertido (800) Web Account Joseph Vertido 123 Main St,

12 The Golden Record Common/Generic Practices Most Reliable Source Web Account Joseph Vertido 123 Main St, Billing Joseph Vertido 123 Main St, Los Angeles CA Customer Service Joseph Vertido (800)

13 The Golden Record Common/Generic Practices Most Recently Created or Updated Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido Empresa Rancho Santa Margarita CA (800) /7/2008 JOSEPH Vertido Avenida empresa Rancho Santa Margarita CA Vertido, Joseph Avenida empressa **

14 The Golden Record Common/Generic Practices Most Complete Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido Empresa Rancho Santa Margarita CA (800) /7/2008 JOSEPH Vertido Avenida empresa Rancho Santa Margarita CA Vertido, Joseph Avenida empressa **

15 The Golden Record Common/Generic Practices Longest String Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido Empresa Rancho Santa Margarita CA (800) /7/2008 JOSEPH Vertido Avenida empresa Rancho Santa Margarita CA Vertido, Joseph Avenida empressa **

16 The Golden Record Common/Generic Practices Longest String 10/25/2012 Joseph Vertido Empresa Rancho Santa Margarita CA (800) /7/2008 JOSEPH Vertido Avenida empresa Rancho Santa Margarita CA Vertido, Joseph Avenida empressa **

17 The Golden Record Common/Generic Practices Most Frequent Occurrence Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido Empresa Rancho Santa Margarita CA (800) /7/2008 JOSEPH Vertido Avenida empresa Rancho Santa Margarita CA Vertido, Joseph Avenida empressa Vertido, Joseph Avenida empressa

18 Survivorship 1) Understand Your Business Rules 2) Determine and Implement Column Level Rules Based Survivorship 18

19 Survivorship Column Level Survivorship Most Frequent Occurrence ID Name Address City State Zip Phone C-2119 Joseph Vertido Empresa Rancho Santa Margarita CA (800) JOSEPH Vertido Avenida empresa Rancho Santa Margarita CA Vertido, Joseph Avenida empressa Vertido, Joseph Avenida empressa Determine Surviving Records Not Null or Blank Most Frequent Occurence 19

20 Survivorship Column Level Survivorship ID Name Address City State Zip Phone C-2119 Joseph Vertido Empresa Rancho Santa Margarita CA (800) JOSEPH Vertido Avenida empresa Rancho Santa Margarita CA Vertido, Joseph Avenida empressa C-2119 Vertido, Joseph Avenida empressa Rancho Santa Margarita CA Determine Surviving Records 20

21 Survivorship The Problem ID Name Address City State Zip Phone Date C-2119 Joseph Vertido Empresa Rancho Santa Margarita CA C-2119 Joe Vertido AAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAA AA AAAA AAAAAAAAA AAAAAAA C-2119 Vertido, J 123 Fake St /15/2012 C-2119 Vertido Avenida empressa /24/2009 Generic Most Complete Recent Rule Based Record Survivorship Only Goes So Far The Answer: Reference Data 21

22 Knowledge Centric Survivorship Validate critical information and other sensitive information in our data such as: Social Security, Address, Phone, and Name Having the knowledge of which data are good and bad changes the perspective for generating Survivorship Rules. Reference Data is not limited to third party data. One can also create their own knowledgebase (eg. Product Codes). 22

23 Knowledge Centric Survivorship Most Reliable Frequent Source Date Stamp Name Address City State Zip Phone 10/25/2008 Joseph Vertido 123 Fake St Fake City CA Most Recent Frequent Date Stamp Name Address City State Zip Phone 3/7/2012 Joseph Vertido 123 Fake St Fake City CA (800) Valid Address Valid Phone Date Stamp Name Address City State Zip Phone Joseph Vertido Avenida Empressa Rancho Santa Margarita CA a 23

24 Knowledge Centric Survivorship Date Stamp Name Address City State Zip Phone 10/29/2012 Joseph Vertido Avenida Empressa Rancho Santa Margarita CA (800)

25 Establishing Relationships ID Name Address City State Zip Phone C-2119 Joseph Vertido AvenidaEmpresa Rancho Santa Margarita CA C-2119 Joseph Vertido 2205 La Palma Ave Anaheim CA

26 Summary Survivorship is just as important as any process in the spectrum of data quality. Generic Survivorship < Column Based Survivorship < Knowledge Centric Survivorship Reference Data is your ally! 26

27 Thank You Bud Walker Director of Data Quality Solutions Joseph Vertido Data Quality Analyst