ECCMA Presentation The Golden Record and Survivorship Bud Walker Director of Data Quality Solutions Joseph Vertido MD-MVP Channel Manager / Data Quality Analyst 1
Data Quality Problems REPORTING ANALYTICS DASHBOARD DATA WAREHOUSE 2
Data Quality Problems Introduction of Inconsistencies Without any form of Data Quality control, allowing data to come in as-is introduces problems. 3
Data Quality Problems Standardization Format and Ordering Misspelling Misplacement Casing Validity Accuracy Stale Data 4
The Data Quality Cycle GOLDEN RECORD AND SURVIVORSHIP 5
Data Quality Problems REPORTING ANALYTICS DASHBOARD DATA WAREHOUSE 6
Data Quality Problems Issues with Consolidation Duplicates Fuzzy Duplicates Domain Specific Fuzzy Duplicates DATA WAREHOUSE 7
Data Quality Problems Issues with Consolidation Golden Record and Survivorship DATA WAREHOUSE 8
The Golden Record / Survivorship What is the Golden Record / Survivorship? The Golden Record or as many refer to as the Survivor Record, is the single accurate representation of the truth Why is it important? The implementation of Survivorship techniques can be long, complicated and tedious. It is important that a business come to an agreement on what Survivorship Criteria to implement. Failure to successfully implement Survivorship Schemas become detrimental to the efficiency and quality of the deduplication process. Ultimately results in loss of information and inaccurate Business Intelligence 9
The Golden Record Billing Joseph Vertido 123 Main St, Los Angeles CA 92688 Customer Service Joseph Vertido (800) 800 6245 Web Account Joseph Vertido 123 Main St, 92688 10
The Golden Record Billing Joseph Vertido 123 Main St, Los Angeles CA 92688 Customer Service Joseph Vertido (800) 800 6245 Web Account Joseph Vertido 123 Main St, 92688 11
The Golden Record Common/Generic Practices Most Reliable Source Web Account Joseph Vertido 123 Main St, 92688 Billing Joseph Vertido 123 Main St, Los Angeles CA 92688 Customer Service Joseph Vertido (800) 800 6245 12
The Golden Record Common/Generic Practices Most Recently Created or Updated Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA (800) 800-6245 3/7/2008 JOSEPH Vertido 22382 Avenida empresa Rancho Santa Margarita CA 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 **8008006245 13
The Golden Record Common/Generic Practices Most Complete Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA (800) 800-6245 3/7/2008 JOSEPH Vertido 22382 Avenida empresa Rancho Santa Margarita CA 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 **8008006245 14
The Golden Record Common/Generic Practices Longest String Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA (800) 800-6245 3/7/2008 JOSEPH Vertido 22382 Avenida empresa Rancho Santa Margarita CA 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 **8008006245 15
The Golden Record Common/Generic Practices Longest String 10/25/2012 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA (800) 800-6245 3/7/2008 JOSEPH Vertido 22382 Avenida empresa Rancho Santa Margarita CA 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 **8008006245 16
The Golden Record Common/Generic Practices Most Frequent Occurrence Date Stamp Name Address City State Zip Phone 10/25/2012 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA (800) 800-6245 3/7/2008 JOSEPH Vertido 22382 Avenida empresa Rancho Santa Margarita CA 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 8008006245 17
Survivorship 1) Understand Your Business Rules 2) Determine and Implement Column Level Rules Based Survivorship 18
Survivorship Column Level Survivorship Most Frequent Occurrence ID Name Address City State Zip Phone C-2119 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA (800) 800-6245 JOSEPH Vertido 22382 Avenida empresa Rancho Santa Margarita CA 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 8008006245 Determine Surviving Records Not Null or Blank Most Frequent Occurence 19
Survivorship Column Level Survivorship ID Name Address City State Zip Phone C-2119 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA (800) 800-6245 JOSEPH Vertido 22382 Avenida empresa Rancho Santa Margarita CA 92688 8008006245 Vertido, Joseph 22382 Avenida empressa 92688 8008006245 C-2119 Vertido, Joseph 22382 Avenida empressa Rancho Santa Margarita CA 92688 8008006245 Determine Surviving Records 20
Survivorship The Problem ID Name Address City State Zip Phone Date C-2119 Joseph Vertido 22382 Empresa Rancho Santa Margarita CA 9495895200 C-2119 Joe Vertido AAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAA AA AAAA AAAAAAAAA AAAAAAA C-2119 Vertido, J 123 Fake St. 11111 9495895200 6/15/2012 C-2119 Vertido 22382 Avenida empressa 92688 8008006245 1/24/2009 Generic Most Complete Recent Rule Based Record Survivorship Only Goes So Far The Answer: Reference Data 21
Knowledge Centric Survivorship Validate critical information and other sensitive information in our data such as: Social Security, Address, Phone, Email and Name Having the knowledge of which data are good and bad changes the perspective for generating Survivorship Rules. Reference Data is not limited to third party data. One can also create their own knowledgebase (eg. Product Codes). 22
Knowledge Centric Survivorship Most Reliable Frequent Source Date Stamp Name Address City State Zip Phone 10/25/2008 Joseph Vertido 123 Fake St Fake City CA 11111 Most Recent Frequent Date Stamp Name Address City State Zip Phone 3/7/2012 Joseph Vertido 123 Fake St Fake City CA 11111 (800) 800-6245 Valid Address Valid Phone Date Stamp Name Address City State Zip Phone Joseph Vertido 22382 Avenida Empressa Rancho Santa Margarita CA 92688 123123123a 23
Knowledge Centric Survivorship Date Stamp Name Address City State Zip Phone 10/29/2012 Joseph Vertido 22382 Avenida Empressa Rancho Santa Margarita CA 92688 (800) 800-6245 24
Establishing Relationships ID Name Address City State Zip Phone C-2119 Joseph Vertido 22382 AvenidaEmpresa Rancho Santa Margarita CA 92688 8008006245 C-2119 Joseph Vertido 2205 La Palma Ave Anaheim CA 92806 8008006245 25
Summary Survivorship is just as important as any process in the spectrum of data quality. Generic Survivorship < Column Based Survivorship < Knowledge Centric Survivorship Reference Data is your ally! 26
Thank You Bud Walker Director of Data Quality Solutions bud@melissadata.com Joseph Vertido Data Quality Analyst joseph@melissadata.com