42% of companies blame multiple databases for their data quality issues. 75% of companies waste an average of 14% of revenue due to bad data quality

Transcription

1 Better BI through Master ment 42% of companies blame multiple databases for their data quality issues Jeremy Pritchard Photo Credit : NASA Photo Credit : Tim Dobbelaere Experian Survey 75% of companies waste an average of 14% of revenue due to bad data quality Master ment improves business performance Photo Credit : Howard Lake Experian Survey Photo Credit : AP What is Master? What is Master ment? Master data is data that is shared by multiple computer systems. The Information Difference Master data is the consistent and uniform set of identifiers and extended attributes that describes the core entities of the enterprise including customers, prospects, citizens, suppliers, sites, hierarchies and chart of accounts. Gartner Master data is information that is key to the operation of a business persistent, non-transactional data that defines a business entity for which there is, or should be, an agreed-upon view across the organisation. Wikipedia Master data is often one of the key assets of a company. Microsoft Master data management is a technology-enabled discipline in which business and IT work together to ensure the uniformity, accuracy, stewardship, semantic consistency and accountability of the enterprise s official shared master data assets. The creation of: Gartner Master ment comprises a set of processes, governance, policies, standards and tools that consistently defines and manages the master data. The Golden Record Single Version of the Truth Wikipedia Information ers 1

2 Types of data in an organisation Understanding Master Think of nouns and verbs Bob Smith buys a widget (SKU #A1234) and ships it to his home address Unstructured Transactional Metadata Hierarchical Master The master data elements are the nouns and are people, things, and places The transactional data elements are verbs that describe what happens to those people, places, and things. CRM Marketing ERP WMS Financial The What, Why, and How of Master ment Microsoft November 2006 Deciding what Master should be d ment Concept Cardinality Volatility Lifetime Value Reuse ment Concept It embodies: quality management policies Business process management Risk management People governance is a quality control discipline for: Assessing Managing Using Improving Monitoring Maintaining Protecting organisational information Process Technology Information ers 2

3 ment Concept - Batch Real Time ment Concept Profiling Monitoring Profiling Basic Analysis Patterns / Masking Extremes Quantities Frequency Analysis Monitoring Advanced Profiling Custom analysis of data Defined by user and relevant to data context Output is Binary (true/false) Indicators Information ers 3

4 - Monitoring ment Concept Portal DQ plan Profiling/DQIs Reports Cleansing Scoring Cleansing Parsing Validation Parsing parsed into components (pattern based) E.G. Jim Smith -> Jim + Smith Validation Validation of against rules Validation of against reference tables ment Adding data Standardisation Transformation into standard format (16-Feb-75 > 16/02/1975) Standard and nonstandard abbreviations (Str. -> Street) Language-specific replacements Cleansing Parsing Validation ment ment Standardisation Standardisation Before and After Cleansing Issue Resolution ANNE CHRISTINE HALL IAN SCOTT Name Gender Name PHILLIPS JOHN IAN SCOTT SMITH JOHN SMITH out_first_name John Ian F N Male Date of Birth 14/11/ /12/ Oct /01/1971 out_last_name Smith Scott Telephone Address Gender Line 1 phillips87@live.co.uk BOOTON Male COURT JS@hotmail.com out_gender 22 RINGMORE M christine.hall@gmail.com ian@@dfgmail.-.com 6 56C HORNCHURCH ROAD STREET 56 WOULD LANE Address Line 2 Address Date Line 3 of Birth 10/01/ Oct.1956 out_birthdate 10/01/ /10/1956 Address Line 4 Address Telephone Line 5 KIDDERMINSTER PORCESTERSHIRE PLYMUTH DEVON out_telephone LONDON ISLEWORTH MIDDLESEX Postcode DY102YZ PL5 2TF JS@hotmail.com ian@@dfgmail.-.com SE233DE out_ TW7-5ED JS@hotmail.com Score Explanation Address Line 1 0 ADDRESS_VALID RINGMORE WOULD LANE STREET GENDER_TAKEN_FROM_NAME 300 out_address_line_1 GENDER_TAKEN_FROM_NAME RINGMORE WOOD _INV LANE RISE Address Line 2 ADDRESS_CORRECTED_MINOR ADDRESS_CORRECTED_MINOR out_address_line_2 DATE_STANDARDIZED GENDER_STANDARDIZED Address Line 3 out_address_line_3 TELEPHONE_STANDARDIZED ADDRESS_CORRECTED_MAJOR Address Line 4 ISLEWORTH out_address_line_4 out_first_name out_last_name Address Line 5 LONDON MIDDLESEX John out_post_town Smith Ian LONDON ISLEWORTH Scott Anne Christine Phillips Hall out_gender out_birthdate Postcode 14/11/1987 SE233DE TW7-5ED M out_postcode 10/01/1971 M SE23 TW7 5ED 3DE 28/10/1956 F F 10/12/1940 out_telephone out_ phillips87@live.co.uk christine.hall@gmail.com js@hotmail.com out_address_line_1 6 BOOTON COURT Score 56C HORNCHURCH ROAD RINGMORE RISE 56 WOOD LANE out_address_line_2 out_address_line_3 _INV GENDER_TAKEN_FROM_NAME Explanation DATE_STANDARDIZED out_address_line_4 out_post_town KIDDERMINSTER PLYMOUTH ADDRESS_CORRECTED_MINOR LONDON ISLEWORTH out_postcode DY10 Explanation 2YZ PL5 2TF GENDER_STANDARDIZED SE23 3DE TW7 5ED TELEPHONE_STANDARDIZED ADDRESS_CORRECTED_MAJOR Is the score lower than the threshold? No Yes Information ers 4

5 - Issue ment ment Concept Portal DQ plan Profiling/DQIs Reports Issue data Issue base Issue List Workflow Exception Mgt Master ment Master ment Architectures Name: Bob Smith Tel: DOB: Gender: M Name: B Smith Tel: DOB: 23/10/71 Gender: M Name: Bob Smith Tel: DOB: 23/10/71 Gender: M Name: Bob Smith Tel: DOB: 23/10/71 Gender: Consolidated Master is Single Version of Truth at Master Updates occur at Sources Updates propagated to Master Coexistence + Master is Single Version of Truth is ongoing Updates occur at Sources or Master Updates propagated to other Sources Name: Bob Smith Tel: DOB: Gender: Male Name: B Smith Tel: (0) DOB: 23-Oct-71 Gender: M Name: Smith, Bob Tel: (01283)56982 DOB: 23/10/1971 Gender: Registry + + Multiple Versions of Truth is ongoing Updates occur at Sources Centralised + Master is Single Version of Truth at Master CRM Marketing ERP WMS Financial Keys and Metadata in Registry Updates optionally propagated to other Sources Updates occur at Master Updates propagated to Sources ing How many people are here? & Goal: Identify groups of records that in reality represent a single client or entity. Cleansed data John Smith M 16/12/ Ringmore Rise, London, SE23 3DE John Smith M /12/ Ringmore Rise, London, SE23 3DE John Smith M Arnold Street, Boldon Colliery, Bolton, NE35 9BD Smith M 16/11/1978 John Smith M /11/ Catalina Avenue, Pembroke Dock, SA72 6YB John Smith M 16/11/ Catalina Avenue, Pembroke Dock, SA72 6YB John Smiht M /11/1978 Jane Watson F Catalina Avenue, Pembroke Dock, SA72 6YB Jane Watson F /01/ Catalina Avenue, Pembroke Dock, SA72 6YB Jane Smith F /01/1982 J. Smith Information ers 5

6 Merging Cleansed data John Smith M 16/12/ Ringmore Rise, London, SE23 3DE John Smith M /12/ Ringmore Rise, London, SE23 3DE John Smith M Arnold Street, Boldon Colliery, Bolton, NE35 9BD Smith M 16/11/1978 John Smith M /11/ Catalina Avenue, Pembroke Dock, SA72 6YB John Smith M 16/11/ Catalina Avenue, Pembroke Dock, SA72 6YB John Smiht M /11/1978 Jane Watson F Catalina Avenue, Pembroke Dock, SA72 6YB Jane Watson F /01/ Catalina Avenue, Pembroke Dock, SA72 6YB Jane Smith F /01/1982 J. Smith & Creating the Golden Record Can cherry pick the best fields or even the best record For example: The one from the reference system The newest one The one of highest quality Cleansed data John Smith M 16/12/ Ringmore Rise, London, SE23 3DE John Smith M /12/ Ringmore Rise, London, SE23 3DE John Smith M Arnold Street, Boldon Colliery, Bolton, NE35 9BD Smith M 16/11/1978 John Smith M /11/ Catalina Avenue, Pembroke Dock, SA72 6YB John Smith M 16/11/ Catalina Avenue, Pembroke Dock, SA72 6YB John Smiht M /11/1978 Jane Watson F Catalina Avenue, Pembroke Dock, SA72 6YB Jane Watson F /01/ Catalina Avenue, Pembroke Dock, SA72 6YB Jane Smith F /01/1982 J. Smith Cleansed data John Smith M 16/12/ Ringmore Rise, London, SE23 3DE John Smith M /12/ Ringmore Rise, London, SE23 3DE John Smith M Arnold Street, Boldon Colliery, Bolton, NE35 9BD Golden record The The newest most permanent frequent address ment Concept Information ers 6

7 Intelligence Operational vs. Analytical Master ment Operational vs. Analytical Master ment Operational MDM Operational Operational data is fundamental to the running of an organisation Global Cruise Company Customer held in multiple systems Booking Legacy CRM Marketing Ship Operational MDM centres on assuring single view of master data in the core systems used by business users Sales, service, order management, manufacturing, purchasing, billing, accounts receivable, accounts payable, payroll, etc. Rely heavily on integration technologies to keep systems in sync Customer data can be entered in all systems (except Marketing) No real-time checking of existing customer in other systems No proper link between customers in systems Business performance is impacted: Customer service reps having to use two systems (legacy and CRM) to deal with customers Passport information captured in Ship System but not shared for future use Problems with loyalty scheme Duplicates fed into Marketing system customers being marketed to multiple times Operational vs. Analytical Master ment Analytical MDM Analytical Analytical data is used to support a company's decision making Global accountancy training company Sell to individuals not companies Analytical MDM centres on assuring single view of master data in the downstream data warehouse used most often to supply the data for a business intelligence (BI) solution for historical and predictive analysis Have no idea how much companies are spending with them Would like to be able to build stronger relationships with organisations Any data cleansing done inside an Analytical MDM solution is invisible to the transactional applications Information ers 7

8 Master ment - Value Across the Enterprise Operational Analytical Single Version of Truth = Better System synchronisation Consistency in transactional data Party/product data across all systems System integration/migration Cost reduction within the business process aggregation & analysis Marketing segmentation & analysis Risk management Financial reporting Cost reduction and time savings in analysis The Reality Maximum business value comes from managing both operational and analytical master data Adoption and Plans for MDM Reported Success of MDM Programs Information Difference 2013 Information Difference 2013 The Right Way to Implement Your MDM Program Executive Sponsor Business Case Scope Plan for change Photo Credit : photoeverywhere Information ers 8