Proven Strategies for Data Governance Master Data Management and Data Quality. Wayne Pullam Product Specialist

Transcription

1 Proven Strategies for Data Governance Master Data Management and Data Quality Wayne Pullam Product Specialist 1

2 2

3 Business Drivers Drive IT Initiatives Key to Success is a Focus on Quality Master Data Key Business Drivers Revenue Growth Operational Efficiency Regulatory Compliance Key Strategic Imperatives Improve Decisions Modernize Business Improve Efficiency & Reduce Costs Mergers Acquisitions & Divestitures Effectively Segment Customer Base Build & Extend Product Portfolios Governance Risk Compliance Increase Partner Network Efficiency Increase Business Agility IT Initiatives Enterprise Applications Data Warehousing MDM BI SaaS Data Governance 3

4 The Challenge Deliver Trusted Data to the Business IT Initiatives Enterprise Applications Data Warehousing MDM BI SaaS Data Governance Data Complexity Sustaining Data Quality Lack of Data Governance No Single View Business IT Data Sources Internal/External Transactional/Analytical/Social 4

5 According to a recent survey of over 800 companies, when asked Has your organization suffered problems due to poor quality master data?, an alarming 83% of respondents said yes. -- The Data Warehouse Institute 5

6 What is Data Governance? Aligning People, Processes, & Technology LAW Establish Legislative Powers, Processes, And Policy-Making ORDER Codify, Monitor, And Enforce Data Governance Policies Across Key Information Assets PEOPLE PROCESSES TECHNOLOGY 6

7 Why is this so hard? Barriers to Data Governance 7

8 8

9 Lack of Quality Foundation Lack of a Quality Foundation Organizations don t know where to start with data governance efforts and lack the tools for ongoing tracking against quality goals Impact: Focus on the wrong problem Ongoing churn and false starts Data Profiling was cited as THE reason why they have a Data Governance council today. With data profiling, we were able to identify the data issues that mattered most and build the case for data governance across the business. Business Analyst State Department of Licensing 9

10 Data Quality Process Role-Based Data Quality Process Easily & Quickly Identify Data Anomalies 1. Profile & Discover Uncover Hidden Details and Relationships Analyst & Data Steward Enable Business Users and Reduce Reliance on IT 5. Monitor DQ Metrics 2. Define & Design Metrics and Rules Increase Productivity and Data Accuracy Business Manager Expand Data Quality Awareness Support Data Governance 4. Manage Exceptions Provide Timely, Trustworthy Data to All Stakeholders Reduce Risk and Exposure 3. Design & Implement Rules and Routines Developer & Architect 10

11 11

12 Lack of Business Buy-In Lack of Business Buy-In Data Governance programs struggle for acceptance with the business or fail outright due to lack of attention to data quality issues Impact: Lack of business support and sponsorship Continued IT dependency for resolving DQ issues Informatica is helping KPN raise the bar in terms of customer satisfaction, target customers with compelling cross-sell and up-sell opportunities, and reduce marketing and sales time to market. Thomas Reichel Sr. IT Architect KPN 12

13 Building the Business Case Financial What impact does bad data have on missing expectations associated with costs, financial management, and revenues? Business Value Productivity How does bad data impact our operational efficiency? Risk How does bad data impact our ability to manage compliance risks, financial risks, or our ability to execute in the marketplace? Trust How does data impact to meet expectations in the marketplace or satisfying our internal ability to execute against strategy? 13

14 14

15 Lack of Business and IT Alignment Lack of Business and IT Alignment IT and business look to each other to resolve data issues, with neither willing to step up and take ownership Impact: Repeated attempts to fix the same issue Business not able to participate Our data quality and governance project has been viewed as a huge success and is getting visibility by the senior executives all the way up to the company s Senior VP of Supply Chain and CFO. Barbara Latulippe enterprise data architect Smith & Nephew 15

16 Bridging the Business and IT Gap Understand the Impact of Bad Data on the Business Data Impact Individual address data has an information quality decay rate of 17% per year in the U.S. Business Impact Unable to effectively attract and retain customers The cost of bad data for an individual business can be put at 15% to 20% of revenues 64% of all data migration projects were delivered late and 37% were overbudget 2/3 of CIOs say they lack a single view of legacy systems for compliance reporting Process inefficiencies and increased productivity costs Ineffectively manage mergers and acquisitions Failure to mitigate governance, risk and compliance pressures 16

17 Master Data Management and Data Quality Why you need both 17

18 Recommended Paths to Success Data Discovery and Profiling Find hidden data problems wherever they are, enabling organizations to discover and analyze all anomalies across all data sources Pervasive Data Quality Reusable rules and tools in one development environment for managing data quality across applications, domains and projects Integrated DQ and MDM Speed up the specification and review cycle between IT and the business via collaboration tools and a common project environment 18

19 Examples of bad DQ and bad MDM 19

20 Data Quality Because poor data can be worse than no data 20

21 Frequent Requirements Data Analysis Parsing Address Matching Monitoring & & Discovery and Validation De-duplication & Standardization Reporting And do this for all data types 21

22 Data Analysis and Discovery Identify patterns, formats, Be able to identify patterns, formats, schema, and data and quality data issues quality Drill issues down into actual data Create rules by example as you profile the data Drill down into actual data Create rules by example as data is profiled 22

23 Using Informatica Analyst to Profile Your Data 100% browser-based Drill-through analysis Data Steward Right-click to create data quality scorecards Increase productivity and efficiency by enabling the business to proactively take responsibility for data quality and reduce their reliance on IT. 23

24 Parsing & Standardization Transform and parse data to multiple fields Correct completeness, conformity, and consistency problems Standardize field formats 24

25 Parsing & Standardization ContactName Phone Judy Dent // Bob s Assistant FirstName MiddleName LastName Title Phone Judy Dent Bob s Assistant (415)

26 Address Validation Validate and correct addresses for over 240 countries in one environment Have continuously maintained reference data from international postal agencies 26

27 Address Validation (before and after) Address1 Address2 Address3 Address4 Address KATY FRWY SUITE 333 HOUSTEN TX Street City County StateCode StateName ZIP ZIP4 Latitude Longitude 7887 Katy Freeway Suite 333 Houston Harris TX Texas Valid addresses keep costs down and helps ensure compliance 27

28 Match and De-Duplicate All Data Types Highly accurate matching requires consideration of multiple attributes using multiple rule sets Use confidence levels to automate life cycle processes Consider over 60 cultural variations for name matching Match data in spite of poor quality 28

29 The Problem Error and variation Identity data is subject to considerable error and variation More than any other type of data Can never be completely avoided New names invented every day Simple data cleansing is not a complete answer Raw data has inherent value for matching Some data cannot be cleaned Databases about people & organisations often contain international data Harder to find and match Basic search techniques frequently miss matches or find too much Examples Mark Douglas or Douglas Mark 117 Jacson, Room 2a or 2a Jackson Rd Browne Brown Johnson, Jhonson Hannah, Hamah Mary Anne, Maryanne Easthartford, Hartford East, Hartford Chris Christine, Christopher, Tina Full stops, dashes, slashes, titles, apostrophes Not Known; Unknown; Deceased; XXXXXX, No Middle Name; ; Wlm/William, Mfg/Manufacturing IBM/International Business Machines Fedex/Federal Express Credit Suisse First Bost Chruschtschow, Chroesjtsjov, Krusciov Khader AL Ghamdi, Khadir A. AlGamdey 29

30 Monitoring Quality Stakeholders need to be aware Current quality metrics Alerts if quality thresholds are not being met Delivery of reports and alerts must be web based 30

31 Monitoring the quality Easy-to-share browser-based scorecards for line of business managers Browser-based scorecards enabling you to: View and share data quality scorecards Drilldown to the actual records Line of business manager Take action to reduce the business impact Zero learning curve for business users to review and track data quality metrics, enabling data quality for the masses. 31

32 Master Data Management Single view of the truth 32

33 How Does MDM fit Inside the IT Landscape? Data Integration Applications Master Data Management Deliver Timely, Trusted, Relevant Master Data Operational Applications RECOGNIZE RESOLVE RELATE CIF Legacy Legacy Systems Third Party Data Luis Valdez CRM RECOGNIZE Data Integration Louis Valdes ERP Luis Valdez CRM Data Profiling RESOLVE Louis Valdes ERP Luis Valdez Spouse Owns Data Quality Julia Smith Product RELATE Master Data Foundation Deliver Timely, Trusted, Relevant Data Data Integration Data Integration Analytical Portal/ Dashboard Legacy Systems Data Marts Business Data Intelligence Warehouse 33

34 Need: Recognize Similar Data Luis Valdez RECOGNIZE Louis Valdes Louis R. Valdez Work: 124 Main Street NO New York, NY Invoice: Web Company: ABC Corp. Amt: $ Newark, NJ Contact: YES Work: Creation: Louis Valdes Work: 124 Main Street New York, NY Company: ABC Corp. Newark, NJ ERP: ERP111 CRM: AA333 WEB: DB CRM: NO AA333 Value: High Contact: YES Work: DOB : 5/17/1970 lvaldez@abccompany.com Julia Smith CUSTOMER IDENTIFIERS: ERP: ERP111 WEB: 3456 Home: 5412 Crater Street Order: 4572 Queens, NY Date: Home: Item: jsmith@gmail.com Qty: 1 Amt: $ Ship: No Louis R. Valdez WEB: DB2222 Luis Luis Raul Raul Valdez Home: Louis 5412 Valdes Crater Ave Order: 2790 Queens, NY Date: Home: Item: Work: ABC Company Qty: 2 Newark, NJ Amt: $ luisvaldes@abc.com Ship: No XREF XREF Identify candidates for matches across multiple systems based on configurable rules Determine match, non-match based on user defined thresholds Queue questionable matches for manual review RECOGNIZE the customer (or other entity) accurately across multiple systems 34

35 Need: Resolve Data Conflicts Luis Valdez RESOLVE Louis Valdes CUSTOMER ERP: ERP111 Louis R. Valdez IDENTIFIERS: ERP: ERP111 WEB: 3456 Julia Smith Work: 124 Main Street NO WEB: DB2222 Home: 5412 Crater Street Order: 4572 New York, NY Invoice: Web CRM: AA333 Queens, NY Date: Company: ABC Corp. Amt: $ Home: Item: Newark, NJ Contact: Luis YES Valdez Qty: 1 Work: Creation: W: 124 Main Street Amt: $ lvaldez@abc.com New York, NY Work: Ship: No CRM ERP CRM: AA333 H: 5412 Crater Ave Queens, NY WEB: DB2222 Louis Valdes Work: 124 Main Street Home: Luis Raul Valdez lvaldez@abccompany.com NO Home: 5412 Crater Ave Order: 2790 New York, NY Value: High Queens, NY Date: Amt: $ Value: High Company: ABC Corp. Contact: YES Home: Item: Creation: DOB: Newark, NJ Invoice History Last Work: Contact ABC Company Qty: 2 Work: DOB - date/details : 5/17/ Newark, NJ Amt: $ lvaldez@abccompany.com - date/details Call Records luisvaldes@abc.com Ship: No - date/details XREF Luis Valdez As necessary, matched records are merged to a golden record The most reliable attributes are selected to survive, at the cell level, based on configurable rules All history and lineage of changes is kept RESOLVE data conflicts among the systems into a trusted source of truth 35

36 Need: Relate Data Across Applications CUSTOMER Luis Valdez RELATE Spouse Employed by Julia Smith ABC Corp IDENTIFIERS: ERP: ERP111 Louis R. Valdez ERP: ERP111 Work: 124 Main WEB: Street DB NO New York, NY CRM: AA333 Invoice: Web Company: ABC Corp. Amt: $ Newark, NJ Luis Valdez Contact: YES Work: Creation: W: 124 Main Street New York, NY Work: H: CRM: 5412 Crater AA333 Ave Queens, NY Louis Valdes Home: Work: 124 Main Street NO New York, NY Value: High Value: High Company: Amt: $ ABC Corp. Contact: YES DOB: Newark, Creation: NJ Last Contact Work: Invoice History DOB : 5/17/ date/details lvaldez@abccompany.com Call Records - date/details - date/details Julia Smith WEB: 3456 Home: 5412 Crater Street Order: 4572 Queens, NY Date: Home: Item: jsmith@gmail.com Qty: 1 Amt: $ Ship: No WEB: DB2222 HOUSEHOLDED w/ CRM AA333 Luis Raul Valdez Home: Julia 5412 Smith Crater Ave Value: Order: Med 2790 Queens, NY Since Date: H: 5412 Crater Ave DOB: Home: Queens, NY Item: Last Contact Home: Work: ABC Company Qty: 2 Newark, NJ Call Amt: Records $ luisvaldes@abc.com - date/details Ship: No XREF Luis Valdez Bring together relationships from different applications and systems Create and manage a rich set of multiple hierarchies for different purposes Maintain history and lineage of relationship changes RELATE people, locations, products and organizations from different sources 36

37 37

38 Shameless Plug Informatica Data Quality Source: Gartner (October 2013) 38

39 39

40 4 40