Data Quality Through Curation at Big Data Scale Andy Palmer, CEO & Co-Founder, Tamr, Inc. MIT CDOIQ Symposium, July 23 & 24, 2014

Transcription

1 Data Quality Through uration at Big Data Scale Andy Palmer, EO & o-founder, Tamr, Inc. MIT DOIQ Symposium, July 23 & 24, 2014

2 Top 3 Issues for Data Quality in Enterprise 1. Always start with questions and context 2. Always start with questions and context 3. Always start with questions and context Data 1,000 Questions Structured Prescriptive Internal External Agile/Rapid Dynamic Reproducible Scalable Predictive Descriptive Unstructured

3 Three Flavors of Analytical ontext Prescriptive Healthcare providers use automated data analysis to predict patient issues and intervene with remediation plans Predictive Banks use multiple sources of background data to improve lending decisions Descriptive Retailers use data variety to create a more complete view of their customer s buying decisions to improve online offers & increase sales Solid foundation of connected, complete, and high-quality data

4 Big Data Opportunity in the Enterprise On average Large Enterprises have 2,000+ data sources Of those, on average, <10% of data sources are getting into master data, data warehouses, data marts, and ETL Many organizations purchase the same data for the same vendors once even 3-4 times! Need to optimize analytic return on data investment - both purchased and produced Individual Analytics may require <2-3 sources but need to ensure that it s the correct sources - not just the familiar sources

5 The Data Source Problem The centralization of systems (SAP, Oracle, etc.) has failed to achieve a unified view of enterprise data Silos are a reality of every business Internally- vs. Externally-generated data entralized vs. Distributed organizational structures and pendulum of change over decades Use of systems to reinforce organizational models Natural to end up with radical heterogeneity What is a source? Database? Instance? Table? Spreadsheet? JSON object? Each source contains attributes, records and content Structured, unstructured, and semi-structured

6 Evolution of Enterprise Data Quality and uration Structured Info service providers embrace practices of modern public internet search companies Extreme top-down and deterministic approach to improving context and analytical use Internal Application of probabilistic and collaborative approaches to curate and leverage structured and unstructured enterprise data sources External Extreme bottom-up probabilistic approach to improving context and use of content Use of more unstructured and semi-structured sources in enterprise Unstructured

7 Evolution of Enterprise Data Quality and uration Structured Info service providers embrace practices of modern public internet search companies External Internal Use of more unstructured and semi-structured sources in enterprise Unstructured

8 Bottom-up Probabilistic and ollaborative uration: Only Viable Way to Tame Enterprise Data Variety Back to the Future 1990 s web: Probabilistic search and website connection - unstructured public data 2010 s enterprise: Bottom-up probabilistic data source connection and curation of structured and semi-structured data

9 Data Quality and uration in the Enterprise OLD NEW 1. Master data management, ETL, data warehouses, and marts 1. Probabilistic, authoritative Data uration 2. Small 10 s of data architects and curators inside of IT 2. Hundreds/thousands of stakeholders from across enterprises engaged in collaborative data curation 3. Highly engineered Deterministic ETL into warehouses 3. Bottom-up probabilistic mapping and matching complemented with impact from many data experts 4. Pre-defined, static reporting 4. Agile analytics with radical contextual diversity and rich visualizations 5. Need to know information access controls 5. Privacy through control of information use 6. Bimodal - either highly scalable and reproducible or ad-hoc 6. Reproducibility of analytics and pipelines for all users

10 New Approach Required to omplement Traditional MDM, ETL, Data Profiling and Data Quality Many Next Generation Probabilistic and ollaborative Data uration Enterprise Analysts/Users Legacy Data Integrators Individual uration Tools Few Few Many Enterprise Sources & Variety

11 Big Data Example: How well am I doing retaining my best customers? Mobile App DB DB Twitter/Facebook Feeds Retail Sales DB RM Online Sales DB Google Analytics redit ard DB

12 ontextual Example: How well am I doing retaining my best customers? The High Spender segment s online sales are down due to drop in conversion rate. onversion rates increase when presented coupon over indexing in retail stores. ustomer engagement in my app and social feeds are increasing, key influencers contribute only 1% of sales, but their followers 20%. ustomers acquired through Facebook channels become the best customers ($50 ALTD, 50% spend using gift cards). Unified View

13 Making ontext Enterprise-wide is a hallenge No single definition of data - each system has own reference No way to tap into humans to collaborate on context ustom bridges across data sources don t scale (more complexity) onstantly changing applications with data variety Data variance across sources (jsmith@apple.com jsmith@tamr.com) No metadata catalog

14 Building Rich ontext Requires a New Approach Identify relationships across your sources using a machine learning bottom up approach ontinuous active learning combining machine/human insight ost effective as you unify more sources - marginal cost of new source = at least linear. Deploy context-rich sources for the different LOBs across the enterprise Enterprise metadata catalog - all your attributes, all your sources Services (e.g. APIs) can also be directly deployed in data warehouses/lakes and operational workflows In-situ curation of large sources

15 As Sources are Unified, Data Quality Improves ontinuously BEFORE AFTER

16 Top 3 Issues for Data Quality in Enterprise 1. Always start with questions and context 2. Always start with questions and context 3. Always start with questions and context Data 1,000 Questions Structured Prescriptive Internal External Agile/Rapid Dynamic Reproducible Scalable Predictive Descriptive Unstructured