Metadata Management as part of Data Governance and Data Stewardship at ebay, Inc. Mark Uksusman Sr. Manager, Enterprise Data Architecture, ebay, Inc.

Transcription

1 Metadata Management as part of Data Governance and Data Stewardship at ebay, Inc. Mark Uksusman Sr. Manager, Enterprise Data Architecture, ebay, Inc.

2 50,000 Categories of Products on ebay.com

3 Scale at ebay $60 billion per year in goods are sold on ebay >100k data elements >50 TB/day new data >100 Trillion pairs of informagon >100 PB/day >50k chains of logic 200 million items listed for sale AcGve/AcGve x7x Always online >7500 business users & analysts Big Data turning over a TB every second Millions of queries/day % Availability Near- Real- Gme

4 We manage more than 100 billion database calls per day 5 petabytes of live OLTP site data 500 Oracle based servers 80 petabytes of space available for analygcal data Teradata Hadoop With developers changing live schema every day! 4

5 ebay Data Warehouse 500+ concurrent users Analyze & Report OperaGonal AnalyGcs TransacGonal AnalyGcs High volume ad hoc queries 100+ concurrent users Discover & Explore Compare User AcGvity against last year Trending and Forecast Analysis (large history) >5 concurrent users Image FingerprinGng Image ClassificaGon Pabern RecogniGon Production Data Warehousing Large Concurrent User-base Enterprise- class System Structural Data Contextual-Complex Analytics Deep, Seasonal, Consumable Data Sets Low End Enterprise- class System Semi- Structural Data Structure the Unstructured Detect Patterns Java /C Developer System Teradata EDW/ODW 8+PB Teradata / Singularity 44+ PB Hadoop 50+ PB 450 Data Models; 7,000 tables and 100,000 columns

6 Data Warehouse SoluRon PorSolio It is not a Silver Bullet for a DW soluron Workload Management IO Concurrency CPU Storage EDW Singularity Hadoop Flexibility Governance

7

8 Virtual Data Mart Request

9 Why Metadata Management? Before: MulRple, Unrelated Metadata Now: One, Integrated Global Directory

10 Why Data Governance? The organizaron without data governance

11 Why Data Stewardship and Global Directory? Data TransformaGons from system to system Data and System Availability Data DefiniGon problems Business requests, Self Service AnalyGcs Source data (SOR) problems Lack Lack of of accountability + Failure to communicate Failure to communicate + Availability problems Security Data ClassificaGons Data Surprises! Lack of trust in the data = (and results)

12 Data Governance and Data Management Data Management Data Architecture Business Domain Modeling Physical Data Modeling Logical Data Modeling Database Design Data Governance and Stewardship Metadata Management Data Governance Enterprise InformaRon IntegraRon Data lineage

13 Data Architecture and Data Governance Framework Data Governance commicee Compliance Monitoring and Enforcement ExecuRon and Decision- Making Leadership Data PlaSorm and MDM Policy DefiniRon Planning and CoordinaRon Data Architecture, Data Quality and Stewardship Data Model / Metadata SOR / Data Topology / Data Flow Data Quality Assessment Ongoing Data Cleansing and Conversion Business Terms /Metrics / Data Quality Data Governance The formal orchestragon of people, processes, and technology to enable an organizagon to leverage data as an enterprise asset. Arbitrate issues and enforce the rules CoordinaGon and compliance Establish policies, procedures, success metrics and processes to maintain quality data IdenGfy all business and applicagon stakeholders data owners (stewards ) Conduct audit and control CommunicaGon and change management Coordinate creagon of data definigons and data lineage

14 Global Directory System IntegraGon

15 ebay Data PlaSorm and Global Directory

16 Data Stewardship and Data Stewards objecrves Data Stewardship is opera5onal aspect of data governance A Data Stewardship is a key part of an overall Data Governance program. Data Stewardship is fully integrated with creagon of AnalyGcal Reports and product development life cycle. The Data Stewardship team is allocagng required Data Stewards to finalize solugon and document findings in MDR Data Steward is a role assigned to a person that is responsible to define and maintain metrics, business definigons and system of record (SOR) for business data elements. The data stewards are decision- makers about the data Data Steward ensures that each assigned data element and metric are defined in MDR Data Stewards are creagng Business and Technical Metadata in MDR.

17 Data Governance OrganizaRon The diagram shows the organizaron structure of the Data Governance Body, relaronship between the Data Governance Lead, Data Owners, Enterprise Data Stewards and Domain Data Stewards. Data Governance commicee Business and Technology sponsors and main stakeholders Chief Data Steward / Manager of Data Stewardship team Enterprise Data Stewards Team of Data Stewards for each Business Unit / Business Domain - Business and Technology stewards Takes care of data assets from Business and Technology perspecgves Data Owner Owns physical data assets Business Data Owners Technology Data Owners Domain & Project Data Stewards Experts in specific domains and data areas Data Quality, SAE, PM, OPS, BSA, Data Architects, Domain Architects, Data Analysts

18 ebay, Inc. Master Data Management Program Link customer data across ebay, Inc. Making data more accessible to business Integrated Customer Integrate data sources across enterprise Enterprise data integration

19 MDM and 360 Customer view ebay, Inc. Customer Multiple Accounts Multiple Organizations Multiple Data Sources AWs357 AbGrWe11 GrWHockey24 ebay Shopping Bill Me Later Milo GSI Master Data Transactions Contacts/Event Master Data Transactions Contacts/Event Where PayPal Master Data Transactions Contacts/Event Stub Hub!

20 360 Customer Data Enrichment- What is Acxiom Data? Demographic data about US users Individual and household informagon Enrichment of ebay, PayPal and StubHub customer data from Acxiom 20

21 Which users are enriched with Acxiom data? U.S. ebay members who have been acgve from Jan 2004 PLUS all US users since Jan 2007 New users sent to Acxiom monthly to obtain demographic data and account linking. Full file is refreshed annually Quarterly updates (address changes) are received 21

22 IdenRficaRon Individual ID Sample name/address Household ID Account and User ID ebay / PayPal / StubHub / Shopping / Milo / Where 22

23 Household ComposiRon Marital Status Children Number of Children Number of Adults Household Size 23

24 Demographics Age in Two- Year Increments - Input Individual Gender - Input Individual Adult Age Ranges Present in Household Children s Age Ranges Present in Household 24

25 Residence InformaRon Home Owner/Renter- Premier Length of Residence Dwelling Type Property Type 25

26 Professional / EducaRonal Business Owner OccupaGon Income Household EducaGon 26

27 Buyer Behavior Mail Order Responder Categories Mail Order Buyer (various categories) Credit Card Users PayPal Users 27

28 Lifestyle / Interests Fashion Photography VCR / LD / DVD Movie Collector Pets - Dog Owner CollecGbles - General Sports Grouping Outdoors Grouping Travel Grouping Reading Grouping Cooking / Food Grouping Exercise / Health Grouping Stereo / Video Grouping Electronics / Computers Grouping Home Improvement Grouping InvesGng / Finance Grouping CollecGbles - AnGques - Grouping

29 MDM and 360 view of the customer Account: Gwin357 1 account GMB: $795 (C) GMV: $0 Primary Cat: Auto Parts Pays with PayPal. BBE: INR on 7/5/08 Individual: Greg Wingard 4 accounts (2 ebay, 2 PP) Male, 35 years old. GMB: $2,550 (B) GMV: $22,250 (Silver) TPV: $32,575 (A) Primary Cat: + Sports Sells Collectibles Household: Greg & Abigail Wingard 5 accounts (3 ebay, 2 PP) Married, 2 children Income: $250k Net Worth: $850k Own Home, $900k value, suburbs GMB: + $3,200 (B) Primary Category : + Fashion Off-eBay purchases: Media Interests: Fashion, Sports and Finance

30 Business Domain Model and Data Stewardship

31 Metadata Management and Agile Data Warehousing TransacRonal / Behavioral / Experimental / OperaRonal Data Agile Model Driven Development (AMDD) methodology is uglized Data Models are fully integrated with Global Directory Technical and Business Metadata are fully integrated Cross- domain analysis is available. Virtual Data Marts are generated in real Gme Developed powerful methodology for Big Data AnalyGcs

32 Keys to success of Data Stewardship, Metadata Management and MDM program Processes and Tools Data Architecture Steering Commibee Data Governance Agile Data Modeling Master Data Management Product Development Common Dimensional Modeling Metadata Management Technical and Business Metadata Compliance ReporGng System integragon ASG Rochade technology is used for Technical and Business Metadata

33 Global Directory / MDR Data Browsing Metadata is organized by Business Domains Lists of Data Models

34 Compliance ReporRng: Data Models vs. Systems Compliance report is automagcally generated and sent to distribugon list of execugves, managers and developers Teradata Data Model Compliance Audit Report Date Compliance Missing Tables Different Tables % Compliance 1/4 0.0 Report Date: Oct 24, :57:31 AM 8/ Wildcat Scan Date: Oct 24, :37:10 AM 8/ Caracal Scan Date: Oct 24, :36:21 AM / / Overall Audit Report Compliance: 98.4% 9/ / View Compliance by Subject Area 10/ Data Model Data Warehouse In Model but not in Data Warehouse Data Type Difference Between Data Model and Data Warehouse In Data Warehouse but not in Data Model 10/ / Comments A B C D E / Table 3,700 3, Count Column Count 57,793 53, A B C D E Comments Total count of tables (including persistent tables and indices) or columns in the data model. Total count of tables (including persistent tables and indices) and columns that are in the data warehouse. Count of tables (including persistent tables and indices) and columns that occur in the model but not in the data warehouse. Primarily consists of projects that are modeled but have not been released into production. Count of columns (and the associated tables) that occur in both the data model and the data warehouse, but the data types are different. The column must exist in both the data model and the data warehouse to be compared. Count of all objects in the data warehouse production databases, entire tables, persistent tables and indices and the associated columns that are missing from the data model plus a count of columns that are missing in the data model from tables that exist in both the data model and the data warehouse. Views and working databases are not included Number of Missing Tables /22 8/29 9/5 9/12 9/19 9/26 10/3 10/10 10/17 10/ Date Table Audit Trend 8/22 8/29 9/5 9/12 9/19 9/26 10/3 10/10 10/17 10/24 Date Number of Different Tables Missing Tables Different Tables

35 Wiki and Global Directory integration! MDR Wiki Real Rme metadata updates IntegraRon of Wiki and Metadata Management One single source of metadata Dynamic Wiki pages

36 Global Directory metaglossary ApplicaRon Business Glossary Business Terms Business Metrics Approval workflow Advanced search Contains Links to: External Docs(Wiki) Logical data objects Physical data objects Reports and metrics The total number of feedback comments received The total number of feedback comments received The total number of feedback comments received The number of posigve comments displayed as a percent of The number of posigve comments displayed as a percent of

37 Global Directory and Agile Data Architecture Guided Reports Dimensional Models MicroStrategy cubes

38 Questions?

39 Contact InformaGon If you have further quesgons or comments: Mark Uksusman Sr. Manager, Enterprise Data Architecture ebay, Inc