The Importance of Meta Data and Data Governance in Process and Data Management

Transcription

1 EWSolutions The Importance of Meta Data and Data Governance in Process and Data Management By David Marco President EWSolutions 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 1

2 EWSolutions Background EWSolutions is a Chicago-headquartered strategic partner and full life-cycle systems integrator providing both award winning strategic consulting and fullservice implementation services. This combination affords our clients a full range of services for any size enterprise information management, meta data management, data governance and data warehouse/business intelligence initiative. Our notable client projects have been featured in the Chicago Tribune, Federal Computer Weekly, Crain s Chicago Business, and won the 2004 Intelligent Enterprise s RealWare award, 2007 Excellence in Information Integrity Award nomination and DM Review s 2005 World Class Solutions award Excellence in Information Integrity Award Nomination Best Business Intelligence Application Information Integration Client: Department of Defense World Class Solutions Award Data Management For more information on our Strategic Consulting Services, Implementation Services, or World-Class Training, call toll free at 866.EWS.1100, , main number or us at [email protected] 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 2

3 EWSolutions Partial Client List Arizona Supreme Court Bank of Montreal BankUnited Basic American Foods Becton, Dickinson and Company Blue Cross Blue Shield companies Branch Banking & Trust (BB&T) British Petroleum (BP) California DMV California State Fund Capella University Cigna College Board Comcast Corning Cable Systems Countrywide Financial Defense Logistics Agency (DLA) Delta Dental Department of Defense (DoD) Driehaus Capital Management Eli Lilly and Company Environment Protection Agency Farmers Insurance Group 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 3 Federal Aviation Administration Federal Bureau of Investigation (FBI) Fidelity Information Services Ford Motor Company GlaxoSmithKline Harbor Funds Harris Bank The Hartford Harvard Pilgrim HealthCare Health Care Services Corporation Hewitt Associates HP (Hewlett-Packard) Information Resources Inc. International Paper Janus Mutual Funds Johnson Controls Key Bank LiquidNet Loyola Medical Center Manulife Financial Mayo Clinic Microsoft NASA National City Bank Nationwide Neighborhood Health Plan NORC Physicians Mutual Insurance Pillsbury Quintiles Sallie Mae Schneider National Secretary of Defense/Logistics Singapore Defence Science & Technology Agency Social Security Administration South Orange County Community College SunTrust Bank Target Corporation The Regence Group Thomson Multimedia (RCA) United Health Group United States Air Force United States Army United States Department of State United States Navy United States Transportation Command University of Michigan University of Wisconsin Health USAA US Cellular Waste Management Wells Fargo Wisconsin Department of Transportation Zurich Cantonal Bank For more information on our Strategic Consulting Services, Implementation Services, or World-Class Training us at [email protected]

4 Professional Profile/Contact Information Best known as the world s foremost authority on meta data management, he is an internationally recognized expert in the fields of data warehousing, data governance and enterprise information management. In 2004 David Marco was named the Melvil Dewey of Metadata by Crain s Chicago Business as he was selected to their very prestigious Top 40 Under 40 list. David Marco has authored several books including the widely acclaimed Universal Meta Data Models (Wiley, 2004) and the classic Building and Managing the Meta Data Repository: A Full Life- Cycle Guide (Wiley, 2000). Selected to the prestigious 2004 Crain s Chicago Business Top 40 Under DAMA Data Management Hall of Fame (Professional Achievement Award) Chairman of the Enterprise Information Management Institute (EIMInstitute.ORG) 2007 DePaul University named him one of their Top 14 Alumni Under 40 Presented hundreds of keynotes/seminars across four continents Published hundreds of articles on information technology Author of several best selling information technology books Taught at the University of Chicago and DePaul University [email protected] 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 4

5 Acknowledgements Session materials adapted from the books... Universal Meta Data Models (Wiley, 2004) Building and Managing the Meta Data Repository: A Full Life-Cycle Guide (Wiley, 2000) 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 5

6 Marco Masters Series Teaching the full 3 day version of our Meta Data Management course from June 6 8, 2011 in Chicago, IL Teaching the full 3 day version of our Data Governance course from October 3 5, 2011 in Chicago, IL Visit for details 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 6

7 Fundamentals and Defining Terms 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 7

8 Data Governance Fundamentals Enterprise Information Management (EIM): The systematic processes and governance procedures for applications, processes, data, and technology at a holistic enterprise perspective The purpose of enterprise information management is to bring enterprise order, purpose, structure, efficiency, and performance to applications, processes, data, meta data and technology EIM is not a single technology or component, but a coordinated framework of disciplines for managing data, meta data and information assets throughout the organization Data Does Not Manage Itself!! 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 8

9 EIM Focus Areas Data Management is the foundation for all of the other EIM focus areas. Regardless of which focus area you target first, you will need to do Data Management Process Management Enterprise Information Management Data Architecture Information Quality IT Portfolio Management Master Data Management Information Delivery Information Security Data Management 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 9

10 Process Management Process Management: is the application of knowledge, skills, tools, techniques and systems to define, visualize, measure, control, report and improve processes with the goal to meet customer requirements Why do we need to manage processes? What does one process look like for a large corporation? 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 10

11 Process Management Process management: looks to holistically manage the Information Technology (IT) and business processes that exist within the organization in order to streamline, optimize, historically track, ensure quality and prevent redundancy of the IT processes at an enterprise level. Process (people & technical) Management Target Areas What processes exist? What processes are used? What function does the process perform? Who uses the process? What data is utilized by the process? What inputs does the process get from which people? Which processes are used in which applications? Workflow (process dependencies and scheduling) Process performance Vital for Service Oriented Architectures (SOA) Process Management 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 11

12 Process Management Why do we need to manage processes? What does one process look like for a large corporation? 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 12

13 Fortune 100 One Process Sales Request Customers Shopping/ Quoting Field Admin Astro - NA - HP Profiler - HP Explorer4 - Support Pricing Tool - Sales Request Sales Force Sales Request QMS AQS - - NA - Sales Request SBW - HP Quick Quote - Watson - HP Call Centers - Rockville, Roseville, Cupertino, Ontario Siebel echannel BPS - PWEB - US - Parts Repair Customers and Partners Active Answers Product Advisor Cybrant - HP ITRC - SUM - NA - HP SW Depot - Shopping - HP COPE/BET Remedy US - HP Siebel Call Center US - Les Web - CPL - Prod. & Price Golden Eggs - VISA - Sales Request NCAS - QL - LSO - HP Web/ /Fax Web/ /Fax Web/ /Fax eprime - HP Rack Assistant WW Comcat - Quoters WB - Enterprises Corporate Relationships Web/ /Fax Global Biz Link (Extranets) - NA - CEI's Omaha Call Center, or Littleton OLS - Chameleon - Business Advantage - Web/ /Fax Web/ /Fax Web/ /Fax Catalog Procurement - CGBX - NA - Vendor emarket Place - IPS Only Agents Reseller Web - HP Web/EDI Web/EDI Commerci al Web/EDI Web/EDI Fast-Web Partner Kylor (AA) - & Ptnr Direct - Prime - ecat - Rack Builder - ESN - Prophecy - US - Descartes EDI Gateway - NA - CSN - NA - CPN - CDAT - GEM Commercial Tele/Web Resellers SMB Channel Tele/Web Links Customer EDI/XML Advantage Gateway - NA/AP eprocurement - - HP Customer Advantage Direct Plus - IPS Only B2B Integ SVR / CEI's Web Svr - Biz-Store - HP Customer Advantage Extranets - Kiosk Order Kiosk - SAP - XRS - IPC - HP e-store - Canada - HP Buysite & Market Site.com Retailer Kiosk Order EDI Order Kiosk Order e- Pack - Kiosk - HP SAP - IPC D7 - Consumer s / Micro Business PC, Servers, Tele/Web LJ Call Center - US Consum er Call Center - HP At Home Shopping - Virtual Sales Rep - HP CAFE - Game DB - Compaq GEMS - Direct NA - Tele/Web Affiliates estores - Factory Outlet - EDR - Order Management CPSA/MSS - ACS - SAP - HPS - NA - OM - HP System T - CCSU SMART db - Core - Eclipse - HP FOCUS - Conrad - HP Pears - NA - SAP - Fusion - NA - OM - HP NCAS - NA - PPS - SBS - SW Depot - OM - HP GPSY CPL - SCA - NA HP - HP SAP - XRS - CMG - NA - HP SAP - XRS - Matl - HP WW CISYS - HP Legacy Parts - OM - HP Compucom - OM WWPAK WWOMS - HP Canada - HP eparts - HP Allegro - NA - HP WWOMS US - HP ICON - HP Primus - NA - etracking - OSS - HP SAP EPH - OM - FOCUS GOLD (252) - - Vista - OM - STO SRS - SAP - CaLado - OM - HP Fireman - NA - DAC eparts epo - NA - HP SAM - NA - HP E-Eureka - NA - HP Earl - HP RosettaNet Portal - HP Resource Marketplace - NA - HP SAP - Consumer Direct SV - OM - HP SSIM - HP SAP - HPS - NA - Upfront - SAP - Replenishment D7 - OM - HP Finance SAP - HPS - NA - OM Finance - HP WWFTP - PM Tools HP - HP SIS - Heart - HP Tiger - NA - HP TIM - HP HPFO Legacy Vista - Finance - CTO products SORDS - NA - HP SACS - NA - HP SAP - HPFO - NA - HP CyberS Armada - NA - Financial Services MAT - NA - HP HP Shopping Financial Institution Plus - SmartCube NA - - HP HP Legacy OM - Finance - HP SAP - Consumer Direct SV - Finance - HP Supply Chain SAP - HPS - NA - Virtual Sourcing - HP CCDB - NA - HP FAI - NA - HP EIA - NA - HP Clear Contract - Clarify - Pulse - SW Depot - Fulfillment - HP MAXCIM - HP GDS - NA - HP SAP - Fusion - NA - Fulfillment - HP Fast - OM Server - NA - HP SFDM (M1/M2) - Plato - NA - HP Microsoft SAP SW Supply Chain - HP Compucom SWAT - HP Legacy Parts - SC - HP APOGEE - Storage - HP APOGEE/SHIP US - Services - HP SAP EPH - Fulfillment - efls - RMS - Power - SAP EPH ORS - - Finance - Vista - Fulfillment - SAP - Replenishment D7 - Fulfillment - HP ASM - SAP - CaLado - Fulfillment - HP IOM - NA - HP Xelus Plan - NA - HP PROMIS - NA - HP SPD - NA - HP Norm - NA - HP Entitlement - NA - HP CDO - NA - HP Knight - SAP - Education - NA - HP Aurora Mgr - NA - HP RCM - HP Backplane/Shopfloor - NA - HP Clarify - WFM - NA - HP AUE - NA - HP DAC Fulfillment WWSNRS - Ryder MM HP SAP - Consumer Direct SV - Fulfillment - HP Roseville Houston M1/M2 Ingram Micro Tech Data AGS Symnex Daisy Tech Hitachi Houston M5 FIC Q-run MSL Printer CQ Direct Ryder Suppliers Factory Woodland IJ Ontario LJ Completion Memphis IJ Completion Menlo & Supplies Completion & Supplies Logistics DC & Supplies Projector DC CQ 3PO Richmond DC Supplier Supplies Tatung Completion & DC Fremont CQ Alliance Partners Tijuana LJ SCI - HP 3PO Quanta Completion Alabama - HP Venture & Supplies Irish DC Express SCI Sandston Compal Power OHL LJ Comp. & - HP Packaging Memphi Supplies s DC DC Genco Comm. Resellers 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 13

14 Data Management Data Management: Data Resource Management is the development and execution of architectures, policies, practices and procedures that properly manage the full data lifecycle needs of an enterprise (DAMA International) 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 14

15 Data Management Data Management: is the function of managing the data within an organization This is the keystone focus area Without this area it is almost impossible to address the other focus areas Business requirements will drive this initiative Data Management 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 15

16 Data Management Objectives Data Management looks to answer questions on the data in a company: What does it mean (data lineage)? What is its source (data heritage)? What are the valid values? What formulas were used to calculate it? What are its business rules? What are its technical rules? Subject area definition Define business entities Enterprise conceptual model Enterprise logical model 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 16

17 Data Management Maybe data management looks better than process management? 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 17

18 Islands of Data Operational Applications TOPPS ES-9000 HPE-980 RX42 VAX Cluster AMRS-Advanced Medical RIS- Record Radiology System Information System LAB/LIS- Laboratory Information System VAX(Burlington) IPS- Inter Practice System PSIMED HPE-980 North E/B AS/400 MGD Claims Query AS/400 Pharmacare (outside vendor) Pharmaview (outside vendor) HP-992 NEDClinical Computer NED E/B ACPS- Automatic Claims Processing PASS - Patient System Appointment Scheduling ENCOUNTER System FFS - Fee For Service REF - Referrals Pharmacy DB Provider Master HP (9 series) MARS AMISYS Hospital Summary Quality Assurance Request Network & Medical Request Actuarial End User Request Claims Prov DB Drug DB Islands of Data Bulletin Board System (BBS) Pentium PC Healthchex AST 486 server MHUM - Mental Health Utilization MS SQL Management Server ASAP - Actuarial System Analysis ES-9000 Program GL GMIS-Claim Check SAS, SAS screens, DB2 Viewpoint and SAS dataset Decision Analyzer Actuarial SAS dataset HP937 NED Datawarehouse HSA dept. Lan Server NED Multiview GL and AP PHC Multi-view GL. PHC JV Finance Server PHC Network Development Server MAMSYS/PAPSYS (14) Foxpro Applications PHC Medical Services Server PHC Actuarial Analysis Server Dec Alpha Quantum Analytic Database PHC Pharmacy Server Medical Groups End User Reporting Systems Financial Analysis Standard Adhoc Pharmacy Standard Adhoc Claims Standard Adhoc Health & Medical Standard Services (CQM, HSA) Adhoc Utilization Standard Adhoc Actuarial & Underwriting Standard Analysis Adhoc Sales & Mktg / Standard Network Development Adhoc Membership/Enrollment & Standard Billing Adhoc Legend HCD MGD NED PHC Multiple colors indicate the system is used by multiple divisions Enterprise Warehousing Solutions, Inc. (EWSolutions) 18

19 Process & Data Management Clearly we can see that there is a problem Question: What are the key disciplines to resolve this problem? Answer: Data Governance (people) and Meta Data Management (technology) 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 19

20 Data Governance Background & Fundamentals 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 20

21 Data Governance Defined Data Governance: defines the people, processes, framework and organization necessary to ensure that an organization s information assets (data and meta data) are formally, properly, proactively and efficiently managed throughout the enterprise to secure its trust, accountability, meaning and accuracy 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 21

22 Understanding Data Governance Data Misunderstood Inaccurate Misleading Data Governance Policies, Procedures, Consensus, Knowledge, Information, Data, Meta Data Data Stewards Understood Accurate Consistent T r a n s f o r m D a t a I n t o I n f o r m a t i o n Data Stewards 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 22 Actionable Information

23 How Do You Manage Information Assets? This is all Data Governance You cannot manage what you do not measure You cannot measure what you do not understand You do not understand Enterprise Warehousing Solutions, Inc. (EWSolutions) 23

24 Data Governance ROI 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 24

25 Data Governance Metrics Defined hard and soft dollar savings and earnings Testimonials from participating areas (lines-of-business, divisions, etc.) Improvements in process and data performance (limiting redundancy, increasing reuse, improving performance, etc.) Documented improvements in information quality Number of times meta data is read/updated/added/deleted from the MME Number of participating Data Stewards Number of defined Subject Areas Number of entities, attributes and relationships actively managed Number of entities, attributes and relationships with corresponding meta data 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 25

26 The Cost of Redundancy Large healthcare insurance company Has a $1.6 billion IT budget They estimate it costs them $2 per month to store each gigabytes of data $8 per month if you add in services and maintenance They estimate that they have 1.6 petabytes of redundant data What does this cost them yearly? Simple math $8 x 12 months x 1,000,000 (1.6 petabytes) = $153,600, Enterprise Warehousing Solutions, Inc. (EWSolutions) 26

27 How Does a Lack of Data Governance Impact IT Development? 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 27

28 Case Study NASA Problem NASA has a history of financial mismanagement. The agency s contract-management function has earned a spot on the GAO s high risk watch list every year since 1990 In early 2004 NASA s auditor (PricewaterhouseCoopers) proclaimed several issues with NASA s 2003 financial statements NASA couldn t adequately document more than $565 billion billion in year end adjustments Because of the lack of a sufficient audit trail it was not possible to complete further audit procedures NASA has a $204 million line item called Other that could not be explained or supported, indicating that NASA had not correctly reconciled its budgetary resources to its net cost of operations NASA s stated fund balance was $2 billion more than the balance in the treasury account NASA s proposed 2005 budget is $ billion (source: NASA) Source: CFO magazine, NASA, We have a problem, May, Enterprise Warehousing Solutions, Inc. (EWSolutions) 28

29 Case Study NASA Why does this problem exist? NASA says this problem is caused by enterprise software implementation called Integrated Financial Management Program (IFMP) NASA s CFO Gwendolyn Brown said the conversion to the new system caused the problem with the audit, specifically the agency had great difficultly converting the historical financial data from 10 legacy systems to the new system NASA has a stovepipe structure, in which each center behaves as an independent entity with a unique history and culture that is loath to brook outside interference from other parts of NASA For example, NASA has 10 centers, each with a different financial reporting system It s like a dozen dueling fiefdoms, says Keith Cowing, editor of NASA Watch 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 29

30 Information As A Corporate Asset 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 30

31 Information as a Corporate Asset Information and knowledge are the primary resources of the knowledge society of the 21 st century. P. Drucker, 1992 Organizations that do not understand the overwhelming importance of managing data and information as tangible assets in the new economy will not survive. T. Peters, Enterprise Warehousing Solutions, Inc. (EWSolutions) 31

32 Data Governance Components Thought Ware: Mission/Core Values, Goals/Objectives, Charters/Principles, Critical Success Factors, Plans, Documents/Policies, Communication Plan (messages and vehicles), Roles/Functions and Responsibilities Definitions, Accountability Matrix, Organizational Interdependencies, Workflow People Ware: Structures, Organizations, Committees, Teams/Groups, People Work Ware: Managed Meta Data Environment, Software, Training and Education, References, Templates, Standards Artifacts: Meta Data, Data Rules and Definitions, Decision Rights, Accountabilities, Controls 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 32

33 Illustration of the Four Components Thought Ware The true tangible value/measure of DG. when the artifacts are used successfully Guides People Ware Work Results Assists Work Ware Kept in Tools Artifacts 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 33

34 Governance and Strategy Data governance is the method for connecting information management and the corporate business strategy 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 34

35 Data Governance Organization 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 35

36 Data Governance Organization Every organization forms their data governance organization a little differently Some have a more or less complex organization What is critical is that the organization: is actively using the MME has clear lines of communication has a defined and well understood decision making process well defined feedback loop 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 36

37 Subject Area A logical grouping of items of interest to the enterprise, or areas of interest within the company About Subject Areas in an organization The nouns of an entity. Examples: Legal Entity Cost Center Account Product Customer Sale 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 37

38 Data Governance Organization Requirements Subject Area Groups Subject Area User Group #1 Chief Steward Business Steward(s) Technical Steward(s) Interested Parties Subject Area User Group #2 Chief Steward Business Steward(s) Technical Steward(s) Interested Parties Subject Area User Group #3 Chief Steward Business Steward(s) Technical Steward(s) Interested Parties 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 38 Data Stewardship Coordination Group Program Manager Chief Stewards EIM Focus Area/ Project #1 Steward Team EIM Focus Area/ Project #2 Steward Team EIM Focus Area/ Project #3 Steward Team Recommendations Enterprise Oversight Data Governance Council Members: Executive Sponsor (s) Program Manager Chief Stewards CIO Key business staff Key IT staff Managed Meta Data Environment Data Custodian Team Information Technology Policies, Procedures, Standards, etc. Technical Stewards

39 Data Governance Organizations <5% Strategic Data Governance Executive Committee Data Governance Program Mgr. Various Data Stewards < 20% Tactical Data Stewardship Groups Data Governance Program Mgr. Chief Stewards Business Stewards Technical Data Stewards 80-85% Conflicts Resolved at this level Subject Area Groups Chief Stewards Business Data Stewards Technical Data Stewards Operational by Subject Area, not by LOB 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 39

40 Meta Data Management 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 40

41 Meta Data vs. Data Meta Data: Meta data contains the knowledge that a 1) field is called Customer_Name, is 40 characters in length, and exists in systems A, B, and C; 2) that our company has 3 systems which contain customer master data. These systems are Data: Data would be a specific instance of Customer_Name equaling John Doe Information: Data that is meaningful to a business user. They understand it and they know what to do with it 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 41

42 Data Governance Fundamentals (content) (context) 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 42

43 What is Meta Data? Meta Data By definition meta data is So What!! 1. Data about data. 2. Everything that data is not Enterprise Warehousing Solutions, Inc. (EWSolutions) 43

44 What is Meta Data? Meta Data Definition All physical data (contained in software and other media) and knowledge (contained in employees and various media) from within and outside an organization, containing information about your company s physical data, industry, technical processes, and business processes. Meta Data Is Knowledge 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 44

45 Managed Meta Data Environment ROI The key to your company s prosperity is how well you gather, retain and disseminate knowledge Managed meta data environments are the key to gathering, retaining and disseminating knowledge 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 45

46 Managed Meta Data Environment ROI Meta Data for the Business (business meta data) Meta Data for the IT Department (technical meta data) 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 46

47 MME ROI Intel finds huge ROI in managing meta data Estimates $6 in savings for every $1 spent Emphasis on reducing developer s average research time of 30% Uses Centralized architecture Key to success is based on regular, frequent scan updates Source: Computer World magazine July 2005 A Canadian government agency achieved: More than 90 percent reuse of existing data definitions 85 percent improvement in application integration 25 percent reduction in DW analysis and design Impact analysis study in 2 hours vs. 36 person-days of consulting Knowledge workers spend up to 2.5 hours each day looking for information but find what they are looking for only 40% of the time. Kit Sims Taylor Economist and Researcher in Economics of information and networks Bellevue Community College, WA 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 47

48 Business Meta Data 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 48

49 Managed Meta Data Environment ROI Meta Data for the Business (business meta data) Provides the semantic layer between a company s systems (operational and business intelligence) and their business users 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 49

50 Meta Data for the Business Reduces training costs Makes strategic information (e.g. data warehousing, CRM, SCM, etc.) much more valuable as it aids analysts in making more profitable decisions Create actionable information Limits incorrect decisions Assists business analysts in finding the information they need, in a timely manner Bridges the gap between business users and IT professionals Increases confidence in the IT system data 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 50

51 Business Meta Data In Action 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 51

52 Example without Meta Data 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 52

53 Example with Meta Data 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 53

54 Meta Data Providing the Semantic Layer 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 54

55 Search Enter Your Search Terms Below Customer Relationship & Sales Management Reports Monthly Product Sales Marketing Reports Finance Reports 1. Global Sales by Month This report shows a years worth of U.S., international, and Totals, of summarized sales figures by product category, on a monthly basis. 2. Global Sales by Region, by Month Logistics Reports This report shows a years worth of U.S., international, and Totals, of summarized sales figures by product category, on a monthly basis by region. 3. Global Product Sales by Region, by Month Ad-Hoc Reporting This report shows a years worth of U.S., international, and Totals, of detailed product sales figures, on a monthly basis by region. Search 1 of 4 pages, 17 total documents found 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 55

56 Meta Data for the Business 2007 Monthly Global Sales Report February 7, 2008 Month Product Category Sales $ (in thousands) U.S December TV 22,101 VCR 11,190 Cellular Phone 12,190 Digital 4,002 Miscellaneous 1,209 November TV 42,000 VCR 21,190 Cellular Phone 28,193 Digital 8,901 Miscellaneous 2,730 October TV 70,100 VCR Cellular Phone 31,900 41,700 Digital 20,000 Miscellaneous 4,850 Sales $ (in thousands International 10,200 4,300 7,193 1, ,200 9,878 12,193 2,901 1,530 32,950 14,878 17,550 4,100 2,850 Sales $ (in thousands Total 32,301 15,490 19,383 5,303 2,079 64,200 31,068 40,386 11,802 4, ,050 46,778 59,250 24,100 7, Enterprise Warehousing Solutions, Inc. (EWSolutions) 56 Sales $ U.S. is comprised of aggregated sales revenues from the United States, Canada, and Mexico, but does not subtract sales dollars from returned orders

57 Meta Data for the Business Carrier/Usage Summary Report NoTeleCo November 20, 2008 Month Carrier Name Usage Type Regular Usage (M seconds) Discounted Usage (M seconds) Total Usage (M seconds) October BigTeleCo Long Discounted DistanceUsage: Any local or long 7,201 4,288 11,489 Local distance phone usage that has a discount 72,033 42, ,033 TeleBell Long applied Distance to it. Discounts include nonprime, ,407 holiday and rate specials. Local 23,000 17,255 40,255 NewBell Long Last Distance Updated: 3/31/2004, Bob Jones Local 1, ,857 September BigTeleCo Long Distance 6,400 4,000 10,400 Non-Prime Usage: Any local or Local long 73,450 42, ,152 distance TeleBell phone usage that occurs Long Distance ,395 between the time of 8:00pm 7:00am. Local 23,500 17,923 41,423 Last Updated: NewBell 1/08/2005, Tony Long Ragone Distance Local 1, ,878 August BigTeleCo Long Distance 6,220 4,010 10,230 Local 71,207 41, ,125 TeleBell Long Distance ,406 Local 20,010 15,500 35,510 NewBell Long Distance Local 1, , Enterprise Warehousing Solutions, Inc. (EWSolutions) 57

58 Meta Data Makes for Better Decisions 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 58

59 Meta Data for the Business 2007 Monthly Global Sales Report February 7, 2008 Month Product Category Sales $ (in thousands) U.S December TV 22,101 VCR 11,190 Cellular Phone 12,190 Digital 4,002 Miscellaneous 1,209 November TV 42,000 VCR 21,190 Cellular Phone 28,193 Digital 8,901 Miscellaneous 2,730 October TV 70,100 VCR Cellular Phone 31,900 41,700 Digital 20,000 Miscellaneous 4, Enterprise Warehousing Solutions, Inc. (EWSolutions) 59 Sales $ (in thousands International 10,200 4,300 7,193 1, ,200 9,878 12,193 2,901 1,530 32,950 14,878 17,550 4,100 2,850 Sales $ (in thousands Total 32,301 15,490 19,383 5,303 2,079 64,200 31,068 40,386 11,802 4, ,050 46,778 59,250 24,100 7,700 Information Quality Tracking Statistics 8.4% of the dollar values were not loaded 1.7% of the records were not loaded

60 Technical Meta Data 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 60

61 Managed Meta Data Environment ROI Meta Data for the IT Department (technical meta data) Help IT departments better manage, maintain and grow their IT systems and assets 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 61

62 Meta Data for the IT Department Dramatically reduces the probability of project failure Speeds system s time-to-market Reduce system development life-cycle time Limit redundant data Limit redundant processes Managing IT portfolios Leverage work done by other teams Reduced rework Reduce research time Reduce unproductive work Lowers the impact of staff turnover 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 62

63 Technical Impact Analysis 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 63

64 Meta Data for the IT Department Question: Show all decision support tables/files, programs, and fields impacted by a change to the CUST table in the Order Entry system Impact Analysis Report January 7, 2008 Source System Source Table Impact Field Program Impacted Tables/Files Impacted Table Type Fields Impacted Order Entry CUST Customer_Name CUSTOMER_PR02 DW_CUSTOMER T Cust_Name_First Cust_Name_Middle Cust_Name_Last CUSTOMER_PR01 I02_CUSTOMER I Cust_Name_First Cust_Name_Middle Cust_Name_Last Customer_Addr CUSTOMER_PR02 DW_CUSTOMER T Cust_Name_Address Cust_Name_City Cust_Name_State Cust_Name_Zip CUSTOMER_PR01 I02_CUSTOMER I Cust_Name_Address Cust_Name_City Cust_Name_State Cust_Name_Zip 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 64 *Legend T = Target I = Intermediate S = Source

65 Meta Data for the IT Department Question: Show all systems, tables/files, fields, and their domains impacted by a change to the length of all occurrences of the Customer_Name field Impact Analysis Report January 7, 2008 Field System Tables/Files Fields Domain Customer Name Order Entry CUSTOMER_BILL_TO CUST_NAME Alphanumeric 20 CUSTOMER_SELL_TO CUST_NAME Alphanumeric 20 CUSTOMER_SHIP_TO CUST_NAME Alphanumeric 20 ORDER_HEADER CUST_NAME Alphanumeric 20 ORDER_DETAIL CUST_NAME Alphanumeric 20 General Ledger CUSTOMER Cust_Name Alphanumeric 35 EXPENSES Cust_Name Alphanumeric 35 CUST_ACCOUNTS Cust_Name Alphanumeric 35 Data Warehouse DW_CUSTOMER Cust_Name Alphanumeric 20 I01_CUSTOMER Cust_Name Alphanumeric 20 I02_CUSTOMER Cust_Name Alphanumeric 20 I03_CUSTOMER Cust_Name Alphanumeric 20 Data Mart - Marketing DM_CUSTOMER DM_Cust_Name Alphanumeric 20 I01_DM_CUSTOMER DM_Cust_Name Alphanumeric 20 I02_DM_CUSTOMER DM_Cust_Name Alphanumeric Enterprise Warehousing Solutions, Inc. (EWSolutions) 65

66 MME For Systems Consolidation 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 66

67 Meta Data for the IT Department Systems Consolidation Report BigCity Bank BigCity Bank Small Town Bank August 15, 2008 Attribute Name Attribute Definition Entity Name System Name Attribute Name Attribute Definition Entity Name System Name Cust_Nbr Cust_Nbr is the attribute of record for BigCity Bank customer numbers Cust_Tbl Central Customer System CUSTNUM Purchase_No Customer numbers from the deposit system. Customer numbers from the purchase in the legacy deposit system CUSTTABLE Purch_Tbl CUSTAPPL CUSTSYS Borwr_No Customer numbers from the loan system. Borrower_File LoanSys Cust_Type Cust_Type is the attribute of record for BigCity Bank customer types (affluent, upward, standard, high risk). Cust_Tbl Central Customer System CUSTCDE Customer types from the general ledger system. GL_CUST GLAPPL Cust_Card_Ind Cust_Card_Ind is the attribute of record for BigCity Bank customer s that have a BCB credit card. Cust_Tbl Central Customer System None applicable Cust_Crdt_Ratg Cust_Crdt_Ratg is the attribute of record for BigCity Bank customer credit ratings (Superior Risk, Low Risk, Standard Risk, High Risk, Extreme Risk). Cust_Tbl Central Customer System Credit_Rate Customer rate is from the general ledger system and refers to the credit rating/worthiness of a customer. GL_CUST GLAPPL 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 67

68 Meta Data for the IT Department Systems Consolidation Report BigCity Bank BigCity Bank Small Town Bank August 15, 2008 Entity Name Attribute Definition Attribute Name Domain Value Transformation Rules Attribute Name Domain Value Entity Name Cust_Tbl Cust_Type is the attribute of record for BigCity Bank customer types: 1 = affluent 2 = upward 3 = standard 4 = high risk Cust_Type 1 Cust_Type = 1 WHEN CUSTCDE CUSTCDE = 3 AND CUSTBAL > 500,000 CUSTBAL 2 Cust_Type = 2 WHEN CUSTCDE CUSTCDE = 4 AND CUSTBAL <= 500,000 AND CUSTBAL CUSTBAL > 200,000 3 Cust_Type = 3 WHEN CUSTCDE CUSTCDE = 1 or 2 AND CUSTBAL <= 200,000 AND CUSTBAL CUSTBAL > 75,000 3 GL_CUST High cardinality field GL_CUST 3 GL_CUST High cardinality field GL_CUST 3 GL_CUST GL_CUST High cardinality field 4 Cust_Type = 4 WHEN CUSTCDE CUSTCDE = 0 AND CUSTBAL < 75,000 AND CUSTBAL Credit_Rate < 22 Credit_Rate 3 GL_CUST GL_CUST High cardinality field GL_CUST High cardinality field Cust_Card_Ind Cust_Card_Ind is the attribute of record for BigCity Bank customer s that have a BCB credit card. Cust_Tbl 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 68

69 Managed Meta Data Environment 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 69

70 Managed Meta Data Environment When implementing a meta data management system there is a lot more to it than just a meta data repository Important: The MME is an operational system, not a data warehouse 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 70

71 Managed Meta Data Environment Managed Meta Data Environment (MME): The managed meta data environment represents the architectural components, people and processes that are required to properly and systematically gather, retain and disseminate meta data throughout the enterprise Enterprise Warehousing Solutions, Inc. (EWSolutions) 71

72 Meta Data Sourcing Layer Software Tools Documents/ Spreadsheets Messaging/Transactions (EAI, web services, XML, etc.) Meta Data Extract Meta Data Extract Meta Data Extract Applications (CRM, ERP, data warehouses, etc.) Websites/ E-Commerce Meta Data Extract Meta Data Extract Third Parties (business partners, vendors, customers, government agencies) Managed Meta Data Environment Meta Data Extract M e t a D a t a I n t e g r a t i o n L a y e r Meta Data Management Layer Meta Data Repository End Users (business and technical) Data Warehouse/ Data Mart(s) Applications (CRM, ERP, etc.) Meta Data Marts Meta Data Delivery Layer Websites/E-Commerce Third Parties (vendors, customers, government agencies) Messaging/Transactions (EAI, web services, XML, etc.) Business Users End Users (business and technical) End Users (business and technical) Software Tools 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 72

73 Meta Data Management in Governance Stewards manage data (instances of data values) and meta data (information concerning the data) Meta data management is the key technical enabler for the performance of successful governance Difficult to do governance successfully without managed meta data environment 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 73

74 Meta Data Subject Areas Enterprise Architecture Data Governance Policies, Standards, Programs, Roles, Organizations, Stewardship Assignments Data Models & Business Definitions Entities, Attributes, Relationships and Rules, Business Definitions Process Models Functions, Activities, Roles, Inputs/Outputs, Workflow, Business Rules System Portfolio & IT Governance Databases, Applications, Projects, Integration Roadmap Business Architecture Organizations, Strategies, Alignment, Performance Reference Data Values Internal & External Codes, Domain Values and Meanings Data Structures Files, Tables, Columns, Views, Business Definitions, Indexes, Usage, Performance, Change Management Legacy Systems Understanding VSAM, COBOL, Impact Analysis, Restructuring, Reuse, Componentization System Design & Development Requirements, UML, Java, EJB, Legacy System Wrappers, Code/Component Reuse Service Oriented Architecture (SOA) Web Services, MQ, XML Technologies, Enterprise Service Bus Data Quality Defects, Metrics, KPIs, Ratings Data Integration Sources, Targets, Transformations, Lineage, ETL Performance, EAI, EII, Migration / Conversion Analytics Data, Definitions, Reports, Users, Usage, Performance Data Security Classifications, Users, Groups, Filters, Authentication, Audits, Privacy, Risk Management, Compliance Document Content Management Unstructured Data, Documents, Taxonomies, Ontologies, Name Sets, Legal Discovery, Search Engine Indexes 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 74

75 Managed Meta Data Environment ROI We Build Systems To Manage Every Aspect Of Our Business, Except One To Manage The Systems Themselves. A Managed Meta Data Environment Is A System That Manages Our Systems Enterprise Warehousing Solutions, Inc. (EWSolutions) 75

76 Understanding the Big Picture 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 76

77 EIM Functional Framework Data Stewardship Information Quality Management Reference & Master Data Management Data Governance Data Architecture Data Warehousing & Business Intelligence Meta Data Management Database Management Information Security Management Unstructured Data Management EIM Services Reference & Master Data Transaction Data Business Intelligence Data 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) EWolutions All Rights Reserved

78 The DAMA-DMBOK Framework Version 3 Data Quality Management Data Architecture Management Data Development Meta Data Management Document & Content Management Data Governance Database Operations Management Data Security Management Data Warehousing & Business Intelligence Management Reference & Master Data Management 2008 DAMA International Enterprise Warehousing Solutions, Inc. (EWSolutions) 78

79 EIM Maturity Model Redundant, undocumented data. Disparate databases without architecture. Little or no business meta data. Diverging semantics. Minimal data integration.. Minimal data cleansing. Dependent on a few skilled individuals. Responsibilities assigned across separate IT groups. Few defined IT roles. Some commonly used approaches but with no commitment to their use. Some management awareness, but no enterprise-wide buy-in. Little or no business involvement, no defined business roles. General purpose tools used as point solutions. Reactive monitoring and problem solving. Data regarded as a minor by-product of business activity, with no estimated business impact. Level 1 Informal Processes Growing intuitive executive awareness of the value of data assets in some business areas. Initial forays in data stewardship and governance but roles are unclear and not ongoing. Initial efforts to implement enterprise-wide management, but with contention across groups with differing perspectives. New skills requirements are recognized and addressed with training. Enterprise architecture and MME projects underway. Data Distribution Services are deployed as strategic solutions Some processes are repeatable. Level 2 Emerging Processes Active executive Involvement across the enterprise. Ongoing, clearly defined business data stewardship. Central EDM organization. Standard processes, metrics, and tools used enterprise wide. Enterprise data architecture guides implementations. Centralized meta data management. Quality SLA s are defined and monitored regularly. Commitment to continual skills development. Periodic audits and proactive monitoring. Level 3 Engineered Processes Measurable process goals are established for each defined process Measurements are collected and analyzed. Quantitative (measurement) analysis of each process occurs Beginning to predict future performance Defects are proactively identified and corrected. Level 4 Controlled Processes Quantitative and qualitative understanding used to continually improve each process. Value is monitored continuously. Understanding of how each process contributes to the business strategies and goals of the enterprise. Level 5 Optimizing Processes 2010 EWSolutions All Rights Reserved 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 79

80 Conclusion 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 80

81 Conclusion Data & Process Management are enterprise activities that address the structured management of our informational systems Data governance & Meta Data Management are foundational activities Master Data Management 360 degree view of customer Cross sell These disciplines are not easy but what enterprise activity is? Focus on the best practices Stay focused You WILL have a big success!! 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 81

82 Questions 2011 Enterprise Warehousing Solutions, Inc. (EWSolutions) 82