Building a Custom Data Warehouse Tom Connolly, BizTech Session #11976
Agenda Presentation Overview Project Methodology for the DDW Phase 1 Project Definition (Planning) Phase 2 Development Phase 3 Operational Support & Business Validation Concluding Remarks
Presentation Overview Tom Connolly, Partner, Director of Technical Services Professional Summary Engineering degrees from Notre Dame and Villanova Oracle Technologist (since 1989) 1 real job managing a large government data center Two years with Coopers & Lybrand Adjunct Professor at Villanova, Babson Fifteen years as Entrepreneur / Consultant / Executive Qualification Summary Oracle Consulting Project Manager with 25 years of experience as Applied Technologist with Oracle technologies including EBS, Database Admin, Business Intelligence and java development Experienced in all phases of the system development lifecycle, Project management and implementation methodologies Industry experience in professional services, business services, government contracting, higher education, telecommunication, nonprofit, and manufacturing
Presentation Overview About BizTech Leading Regional IT Services firm focused exclusively on Oracle applications and technology solutions Oracle Platinum Partner Over 400 successful Oracle implementations over the past 15 years Active in regional and national Oracle and industry conferences Comprehensive Service Offerings Advisory Services Oracle E-Business Applications Services Oracle Technology Services (Business Intelligence and EPM) Cloud Hosting and Managed Services Nine BizTech consultants presenting at Collaborate!
Presentation Overview Sanity Check why are we here discussing data warehousing? Anyone? Typically, organizations develop a Dimensional Data Warehouse (DDW) as a foundation for Understanding Complex Business Activities and, ultimately, Improving Enterprise Performance (through actionable business intelligence )
Presentation Overview Business Intelligence Concepts and methods to improve business decision making by using factbased support systems Analytical vs. Operational Reporting Operational Reporting: Detail oriented, point-in-time view of data (e.g. What are the open orders) Analytical Reporting (BI): Intended to provide business insight by looking at data in aggregate or over time (e.g., which customers are generating the most orders?)
Presentation Overview Why Business Intelligence?
Presentation Overview But beware, a little Business Intelligence, A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: for shallow draughts intoxicate the brain, but drinking deeply sobers one again* Meaning - A small amount of knowledge can cause people to think they are more expert than they really are. Or, in the data warehousing realm, complex systems supporting complex business operations require more than a little learning to derive insight *Origin - First used by Alexander Pope (1688-1744) in An Essay on Criticism, 1709:
Presentation Overview BI Maturity Model Opportunistic Tactical Strategic Type A Type B Type C Focused: Process efficiency or cost reduction Scope: Department Reporting Operational: Improve business effectiveness Scope: Multidepartment CPM (dashboards, KPIs) Strategic: Integrated business execution and management Scope: Enterprise, Partners, Customers Cross-business analytics Pervasive: Agility with change and innovation Scope: Users, Executives, Enterprises, Departments, Partners, Customers Available in a wide range of applications, tools, mediums Measure Optimize Decide Manage Lead Discover Innovate Increasing business value Source: Gartner, Business Intelligence Scenarios: Pervasive BI, Gartner Symposium ITxpo 2006
Presentation Overview More simply, the goal of a dimensional data warehouse is to provide answers to critical business questions Which business unit or location had the greatest increase in clients last week? Last quarter? How many major reportable events were recorded last quarter verses the same quarter last year? Who are our top suppliers overall? By supplier classification? What is the distribution of treatment service types by center? By diagnostic category? What if our fund raising group was doubled in size? When (during which periods) did our Length of Stay trend increase at a faster rate than our headcount? Where do our clients reside (distribution by state)?
Presentation Overview What is a Data Warehouse? Simple perspective, three components Source Systems (operational applications) Information Storage Area (the warehouse) Reporting Marts Also, three stages in the Information Life Cycle A B C
Presentation Overview What about that word Dimensional? A dimensional data warehouse is one which organizes data into Facts and Dimensions Facts quantifiable data elements, or measures, indicative of specific transactions or events (i.e. dollars, quantities, counts) For example, product sales, quantity shipped, # service requests Dimensions descriptive data elements or attributes associated with the data measures (who, when, where, ) For example, sales by customer, by quarter, by region Actually, all data warehouses are dimensional (or they should be) but it is helpful to explicitly acknowledge this as a key factor in organizing the data Facts and dimensions are stored as tables within the warehouse and are ready-made for analytical reporting
Presentation Overview Data within the dimensional warehouse is typically depicted in a diagram called the star schema Customer Time Sales Location Product Let s take another look at how the warehouse is assembled
DW Architecture Source Applications Transactional Data Data Warehouse DW ETL Data Marts Reporting Business Intelligence Analytical Data Business Areas Legal Ext. Affairs Center Ops, HR Finance, IT Clinical
DW Architecture Source Applications Data Marts Reporting Business Areas Marketing Contracts Clinical Billing HR Finance DW Client Fiscal Staff Business Intelligence Legal Ext. Affairs Center Ops, HR Finance, IT Clinical 1 1 2 1 2 3 2 2 3 Risk Mgmt Quality 1 2 1 2 ETL 3 3 3 Transactional Data Analytical Data
Presentation Overview Seems complex How to get started? How to ensure success? Start with a Plan Consider purchasing a pre-built warehouse or reference model Drive development with a proven methodology Hint today s presentation Let s take a look at how Oracle has matured its reporting strategy over time
Presentation Overview Oracle Legacy 2003 1999 1994 5 3 2 Fusion Intelligence (OBI Dashboards) DBI Dashboards Discoverer Reports 2003 2005 4 6 Oracle EBS AP DBI OBIA Dashboards EPM Applications Oracle Reports GL Disco EUL 1 1990 AR Enterprise Data Warehouse Hyperion Essbase 7 Latest: Fusion Apps w/ OBIA and OTBI 2010
Presentation Overview Pre-built Warehouse (OBIA assets) 1 Pre-built warehouse with 16 star-schemas designed for analysis and reporting on financial analytics 3 Pre-mapped metadata, including embedded best practice calculations and metrics for financial, executives and other business users Presentation layer Logical business model Physical sources 2 Pre-built ETL to extract data from over 3,000 operational tables and load it into the DW, sourced from SAP, PSFT, Oracle EBS and other sources 4 A best practice library of over 360 pre-built metrics, 30 intelligent dashboards, 200+ reports and several alerts for CFO, Finance Controller, Financial Analyst, AR/AP Managers and Executives
Presentation Overview But, this is a presentation on Custom Data Warehousing. Why talk about the pre-built warehouse? OBIA follows standard Warehousing design and serves as a very good model to follow when building a custom warehouse Can also serve as a starting point, from which an organization can then extend to suit business needs But, regardless of whether you start with a pre-built model or develop from scratch, it is important to follow a proven methodology
Agenda Presentation Overview Project Methodology for the DDW Phase 1 Project Definition (Planning) Phase 2 Development Phase 3 Operational Support & Business Validation Concluding Remarks
Phase 1 Project Definition How do you eat a 2,000 pound elephant? One bite at a time How do we build out the DDW architecture complete with dimensional model, ETL programs, data marts, and BI dashboards? One subject area, one star schema, one source system ETL, and one dashboard at a time The Project Definition phase is focused on identification and prioritization of mini project packets bite size pieces of the DDW mapped to specific business objectives The project team can then iterate through one project packet at a time in an agile manner Larger project teams can tackle overlapping project packets to achieve more aggressive business objectives in a consistent manner with confidence
Phase 1 Project Definition ID Name 1 Project Definition 2 Determine Business Information Needs (BIN) 3 Research 4 Strategic objectives, CSF's, metrics, subject areas, business processes, systems 5 Develop Information Packets 6 Metrics, analytical questions, subject areas, facts, dimensions, hierarchies 7 Describe expected audience and uses for this information 8 Include discussion of roll-ups, summaries and drill-downs 9 Impact Analysis 10 Business Impact 11 Technical impact (to existing DW or other systems) 12 Document high-level data model 13 Likely data source (i.e. legacy app, access database, excel, third-party provider) 14 Data Issues (missing, incomplete, inconsistent, or inaccurate data) 15 Map out intended Data Lineage (source-->ods-->ddw-->mart-->dashboard) 16 Feedback to PMO for cross project coordination
Phase 2 Development Three stages in the Development Life Cycle Cycle A Operations to Warehouse Cycle B Warehouse to Data Mart Cycle C Business Delivery A B C
Phase 2 Development ID Name 17 Development Cycle A - Operations to W arehouse or ODS 18 Analysis 19 Confirm data requirements with Business lead 20 Analysis of Data Architecture (DataWarehouse Reqts) 21 Identify Source (Application) Tables 22 Design Database Changes For Source Application(s) (if necessary) 23 Identify Target (Warehouse/ODS) Tables and transformations 24 Review/Approve DWR; update operational data dictionary 25 Technical System Design (TSD) 26 Update Warehouse Data Model (WDM) 27 Design/Map ETL Program(s) - extraction, staging and required data transformations 28 Conduct Design Reviews 29 Build & Test 30 Develop ETL programs and operational controls for deployment 31 Validate data mappings 32 Deploy 33 Populate the Warehouse 34 Provide Feedback on Operational Systems
Phase 2 Development ID Name 35 Development Cycle B - W arehouse to Data Mart 36 Analysis 37 Determine DataMart Requirements (similar to DWR) 38 Review/Approve DMR 39 Technical System Design (TSD) 40 Design/Update Dimensional Data Model (DDM) 41 Design Data Mart Load Program(s) 42 Build & Test 43 Develop ETL programs and/or metadata logic 44 Validate 45 Deploy 46 Populate the Data Mart Cube 47 Provide Feedback on Warehouse
Phase 2 Development ID Name 48 Development Cycle C - Business Delivery 49 Analysis 50 Select/Expand Sample Data 51 Identify/Update Derived Data 52 Analytical Requirements 53 Requirements Review 54 Prototyping / Design 55 Analyze Data / Prototype 56 Prototype Review/Approval 57 Build & Test 58 Build or extend dashboards 59 Validate BI Reports 60 Deploy 61 Setup End-user roles and enable access 62 Provide Feedback
Phase 3 Op Suppt, Validation ID Name 63 Operate the W arehouse 64 Develop/Update Warehouse/ODS Operations Infrastructure 65 Develop/Update Data Mart Operations Infrastructure 66 Publish Warehouse Operational Reports (Periodically) 67 Collect Feedback on Analytical Components & Assess Business Impact Ongoing support requires a skilled team
Project Team Roles End User SOURCE SYSTEMS OLTP & ODS (Oracle, SAP, Others) Technical ETL Developer OWB ODI Informatica DAC Scheduler ETL Options Oracle BI Admin Server RPD Metadata Functional Business Analyst Oracle BI Admin Server Oracle BI Presentation Server BI Publisher Dashboards and Reports XML/Office ETL Repository Physical Business Presentation Answers (Adhoc) Technical Dashboard Developer MS Integration DATA WAREHOUSE Technical Data Architect
Agenda Presentation Overview Project Methodology Phase 1 Project Definition (Planning) Phase 2 Development Phase 3 Operational Support & Business Validation Concluding Remarks
Concluding Remarks The Dimensional Data Warehouse represents an important component in an overall reporting strategy BI BI Apps EPM ERP Database Operational Data Store Data Warehouse Essbase Nightly ETL Data Flow Reporting Access
Questions? Comments?
THANK YOU Tom Connolly, Tconnolly@BizTech.com