Data Warehousing Fundamentals

Transcription

1 Data Warehousing Fundamentals Volume 1 Student Guide GC20 Production 2.0 May 1999 M08761

2 Authors Chon S. Chua Richard Green Technical Contributors and Reviewers Jackie Collins Jennifer Jacoby Mike Schmitz John Haydu Russ Pitts Lauran Serhal Brian Pottle Donna Corrigan Patricia Moll Harry Penbert SuiWah Chan Joel Barkin Steve Dressler Publisher Tony McGettigan This documentation contains proprietary information of Oracle Corporation. It is provided under a license agreement containing restrictions on use and disclosure and is also protected by copyright law. Reverse engineering of the software is prohibited. If this documentation is delivered to a U.S. Government Agency of the Department of Defense, then it is delivered with Restricted Rights and the following legend is applicable: Restricted Rights Legend Use, duplication or disclosure by the Government is subject to restrictions for commercial computer software and shall be deemed to be Restricted Rights software under Federal law, as set forth in subparagraph (c) (1) (ii) of DFARS , Rights in Technical Data and Computer Software (October 1988). This material or any portion of it may not be copied in any form or by any means without the express prior written permission of Oracle Corporation. Any other copying is a violation of copyright law and may result in civil and/or criminal penalties. If this documentation is delivered to a U.S. Government Agency not within the Department of Defense, then it is delivered with Restricted Rights, as defined in FAR , Rights in Data-General, including Alternate III (June 1987). The information in this document is subject to change without notice. If you find any problems in the documentation, please report them in writing to Education Products, Oracle Corporation, 500 Oracle Parkway, Box SB-6, Redwood Shores, CA Oracle Corporation does not warrant that this document is error-free. Data Warehouse Method A Methodology for Designing Data Warehouse, SQL*Loader, PL/SQL, Pro*C, Oracle7, Oracle8, and Oracle8i, Distributed Option, Parallel Query Option, Parallel Server Option, Media Server, Spatial Data Option, ConText Option, Video Server, Text Server, WebServer, Oracle Universal Server ROLAP Option, Express Server, Web-enabled Express Server, SQL*Net, Developer/2000, Relational Access Manager, Discoverer, Designer/2000, SQL*Bridge, Transparent Gateway Developer s Kit, Procedural Gateway Developer s Kit, Express, Express Analyzer, Express Objects, Sales Analyzer, and Financial Analyzer are product names, trademarks, or registered trademarks of Oracle Corporation. All other products or company names are used for identification purposes only and may be trademarks of their respective owners.

3 Contents Preface Profile xi Related Publications xiv Typographic Conventions xv Lesson 1: Introduction Course Objectives 1-3 Agenda 1-5 Questions About You 1-9 Lesson 2: Meeting a Business Need Overview 2-3 Unsuitability of OLTP Systems for Complex Analysis 2-5 Management Information Systems and Decision Support 2-7 Data Extract Processing 2-9 Business Drivers for Data Warehouses 2-15 Current Situation and Growth of Data Warehousing 2-19 Typical Uses of a Data Warehouse 2-21 Summary 2-23 Practice Lesson 3: Defining Data Warehouse Concepts and Terminology Overview 3-3 Data Warehouse Definition 3-5 Data Warehouse Properties 3-7 Data Warehouse Terminology 3-21 Components of a Data Warehouse 3-25 Oracle Warehouse Vision, Products, and Services 3-31 Summary 3-41 Practice Lesson 4: Driving Implementation Through a Methodology Overview 4-3 Warehouse Development Approaches 4-5 The Need for an Iterative and Incremental Methodology 4-13 Data Warehousing Fundamentals iii

4 Contents Oracle Data Warehouse Method 4-15 DWM Fundamental Elements 4-19 Oracle Warehouse Technology Initiative (WTI) 4-57 Summary 4-61 Practice Lesson 5: Planning for a Successful Warehouse Overview 5-3 Managing Financial Issues 5-5 Obtaining Business Commitment 5-9 Managing a Warehouse Project 5-15 Identifying Planning Phases 5-29 Identifying Warehouse Strategy Phase Deliverables 5-31 Identifying Project Scope Phase Deliverables 5-35 Summary 5-41 Practice Lesson 6: Analyzing User Query Needs Overview 6-3 Types of Users 6-5 Gathering User Requirements 6-7 Managing User Data Access 6-9 Security 6-21 OLAP 6-25 Query Access Architectures 6-47 Summary 6-51 Practice Lesson 7: Modeling the Data Warehouse Overview 7-3 Data Warehouse Database Design Phases 7-5 Phase One: Defining the Business Model 7-7 Phase Two: Creating the Dimensional Model 7-17 Data Modeling Tools 7-39 iv Data Warehousing Fundamentals

5 Contents Summary 7-41 Practice Lesson 8: Choosing a Computing Architecture Overview 8-3 Architecture Requirements 8-5 The Hardware Architecture 8-7 Database Server Requirements 8-29 Parallel Processing 8-33 Summary 8-39 Practice Lesson 9: Planning Warehouse Storage Overview 9-3 The Server Data Architecture 9-5 Protecting the Database 9-17 Summary 9-27 Practice Lesson 10: Building the Warehouse Overview 10-3 Extracting, Transforming, and Transporting Data 10-5 Extracting Data Examining Data Sources Extraction Techniques Extraction Tools Summary Practice Lesson 11: Transforming Data Overview 11-3 Importance of Data Quality 11-5 Transformation Transforming Data: Problems and Solutions Transformation Techniques Data Warehousing Fundamentals v

6 Contents Transformation Tools Summary Practice Lesson 12: Transportation: Loading Warehouse Data Overview 12-3 Transporting Data into the Warehouse 12-5 Building the Transportation Process Transporting the Data Postprocessing of Loaded Data Summary Practice Lesson 13: Transportation: Refreshing Warehouse Data Overview 13-3 Capturing Changed Data 13-5 Limitations of Methods for Applying Changes Purging and Archiving Data Final Tasks Selecting ETT Tools Summary Practice Lesson 14: Leaving a Metadata Trail Overview 14-3 Defining Warehouse Metadata 14-5 Developing a Metadata Strategy Examining Types of Metadata Metadata Management Tools Common Warehouse Metadata Summary Practice Lesson 15: Supporting End-User Access Overview 15-3 vi Data Warehousing Fundamentals

7 Contents Business Intelligence 15-5 Multidimensional Query Techniques 15-7 Categories of Business Intelligence Tools 15-9 Data Mining in a Warehouse Environment Oracle Data Mining Partners Summary Practice Lesson 16: Web-Enabling the Warehouse Overview 16-3 Accessing the Warehouse Over the Web 16-5 Common Web Data Warehouse Architecture 16-9 Issues in Deploying a Data Warehouse on the Web Evaluating Web-Based Tools Summary Practice Lesson 17: Managing the Data Warehouse Overview 17-3 Managing the Transition to Production 17-5 Managing Growth Managing Backup and Recovery Identifying Data Warehouse Performance Issues Summary Appendix A: Practice Solutions Practice 2-1 A-2 Practice 3-1 A-4 Practice 4-1 A-7 Practice 5-1 A-11 Practice 6-1 A-12 Practice 7-1 A-13 Practice 8-1 A-14 Practice 9-1 A-15 Data Warehousing Fundamentals vii

8 Contents Glossary Practice 10-1 A-18 Practice 11-1 A-20 Practice 12-1 A-21 Practice 13-1 A-23 Practice 14-1 A-24 Practice 15-1 A-26 Practice 16-1 A-28 viii Data Warehousing Fundamentals

9 Preface...

10

11 Profile Profile Before You Begin This Course This course is the entry-level course in the Data Warehousing curriculum. Therefore, there are no prerequisites to this course. Prerequisites There are no prerequisites for this course. How This Course Is Organized Data Warehousing Fundamentals is an instructor-led course featuring lecture and paper and pencil exercises as well as group discussions to reinforce the concepts and skills introduced. Lesson Aim Lesson 1: Introduction Lesson 2: Meeting a Business Need Lesson 3: Defining Data Warehouse Concepts and Terminology Lesson 4: Driving Implementation Through a Methodology Lesson 5: Planning for a Successful Warehouse In this lesson, the class format is reviewed, the class agenda is described, and students introduce themselves. Because this class is expected to appeal to a broad audience, the introduction will give the instructor an idea of the composition of the class in terms of data warehouse knowledge, Oracle knowledge, and the specific role that each student plays with regard to data warehousing. This lesson examines how data warehousing has evolved from early management information systems to today s decision support systems. The primary motivating factors for data warehouse creation are explored. The types of industries employing data warehouse are considered. This lesson introduces the Oracle definition of a data warehouse. The lesson offers a general description of the properties of a data warehouse. The standard components and tools required to build, operate, and use a data warehouse are identified. This lesson introduces the Oracle Data Warehouse Method (DWM), a methodology employed by Oracle Consulting Services for incremental development of a total warehouse solution using a phased development approach. Partnering initiatives launched by Oracle are described. This lesson introduces the planning that is critical to the success of a data warehouse project. Planning phases, deliverables, and project roles are identified. Overall warehouse strategy and project scope are defined. Data Warehousing Fundamentals xi

12 Preface Lesson Lesson 6: Analyzing User Query Needs Lesson 7: Modeling the Data Warehouse Lesson 8: Choosing a Computing Architecture Lesson 9: Planning Warehouse Storage Lesson 10: Building the Warehouse Lesson 11: Transforming Data Lesson 12: Transportation: Loading Warehouse Data Lesson 13: Transportation: Refreshing Warehouse Data Aim This lesson identifies the analysis required to identify and categorize users that may need to access data from the warehouse, and how their requirements differ. Data access and reporting tools are considered. This lesson examines the role of data modeling in a data warehousing environment. The lesson presents a very high level overview of warehouse modeling steps. You consider the different types of models that can be employed, such as the star schema. Tools available for warehouse modeling are introduced. This lesson examines the computer architectures that commonly support data warehouses. The benefits of each hardware architecture and reasons for using distributed warehouses are examined. Students examine the technology requirements of a database server for warehousing. This lesson examines the database setup and management issues such as partitioning, indexing, and ways to protect your database. In this lesson, you explore the sources of data for the data warehouse data. You consider how the extraction and transformation processes take data from source systems and change it into data that is acceptable to the users of the data warehouse. The lesson also describes typical data anomalies and looks at ways to eliminate them. In this lesson, you explore how the transformation process transforms data from source systems into data suitable for end user query and analysis applications. In this lesson, you examine how the extracted and transformed data is transported into the warehouse. In this lesson, you examine methods for updating the warehouse with changed data, after the first-time load. xii Data Warehousing Fundamentals

13 Profile Lesson Lesson 14: Leaving a Metadata Trail Lesson 15: Supporting End-User Access Lesson 16: Web- Enabling the Warehouse Lesson 17: Managing the Data Warehouse Aim This lesson focuses on the concept of warehouse metadata, and the role it plays in a well-developed and managed warehousing environment. This lesson investigates the ways that users may access the data in the data warehouse. Students are introduced to the concept of business intelligence. The lesson discusses the discovery model used by mining tools, and the reasons enterprises are looking at data mining solutions for discovery of information. This lesson discusses how to take advantage of the Web to deploy data warehouse information. It addresses internal and external access, as well as the advantages of Web-enabling a data warehouse. The lesson outlines the steps involved in deploying a Web-enabled data warehouse. Challenges in deploying a Webenabled data warehouse are also discussed. This lesson explores the management issues, critical success factors, and challenges to successful data warehouse implementation. The lesson addresses issues pertaining to the management of the entire warehouse life cycle. Data Warehousing Fundamentals xiii

14 Preface Related Publications Oracle Publications Title Oracle8i for Data Warehousing: Fast and Simple for More Data and More Users (Nov 1998) Large Scale Data Warehousing with Oracle8i, Winter Corporation Sponsored Research Program DWM Handbook V1.0.0 URL websight.us.oracle.com websight.us.oracle.com Additional Publications Oracle DBA Handbook, Loney, Kevin, Osborne McGraw-Hill; ISBN: Oracle: The Complete Reference, Koch, George and Kevin Loney; Oracle Press; ISBN: X. The Data Warehouse Toolkit, Kimball, Ralph; John Wiley & Sons; ISBN: Building the Data Warehouse, Inmon, W.; John Wiley & Sons; ISBN: Oracle8 Data Warehousing, Dodge, Gary and Gorman, T.; John Wiley & Sons; ISBN: The Data Warehouse Lifecycle Toolkit: Expert Methods for Designing, Developing, and Deploying Data Warehouses, Kimball, Ralph and others; John Wiley & Sons, 1998; ISBN: Data Warehouse Design Solutions, Adamson, C. and Venerable, M.; John Wiley & Sons, 1998; ISBN X. Data Warehousing:Architecture and Implementation, Humphries, M. et. al., Prentice Hall PTR, 1999; ISBN: Web Sites Data Warehouse Institute Web site, at index.htm The Data Warehouse Information Center Web site, at pwp.starnetinc.com/larryg/index.html The Data Warehouse.com Web site, at The Data Warehouse Knowledge Center Web site, at xiv Data Warehousing Fundamentals

15 Typographic Conventions Typographic Conventions Typographic Conventions in Text Convention Element Example Bold italic Glossary term (if The algorithm inserts the new key. there is a glossary) Caps and lowercase Buttons, Click the Executable button. check boxes, Select the Can t Delete Card check box. triggers, Assign a When-Validate-Item trigger... windows Open the Master Schedule window. Courier new, Code output, Code output: debug.seti( I,300); case sensitive directory names, Directory: bin (DOS), $FMHOME (UNIX) (default is filenames, Filename: Locate the init.ora file. lowercase) passwords, pathnames, Password: Use tiger as your password. URLs, user input, usernames Pathname: Open c:\my_docs\projects URL: Go to User input: Enter 300 Initial cap Italic Quotation marks Uppercase Graphics labels (unless the term is a proper noun) Emphasized words and phrases, titles of books and courses, variables Interface elements with long names that have only initial caps; lesson and chapter titles in cross-references SQL column names, commands, functions, schemas, table names Username: Log on as scott Customer address (but Oracle Payables) Do not save changes to the database. For further information, see Oracle7 Server SQL Language Reference Manual. Enter user_id@us.oracle.com, where user_id is the name of the user. Select Include a reusable module component and click Finish. This subject is covered in Unit II, Lesson 3, Working with Objects. Use the SELECT command to view information stored in the LAST_NAME column of the EMP table. Data Warehousing Fundamentals xv

16 Preface Convention Element Example Arrow Menu paths Select File >Save. Brackets Key names Press [Enter]. Commas Key sequences Press and release these keys one at a time: [Alt], [F], [D] Plus signs Key combinations Press and hold these keys simultaneously: [Ctrl]+[Alt]+[Del] Typographic Conventions in Code Convention Element Example Caps and lowercase Oracle Forms When-Validate-Item triggers Lowercase Column names, table names SELECT last_name FROM s_emp; Passwords DROP USER scott IDENTIFIED BY tiger; PL/SQL objects OG_ACTIVATE_LAYER (OG_GET_LAYER ( prod_pie_layer )) Lowercase italic Syntax variables CREATE ROLE role Uppercase SQL commands and functions SELECT userid FROM emp; Typographic Conventions in Navigation Paths This course uses simplified navigation paths, such as the following example, to direct you through Oracle Applications. (N) Invoice >Entry >Invoice Batches Summary (M) Query >Find (B) Approve This simplified path translates to the following: 1 (N) From the Navigator window, select Invoice >Entry >Invoice Batches Summary. 2 (M) From the menu bar, select Query >Find. 3 (B) Click the Approve button. N = Navigator, M = Menu, B = Button xvi Data Warehousing Fundamentals

17 1... Introduction

18 Lesson 1: Introduction Course Objectives After completing this course, you should be able to do the following: Explain why data warehousing is a popular solution Describe data warehousing terminology Identify components of an implementation Explain the important of employing a method Identify modeling concepts Identify the management and maintenance processes Course Objectives Identify the hardware platforms that can be employed with a data warehouse Identify the features of the database server Identify tools that can be employed at each stage Describe user profiles and techniques for querying the warehouse Identify data warehouse implementation issues and challenges Position the products for the Oracle warehouse 1-2 Data Warehousing Fundamentals

19 Course Objectives Course Objectives After completing this course, you should be able to the following: Explain why data warehousing is a popular solution in today s information technology environment Describe the terminology used with data warehousing Identify the standard components of a data warehouse implementation Explain the importance of using a methodology for development, and specifically identify the phases of the Oracle Data Warehouse Method Identify and use data warehouse modeling concepts Identify the different processes required to manage and maintain the warehouse Identify the hardware platforms that can be employed with a data warehouse Identify the features required of a database server for a warehouse implementation Identify the tools that can be used at each phase during the data warehouse development cycle Describe user profiles and the techniques users may employ for querying the warehouse Identify data warehousing implementation issues and challenges Position the products for the Oracle warehouse Data Warehousing Fundamentals 1-3

20 Lesson 1: Introduction Data Warehousing Fundamentals Day 1 Lesson 1 Introduction Lesson 2 Meeting a Business Need Lesson 3 Defining Data Warehouse Concepts and Terminology Lesson 4 Driving Implementation Through a Methodology Lesson 5 Planning for a Successful Warehouse Lesson 6 Analyzing User Query Needs Data Warehousing Fundamentals Day 2 Lesson 7 Modeling the Data Warehouse Lesson 8 Choosing a Computing Architecture Lesson 9 Planning Warehouse Storage Lesson 10 Building the Warehouse Lesson 11 Transforming Data Lesson 12 Transportation: Loading Warehouse Data 1-4 Data Warehousing Fundamentals

21 Agenda Agenda Day 1 Lesson 1: Introduction Lesson 2: Meeting a Business Need Lesson 3: Defining Data Warehouse Concepts and Terminology Lesson 4: Driving Implementation Through a Methodology Lesson 5: Planning for a Successful Warehouse Lesson 6: Analyzing User Query Needs Day 2 Lesson 7: Modeling the Data Warehouse Lesson 8: Choosing a Computing Architecture Lesson 9: Planning Warehouse Storage Lesson 10: Building the Warehouse Lesson 11: Transforming Data Lesson 12: Transportation: Loading Warehouse Data Data Warehousing Fundamentals 1-5

22 Lesson 1: Introduction Data Warehousing Fundamentals Day 3 Lesson 13 Transportation: Refreshing Warehouse Data Lesson 14 Leaving a Metadata Trail Lesson 15 Supporting End-User Access Lesson 16 Web-Enabling the Warehouse Lesson 17 Managing the Data Warehouse 1-6 Data Warehousing Fundamentals

23 Agenda Day 3 Lesson 13: Transportation: Refreshing Warehouse Data Lesson 14: Leaving a Metadata Trail Lesson 15: Supporting End-User Access Lesson 16: Web-Enabling the Warehouse Lesson 17: Managing the Data Warehouse Data Warehousing Fundamentals 1-7

24 Lesson 1: Introduction Questions About You To tailor the class to your specific needs and to encourage dialog among all, please answer the following questions: What is your name and company? What is your role in your organization? What is your level of Oracle expertise? Why are you building a data warehouse or data mart? What do you hope to get out of this class? 1-8 Data Warehousing Fundamentals

25 Questions About You Questions About You You will get a lot more out of this class if you are aware of the background of your classmates and the issues that they face in the development of a data warehouse. Each student has a unique perspective and an experience and knowledge set from which we can learn. Because this class is expected to appeal to a broad audience, the introduction will give the instructor an idea of the composition of the class in terms of data warehouse knowledge, Oracle knowledge, and the specific role that each student plays with regard to data warehousing. Data Warehousing Fundamentals 1-9

26 Lesson 1: Introduction 1-10 Data Warehousing Fundamentals

27 2... Meeting a Business Need

28 Lesson 2: Meeting a Business Need Overview Defining DW Concepts & Terminology Choosing a Computing Architecture Planning Warehouse Storage Planning for a Successful Warehouse Meeting a Business Need Modeling the Data Warehouse ETT (Building the Warehouse) Managing the Data Warehouse Analyzing User Query Needs Supporting End User Access Project Management (Methodology, Maintaining Metadata) Objectives After completing this lesson, you should be able to do the following: Describe why an online transaction processing (OLTP) system is not suitable for complex analysis Describe how extract processing for decision support querying led to data warehouse solutions employed today Explain why businesses are driven to employ data warehouse technology Identify some of the industries that employ data warehouses 2-2 Data Warehousing Fundamentals

29 Overview Overview The top slide on the facing page is a road map representing the flow of the course. The vertical box entitled Meeting a Business Need emphasizes that the warehouse is business driven. The determination of the warehouse architecture, data model, and user query needs all stem from business requirements. The horizontal box running across the bottom represents the ongoing project management throughout the warehouse lifecycle. This lesson examines how data warehousing has evolved from early management information systems to today s decision support systems. The primary motivating factors for data warehouse creation are explored. The types of industries employing data warehouse are considered. Objectives After completing this lesson, you should be able to do the following: Describe why an online transaction processing (OLTP) system is not suitable for complex analysis Describe how extract processing for decision support querying led to data warehouse solutions employed today Explain why businesses are driven to employ data warehouse technology Identify some of the industries that employ data warehouses Data Warehousing Fundamentals 2-3

30 Lesson 2: Meeting a Business Need Characteristics of OLTP Systems Characteristic Typical operation Level of analytical requirements Screens Amount of data per transaction Data level Age of data Orientation OLTP Update Low Unchanging Small Detailed Current Records Why OLTP Is Not Suitable for Complex Analysis OLTP Information to support day-to-day service Data stored at transaction level Database design: Normalized Complex Analysis Historical information to analyze Data needs to be integrated Database design: Denormalized, star schema 2-4 Data Warehousing Fundamentals

31 Unsuitability of OLTP Systems for Complex Analysis Unsuitability of OLTP Systems for Complex Analysis Operational systems largely exist to support transactions, for example, the booking of an airline ticket. Decision support, which is a type of complex analysis, is very different from OLTP. Most OLTP transactions require a single record in a database to be located and updated or an addition of one or more new records. Even a simple decision support query such as How many luxury cars did we sell in Boston for January 1999 requires very different operations at the database level to an OLTP transaction. A potentially large number of records must be located, and there are no update operations at all. Characteristics of OLTP Systems The characteristics of OLTP systems are described below. Characteristic OLTP Typical operation Update Level of analytical requirements Low Screens Unchanging Amount of data per transaction Small Data level Detailed Age of data Current Orientation Records Why OLTP Is Not Suitable for Complex Analysis OLTP databases are fully normalized and are designed to consistently store operational data, one transaction at a time. Complex analysis, on the other hand, requires database design that even business users find directly usable. To achieve this, a different database design techniques are required, for example the use of dimensional and star schemas with highly denormalized dimension tables. OLTP focuses on recording and completing different types of business transactions but is unable to provide decision makers with the information they need. The data needed for such complex analysis is scattered throughout different OLTP systems and must first be carefully integrated before the information needed can be obtained. Extracting the data from these OLTP systems demands so much of the system resources that the IT professional must wait until nonoperational hours before running the queries required to produce the report. Thus OLTP systems are not suitable for complex analysis because the database design is not optimized to run such queries. Additionally, OLTP systems do not have an integrated pool of data from all the operation systems within the enterprise in order for business users to derive complex analysis. Also, OLTP systems do not store historical data that is needed for complex analysis. Data Warehousing Fundamentals 2-5

32 Lesson 2: Meeting a Business Need Management Information Systems and Decision Support Production platforms Ad hoc access Operational reports Decision makers MIS systems provided business data Reports were developed on request Reports provided little analysis capability Decision support tools gave personal ad hoc access to data Analyzing Data from Operational Systems Data structures are complex Systems are designed for high performance and throughput Data is not meaningfully represented Data is dispersed OLTP systems may be unsuitable for intensive queries Production platforms Operational reports 2-6 Data Warehousing Fundamentals

33 Management Information Systems and Decision Support Management Information Systems and Decision Support Early Management Information Systems Early Management Information Systems (MIS) provided management with reports to assess the performance of the business. Report requirements were submitted as a request to the MIS development team, who developed the report and made it available to the user some time afterward days, weeks, or even months later. The data in the reports was made available in a way that was difficult to use for analysis and forecasting. Personal Computing With the advent of personal computing and 4GL programming techniques, MIS became known as decision support (decision support systems or DSS). DSS was judged to support business users better, by giving them direct access to the operational data for additional ad hoc querying, which provided more flexible reporting as the information was needed. Analyzing Data from Operational Systems Although decision support tools are friendly, intuitive, and easy to use, often the structure of data in the online transaction processing systems does not support the user s real analytical requirements. The structure of the operational data is often complex and too highly structured (3NF). The system was designed for high performance high throughput online transaction processing rather than CPU-intensive analysis of information. The data is not always meaningfully presented to the end user query tool. The same data elements may be defined differently for each operational system. For example, a customer record may hold the customer telephone number. In one system this number is stored as a 15-digit number, and on another as a 20 alphanumeric character value. Data is dispersed on multiple and diverse systems, leading to data redundancy and the inability to coordinate data between systems to provide a global picture of the business. Running online transaction processing and decision support concurrently on one machine degrades performance of the operational system, response time to users, and performance of networks. The overall impact on the operational system may be too great. Data Warehousing Fundamentals 2-7

34 Lesson 2: Meeting a Business Need Data Extract Processing Operational systems Extracts Decision makers End user computing offloaded from the operational environment User s own data Management Issues Operational systems Extracts Decision makers Extract explosion 2-8 Data Warehousing Fundamentals

35 Data Extract Processing Data Extract Processing DSS and Degradation The problem of performance degradation was partially solved by using extract processing techniques, which select data from one environment and transport it to another environment for user access (a data extract). Data Extract Program The data extract program searches through files and databases, gathering data according to specific criteria. The data is then placed into a separate set of files, which may reside on another environment, for use by analysts for decision support activities. Extract processing was a logical progression from decision support systems. It was seen as a way to move the data from the high-performance, high throughput online transaction processing systems onto client machines dedicated to analysis. Extract processing also gave the user ownership of the data. Management Issues with Data Extract Programs Although the principle of extracts appears logical, and to some degree represents a model similar to the way a data warehouse works, there are problems with processing extracts. Extract programs may become the source for other extracts, and extract management can become a full-time task for information systems departments. In some companies hundreds of extract programs are run at any time. Data Warehousing Fundamentals 2-9

36 Lesson 2: Meeting a Business Need Productivity Issues Duplicated effort Multiple technologies Obsolete reports No metadata Data Quality Issues No common time basis Different calculation algorithms Different levels of extraction Different levels of granularity Different data field names Different data field meanings Missing information No data correction rules No drill-down capability 2-10 Data Warehousing Fundamentals

37 Data Extract Processing Data Extract Program (continued) Productivity Issues with Extract Processing The productivity issues in an extract processing environment are listed below: Extract effort is duplicated, because multiple extracts access the same data and use mainframe resources unnecessarily. The program designed to access the extracted data must encompass all technologies employed by the source data. A report cannot always be reused, because business structures change. There is no common metadata providing a standard way of extracting, integrating, and using the data. Data Quality Issues with Extract Processing The data quality issues in an extract processing environment are listed below: The data has no time basis and users cannot compare query results with confidence. The data extracts may have been taken at a different point-in-time. Each data extract may use a different algorithm for calculating derived and computed values. This makes the data difficult to evaluate, compare, and communicate by managers who may not know the methods or algorithms used to create the data extract or reports. Data extract programs may use different levels of extraction. Access to external data may not be consistent, and the granularity of the external data may not be well defined. Data sources may be difficult to identify, and data elements may be repeated on many extracts. The data field names and values may have different meanings in the various systems in the enterprise (lack of semantic integrity). There are no data correction rules to ensure that the extracted data is correct and clean. The reports provide data rather than information, and no drill-down capability. Data Warehousing Fundamentals 2-11

38 Lesson 2: Meeting a Business Need From Extract to Warehouse DSS Internal and external systems Data warehouse Controlled Reliable Quality information Single source of data Decision makers Advantages of Warehouse Processing Environment No duplication of effort No need for tools to support many technologies No disparity in data, meaning, or representation No time period conflict No algorithm confusion No drill-down restrictions 2-12 Data Warehousing Fundamentals

39 Data Extract Processing Transitioning from Extract Processing Environment to Warehouse Processing Environment There was a transition from decision support using data extracts to decision support using the data warehouse. The data warehouse is a complete environment that requires skill, knowledge, and commitment to put together, particularly for the very large scale enterprise implementation. The data warehouse environment is more controlled and therefore more reliable for decision support than an extract environment. The data warehouse environment supports your entire decision support requirements by providing high-quality information, made available by accurate and effective cleansing routines and using consistent and valid data transformation rules and documented presummarization of data values. It contains one single source of accurate, reliable information that can be used for analysis. Advantages of the Warehouse Processing Environment over the Extract Processing Environment The advantages of the warehousing processing environment are listed below: No duplication of effort No need to consider using a query and reporting tool that supports more than one technology No disparity with the data and its meaning No disparity with the way data is represented No conflict over the time periods employed No contention over the algorithms that have been used No restriction on drill-down capabilities Data Warehousing Fundamentals 2-13

40 Lesson 2: Meeting a Business Need Business Motivators Know the business Reinvent to face new challenges Invest in products Invest in customers Retain customers Invest in technology Improve access to business information Be profitable Provide superior services and products Business Motivators Provide supporting information systems Get quality information Reduce costs Streamline the business Improve margins 2-14 Data Warehousing Fundamentals

41 Business Drivers for Data Warehouses Business Drivers for Data Warehouses Businesses in the nineties face challenges such as regulatory control, competition, market maturity, product differentiation, customer behavior, and accelerated product life cycles, all of which require businesses to develop market awareness, responsiveness, adaptability, innovation, efficiency, and quality. Critical Success Factors for a Dynamic Business Environment In order to succeed in an ever-changing business environment a company must: Know both the market they are in and their business (internally and externally). Reinvent themselves to face new challenges. This may be changing product requirements, diverse and effective services, or even changes in internal organizational structures. Invest in research and development of new product channels. Invest in high-value customers who contribute greater returns to the business. Retain existing customers and attract new customers. Invest in new technology to support business needs. Improve access to information so that they can make rapid decisions, based on an accurate picture of the business. Be profitable. At the same time, they must be able to invest in resources for the future, such as technology and people. Provide superior services and products to keep market share and maintain income. Information Needed to Ensure Success To support these strategies, a business needs to have: Access to consistent and high-quality information on the behaviors of the business and the external markets, so that they can constantly monitor the state of the business. Information that can help to reduce costs, streamline the business, and improve margins. Data Warehousing Fundamentals 2-15

42 Lesson 2: Meeting a Business Need Technological Advances Parallelism Hardware Operating system Database Query Index Applications 8i Large databases 64-bit architectures Indexing techniques Affordable, cost-effective open systems Robust warehouse tools Sophisticated end user tools 2-16 Data Warehousing Fundamentals

43 Business Drivers for Data Warehouses Technology Needed to Support the Business Needs Today s information technology climate provides you with cost-effective computing resources in the hardware and software arena, Internet and intranet solutions, and databases that can hold very large volumes of data for analysis, using a multitude of data access technologies. Technological Advances Enabling Data Warehousing Technology (specifically open systems technology) is making it affordable to analyze vast amounts of data, and hardware solutions are now more cost-effective. Parallelism Recent advances in parallelism have benefited all aspects of computing: Hardware environment Operating system environment Database management systems and all associated database operations Query techniques Indexing strategies Applications Other Factors Very large volumes of data can be managed for warehouses greater than one terabyte in size. Recently introduced 64-bit architectures are increasing server capacity and speed. Improved indexing techniques (bitmap index, hash index, star join) provide rapid access to data. Warehouse tools are becoming more robust and less expensive. Licensing strategies are more effective and affordable. Open systems are available. Sophisticated, user-friendly, and intuitive tools are available to the user community for all types of data warehouse access. Data Warehousing Fundamentals 2-17

44 Lesson 2: Meeting a Business Need Current Situation and Growth USA Europe APAC Other Revenue USA Europe APAC Other Projected Growth Current Revenue Growth Motivators and Inhibitors Successful implementations Decreased risk Robust extraction software Improving price to performance ratios Improved staff training Year 2000 compliance Skills shortage Lack of integrated metadata Data cleaning cost 2-18 Data Warehousing Fundamentals

45 Current Situation and Growth of Data Warehousing Current Situation and Growth of Data Warehousing Data warehouses are becoming increasingly popular. The statistics for the estimated growth of data warehousing are compelling. These figures are not specific to Oracle but are industry wide. Revenues A recent report has shown that in 1996 data warehouse revenues (which include hardware, software, and people-provided services) netted $8 billion (US). It is forecast that in 2001 this figure will rise to $23 billion (U.S.), assuming a compound annual growth rate of around 20% per year. Geography Most data warehouse implementations exist in the U.S., with Europe following close behind, and then Asia Pacific. Growth Motivators These include: Increased successful implementations Decreased risk with vendors supplying a total solution More robust and functional extraction software Improved (and improving) price-to-performance equipment ratios Improved training for IT staff Growth Inhibitors These may include: Year 2000 compliance Shortage of skills in specific areas of data warehousing The lack of integrated metadata components The labor-intensive commitment to the data cleaning function and its corresponding dollar and time cost Enterprisewide Implementations and Data Marts Enterprise data warehouses are in position to dominate the business, compared with the smaller data mart implementations that are specific to departments or specific functional requirements. Data Warehousing Fundamentals 2-19

46 Lesson 2: Meeting a Business Need Typical Uses of a Data Warehouse Financial Manufacturing Telecom Retail Others Percentage Market Coverage Airline Banking Health care Investment Insurance Retail Telecommunications Manufacturing Credit card suppliers Clothing distributors 2-20 Data Warehousing Fundamentals

47 Typical Uses of a Data Warehouse Typical Uses of a Data Warehouse The requirements of a business can be met by employing a data warehouse solution, which collects data from internal business operations and external data from outside organizations to provide a single source of reliable data for analysis. Typical Users of a Data Warehouse There are many industries that employ data warehouses: Airlines for aircraft deployment, analysis of route profitability, frequent flyer promotions, and maintenance Banking for trend analysis, promotion of products and services, and customer service Health care for analysis and cost reduction Investment and insurance companies for planning, customer analysis, risk assessment, and portfolio management Retail stores for trend analysis, buying pattern analysis, promotions, customer profiling, and pricing Telecommunications for analysis and for product and service promotions Other industries that currently use data warehouse solutions are manufacturers, credit card issuers, and clothing distributors Figures show that the highest proportion of revenues in data warehousing is spent by the financial services, retail, telecommunications, and manufacturing industries Data Warehousing Fundamentals 2-21

48 Lesson 2: Meeting a Business Need Summary This lesson covered the following topics: Describing why an online transaction processing (OLTP) system is not suitable for complex analysis Describing how extracting processing for decision support querying led to data warehouse solutions employed today Explaining why businesses are driven to employ data warehouse technology Identifying some of the industries that employ data warehouses 2-22 Data Warehousing Fundamentals

49 Summary Summary This lesson covered the following topics: Describing why an online transaction processing (OLTP) system is not suitable for complex analysis Describing how extracting processing for decision support querying led to data warehouse solutions employed today Explaining why businesses are driven to employ data warehouse technology Identifying some of the industries that employ data warehouses Data Warehousing Fundamentals 2-23

50 Lesson 2: Meeting a Business Need Practice 2-1 Overview The practice covers answering questions and discussing how data warehousing meets business needs 2-24 Data Warehousing Fundamentals

51 Practice 2-1 Practice OLTP databases hold up-to-the-minute information and are most commonly designed as read-only databases. True False 2 In the scenario below, state whether it refers to an operational system or an analytical processing system. Show me how a specific brand of printer is selling throughout different parts of the United States and how this specific brand of printer is selling since it was first introduced into my stores. This scenario refers to: a An operational system b An analytical processing system 3 Who is the target audience for the data warehouse? a The business community in the organization b IT professionals c Data-entry clerks d None of the above e All of the above 4 Are the following statements true or false? a Operational systems display the following qualities: Good performance Static data contents High availability Unpredictable CPU use b Identify the reasons why business analysis is not easy with operational systems. Data is not structured for drill-down capablity. The system is not designed for querying. Data analysis can be CPU-intensive. Data is not integrated between systems. 5 In groups of three or four, discuss the questions below and present your points to the class at the end of the discussion. a List some of the reasons that your company is considering implementing a data warehouse or data mart. Data Warehousing Fundamentals 2-25

52 Lesson 2: Meeting a Business Need b c What are some of the business problems that your company is trying to answer? Why is the business community in your organization unable to find the answers to their business questions based on the existing information systems? 2-26 Data Warehousing Fundamentals

53 3... Defining Data Warehouse Concepts and Terminology

54 Lesson 3: Defining Data Warehouse Concepts and Terminology Overview Defining DW Concepts & Terminology Choosing a Computing Architecture Planning Warehouse Storage Planning for a Successful Warehouse Meeting a Business Need Modeling the Data Warehouse ETT (Building the Warehouse) Managing the Data Warehouse Analyzing User Query Needs Supporting End User Access Project Management (Methodology, Maintaining Metadata) Objectives After completing this lesson, you should be able to do the following: Identify a common, broadly accepted definition of a data warehouse Recognize some of the operational properties of a data warehouse Recognize common data warehousing terminology Identify the functionality associated with each component required for a successful data warehouse implementation Identify and position the Oracle Warehouse vision, products, and services 3-2 Data Warehousing Fundamentals

55 Overview Overview The previous lesson covered how data warehousing has evolved from early management information systems to today s decision support systems that meets a business need. This lesson defines data warehouse concepts and terminology. Note that the Defining Data Warehouse Concepts and Terminology block is highlighted in the course road map on the facing page. Specifically, this lesson introduces the Oracle definition of a data warehouse. The lesson offers a general description of the properties of a data warehouse. The standard components and tools required to build, operate, and use a data warehouse are identified. Objectives After completing this lesson, you should be able to do the following: Identify a common, broadly accepted definition of a data warehouse Recognize some of the operational properties of a data warehouse Recognize common data warehousing terminology Identify the functionality associated with each component required for a successful data warehouse implementation Identify and position the Oracle Warehouse vision, products, and services Data Warehousing Fundamentals 3-3

56 Lesson 3: Defining Data Warehouse Concepts and Terminology Definition of a Data Warehouse An enterprise structured repository of subjectoriented, time-variant, historical data used for information retrieval and decision support. The data warehouse stores atomic and summary data. Oracle Data Warehouse Method 3-4 Data Warehousing Fundamentals

57 Data Warehouse Definition Data Warehouse Definition This definition of a data warehouse from the Oracle Data Warehouse Method describes many of the most significant characteristics of a data warehouse. The Oracle Data Warehouse Method was developed using experiences gained from successful data warehouse projects carried out by Oracle Consulting Services. This method is discussed in Lesson 4. Subject-Oriented While the data in an OLTP system is stored to support a specific business process (for example, order entry, campaign management, and so on) as efficiently as possible, data in a data warehouse is stored based on common subject areas (for example, customer, product, and so on) for ease of access. That is because the complete set of questions to be posed to a data warehouse are never known. Every question the data warehouse answers spawns new questions. Thus, the focus of the design of a data warehouse is providing users easy access to the data so that current and future questions can be answered. Time-Variant The data warehouse contains slices of data across different periods of time. With these data slices, the user can view reports from now and in the past. Historical A data warehouse typically contains several years worth of data. This is necessary to support trending, forecasting, and time-based performance reporting (for example, current year versus previous year). Information Retrieval and Decision Support A data warehouse is a facility for getting at information to answer questions. It is not meant for direct data entry; batch updates are the norm for refreshing data warehouses. Atomic and Summary Data Depending on the purpose of the data warehouse, it may contain atomic data, summary data, or both. Data Warehousing Fundamentals 3-5

58 Lesson 3: Defining Data Warehouse Concepts and Terminology Data Warehouse Properties Subject Oriented Integrated Data Warehouse Non Volatile Time Variant Subject-Oriented Data is categorized and stored by business subject rather than by application. OLTP Applications Data Warehouse Subject Equity Plans Insurance Loans Shares Savings Customer financial information 3-6 Data Warehousing Fundamentals