Data Warehouse Architecture

Transcription

1 Anwendungssoftwares a -Warehouse-, -Mining- und OLAP-Technologien Warehouse Architecture

2 Overview Warehouse Architecture Sources and Quality Mart Federated Information Systems Operational Store Metadata Metadata Repository Metadata in Warehousing 2

3 Architecture End User Access data flow control flow Warehouse Warehouse Manager Metadata Manager Metadata Repository Load Staging Area Transformatio n Extraction Staging Area Monitor Warehouse System Sources (A. Bauer, H. Günzel: Warehouse Systeme, 2001) 3

4 Sources Characteristics of source systems: narrow, "account-based" queries no queries in a broad and unexpected way, like DW maintain little historical data no conformed dimensions (product, customer, geography, ) with other legacy systems use keys (production keys) to make certain things unique (product, customer, ) Important issues in selecting data sources: Purpose of the data warehouse Quality of data sources (consistency, correctness, completeness, exactness, reliability, understandability, relevance) Availability of data sources (organizational prerequisites, technical prerequisites) Costs (internal data, external data) Sources 4

5 Quality consistency correctness completeness Are there contradictions in data and/or metadata? Do data and metadata provide an exact picture of the reality? Are there missing attributes or values? exactness reliability understandability relevance Are exact numeric values available? Are different objects identifiable? Homonyms? Is there a Standard Operating Procedure (SOP) that describes the provision of source data? Does a description for the data and coded values exist? Does the data contribute to the purpose of the data warehouse? 5

6 Dimensions of Sources origin time usage internal vs. external data current vs. historic data data vs. metadata type character set orientation confidentiality number, string, time, graphic, audio, video, numeric, alphanumeric, boolean, binary, ASCII, EBCDIC, UNICODE, left to right, right to left, top-down strictly confidential, confidential, public, 6

7 Monitoring Goal: Discover changes in data source incrementally Approaches: Trigger Replica Timestamp Log Snapshot Based on triggers defined in source DBMS replication support of source DBMS timestamp assigned to each row log of source DBMS periodic snapshot of data source Changes identified by trigger writes a copy of changed data to files replication provides changed rows in a separate table use timestamp to identify changes (supported by temporal DBMS) read log compare snapshots 7

8 Staging Area (DSA) Staging Area: A storage area and a set of processes that clean, transform, combine, deduplicate, household, archive, and prepare source data for use in the data warehouse. (R. Kimball et al: The Warehouse Lifecycle Toolkit, 1998) Load is temporarily stored in the data staging area before it is loaded into the data warehouse. All transformations are performed in the DSA. Preprocessing does not influence data sources or data warehouse DSA is the central repository for ETL (Extraction - Transformation - Load) processing. Staging Area Transformatio n Extraction Monitor 8

9 Extraction Transfer data from data source into the data staging area. Extracted subset of data sources and schedule of the extraction depends on the kind of analysis that should be supported. Method depends on the monitoring strategy used: Read data from a file written by triggers. Read data from replication tables. Select data based on the timestamp. Read data from log. Read output of snapshot comparison. Multiple extract types: periodic started by the admin/user event-driven immediate after changes in data sources Staging Area Extraction Sources 9

10 Transformation Convert the data into something representable to the users and valuable to the business. Transformation of structure and content Typical transformations: denormalization, normalization data type conversion calculation, aggregation standardization of strings and date values Staging Area conversion of measures cleansing (missing, wrong, and inconsistent values) Transformatio n 10

11 Load Transfer data from the data staging area into the data warehouse. in the warehouse is rarely replaced. The history of values/changes is stored instead. Mainly based on bulk load tools of the DBMS. Offline vs. online load. Parallel load may be required. Warehouse Load Staging Area 11

12 Warehouse Manager Controls all components of the data warehouse system: Monitor: Discover changes in data sources Extraction: Select and transfer data from data sources to the data staging area Transformation: Consolidate data Load: Transfer data from data staging area to the data warehouse End User Access: Analysis of data in the data warehouse 12

13 Basic Elements of the Warehouse Source Systems Staging Area "The Warehouse" Presentation Servers End User Access extract extract extract Storage: Flat files; RDBMS; other Processing: clean; prune; combine; remove duplicates; household; standardize; conform dimensions; store awaiting replication; archive; export to data marts; No user query services populate, replicate, recover populate, replicate, recover populate, replicate, recover Mart #1: OLAP query services; dimensional! subject oriented; locally implemented; user group driven; may store atomic data; may be frequently refreshed; conforms to DW Bus DW BUS Mart #2: DW BUS Mart #3: Conformed dimensions Conformed facts Conformed dimensions Conformed facts feed feed feed feed Ad Hoc Query Tools Report Writers End User Applications Models forecasting; scoring; allocating; data mining; other downstream systems; other parameters; special UI upload cleaned dimensions upload model results (R. Kimball, et al.: The Warehouse Lifecycle Toolkit, 1998) 13

14 Architecture Clients Warehouse Logical Warehouse Warehouse Central architecture only one data model performance bottleneck complex to build easy to maintain Federal architecture logically consolidated separate physical databases that store detailled data faster response time Tiered architecture physical central data warehouse separate physical databases that store summarized data faster response time (M. Jarke et al., Fundamentals of Warehouses, 2002) 14

15 Marts End User Access End User Access End User Access End User Access End User Access End User Access End User Access End User Access Mart Mart Mart Warehouse Transformatio n Warehouse Mart Mart Mart Mart Load Load Load Load dependent data marts independent data marts 15

16 Marts dependent data marts (tiered architecture) Central data warehouse (DW) is build first Extracts of the data warehouse are provided as data marts (materialized views) Establish ETL process for DW only Consistent analysis on DW and DM independent data marts (federated architecture) Several data marts (DM) are build first marts are integrated by means of a second transformation step Establish ETL process for each DM and the central DW Inconsistent analysis is possible Virtual data warehouse possible (federated architecture) 16

17 Federated Information Systems Federated DBMS Transparent access to a collection of heterogeneous and semiautomonomous data sources. Presentation layer Complete, extensible database engine - Function compensation - Powerful (global) query optimizer (pushdown analysis, cost-based optimization, query rewrite) Federation layer e.g. uniform access language, uniform access schema, uniform metadata set Local Applications Foundation layer (data sources) 17

18 Architecture for a Federated base Server SQL API Wrapper Back-end Source Client Federated base Server Back-end Source Catalog 18

19 Federated DBMS: Processing Scenario Ename & Dname Federated DB SELECT Ename, Dname FROM EMP E, DEPT D WHERE E.Dno = D.Dno AND E.floor = 2 AND D.Mgr = 'Cooke' Knowing what the data source can do is a good idea! Oracle SELECT Ename, Dno FROM EMP WHERE Floor = 2 ORDER BY Dno DB2 SELECT Dname, Dno FROM Dept WHERE Mgr = 'Cooke' ORDER BY Dno 19

20 Operational Store (ODS) Term has taken many definitions. For example: Point of integration for operational systems - refreshed within a few seconds after the operational data sources are updated - very little transformations are performed - Example: Banking environment where data sources keep individual accounts of a large multinational customer, and the ODS stores the total balance for this customer. true operational system separated from the data warehouse Decision support access to operational data - integrated and transformed data are first accumulated and then periodically forwarded to the ODS - involves more integration and transformation processing - Example: Bank that stores in the ODS an integrated individual bank account on a weekly basis part of the data warehouse or separate system? 20

21 Classes of Operational Stores applications ODS DWH class 0 Tables are copied from the operational environment applications ODS DWH class 1 Transactions are moved to the ODS in an immediate manner (range of one to two seconds) applications ODS DWH class 2 Activities in the operational environment are stored, integrated, and forwarded to the ODS applications ODS DWH class 3 ODS is fed aggregated analytical data from the data warehouse applications ODS DWH (W. H. Inmon: ODS Types, 01/2000) class 4 Combination of integrated data from the operational environment and aggregated data from the analytical environment 21

22 Distribution of DW Project Costs DW Design Costs Recurring DW costs acces s/analysis tools 6% DBMS 10% network costs 10% m etadata des ign 5% integration and trans form ation 15% activity m onitor 2% data m onitor 2% disk storage 30% process or cos ts 20% sum m ary table usage analys is 2% m etadata m anagem ent 3% end-us er training 6% m onitoring of avctivity and data 7% servicing data m art requests for data 21% security periodic adm inistration verification of the 1% conform ance to the enterprise data m odel 2% occasional reorganization of data 1% data rchiving 1% capacity planning 1% DW refres hm ent 55% (M. Jarke et al., Fundamentals of Warehouses, 2002) 22

23 Metadata Repository Metadata Manager Metadata Repository A repository is a shared database of information about engineering artifacts, such as software, documents, maps, information systems, and manufactured components and systems. Functions of a repository: Object management Dynamic extensibility Relationship management Notification Version management Configuration management (P. Bernstein: Repositories and Object Oriented bases, 1998) 23

24 Metadata in Warehousing What data is available in the warehouse and where is the data located? dictionary: Definitions of the databases and relationship between data elements flow: Direction and frequency of data feed transformation: Transformations required when data is moved Version control: Changes to metadata are stored usage statistics: A profile of data in the warehouse Alias information: Alias names for a field Security: Who is allowed to access the data Stored in a metadata repository Need for a standard interchange format 24

25 Metadata in Warehousing Criteria to identify important classes of metadata in data warehousing: Type of data Abstraction User Origins Time Usage of metadata in data warehousing: passive active semi-active Main goals: Support development and operation of a data warehouse - system integration - processes for DW administration - flexible application development - access rights Provide information for data warehouse users - quality of data - consistent terminology - support for data analysis 25

26 Metadata Management centralized - decentralized - federated User Access Administration Tool Development Tool Analysis Tool Metadata Manager Local Tool Metadata Repository Metadata Repository Metadata Repository Metadata Repository federated metadata management data flow control flow (A. Bauer, H. Günzel: Warehouse Systeme, 2001) 26

27 Summary Basic Components: Staging Area: Extraction, Transformation, Load Warehouse base Warehouse Manager Metadata Repositories and Metadata Manager Marts: Distributed Warehouse Warehouse vs. Federated Information Systems Metadata is important to: Support development and operation of a data warehouse Provide information for data warehouse users Metadata standards are important to interchange metadata between warehouse tools, warehouse platforms and warehouse metadata repositories. 27