R. Kimball s definition of a DW

Transcription

1 Design of DW R. Kimball s definition of a DW A data warehouse is a copy of transactional data specifically structured for querying and analysis. According to this definition: The form of the stored data (RDBMS, flat file) has nothing to do with whether something is a data warehouse. Data warehousing is not necessarily for the needs of "decision makers" or used in the process of decision making. 1

2 The goals of a Data Warehouse The data warehouse must make an organization's information easily accessible. The data warehouse must present the organization's information consistently. The data warehouse must be adaptive and resilient to change. The data warehouse must be a secure bastion that protects our information assets. The data warehouse must serve as the foundation for improved decision making. The business community must accept the data warehouse if it is to be deemed successful. Key Definitions Business Intelligence refers to reporting and analysis of data stored in the warehouse Data warehouse is the foundation for business intelligence. Data warehouse/business intelligence (DW/BI) refers to the complete end-to-end system. 2

3 Data Warehouse Usage Three kinds of data warehouse applications Information processing supports querying, basic statistical analysis, and reporting using crosstabs, tables, charts and graphs Analytical processing multidimensional analysis of data warehouse data supports basic OLAP operations, slice-dice, drilling, pivoting Data mining knowledge discovery from hidden patterns supports associations, constructing analytical models, performing classification and prediction, and presenting the mining results using visualization tools Integrated Data Collections Of all the aspects of a data warehouse, integration is the most important Data is fed from multiple disparate sources into the data warehouse As the data is fed it is converted, reformatted, resequenced, summarized, and so forth 3

4 The result is that data once it resides in the data warehouse has a single physical corporate image Data is updated in the operational environment as a regular matter of course, but warehouse data exhibits a very different set of characteristics 4

5 Data warehouse data is loaded (usually en masse) and accessed, but it is not updated (in the general sense) Instead, when data in the data warehouse is loaded, it is loaded in a snapshot, static format When subsequent changes occur, a new snapshot record is written. In doing so a history of data is kept in the data warehouse Three Data Warehouse Models Enterprise warehouse collects all of the information about subjects spanning the entire organization Data Mart a subset of corporate-wide data that is of value to a specific groups of users. Its scope is confined to specific, selected groups, such as marketing data mart Independent vs. dependent (directly from warehouse) data mart Virtual warehouse A set of views over operational databases Only some of the possible summary views may be materialized 5

6 A data warehouse is a central repository for all or significant parts of the data that an enterprise's various business systems collect Enables strategic decision making A data mart is a repository of data gathered from operational data and other sources that is designed to serve a particular community of knowledge workers In scope, the data may derive from an enterprise-wide database or data warehouse or be more specialized. 6

7 Data mart is a specific, subject-oriented repository of data that was designed to answer specific questions Usually, multiple data marts exist to serve the needs of multiple business units (sales, marketing, operations, collections, accounting, etc.) Data warehouse is a single organizational repository of enterprise wide data across many or all subject areas. Data warehouse is an enterprise wide collection of data marts History of DWs: The early 90 s W.H. Inmon coins the term data warehousing Widening interest from enterprises Widening interest from vendors Almost ignored in the academic world Some topics from traditional databases: integration of heterogeneous sources materialized views aggregation queries... 7

8 History of DWs: Back to 1995 Start of the Stanford DW Project develop algorithms and tools for the efficient collection and integration of information from heterogeneous sources Widening interest from the academic world dedicated workshops gaining attention in major conferences Dedicated commercial tools for Data Warehousing The CIKM 95 paper by J. Widom Jennifer Widom, Research Problems in Data Warehousing, Int'l Conf. on Information and Knowledge Management, change detection, view maintenance, data scrubbing, optimization, design, evolution Data Warehouse Design Process Top-down, bottom-up approaches or a combination of both Top-down: Starts with overall design and planning (mature) Bottom-up: Starts with experiments and prototypes (rapid) From software engineering point of view Waterfall: structured and systematic analysis at each step before proceeding to the next Spiral: rapid generation of increasingly functional systems, short turn around time, quick turn around 8

9 Lifecycle planning Translation from user requirements into software requirements Transformation of the software requirements into software design Implementation of the design into programming code The sequence of this steps is defined by the lifecyle model Top down Top-down approach The Inmon s approach DW is developed based on the Enterprise wide data model DW as a single repository feeds data into data marts Longer to implement May fail due to the lack of patience and commitment 9

10 Bottom-up Bottom-up approach The Kimball s approach Starts with one data mart (ex. sales); later on additional data marts are added (ex. collection, marketing, etc.) Data flows from source into data marts, then into the data warehouse Faster to implement Implementation in stages Need to ensure consistency of metadata Making sure each data mart calls Apple and Apple.. the Hybrid approach (bottom-up + top down) The Kimball Lifecycle Illustrates the general flow of a DW implementation Identifies task sequencing and highlights activities that should happen concurrently May need to be customized to address the unique needs of your organization Not every detail of every Lifecycle task will be performed on every project 10

11 The Kimball Lifecycle Diagram Program/Project Planning Kimball s view of programs and projects Project refers to a single iteration of the Kimball Lifecycle from launch through deployment Program refers to the broader, ongoing coordination of resources, infrastructure, timelines, and communication across multiple projects a program contains multiple projects In real world, programs do not necessarily start before projects although ideally they should be. 11

12 Program/Project Planning Project planning Scope definition understanding business requirements Tasks identification Scheduling Resource planning Workload assignment The end document represents a blueprint of the project The Kimball Lifecycle Diagram 12

13 Program/Project Management Enforces the project plan Activities: Status monitoring Issue tracking Development of a comprehensive communication plan that addresses both the business and IT units The Kimball Lifecycle Diagram 13

14 Business Requirements Definition Success of the project depends on a solid understanding of the business requirements!!! Understanding the key factors driving the business is crucial for successful translation of the business requirements into design considerations The Kimball Lifecycle Diagram 14

15 What follows the business requirements definition? 3 concurrent tracks focusing on Technology Data Business intelligence applications Arrows in the diagram indicate the activity workflow along each of the parallel tracks Dependencies between the tasks are illustrated by the vertical alignment of the task boxes. The Kimball Lifecycle Diagram 15

16 Technology Track Technical Architecture Design Overall architectural framework and vision Considerations: the business requirements current technical environment planned strategic technical directions Technology Track Product Selection and Installation Based on the designed technical architecture Evaluation and selection of Products that will deliver needed capabilities Hardware platform Database management system Extract-transformation-load (ETL) tools Data access query tools Reporting tools must be evaluated Installation of selected products/components/tools Testing of installed products to ensure appropriate end-to-end integration within the data warehouse environment. 16

17 The Kimball Lifecycle Diagram Data Track Design of the dimensional model The physical design of the model Extraction, transformation, and loading (ETL) of source data into the target models. 17

18 The Kimball Lifecycle Diagram Dimensional Modeling Steps Identify the business process Identify the level of detail needed (grain) Identify the dimensions Identify the facts 18

19 Dimensional Modeling Detailed data analysis of a single business process is performed to identify the fact table granularity, associated dimensions and attributes, and numeric facts. Dimensional models contain the same data content and relationships as models normalized into third normal form, but structured differently. Improve understandability and query performance required by DW/BI Dimensional Modeling Primary constructs of a dimensional model fact tables dimension tables Fact tables Contain the metrics resulting from a business process or measurement event, such as the sales ordering process or service call event Dimensional models should be structured around business processes and their associated data sources, 19

20 Fact table s granularity should be set at the lowest, most atomic level captured by the business process This allows for maximum flexibility and extensibility Dimensional Modeling Dimensional table Contain the descriptive attributes and characteristics associated with specific, tangible measurement events, such as the customer, product, or sales representative associated with an order being placed. Dimension attributes are used for constraining, grouping, or labeling in a query. Hierarchical many-to-one relationships are denormalized into single dimension tables. 20

21 Star Schema A fact table Multiple dimension tables Example: Assume this schema to be of a retail-chain. Fact will be revenue (money). How do you want to see data is called a dimension. Snowflake Schema The snowflake schema is a variation of the star schema used in a data warehouse. The snowflake schema is a more complex schema than the star schema because the tables which describe the dimensions are normalized. 21

22 Snowflake Schema Disadvantages: Fact tables are typically responsible for 90% or more of the storage requirements, so the benefit is normally insignificant. Normalization of the dimension tables ("snowflaking") can impair the performance of a data warehouse. Advantages: If a dimension is very sparse (i.e. most of the possible values for the dimension have no data) and/or a dimension has a very long list of attributes which may be used in a query, the dimension table may occupy a significant proportion of the database and snowflaking may be appropriate. In practice, many data warehouses will normalize some dimensions and not others, and hence use a combination of snowflake and classic star schema. The Kimball Lifecycle Diagram 22

23 Physical Design Defining the physical structures setting up the database environment Setting up appropriate security preliminary performance tuning strategies, from indexing to partitioning and aggregations. If appropriate, OLAP databases are also designed during this process. The Kimball Lifecycle Diagram 23

24 ETL Design and Development The MOST important stage 70% of the risk and effort in the DW project is attributed to this stage ETL system capabilities: Extraction Cleansing and conforming Delivery and management ETL Raw data is extracted from the operational source systems and is being transformed into meaningful information for the business ETL processes must be architected long before any data is extracted from the source ETL system strives to deliver high throughput, as well as high quality output Incoming data is checked for reasonable quality Data quality conditions are continuously monitored Kimball calls ETL a data warehouse back room 24

25 The Kimball Lifecycle Diagram Business Intelligence Application Track Applications that query, analyze, and present information from the dimensional model. BI applications deliver business value from the DW/BI solution, rather than just delivering the data The goal is to deliver capabilities that are accepted by the business to support and enhance their decision making. 25

26 BI Application Design Identify the candidate BI applications and appropriate navigation interfaces to address the users needs and needed capabilities. Produce BI application specification BI Application Development Configuration of the business metadata and tool infrastructure Construction and validation of the specified analytic and operational BI applications and the navigational portal The Kimball Lifecycle Diagram 26

27 Deployment It is crucial that adequate planning was performed to make sure that: the results of technology, data, and BI application tracks are tested and fit together properly Appropriate education and support infrastructure is in place. It is critical that deployment be well orchestrated Deployment should be deferred if all the pieces, such as training, documentation, and validated data, are not ready for production release. The Kimball Lifecycle Diagram 27

28 Maintenance Occurs when the system is in production Includes: technical operational tasks that are necessary to keep the system performing optimally usage monitoring performance tuning index maintenance system backup Ongoing support, education, and communication with business users The Kimball Lifecycle Diagram 28

29 Growth DW systems tend to expand (if they were successful) Is considered as a sign of success New requests need to be prioritized Starting the cycle again Building upon the foundation that has already been established Focusing on the new requirements Lifecycle planning Translation from user requirements into software requirements Transformation of the software requirements into software design Implementation of the design into programming code The sequence of this steps is defined by the lifecyle model 29

30 Spiral Review Determine objectives, Alternatives, and constraints Development Plan Start Realse Risk analysis I II III Prototypes Simulation models Reqirements code test IV Evaluate Spiral Determine objectives, and constraints Identify and resolve risks Evaluate alternatives Develop the deliverables for that iteration, and verify that they are correct Plan next iteration Commit to an approach for the next iteration One of the most important advantages of the spiral model is that as costs increase, risk decrease. The more time and money you spend, the less risk your re taking 30