Introduction to Data Warehouses

Transcription

1 Introduction to Data Warehouses Krzysztof Dembczyński Institute of Computing Science Laboratory of Intelligent Decision Support Systems Politechnika Poznańska (Poznań University of Technology) Intelligent Decision Support Systems Master studies, first semester Academic year 2008/09 (summer course)

2 1 Data Warehouses 2 Information about the Course 3 Summary

3 1 Data Warehouses 2 Information about the Course 3 Summary

4 Goal: supporting decision makers.

5 First, organize your data, then start to analyze them.

6 Decision Support Data are used for decision support. Information processing: querying, basic statistical analysis, reporting using cross-tabs, tables, charts, or graphs, low-cost Web-based accessing tools integrated with Web browsers. Exploratory querying: OLAP operations for multidimensional data view and analysis, finding unexpected facts in databases. Approximate queries: response times are often impractical for large data warehouses; use fast, approximate answers. Knowledge discovery: finding hidden patterns and associations, analytical models for prediction and clustering, visualization.

7 Applications: Business data analysis (market-basket analysis, sales prediction, stock market prediction, direct marketing, CRM). Web mining (personalization, text categorization and clustering, recommender systems). Computer Aided Diagnostic. Computer vision and pattern recognition. Forecast prediction. Prediction of gene structure....

8 Main Players on the Market: IBM, Oracle, Microsoft, Sybase, SAS, Cognos, Informatica, Business Objects, SPSS, Statistica, Insightful (S-Plus), R, Weka, RapidMiner :) but also: Google and Yahoo!

9 Data mining is predicted to be one of the most revolutionary developments of the next decade. Data mining is one of 10 emerging technologies that will change the world. Life after ERP. What now? Your ERP system is in place. Now it s time for intelligence. It s often more important to creatively invent new data sources than to implement the latest academic variations on an algorithm.

10 To be learned in the coming semester...

11 The aim of the course is to get to know how to organize, store and process large volumes of data for intelligent decision support. Two perspectives are presented: Basic skills for: Design, implementation and use of data warehouse systems. Design of algorithms for data processing. Designing, implementing, and use of a data warehouse system. Implementing efficient data processing algorithms for dedicated applications.

12 We are computer scientists, that is why: we need an application, we need to know how to write efficient software.

13 Data Models and Evolution of Database Systems Data models: hierarchical, network, relational, object-oriented, multidimensional. Database systems: operational (OLTP), analytical (OLAP), data mining, dedicated solutions, WWW.

14 Modeling of Data Warehouses Complex entity-relationship diagrams (ERD) for OLTP. Simple star schema for OLAP. Specific approach to data warehouse modeling. Example: mobile phone operator.

15 ETL Process Extraction, transformation and load of data. Heterogeneous data sources: database systems, WWW, services, specific databases,.txt,.doc and.xls files. Data is integrated, transformed and cleansed. Data is load and data warehouse is refreshed.

16 OLAP Systems, Cubes and Queries OLAP provides an effective solution for accessing and processing large volumes of high dimensional data: parallel access to data, sophisticated data structures, optimization. Access through multidimensional reports and query languages like MDX.

17 Processing of Very Large Data Denormalization and summarization. Materialized perspectives. Query re-write. Join processing and partitioning. Indexes. Optimization of query processing. Map-reduce for data processing. Text processing.

18 Association Rules and Sequence Mining Set of items that are frequent. What is the most frequent itemset in Polish chain of small markets? The most frequent sequences.

19 Exploratory OLAP How to find unexpected facts in data? Hypothesis-driven vs. discovery-driven exploration of data cubes.

20 Approximate Query Processing Response times are often impractical for large data warehouses. One approach: fast, approximate answers. Approximate answer: ± 2000 (with 95% confidence) return answer in 10 seconds. Exact answer: return answer in 1 hour.

21 1 Data Warehouses 2 Information about the Course 3 Summary

22 Time and Place Lecture: Wednesday 15.10, room no. E-108. Labs: Monday 15.10, room no. 43. Office hour: Monday and Wednesday , room no. 2.

23 Instructors Krzysztof Dembczyński kdembczynski(at)cs.put.poznan.pl Web site ophelia.cs.put.poznan.pl/webdav/dbdw/ students/

24 Schedule of the Lectures Introduction to Data Warehouses Data models and Evolution of Database Systems Modeling of Datawarehouses I Modeling of Datawarehouses II (case study) ETL Process OLAP Systems, Cubes and Queries Processing of Very Large Data I Processing of Very Large Data II

25 Schedule of the Lectures Processing of Very Large Data III Processing of Very Large Data IV Processing of Very Large Data V Data Analysis I Data Analysis II Data Analysis III Data Analysis IV Test Exam to be announced

26 Schedule of the Laboratories Free time Introduction to MS SQL Modeling of Datawarehouses I Modeling of Datawarehouses II (case study) ETL (MS SQL 2005) OLAP Cubes (MS SQL 2005 sales data) MDX (MS SQL 2005 sales data) ETL, OLAP and MDX (case study)

27 Schedule of the Laboratories MOLAP (MS SQL 2005 recommender system) MOLAP (programming recommender system) MOLAP (recommender system; report) Association Rules (programming product data) Association Rules (programming product data) Association Rules (product data; report) Evaluation

28 Schedule of the Laboratories Send me an (before next Wednesday, 12.00) with a list of students using a format: Family_name \t First_name \t student_id \t

29 Final Evaluation Test: 60 points (min. 50%) Labs: 40 points (min. 50%) Labs Case study: modeling 25 points (min. 50%) Case study: ETL, OLAP and MDX 25 points (min. 50%) MOLAP: report 25 points (min. 50%) Association Rules: report 25 points (min. 50%) Bonus points for all: up to 10 points. Scale 90 points points points points points 3.0 otherwise 2.0

30 Bibliography C.J. Date, Wprowadzenie do systemów baz danych, Wydawnictwa Naukowo-Techniczne Z. Królikowski, Hurtownie danych: logiczne i fizyczne struktury danych, Wydawnictwo Politechniki Poznańskiej 2007 Ch. Todman, Projektowanie hurtowni danych. Zarzadzanie kontaktami z klientami (CRM), Wydawnictwa Naukowo-Techniczne 2003 M. Jarke, M. Lenzerini, Y. Vassiliou, P. Vassiliadis, Hurtownie danych. Podstawy organizacji i funkcjonowania, Wydawnictwa Szkolne i Pedagogiczne 2003 V. Poe, P. Klauer, S. Brobst, Tworzenie hurtowni danych, wspomaganie podejmowania decyzji, Wydawnictwa Naukowo-Techniczne 2000 R. Kimball, L. Reeves, M. Ross, W. Thornthwaite, The Data Warehouse Lifecycle Toolkit: Expert Methods for Designing, Developing, and Deploying Data Warehouses, John Wiley & Sons 1998 R. Kimball, M. Ross, The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling, John Wiley & Sons 2002

31 Bibliography D. Hand, H. Mannila, P. Smyth, Eksploracja danych, Wydawnictwa Naukowo-Techniczne 2006 J. Han, M. Kamber, Data Mining: Concepts and Techniques, Morgan-Kaufmann 2000 Ch. D. Manning, P. Raghavan, H. Schütze, Introduction to Information Retrieval, Cambridge University Press 2008, information-retrieval-book.html

32 1 Data Warehouses 2 Information about the Course 3 Summary

33 Summary Course content has been presented. Information about the course has been given. Questions?