Lecture Data Warehouse Systems

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Lecture Data Warehouse Systems"

Transcription

1 Lecture Data Warehouse Systems Eva Zangerle SS 2013

2 PART A: Architecture Chapter 1: Motivation and Definitions

3 Motivation Goal: to build an operational general view on a company to support decisions in the management Problem: integration of different raw data and raw databases Purchase, sale (scanning tills), personnel administration, salary, accounting etc. Different database systems, or even file systems ( Distributed or federated DBS) Incompatible data types or measurement units ( Repository) 3

4 Goal: Decision Support System 4

5 Motivation Consistent and holistic view of the company Data extraction, consolidation and transformation (possibly data cleaning) Schema integration Architectural design Support of different aggregated queries Single-dimensional relations are no longer adequate for storage In general, multidimensional data- und index structures are used ( Data cube Star schema, snowflake schema Index support: multidimensional index structures) 5

6 Definition of a Data Warehouse 6

7 Definition of a Data Warehouse Definition according to the GI-working group: A Data Warehouse is a database which (in a technical point of view) integrates data from different (heterogeneous) data sources and (in an economic point of view) provides the user with this data for business analysis purposes. Frequently, but not necessarily, a historization of data takes place. DW is centralized database providing a consistent (!) view on all integrated data single point of truth for analysis Data is kept redundant. 7

8 According to Inmon: Definition cont d A data warehouse is a subject-oriented, integrated, non-volatile, and time variant collection of data in support of management decisions. 8

9 Definition cont d Subject orientation: The goal is not only task-oriented (e.g. management of personnel data), a DW rather is designed and optimized for data analysis. Integration: DW integrates data from multiple sources. Non-volatility: Data basis is stable, no changes performed to data once stored. Time variance: Data is stored over certain timespan, goal of analysis is comparison of data at different points of time 9

10 Definition cont d A Data Warehouse system is an information system consisting of: Data acquisition area (extraction, transformation, loading components) Analysis components Metadata manager Data warehouse manager Databases (base database, data warehouse and repository) 10

11 Important differentiation Definition cont d A Data Warehouse System is not a single product, but consists of a number of components. Data Warehousing is the process that describes the dynamic procedure beginning with data acquisition up to the storage and analysis of data within the system. A Data Warehouse also is the DB within the Data Warehouse system where the analysis and evaluations for the users performed. 11

12 Example DW 12

13 Example: Star*Kauf Exemplary data warehouse selling electronic devices in Germany Incorporates personnel data, POS-data, etc. 13

14 Example: Star*Kauf Product Region Time top top top branch family group country mark region district city year quarter month week article store day turnover sales 14

15 Typical Analysis Poor sellers vs. cash cows Shopping basket analysis Marketing strategy Detection of (regional or timely) trends Examine effectiveness of marketing measures 15

16 In-Depth: Technical Aspects 16

17 Relational Representation Product Region Time as sales(article, store, day)? or: sales(store, article, day)? or: sales(day, article, store)? Problem: Linear compound index only supports one direction of queries Possible solution: secondary indexes (beware: many combinations!) DW solution: multidimensional data model and according indexes 17

18 Category Family Group Article product Multidimensional Data Model dimension time classification hierarchy Year Quarter Month Day classification scheme cube Measures: e.g. sales, transaction volume, etc. 18

19 Example: Classification Hierarchy Example: product dimension classification hierarchy (Instance) Top classification scheme Top electronics toys Product branch computers cars Product family portable stationary wood plastic Product group Sony XY Toshiba Z Stunt Star Fire Rider Article 19

20 Queries in Different Classification Directions and Levels of Aggregation Classification relation FACT Classification level Measures Example: Article Group Family Branch Sales Sales Turnover Store City District Region Day Month Quarter Year Week 20

21 Time Time Product Product Geography Geography New MD-Operators: Rotation 6 views of a 3-dimensional cube by 90 rotation (also known as pivotization) Time Time Product Product Geography Geography 21

22 Geography Geography Geography MD-Operators: Dicing & Slicing Slicing: Time Time Dicing: Time 22

23 Roll-up MD-Operators: Drill-down & Roll-up Drill-down Time Time 23

24 Aggregation Aggregation: change of granularity An aggregate function maps a set of values to a single one Example of aggregation Aggregation of all sales on items sorted by product groups More aggregate operators Standard SQL-functions: SUM, AVG, COUNT, MIN, MAX Order based aggregations Cumulation Rankings 24

25 Hierarchies are not always inclusive Example of two parallel hierarchies in the time dimension Top Year Quarter Month Week Day 25

26 Caution Concerning Summability Examples: Stock of inventory at the end of a year is the stock of inventory of the last month, not the sum of the stock of inventory of all months in the year. Tax rates are not summable. Summability: Correct application of an aggregate function on a cube only if (necessary conditions, not necessarily sufficient): Disjunctivity Completeness Type and measure compatibility 26

27 Summability cont. Disjunctivity and completeness are basic features of a classification Completeness: i.e., all product groups together must contain all products Disjunctivity: for instance: no product can belong to more than one group, otherwise it will be counted multiple times Type compatibility of operation and measures Summability dependent on the aggregation type of the measure 27

28 Building up a DW Analysis DWH Data Warehouse Loading Loading Loading Loading Transformation Transformation Transformation Transformation Extraction Extraction Extraction Extraction Store1 Store2 Central Purchasing Central Stock 28

29 Working with Data Marts Analysis Analysis Analysis DWH Data Marts (scalability, distribution, no single point of failure) Loading Loading Loading Loading Transformation Transformation Transformation Transformation Extraction Extraction Extraction Extraction Store1 Store2 Central Purchasing Central Stock 29

30 DW Reference Architecture 30

31 Description of the Architecture The Monitor supports extraction, through selection of relevant data from the Data Sources. The data required for the DW-process are extracted into the (temporary) Working Area. There they are transformed, i.e., cleaned, integrated, etc. Ideally, the data is then loaded into a Base Database, which is modelled independent of applications and is based on integrated data (and schema) of the different data sources. Finally, the data is loaded into a Data Warehouse specifically modelled for an application and is ready for analysis. The Metadata and the procedure in the DW-process are stored in the Repository. 31

32 Control Components Data Warehouse Manager (central control component) Initialisation, control and monitoring of all functions, from data extraction from the source data through to analysis of the data. DW Manager uses information from the Repository for the extraction and analysis steps. Metadata Manager Controls the metadata administration Controls the metadata flow Link between Data Warehouse Manager and Repository 32

33 Repository holds all metadata Repository Information about the setup, maintenance and administration of a Data Warehouse System, i.e. schema data and additional information Descriptive and process-related information Metadata may contain: Description of the whole Data Warehouse System Names, definitions, structure and content description of relations, attributes, measures, Identification of data sources Context and meaning of data Transformation and integration rules for filling the warehouse Transformation and integration rules for end-user rules Operational information such as updates, versions, access rights Usage and performance of the Data Warehouse 33

34 (Short) History of DW 34

35 Historical Development History of decision support systems 60s: Explicit programming of all statistical evaluations. Execution in batch operation. Inflexible and expensive. 70s/80s: Management Information Systems (MIS) allow easier handling using GUIs. Analysis only of the operative dataset 90s: Idea of multidimensional data analysis with the buzzword "Online-Analytical-Processing" (OLAP) by E.F. Codd Codd s Rules; Establishment of the OLAP-Council in 1998: Specification of an interface for multidimensional data access. OLAP-functionality provided by Data Warehouse Systems at the analysis level available. Redundantly stored data in the Data Warehouse is analysed, rather than the OLTP-Systems 35

36 Historical Development cont. History of decision support systems 00s: New hardware trends: cloud storage, real time analysis and web-based decision support systems, distributed solutions Direct relation to big data and scientific databases Emerging of new application areas (i.e., molecular biology) 36

37 OLAP vs. OLTP 37

38 OLAP vs. OLTP OLTP = Online Transactional Processing Time-critical Mostly select, insert, delete and update operations ACID OLAP = Online Analytical Processing Long-lasting computations Mostly select, data only loaded once No transactions necessary 38

39 DW vs. Transactional Systems Database systems in the context of Data Warehousing: Very high data volume Mostly static datasets Extensive correlation between data elements Many NULL values Possibly long-lasting computations (summations/aggregations) 39

40 DW vs. Transactional Systems Conventional database systems Comparatively few, small data objects High update rates Mostly independent data elements No/few NULL values (possibly) Mostly short transaction length 40

41 DW vs. Transactional Systems Comparison of the query characteristics Focus Transaction length and type Query structure Data volume of a query Transactional Read, Write, Update, Delete Short read/write transactions simple Few records Analytical Read, Periodical adding Long read transactions complex Many records Data model Flexible data model in regard to queries Analysis oriented data model 41

42 DW vs. Transactional Systems Comparison of the data characteristics Data sources Properties Transactional Mostly only one Not derived, up-to-date, autonomous, dynamic Analytical Many Long read transactions Data volume Access Megabyte-Gigabyte Single tuple access Gigabyte-Terabyte Range queries 42

43 DW vs. Transactional Systems Comparison of the user characteristics Transactional Analytical User type Number of users Response time Input/output by clerks many ms - s Evaluation by managers, controllers, analysts few s min - hours 43

44 Application Areas 44

45 Application Areas In general, suitable application areas are characterised by a large amount of data which need to be analysed. Business application areas I.e., sales, turnover, disposal analysis of large companies and trade chains Market data analysis by the GfK Scientific application areas I.e., molecular biology Technical application areas I.e., quality control in manufactures 45

46 Business Application Area Example Market research by the GfK (Gesellschaft für Konsum-, Marktund Absatzforschung) Sales numbers of trade companies are collected, cleaned and extrapolated to a representative whole Based on this dataset, statistical characteristics are created and sold to manufacturers for decision support, e.g. in the areas of marketing, product development, etc. This scenario, in which sales numbers for a given product of the retailer at a recorded point or period in time are stored will be looked at more closely during the course. 46

47 Molecular Biology Example Fundamental principle Composition of genetic information as a double-helix consisting of four nucleic acids Method Isolation of roughly 100,000 genes in the 46 chromosomes of the DNA utilising restriction enzymes Analysis of the partial sequences utilising so-called sequencers Decoding of the nucleotid sequence of the DNA by mapping short partial sequences Desired results Prognosis of the shape of complex three-dimensional protein molecules Understanding and targeted manipulation of the genetic information 47

48 Outlook 48

49 Future Topics Reference architecture Multidimensional data model Implementation of MD model Also: new approaches towards a combination of OLAP and OLTP 49

OLAP and OLTP. AMIT KUMAR BINDAL Associate Professor M M U MULLANA

OLAP and OLTP. AMIT KUMAR BINDAL Associate Professor M M U MULLANA OLAP and OLTP AMIT KUMAR BINDAL Associate Professor Databases Databases are developed on the IDEA that DATA is one of the critical materials of the Information Age Information, which is created by data,

More information

(Week 10) A04. Information System for CRM. Electronic Commerce Marketing

(Week 10) A04. Information System for CRM. Electronic Commerce Marketing (Week 10) A04. Information System for CRM Electronic Commerce Marketing Course Code: 166186-01 Course Name: Electronic Commerce Marketing Period: Autumn 2015 Lecturer: Prof. Dr. Sync Sangwon Lee Department:

More information

DATA WAREHOUSE AND OLAP TECHNOLOGIES. Outline. Data Warehouse Data Warehouse OLAP. A data warehouse is a:

DATA WAREHOUSE AND OLAP TECHNOLOGIES. Outline. Data Warehouse Data Warehouse OLAP. A data warehouse is a: DATA WAREHOUSE AND OLAP TECHNOLOGIES Keep order, and the order shall save thee. Latin maxim Outline 2 Data Warehouse Definition Architecture OLAP Multidimensional data model OLAP cube computing Data Warehouse

More information

Week 3 lecture slides

Week 3 lecture slides Week 3 lecture slides Topics Data Warehouses Online Analytical Processing Introduction to Data Cubes Textbook reference: Chapter 3 Data Warehouses A data warehouse is a collection of data specifically

More information

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 29-1

Copyright 2007 Ramez Elmasri and Shamkant B. Navathe. Slide 29-1 Slide 29-1 Chapter 29 Overview of Data Warehousing and OLAP Chapter 29 Outline Purpose of Data Warehousing Introduction, Definitions, and Terminology Comparison with Traditional Databases Characteristics

More information

Introduction to Data Warehousing. Ms Swapnil Shrivastava swapnil@konark.ncst.ernet.in

Introduction to Data Warehousing. Ms Swapnil Shrivastava swapnil@konark.ncst.ernet.in Introduction to Data Warehousing Ms Swapnil Shrivastava swapnil@konark.ncst.ernet.in Necessity is the mother of invention Why Data Warehouse? Scenario 1 ABC Pvt Ltd is a company with branches at Mumbai,

More information

DATA WAREHOUSING AND OLAP TECHNOLOGY

DATA WAREHOUSING AND OLAP TECHNOLOGY DATA WAREHOUSING AND OLAP TECHNOLOGY Manya Sethi MCA Final Year Amity University, Uttar Pradesh Under Guidance of Ms. Shruti Nagpal Abstract DATA WAREHOUSING and Online Analytical Processing (OLAP) are

More information

Part 22. Data Warehousing

Part 22. Data Warehousing Part 22 Data Warehousing The Decision Support System (DSS) Tools to assist decision-making Used at all levels in the organization Sometimes focused on a single area Sometimes focused on a single problem

More information

14. Data Warehousing & Data Mining

14. Data Warehousing & Data Mining 14. Data Warehousing & Data Mining Data Warehousing Concepts Decision support is key for companies wanting to turn their organizational data into an information asset Data Warehouse "A subject-oriented,

More information

OLAP & DATA MINING CS561-SPRING 2012 WPI, MOHAMED ELTABAKH

OLAP & DATA MINING CS561-SPRING 2012 WPI, MOHAMED ELTABAKH OLAP & DATA MINING CS561-SPRING 2012 WPI, MOHAMED ELTABAKH 1 Online Analytic Processing OLAP 2 OLAP OLAP: Online Analytic Processing OLAP queries are complex queries that Touch large amounts of data Discover

More information

Fluency With Information Technology CSE100/IMT100

Fluency With Information Technology CSE100/IMT100 Fluency With Information Technology CSE100/IMT100 ),7 Larry Snyder & Mel Oyler, Instructors Ariel Kemp, Isaac Kunen, Gerome Miklau & Sean Squires, Teaching Assistants University of Washington, Autumn 1999

More information

Terminology and Definitions. Data Warehousing and OLAP. Data Warehouse characteristics. Data Warehouse Types. Typical DW Implementation

Terminology and Definitions. Data Warehousing and OLAP. Data Warehouse characteristics. Data Warehouse Types. Typical DW Implementation Data Warehousing and OLAP Topics Introduction Data modelling in data warehouses Building data warehouses View Maintenance OLAP and data mining Reading Lecture Notes Elmasriand Navathe, Chapter 26 Ozsu

More information

Database Applications. Advanced Querying. Transaction Processing. Transaction Processing. Data Warehouse. Decision Support. Transaction processing

Database Applications. Advanced Querying. Transaction Processing. Transaction Processing. Data Warehouse. Decision Support. Transaction processing Database Applications Advanced Querying Transaction processing Online setting Supports day-to-day operation of business OLAP Data Warehousing Decision support Offline setting Strategic planning (statistics)

More information

www.ijreat.org Published by: PIONEER RESEARCH & DEVELOPMENT GROUP (www.prdg.org) 28

www.ijreat.org Published by: PIONEER RESEARCH & DEVELOPMENT GROUP (www.prdg.org) 28 Data Warehousing - Essential Element To Support Decision- Making Process In Industries Ashima Bhasin 1, Mr Manoj Kumar 2 1 Computer Science Engineering Department, 2 Associate Professor, CSE Abstract SGT

More information

Overview of Data Warehousing and OLAP

Overview of Data Warehousing and OLAP Overview of Data Warehousing and OLAP Chapter 28 March 24, 2008 ADBS: DW 1 Chapter Outline What is a data warehouse (DW) Conceptual structure of DW Why separate DW Data modeling for DW Online Analytical

More information

Mario Guarracino. Data warehousing

Mario Guarracino. Data warehousing Data warehousing Introduction Since the mid-nineties, it became clear that the databases for analysis and business intelligence need to be separate from operational. In this lecture we will review the

More information

Week 13: Data Warehousing. Warehousing

Week 13: Data Warehousing. Warehousing 1 Week 13: Data Warehousing Warehousing Growing industry: $8 billion in 1998 Range from desktop to huge: Walmart: 900-CPU, 2,700 disk, 23TB Teradata system Lots of buzzwords, hype slice & dice, rollup,

More information

Data Warehousing. Read chapter 13 of Riguzzi et al Sistemi Informativi. Slides derived from those by Hector Garcia-Molina

Data Warehousing. Read chapter 13 of Riguzzi et al Sistemi Informativi. Slides derived from those by Hector Garcia-Molina Data Warehousing Read chapter 13 of Riguzzi et al Sistemi Informativi Slides derived from those by Hector Garcia-Molina What is a Warehouse? Collection of diverse data subject oriented aimed at executive,

More information

Data W a Ware r house house and and OLAP Week 5 1

Data W a Ware r house house and and OLAP Week 5 1 Data Warehouse and OLAP Week 5 1 Midterm I Friday, March 4 Scope Homework assignments 1 4 Open book Team Homework Assignment #7 Read pp. 121 139, 146 150 of the text book. Do Examples 3.8, 3.10 and Exercise

More information

DATA CUBES E0 261. Jayant Haritsa Computer Science and Automation Indian Institute of Science. JAN 2014 Slide 1 DATA CUBES

DATA CUBES E0 261. Jayant Haritsa Computer Science and Automation Indian Institute of Science. JAN 2014 Slide 1 DATA CUBES E0 261 Jayant Haritsa Computer Science and Automation Indian Institute of Science JAN 2014 Slide 1 Introduction Increasingly, organizations are analyzing historical data to identify useful patterns and

More information

Data Warehousing. Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de. Winter 2015/16. Jens Teubner Data Warehousing Winter 2015/16 1

Data Warehousing. Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de. Winter 2015/16. Jens Teubner Data Warehousing Winter 2015/16 1 Jens Teubner Data Warehousing Winter 2015/16 1 Data Warehousing Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de Winter 2015/16 Jens Teubner Data Warehousing Winter 2015/16 13 Part II Overview

More information

Data Warehousing. Read chapter 13 of Riguzzi et al Sistemi Informativi. Slides derived from those by Hector Garcia-Molina

Data Warehousing. Read chapter 13 of Riguzzi et al Sistemi Informativi. Slides derived from those by Hector Garcia-Molina Data Warehousing Read chapter 13 of Riguzzi et al Sistemi Informativi Slides derived from those by Hector Garcia-Molina What is a Warehouse? Collection of diverse data subject oriented aimed at executive,

More information

Data Warehousing and Data Mining

Data Warehousing and Data Mining Data Warehousing and Data Mining Part I: Data Warehousing Gao Cong gaocong@cs.aau.dk Slides adapted from Man Lung Yiu and Torben Bach Pedersen Course Structure Business intelligence: Extract knowledge

More information

When to consider OLAP?

When to consider OLAP? When to consider OLAP? Author: Prakash Kewalramani Organization: Evaltech, Inc. Evaltech Research Group, Data Warehousing Practice. Date: 03/10/08 Email: erg@evaltech.com Abstract: Do you need an OLAP

More information

DATA WAREHOUSING - OLAP

DATA WAREHOUSING - OLAP http://www.tutorialspoint.com/dwh/dwh_olap.htm DATA WAREHOUSING - OLAP Copyright tutorialspoint.com Online Analytical Processing Server OLAP is based on the multidimensional data model. It allows managers,

More information

Introduction to Databases, Fall 2004 IT University of Copenhagen. Lecture 6, part 2: OLAP and data cubes. October 8, Lecturer: Rasmus Pagh

Introduction to Databases, Fall 2004 IT University of Copenhagen. Lecture 6, part 2: OLAP and data cubes. October 8, Lecturer: Rasmus Pagh Introduction to Databases, Fall 2004 IT University of Copenhagen Lecture 6, part 2: OLAP and data cubes October 8, 2004 Lecturer: Rasmus Pagh Today s lecture, part II Information integration. On-Line Analytical

More information

Designing a Dimensional Model

Designing a Dimensional Model Designing a Dimensional Model Erik Veerman Atlanta MDF member SQL Server MVP, Microsoft MCT Mentor, Solid Quality Learning Definitions Data Warehousing A subject-oriented, integrated, time-variant, and

More information

BUILDING BLOCKS OF DATAWAREHOUSE. G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT

BUILDING BLOCKS OF DATAWAREHOUSE. G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT BUILDING BLOCKS OF DATAWAREHOUSE G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT 1 Data Warehouse Subject Oriented Organized around major subjects, such as customer, product, sales. Focusing on

More information

Lection 3-4 WAREHOUSING

Lection 3-4 WAREHOUSING Lection 3-4 DATA WAREHOUSING Learning Objectives Understand d the basic definitions iti and concepts of data warehouses Understand data warehousing architectures Describe the processes used in developing

More information

Data Warehouse: Introduction

Data Warehouse: Introduction Base and Mining Group of Base and Mining Group of Base and Mining Group of Base and Mining Group of Base and Mining Group of Base and Mining Group of Base and Mining Group of base and data mining group,

More information

OLAP and Data Mining. Data Warehousing and End-User Access Tools. Introducing OLAP. Introducing OLAP

OLAP and Data Mining. Data Warehousing and End-User Access Tools. Introducing OLAP. Introducing OLAP Data Warehousing and End-User Access Tools OLAP and Data Mining Accompanying growth in data warehouses is increasing demands for more powerful access tools providing advanced analytical capabilities. Key

More information

Advanced Data Management Technologies

Advanced Data Management Technologies ADMT 2015/16 Unit 2 J. Gamper 1/44 Advanced Data Management Technologies Unit 2 Basic Concepts of BI and Data Warehousing J. Gamper Free University of Bozen-Bolzano Faculty of Computer Science IDSE Acknowledgements:

More information

Data Warehousing and Decision Support. Introduction. Three Complementary Trends. Chapter 23, Part A

Data Warehousing and Decision Support. Introduction. Three Complementary Trends. Chapter 23, Part A Data Warehousing and Decision Support Chapter 23, Part A Database Management Systems, 2 nd Edition. R. Ramakrishnan and J. Gehrke 1 Introduction Increasingly, organizations are analyzing current and historical

More information

Search and Data Mining Techniques. OLAP Anna Yarygina Boris Novikov

Search and Data Mining Techniques. OLAP Anna Yarygina Boris Novikov Search and Data Mining Techniques OLAP Anna Yarygina Boris Novikov The Database: Shared Data Store? A dream from database textbooks: Sharing data between applications This NEVER happened. Applications

More information

Namrata 1, Dr. Saket Bihari Singh 2 Research scholar (PhD), Professor Computer Science, Magadh University, Gaya, Bihar

Namrata 1, Dr. Saket Bihari Singh 2 Research scholar (PhD), Professor Computer Science, Magadh University, Gaya, Bihar A Comprehensive Study on Data Warehouse, OLAP and OLTP Technology Namrata 1, Dr. Saket Bihari Singh 2 Research scholar (PhD), Professor Computer Science, Magadh University, Gaya, Bihar Abstract: Data warehouse

More information

European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project

European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project Janet Delve, University of Portsmouth Kuldar Aas, National Archives of Estonia Rainer Schmidt, Austrian Institute

More information

Data Warehousing. Yeow Wei Choong Anne Laurent

Data Warehousing. Yeow Wei Choong Anne Laurent Data Warehousing Yeow Wei Choong Anne Laurent Databases Databases are developed on the IDEA that DATA is one of the cri>cal materials of the Informa>on Age Informa>on, which is created by data, becomes

More information

1960s 1970s 1980s 1990s. Slow access to

1960s 1970s 1980s 1990s. Slow access to Principles of Knowledge Discovery in Fall 2002 Chapter 2: Warehousing and Dr. Osmar R. Zaïane University of Alberta Dr. Osmar R. Zaïane, 1999-2002 Principles of Knowledge Discovery in University of Alberta

More information

Data Warehousing. Outline. From OLTP to the Data Warehouse. Overview of data warehousing Dimensional Modeling Online Analytical Processing

Data Warehousing. Outline. From OLTP to the Data Warehouse. Overview of data warehousing Dimensional Modeling Online Analytical Processing Data Warehousing Outline Overview of data warehousing Dimensional Modeling Online Analytical Processing From OLTP to the Data Warehouse Traditionally, database systems stored data relevant to current business

More information

Course 803401 DSS. Business Intelligence: Data Warehousing, Data Acquisition, Data Mining, Business Analytics, and Visualization

Course 803401 DSS. Business Intelligence: Data Warehousing, Data Acquisition, Data Mining, Business Analytics, and Visualization Oman College of Management and Technology Course 803401 DSS Business Intelligence: Data Warehousing, Data Acquisition, Data Mining, Business Analytics, and Visualization CS/MIS Department Information Sharing

More information

OLAP. Business Intelligence OLAP definition & application Multidimensional data representation

OLAP. Business Intelligence OLAP definition & application Multidimensional data representation OLAP Business Intelligence OLAP definition & application Multidimensional data representation 1 Business Intelligence Accompanying the growth in data warehousing is an ever-increasing demand by users for

More information

OLAP and Data Warehousing! Introduction!

OLAP and Data Warehousing! Introduction! The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still

More information

Basics of Dimensional Modeling

Basics of Dimensional Modeling Basics of Dimensional Modeling Data warehouse and OLAP tools are based on a dimensional data model. A dimensional model is based on dimensions, facts, cubes, and schemas such as star and snowflake. Dimensional

More information

Data Warehousing. Overview, Terminology, and Research Issues. Joachim Hammer. Joachim Hammer

Data Warehousing. Overview, Terminology, and Research Issues. Joachim Hammer. Joachim Hammer Data Warehousing Overview, Terminology, and Research Issues 1 Heterogeneous Database Integration Integration System World Wide Web Digital Libraries Scientific Databases Personal Databases Collects and

More information

DATA WAREHOUSE CONCEPTS DATA WAREHOUSE DEFINITIONS

DATA WAREHOUSE CONCEPTS DATA WAREHOUSE DEFINITIONS DATA WAREHOUSE CONCEPTS A fundamental concept of a data warehouse is the distinction between data and information. Data is composed of observable and recordable facts that are often found in operational

More information

Introduction to Data Mining

Introduction to Data Mining Introduction to Data Mining Jay Urbain Credits: Nazli Goharian & David Grossman @ IIT Outline Introduction Data Pre-processing Data Mining Algorithms Naïve Bayes Decision Tree Neural Network Association

More information

Chapter 5 Business Intelligence: Data Warehousing, Data Acquisition, Data Mining, Business Analytics, and Visualization

Chapter 5 Business Intelligence: Data Warehousing, Data Acquisition, Data Mining, Business Analytics, and Visualization Turban, Aronson, and Liang Decision Support Systems and Intelligent Systems, Seventh Edition Chapter 5 Business Intelligence: Data Warehousing, Data Acquisition, Data Mining, Business Analytics, and Visualization

More information

This tutorial will help computer science graduates to understand the basic-toadvanced concepts related to data warehousing.

This tutorial will help computer science graduates to understand the basic-toadvanced concepts related to data warehousing. About the Tutorial A data warehouse is constructed by integrating data from multiple heterogeneous sources. It supports analytical reporting, structured and/or ad hoc queries and decision making. This

More information

Outline. Data Warehousing. What is a Warehouse? What is a Warehouse?

Outline. Data Warehousing. What is a Warehouse? What is a Warehouse? Outline Data Warehousing What is a data warehouse? Why a warehouse? Models & operations Implementing a warehouse 2 What is a Warehouse? Collection of diverse data subject oriented aimed at executive, decision

More information

New Approach of Computing Data Cubes in Data Warehousing

New Approach of Computing Data Cubes in Data Warehousing International Journal of Information & Computation Technology. ISSN 0974-2239 Volume 4, Number 14 (2014), pp. 1411-1417 International Research Publications House http://www. irphouse.com New Approach of

More information

B.Sc (Computer Science) Database Management Systems UNIT-V

B.Sc (Computer Science) Database Management Systems UNIT-V 1 B.Sc (Computer Science) Database Management Systems UNIT-V Business Intelligence? Business intelligence is a term used to describe a comprehensive cohesive and integrated set of tools and process used

More information

1. OLAP is an acronym for a. Online Analytical Processing b. Online Analysis Process c. Online Arithmetic Processing d. Object Linking and Processing

1. OLAP is an acronym for a. Online Analytical Processing b. Online Analysis Process c. Online Arithmetic Processing d. Object Linking and Processing 1. OLAP is an acronym for a. Online Analytical Processing b. Online Analysis Process c. Online Arithmetic Processing d. Object Linking and Processing 2. What is a Data warehouse a. A database application

More information

OLAP Theory-English version

OLAP Theory-English version OLAP Theory-English version On-Line Analytical processing (Business Intelligence) [Ing.J.Skorkovský,CSc.] Department of corporate economy Agenda The Market Why OLAP (On-Line-Analytic-Processing Introduction

More information

Data Warehousing and OLAP Technology for Knowledge Discovery

Data Warehousing and OLAP Technology for Knowledge Discovery 542 Data Warehousing and OLAP Technology for Knowledge Discovery Aparajita Suman Abstract Since time immemorial, libraries have been generating services using the knowledge stored in various repositories

More information

DATA WAREHOUSE E KNOWLEDGE DISCOVERY

DATA WAREHOUSE E KNOWLEDGE DISCOVERY DATA WAREHOUSE E KNOWLEDGE DISCOVERY Prof. Fabio A. Schreiber Dipartimento di Elettronica e Informazione Politecnico di Milano DATA WAREHOUSE (DW) A TECHNIQUE FOR CORRECTLY ASSEMBLING AND MANAGING DATA

More information

IST722 Data Warehousing

IST722 Data Warehousing IST722 Data Warehousing Components of the Data Warehouse Michael A. Fudge, Jr. Recall: Inmon s CIF The CIF is a reference architecture Understanding the Diagram The CIF is a reference architecture CIF

More information

Decision Support. Chapter 23. Database Management Systems, 2 nd Edition. R. Ramakrishnan and J. Gehrke 1

Decision Support. Chapter 23. Database Management Systems, 2 nd Edition. R. Ramakrishnan and J. Gehrke 1 Decision Support Chapter 23 Database Management Systems, 2 nd Edition. R. Ramakrishnan and J. Gehrke 1 Introduction Increasingly, organizations are analyzing current and historical data to identify useful

More information

Optimizing Your Data Warehouse Design for Superior Performance

Optimizing Your Data Warehouse Design for Superior Performance Optimizing Your Data Warehouse Design for Superior Performance Lester Knutsen, President and Principal Database Consultant Advanced DataTools Corporation Session 2100A The Problem The database is too complex

More information

Chapter 5. Warehousing, Data Acquisition, Data. Visualization

Chapter 5. Warehousing, Data Acquisition, Data. Visualization Decision Support Systems and Intelligent Systems, Seventh Edition Chapter 5 Business Intelligence: Data Warehousing, Data Acquisition, Data Mining, Business Analytics, and Visualization 5-1 Learning Objectives

More information

CS54100: Database Systems

CS54100: Database Systems CS54100: Database Systems Date Warehousing: Current, Future? 20 April 2012 Prof. Chris Clifton Data Warehousing: Goals OLAP vs OLTP On Line Analytical Processing (vs. Transaction) Optimize for read, not

More information

Chapter 3, Data Warehouse and OLAP Operations

Chapter 3, Data Warehouse and OLAP Operations CSI 4352, Introduction to Data Mining Chapter 3, Data Warehouse and OLAP Operations Young-Rae Cho Associate Professor Department of Computer Science Baylor University CSI 4352, Introduction to Data Mining

More information

Data Warehousing & OLAP

Data Warehousing & OLAP Data Warehousing & OLAP Motivation: Business Intelligence Customer information (customer-id, gender, age, homeaddress, occupation, income, family-size, ) Product information (Product-id, category, manufacturer,

More information

Dimensional Modeling for Data Warehouse

Dimensional Modeling for Data Warehouse Modeling for Data Warehouse Umashanker Sharma, Anjana Gosain GGS, Indraprastha University, Delhi Abstract Many surveys indicate that a significant percentage of DWs fail to meet business objectives or

More information

DATA WAREHOUSING APPLICATIONS: AN ANALYTICAL TOOL FOR DECISION SUPPORT SYSTEM

DATA WAREHOUSING APPLICATIONS: AN ANALYTICAL TOOL FOR DECISION SUPPORT SYSTEM DATA WAREHOUSING APPLICATIONS: AN ANALYTICAL TOOL FOR DECISION SUPPORT SYSTEM MOHAMMED SHAFEEQ AHMED Guest Lecturer, Department of Computer Science, Gulbarga University, Gulbarga, Karnataka, India (e-mail:

More information

2. State of the Art of Data Warehouse Research

2. State of the Art of Data Warehouse Research 2. State of the Art of Data Warehouse Research We are drowning in data, but starving for knowledge! (anonymous) Data warehouse systems are a new technology and differ much from traditional transaction-oriented

More information

Data Warehouse Snowflake Design and Performance Considerations in Business Analytics

Data Warehouse Snowflake Design and Performance Considerations in Business Analytics Journal of Advances in Information Technology Vol. 6, No. 4, November 2015 Data Warehouse Snowflake Design and Performance Considerations in Business Analytics Jiangping Wang and Janet L. Kourik Walker

More information

BUILDING OLAP TOOLS OVER LARGE DATABASES

BUILDING OLAP TOOLS OVER LARGE DATABASES BUILDING OLAP TOOLS OVER LARGE DATABASES Rui Oliveira, Jorge Bernardino ISEC Instituto Superior de Engenharia de Coimbra, Polytechnic Institute of Coimbra Quinta da Nora, Rua Pedro Nunes, P-3030-199 Coimbra,

More information

Data Warehousing and OLAP

Data Warehousing and OLAP 1 Data Warehousing and OLAP Hector Garcia-Molina Stanford University Warehousing Growing industry: $8 billion in 1998 Range from desktop to huge: Walmart: 900-CPU, 2,700 disk, 23TB Teradata system Lots

More information

Enterprise Data Warehouse (EDW) UC Berkeley Peter Cava Manager Data Warehouse Services October 5, 2006

Enterprise Data Warehouse (EDW) UC Berkeley Peter Cava Manager Data Warehouse Services October 5, 2006 Enterprise Data Warehouse (EDW) UC Berkeley Peter Cava Manager Data Warehouse Services October 5, 2006 What is a Data Warehouse? A data warehouse is a subject-oriented, integrated, time-varying, non-volatile

More information

Sizing Logical Data in a Data Warehouse A Consistent and Auditable Approach

Sizing Logical Data in a Data Warehouse A Consistent and Auditable Approach 2006 ISMA Conference 1 Sizing Logical Data in a Data Warehouse A Consistent and Auditable Approach Priya Lobo CFPS Satyam Computer Services Ltd. 69, Railway Parallel Road, Kumarapark West, Bangalore 560020,

More information

OLAP (Online Analytical Processing) G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT

OLAP (Online Analytical Processing) G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT OLAP (Online Analytical Processing) G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT OVERVIEW INTRODUCTION OLAP CUBE HISTORY OF OLAP OLAP OPERATIONS DATAWAREHOUSE DATAWAREHOUSE ARCHITECHTURE DIFFERENCE

More information

Why Business Intelligence

Why Business Intelligence Why Business Intelligence Ferruccio Ferrando z IT Specialist Techline Italy March 2011 page 1 di 11 1.1 The origins In the '50s economic boom, when demand and production were very high, the only concern

More information

CS2032 Data warehousing and Data Mining Unit II Page 1

CS2032 Data warehousing and Data Mining Unit II Page 1 UNIT II BUSINESS ANALYSIS Reporting Query tools and Applications The data warehouse is accessed using an end-user query and reporting tool from Business Objects. Business Objects provides several tools

More information

A Comparative Study on Operational Database, Data Warehouse and Hadoop File System T.Jalaja 1, M.Shailaja 2

A Comparative Study on Operational Database, Data Warehouse and Hadoop File System T.Jalaja 1, M.Shailaja 2 RESEARCH ARTICLE A Comparative Study on Operational base, Warehouse Hadoop File System T.Jalaja 1, M.Shailaja 2 1,2 (Department of Computer Science, Osmania University/Vasavi College of Engineering, Hyderabad,

More information

Data W a Ware r house house and and OLAP II Week 6 1

Data W a Ware r house house and and OLAP II Week 6 1 Data Warehouse and OLAP II Week 6 1 Team Homework Assignment #8 Using a data warehousing tool and a data set, play four OLAP operations (Roll up (drill up), Drill down (roll down), Slice and dice, Pivot

More information

Turkish Journal of Engineering, Science and Technology

Turkish Journal of Engineering, Science and Technology Turkish Journal of Engineering, Science and Technology 03 (2014) 106-110 Turkish Journal of Engineering, Science and Technology journal homepage: www.tujest.com Integrating Data Warehouse with OLAP Server

More information

The Quality Data Warehouse: Solving Problems for the Enterprise

The Quality Data Warehouse: Solving Problems for the Enterprise The Quality Data Warehouse: Solving Problems for the Enterprise Bradley W. Klenz, SAS Institute Inc., Cary NC Donna O. Fulenwider, SAS Institute Inc., Cary NC ABSTRACT Enterprise quality improvement is

More information

Understanding Data Warehousing. [by Alex Kriegel]

Understanding Data Warehousing. [by Alex Kriegel] Understanding Data Warehousing 2008 [by Alex Kriegel] Things to Discuss Who Needs a Data Warehouse? OLTP vs. Data Warehouse Business Intelligence Industrial Landscape Which Data Warehouse: Bill Inmon vs.

More information

CHAPTER 5: BUSINESS ANALYTICS

CHAPTER 5: BUSINESS ANALYTICS Chapter 5: Business Analytics CHAPTER 5: BUSINESS ANALYTICS Objectives The objectives are: Describe Business Analytics. Explain the terminology associated with Business Analytics. Describe the data warehouse

More information

CSE 544 Principles of Database Management Systems. Magdalena Balazinska Fall 2007 Lecture 16 - Data Warehousing

CSE 544 Principles of Database Management Systems. Magdalena Balazinska Fall 2007 Lecture 16 - Data Warehousing CSE 544 Principles of Database Management Systems Magdalena Balazinska Fall 2007 Lecture 16 - Data Warehousing Class Projects Class projects are going very well! Project presentations: 15 minutes On Wednesday

More information

Life Cycle of a Data Warehousing Project in Healthcare

Life Cycle of a Data Warehousing Project in Healthcare Life Cycle of a Data Warehousing Project in Healthcare Ravi Verma, Jeannette Harper ABSTRACT Hill Physicians Medical Group (and its medical management firm, PriMed Management) early on recognized the need

More information

Multi-dimensional index structures Part I: motivation

Multi-dimensional index structures Part I: motivation Multi-dimensional index structures Part I: motivation 144 Motivation: Data Warehouse A definition A data warehouse is a repository of integrated enterprise data. A data warehouse is used specifically for

More information

2074 : Designing and Implementing OLAP Solutions Using Microsoft SQL Server 2000

2074 : Designing and Implementing OLAP Solutions Using Microsoft SQL Server 2000 2074 : Designing and Implementing OLAP Solutions Using Microsoft SQL Server 2000 Introduction This course provides students with the knowledge and skills necessary to design, implement, and deploy OLAP

More information

Overview. Data Warehousing and Decision Support. Introduction. Three Complementary Trends. Data Warehousing. An Example: The Store (e.g.

Overview. Data Warehousing and Decision Support. Introduction. Three Complementary Trends. Data Warehousing. An Example: The Store (e.g. Overview Data Warehousing and Decision Support Chapter 25 Why data warehousing and decision support Data warehousing and the so called star schema MOLAP versus ROLAP OLAP, ROLLUP AND CUBE queries Design

More information

Data Warehousing: Data Models and OLAP operations. By Kishore Jaladi kishorejaladi@yahoo.com

Data Warehousing: Data Models and OLAP operations. By Kishore Jaladi kishorejaladi@yahoo.com Data Warehousing: Data Models and OLAP operations By Kishore Jaladi kishorejaladi@yahoo.com Topics Covered 1. Understanding the term Data Warehousing 2. Three-tier Decision Support Systems 3. Approaches

More information

OLAP Systems and Multidimensional Expressions I

OLAP Systems and Multidimensional Expressions I OLAP Systems and Multidimensional Expressions I Krzysztof Dembczyński Intelligent Decision Support Systems Laboratory (IDSS) Poznań University of Technology, Poland Software Development Technologies Master

More information

CSE 544 Principles of Database Management Systems. Magdalena Balazinska Winter 2009 Lecture 15 - Data Warehousing: Cubes

CSE 544 Principles of Database Management Systems. Magdalena Balazinska Winter 2009 Lecture 15 - Data Warehousing: Cubes CSE 544 Principles of Database Management Systems Magdalena Balazinska Winter 2009 Lecture 15 - Data Warehousing: Cubes Final Exam Overview Open books and open notes No laptops and no other mobile devices

More information

Data Warehouses & OLAP

Data Warehouses & OLAP Riadh Ben Messaoud 1. The Big Picture 2. Data Warehouse Philosophy 3. Data Warehouse Concepts 4. Warehousing Applications 5. Warehouse Schema Design 6. Business Intelligence Reporting 7. On-Line Analytical

More information

Student Performance Analytics using Data Warehouse in E-Governance System

Student Performance Analytics using Data Warehouse in E-Governance System Performance Analytics using Data Warehouse in E-Governance System S S Suresh Asst. Professor, ASCT Department, International Institute of Information Technology, Pune, India ABSTRACT Data warehouse (DWH)

More information

An Overview of Data Warehousing, Data mining, OLAP and OLTP Technologies

An Overview of Data Warehousing, Data mining, OLAP and OLTP Technologies An Overview of Data Warehousing, Data mining, OLAP and OLTP Technologies Ashish Gahlot, Manoj Yadav Dronacharya college of engineering Farrukhnagar, Gurgaon,Haryana Abstract- Data warehousing, Data Mining,

More information

ETL-EXTRACT, TRANSFORM & LOAD TESTING

ETL-EXTRACT, TRANSFORM & LOAD TESTING ETL-EXTRACT, TRANSFORM & LOAD TESTING Rajesh Popli Manager (Quality), Nagarro Software Pvt. Ltd., Gurgaon, INDIA rajesh.popli@nagarro.com ABSTRACT Data is most important part in any organization. Data

More information

Databases in Organizations

Databases in Organizations The following is an excerpt from a draft chapter of a new enterprise architecture text book that is currently under development entitled Enterprise Architecture: Principles and Practice by Brian Cameron

More information

MDM and Data Warehousing Complement Each Other

MDM and Data Warehousing Complement Each Other Master Management MDM and Warehousing Complement Each Other Greater business value from both 2011 IBM Corporation Executive Summary Master Management (MDM) and Warehousing (DW) complement each other There

More information

A Design and implementation of a data warehouse for research administration universities

A Design and implementation of a data warehouse for research administration universities A Design and implementation of a data warehouse for research administration universities André Flory 1, Pierre Soupirot 2, and Anne Tchounikine 3 1 CRI : Centre de Ressources Informatiques INSA de Lyon

More information

OLAP OLAP. Data Warehouse. OLAP Data Model: the Data Cube S e s s io n

OLAP OLAP. Data Warehouse. OLAP Data Model: the Data Cube S e s s io n OLAP OLAP On-Line Analytical Processing In contrast to on-line transaction processing (OLTP) Mostly ad hoc queries involving aggregation Response time rather than throughput is the main performance measure.

More information

Oracle9i Data Warehouse Review. Robert F. Edwards Dulcian, Inc.

Oracle9i Data Warehouse Review. Robert F. Edwards Dulcian, Inc. Oracle9i Data Warehouse Review Robert F. Edwards Dulcian, Inc. Agenda Oracle9i Server OLAP Server Analytical SQL Data Mining ETL Warehouse Builder 3i Oracle 9i Server Overview 9i Server = Data Warehouse

More information

Data warehouse and Business Intelligence Collateral

Data warehouse and Business Intelligence Collateral Data warehouse and Business Intelligence Collateral Page 1 of 12 DATA WAREHOUSE AND BUSINESS INTELLIGENCE COLLATERAL Brains for the corporate brawn: In the current scenario of the business world, the competition

More information

IBM WebSphere DataStage Online training from Yes-M Systems

IBM WebSphere DataStage Online training from Yes-M Systems Yes-M Systems offers the unique opportunity to aspiring fresher s and experienced professionals to get real time experience in ETL Data warehouse tool IBM DataStage. Course Description With this training

More information

CHAPTER 3. Data Warehouses and OLAP

CHAPTER 3. Data Warehouses and OLAP CHAPTER 3 Data Warehouses and OLAP 3.1 Data Warehouse 3.2 Differences between Operational Systems and Data Warehouses 3.3 A Multidimensional Data Model 3.4Stars, snowflakes and Fact Constellations: 3.5

More information

CHAPTER 4: BUSINESS ANALYTICS

CHAPTER 4: BUSINESS ANALYTICS Chapter 4: Business Analytics CHAPTER 4: BUSINESS ANALYTICS Objectives Introduction The objectives are: Describe Business Analytics Explain the terminology associated with Business Analytics Describe the

More information