Rocky Mountain Technology Ventures. Exploring the Intricacies and Processes Involving the Extraction, Transformation and Loading of Data

Size: px
Start display at page:

Download "Rocky Mountain Technology Ventures. Exploring the Intricacies and Processes Involving the Extraction, Transformation and Loading of Data"

From this document you will learn the answers to the following questions:

  • What is the second step in the planning and design thread that Kimball and Caserta discussed?

  • What is the main concern of the ETL process?

  • What is the data flow component of the ETL process?

Transcription

1 Rocky Mountain Technology Ventures Exploring the Intricacies and Processes Involving the Extraction, Transformation and Loading of Data 3/25/2006

2 Introduction As data warehousing, OLAP architectures, Decision Support Systems and Business Intelligence concepts become increasingly important to organizations, an augmented level of significance is placed upon Extract-Transform-Load (ETL) processes. An ETL system is the underlying catalyst between typical transactional based systems and architectures that are optimized for Business Intelligence. The common best practice to decouple OLTP and OLAP systems has created the necessity to map and move data between these two homologous constructs in a generalized three-step processes. This three step process consists of First; the data in a transactional based system must be extracted. This consists of connecting to the transactional data source and then moving a copy of the data into the data warehouse. Second it is necessary to transform the data. Transformation involves fixing, cleansing, replacing and aggregating data as a preparatory step prior to presenting business users to the data in a denormalized form. Lastly, the data must be loaded into the target destination. During this phase the data that was prepared during the transformation step is copied into an alternate data warehouse structure that will facilitate business intelligence efforts such as a snowflake or simple star schema. The following sections will outline various perspectives regarding ETL systems. For example, the work that has been done in regards to ETL methodologies and standardizations will be reviewed. The technologies encompassing the ETL process are considered to be relatively new. In addition, an ETL system can be very intricate and complex, hence an attempt is being made to create a methodology that offers a systematic approach that organizes and simplifies the implementation of the said systems. One set

3 of techniques offered by Ralph Kimball will be explored. Next, the intricacies involving the use of ETL tools will be discussed. There have been several pieces of software that have been written that serve the sole purpose of creating and maintaining ETL systems. There are many different types of ETL tools that several of which will be examined below. ETL Methodology The ETL component within a data warehouse project has been compared to an iceberg (Kimball & Ross, 2002). When making this analogy, Kimball was referring to the fact that both an ETL implementation and an iceberg may appear simple and unassuming upon cursory analysis; however upon further inspection, can be extremely complex and overwhelming. The ETL portion of a data warehouse offering is often the source of many problems of which can potentially result in the failure of a given project. This stems from not only the innate technical complexity but the also the often intricate business challenges that are required to be resolved (Earls, 2003). For this reason several individuals have begun to create systematic approaches in order to simplify the potentially daunting task of completing the ETL. One such methodology developed by Ralph Kimball (2004), breaks the ETL implementation process into two distinct groups. First, the ETL process should be understood from a planning and design thread and then also by a data flow thread (Kimball & Caserta, 2004). The planning and design portion of this methodology offers a higher level perspective of the project and consists of four following sections: requirements and realities, architecture, system implementation, along with test and release. The requirements step consists of gathering all end user or business requirements along with

4 any other variables. In addition to establishing the business needs during this step, compliance necessities are investigated such as Sarbanes-Oxley or the Children s Online Privacy Act. Oftentimes specific steps must be taken in order to ensure compliance within the ETL process. The next component within the requirements step consists of performing data profiling analysis. This step is performed to better understand the state, quantity and quality of the data within the system. When analyzing the requirements and realities of a system it is also important to understand security requirements, data interaction issues along with what is required in regards to data expediency and longevity. The second component within the planning and design thread that Kimball and Caserta (2004) discuss is the architecture step. The architecture step consists of developing a high level strategy of how the requirements will be fulfilled. This step can often have large implications in regards to a given project, for many of the technological assets and constraints will be defined here. One of the major components within the architecture step consists of opting to use an ETL tool or hand code the solution. This decision can have a major impact on initial costs and future return on investment (ROI). Also within the architecture step, it is necessary to evaluate whether a streaming data style will be used or some type of batch bulk ETL system will be employed. Other factors that must be taken into consideration during this step include understanding exception handling, deciding how quality handling should be achieved, designing scheduling and automation steps along with applying a plan of attack in order to address any security requirements.

5 The third step in the planning and design thread is system implementation. This involves establishing implementation practices for both hardware and software along with establishing coding and documentation practices and performing quality checks. This step consists of designing an execution plan the other components found within the data design thread.. The last constituent found within the planning and design thread outlined by Kimball and Caserta (2004), consists of the test and release step. This step consists of the blueprint for the development, testing, UAT and production environments. Within this step, promotion and hand-off practices are established. In addition, maintenance policies and procedures are outlined along with the steps needed to carry out any performance tuning or roll back steps. Conversely and in addition to the planning and design thread, Kimball and Caserta (2004), provide a data flow thread perspective in an ETL implementation that consist of the following steps: Extract, Clean, Conform, Deliver and Operations. Essentially, the data flow step consists of the more technical side of delivering the ETL project. Each step that occurred in the planning and design stages will serve as a framework in executing the data flow steps. The extract portion of the data flow thread consists of not only reading the source data but also capturing changed data, intercepting notifications and daemons along with staging the extracted data to a specified area. The cleaning portion of the data flow thread involves regulation of column properties, database structure and value constraints. In addition, business rules must be enforced in this stage. At this point the cleansed data is again extracted and staged to a specified area.

6 The conform step is closely related to the cleaning step as it is another transformation step. During this step, both business labels and metrics are established in a common format. This adheres to the idea that a data assembly centric approach facilitates business intelligence (Kelly, n.d). When dealing with multiple data sources, data with the same meaning can vary greatly. For example, The business label Total Sales may consist of TltSales in one source, and TOTAL_SALES in a second source. Also, the metric collected for total sales may consist of $4.95 in one data source and in a second source. All of these variances must be standardized during the conform step. In addition, any erroneous data or duplicate data is removed from the system and lastly added to a staging environment. The fourth step in the data flow thread consists of delivering the data to the specified final destination. This involves loading the various facts and dimensions into the database structure offered to the business users. This portion of the ETL can also be very difficult due to the wide verity of dimension and aggregated facts within the system. The last component within the data flow thread is the operations element. The operations element is not a specific step in the data flow process, but more of a constituent that oversees the entire data flow process. This component involves issues such as quality inspection, data verification, exception handling, job execution and scheduling. ETL Tools Many organizations complete any necessary ETL work by hand coding the solution with resources within the company. However, other business entities choose to utilize commercial off-the-shelf (COTS) systems. Choosing whether or not to use an

7 ETL tool can have major implications on the project at hand but also on future projects. If a company does indeed decide to utilize an ETL tool, there is an extremely diverse range of tools on the market. The ETL software suites that are currently available fall into one of three generations of tools (Montcheuil & Depupet, n.d.) operating system code to perform ETL processes. First generation tools use native This was oftentimes written in COBAL because these systems were frequently housed on mainframes. First generation tools were usually hard to maintain due to the fact that in-depth knowledge of the code was required and the code differed across platforms. Second generation ETL solutions created a single source where ETL was performed, hence all the code for the ETL occurred in one area. This resulted in a consolidation of the first generation solutions so maintenance became much easier. However, since the ETL processing takes place all in one area with a second generation language, this can potentially result in a bottle neck environment. In addition, second generation languages process the ETL one row at a time, thus performance levels can be mediocre. Third generation languages take advantage of native SQL within a given Database Management System and attempt to perform bulk changes instead of row by row changes as in the second generation solution. The third generation ETL tool also offers a centralized design that facilitates maintenance and has the ability to distribute load, in order to achieve increased scalability. The upfront cost to using a hand coded solution can be much less expensive compared to going with a commercially built tool, however, at some point the financial aspect of the two approaches invert (Earls, 2003). Some companies that have decided to

8 implement their ETL without a tool have found that over time the data source formats and validation rules have changed on a regular basis to the extent where the ETL code must be manually changed an maintained on a frequent interval (Singing, 2004). Despite the fact that opting to use an ETL tool can greatly reduce implementation times, enhance scalability and reduce the efforts needed for maintenance. An ETL tool oftentimes requires a high initial investment. The average entry level investment for a commercially based ETL tool is approximately $75,000 to $100,000. However, upon adding other essentials such as maintenance plans, extra development licenses the average is closer to $200,000 (Earls, 2003). Another option in obtaining an ETL tool is going with an open source solution such as KETL, CloverETL or Enhydra Octopus. This can obviously save a great deal of capital upon the initial cost of the software, however there are other drawbacks that need to be measured when taking into consideration an open source solution. Generally speaking the currently available open source ETL tools are designed for developers who are well versed in ETL intricacies (Howard, 2005). Hence these solutions are often unintuitive and do not include a graphical user interface. Also, some of the open source ETL developers admittedly state that the implementation of their product may require outside assistance (Howard, 2005). Despite the fact that additional resources for tool implementation and specialized skill sets may be necessary to run an open source solution this type of tool could be extremely cost effective and should be considered. Summary Overall, due to the inherent complexity of ETL, some methodologies and best practices need to be established in order to facilitate control over projects. The newly

9 introduced techniques described by Kimball and Caserta (2004) may prove not to be the commonly accepted practices in the future. However the mere establishment of an ETL methodology provides the industry a building block in which to promote progress. In addition, businesses should heavily weigh the implications of the decisions made regarding ETL tools. The presence or absence of ETL software can have major impacts on the profitability of an organization. It has been estimated that over 70% of the resources for a given data warehouse project are allocated to ETL activities (Kimball & Caserta 2004). Due to the overwhelming large demand that ETL processes require, special attention and time should be taken to ensure that ETL development occurs as efficiently and as smoothly as possible.

10 Literature Cited Kimball, R. & Ross M. (2002). The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling Second ed. New York: John Wiley & Sons, Incorporated. Kimball, R. & Caserta J. (2004). The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning Conforming, and Delivering Data. Indianapolis: Wiley Publishing, Incorporated. Earls, A. R. (2003) ETL: Preparation Is the Best Bet. Computer World, International Data Group Incorporated. Retrieved March 25, 2006, from arehouse/story/0,10801,84224,00.html?from=story_picks Songini, M. L. (2004) ETL: Quickstudy. Computer World, International Data Group Incorporated. Retrieved March 25, 2006, from arehouse/story/0,10801,89534,00.html Kelly, C. (n.d) Navigating Business Intelligence An Architected Data Approach. ITNetwork365, Business Intelligence.com. Retrieved March 25, 2006, from m=1 Howard, P. (2005) The Case for Open Source ETL. Bloor Research, Retrieved March 25, 2006, from Montcheuil, Y. & Depupet C. (n.d.) Third Generation ETL: Delivering the Best Performance. Sunopsis Incorporated. Retrieved March 25, 2006, from

An Introduction to Data Warehousing. An organization manages information in two dominant forms: operational systems of

An Introduction to Data Warehousing. An organization manages information in two dominant forms: operational systems of An Introduction to Data Warehousing An organization manages information in two dominant forms: operational systems of record and data warehouses. Operational systems are designed to support online transaction

More information

The Data Warehouse ETL Toolkit

The Data Warehouse ETL Toolkit 2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. The Data Warehouse ETL Toolkit Practical Techniques for Extracting,

More information

THE DATA WAREHOUSE ETL TOOLKIT CDT803 Three Days

THE DATA WAREHOUSE ETL TOOLKIT CDT803 Three Days Three Days Prerequisites Students should have at least some experience with any relational database management system. Who Should Attend This course is targeted at technical staff, team leaders and project

More information

The Benefits of Data Modeling in Data Warehousing

The Benefits of Data Modeling in Data Warehousing WHITE PAPER: THE BENEFITS OF DATA MODELING IN DATA WAREHOUSING The Benefits of Data Modeling in Data Warehousing NOVEMBER 2008 Table of Contents Executive Summary 1 SECTION 1 2 Introduction 2 SECTION 2

More information

East Asia Network Sdn Bhd

East Asia Network Sdn Bhd Course: Analyzing, Designing, and Implementing a Data Warehouse with Microsoft SQL Server 2014 Elements of this syllabus may be change to cater to the participants background & knowledge. This course describes

More information

Chapter 6 Basics of Data Integration. Fundamentals of Business Analytics RN Prasad and Seema Acharya

Chapter 6 Basics of Data Integration. Fundamentals of Business Analytics RN Prasad and Seema Acharya Chapter 6 Basics of Data Integration Fundamentals of Business Analytics Learning Objectives and Learning Outcomes Learning Objectives 1. Concepts of data integration 2. Needs and advantages of using data

More information

COURSE 20463C: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER

COURSE 20463C: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER Page 1 of 8 ABOUT THIS COURSE This 5 day course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse with Microsoft SQL Server

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server Page 1 of 7 Overview This course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse with Microsoft SQL 2014, implement ETL

More information

Course Outline. Module 1: Introduction to Data Warehousing

Course Outline. Module 1: Introduction to Data Warehousing Course Outline Module 1: Introduction to Data Warehousing This module provides an introduction to the key components of a data warehousing solution and the highlevel considerations you must take into account

More information

ETL-EXTRACT, TRANSFORM & LOAD TESTING

ETL-EXTRACT, TRANSFORM & LOAD TESTING ETL-EXTRACT, TRANSFORM & LOAD TESTING Rajesh Popli Manager (Quality), Nagarro Software Pvt. Ltd., Gurgaon, INDIA rajesh.popli@nagarro.com ABSTRACT Data is most important part in any organization. Data

More information

Data Warehousing and Data Mining

Data Warehousing and Data Mining Data Warehousing and Data Mining Part I: Data Warehousing Gao Cong gaocong@cs.aau.dk Slides adapted from Man Lung Yiu and Torben Bach Pedersen Course Structure Business intelligence: Extract knowledge

More information

Course 10777A: Implementing a Data Warehouse with Microsoft SQL Server 2012

Course 10777A: Implementing a Data Warehouse with Microsoft SQL Server 2012 Course 10777A: Implementing a Data Warehouse with Microsoft SQL Server 2012 OVERVIEW About this Course Data warehousing is a solution organizations use to centralize business data for reporting and analysis.

More information

Original Research Articles

Original Research Articles Original Research Articles Researchers Sweety Patel Department of Computer Science, Fairleigh Dickinson University, USA Email- sweetu83patel@yahoo.com Different Data Warehouse Architecture Creation Criteria

More information

MIS636 AWS Data Warehousing and Business Intelligence Course Syllabus

MIS636 AWS Data Warehousing and Business Intelligence Course Syllabus MIS636 AWS Data Warehousing and Business Intelligence Course Syllabus I. Contact Information Professor: Joseph Morabito, Ph.D. Office: Babbio 419 Office Hours: By Appt. Phone: 201-216-5304 Email: jmorabit@stevens.edu

More information

Dimensional Modeling and E-R Modeling In. Joseph M. Firestone, Ph.D. White Paper No. Eight. June 22, 1998

Dimensional Modeling and E-R Modeling In. Joseph M. Firestone, Ph.D. White Paper No. Eight. June 22, 1998 1 of 9 5/24/02 3:47 PM Dimensional Modeling and E-R Modeling In The Data Warehouse By Joseph M. Firestone, Ph.D. White Paper No. Eight June 22, 1998 Introduction Dimensional Modeling (DM) is a favorite

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012

Implementing a Data Warehouse with Microsoft SQL Server 2012 Course 10777A: Implementing a Data Warehouse with Microsoft SQL Server 2012 Length: Audience(s): 5 Days Level: 200 IT Professionals Technology: Microsoft SQL Server 2012 Type: Delivery Method: Course Instructor-led

More information

Implementing a Data Warehouse with Microsoft SQL Server MOC 20463

Implementing a Data Warehouse with Microsoft SQL Server MOC 20463 Implementing a Data Warehouse with Microsoft SQL Server MOC 20463 Course Outline Module 1: Introduction to Data Warehousing This module provides an introduction to the key components of a data warehousing

More information

COURSE OUTLINE MOC 20463: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER

COURSE OUTLINE MOC 20463: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER COURSE OUTLINE MOC 20463: IMPLEMENTING A DATA WAREHOUSE WITH MICROSOFT SQL SERVER MODULE 1: INTRODUCTION TO DATA WAREHOUSING This module provides an introduction to the key components of a data warehousing

More information

SQL Server 2012 Business Intelligence Boot Camp

SQL Server 2012 Business Intelligence Boot Camp SQL Server 2012 Business Intelligence Boot Camp Length: 5 Days Technology: Microsoft SQL Server 2012 Delivery Method: Instructor-led (classroom) About this Course Data warehousing is a solution organizations

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012

Implementing a Data Warehouse with Microsoft SQL Server 2012 Course 10777 : Implementing a Data Warehouse with Microsoft SQL Server 2012 Page 1 of 8 Implementing a Data Warehouse with Microsoft SQL Server 2012 Course 10777: 4 days; Instructor-Led Introduction Data

More information

META DATA QUALITY CONTROL ARCHITECTURE IN DATA WAREHOUSING

META DATA QUALITY CONTROL ARCHITECTURE IN DATA WAREHOUSING META DATA QUALITY CONTROL ARCHITECTURE IN DATA WAREHOUSING Ramesh Babu Palepu 1, Dr K V Sambasiva Rao 2 Dept of IT, Amrita Sai Institute of Science & Technology 1 MVR College of Engineering 2 asistithod@gmail.com

More information

Implement a Data Warehouse with Microsoft SQL Server 20463C; 5 days

Implement a Data Warehouse with Microsoft SQL Server 20463C; 5 days Lincoln Land Community College Capital City Training Center 130 West Mason Springfield, IL 62702 217-782-7436 www.llcc.edu/cctc Implement a Data Warehouse with Microsoft SQL Server 20463C; 5 days Course

More information

Unlock your data for fast insights: dimensionless modeling with in-memory column store. By Vadim Orlov

Unlock your data for fast insights: dimensionless modeling with in-memory column store. By Vadim Orlov Unlock your data for fast insights: dimensionless modeling with in-memory column store By Vadim Orlov I. DIMENSIONAL MODEL Dimensional modeling (also known as star or snowflake schema) was pioneered by

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server This course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse 2014, implement ETL with SQL Server Integration Services, and

More information

Data Warehouse Overview. Srini Rengarajan

Data Warehouse Overview. Srini Rengarajan Data Warehouse Overview Srini Rengarajan Please mute Your cell! Agenda Data Warehouse Architecture Approaches to build a Data Warehouse Top Down Approach Bottom Up Approach Best Practices Case Example

More information

When to consider OLAP?

When to consider OLAP? When to consider OLAP? Author: Prakash Kewalramani Organization: Evaltech, Inc. Evaltech Research Group, Data Warehousing Practice. Date: 03/10/08 Email: erg@evaltech.com Abstract: Do you need an OLAP

More information

Data Warehousing Concepts

Data Warehousing Concepts Data Warehousing Concepts JB Software and Consulting Inc 1333 McDermott Drive, Suite 200 Allen, TX 75013. [[[[[ DATA WAREHOUSING What is a Data Warehouse? Decision Support Systems (DSS), provides an analysis

More information

Understanding Data Warehousing. [by Alex Kriegel]

Understanding Data Warehousing. [by Alex Kriegel] Understanding Data Warehousing 2008 [by Alex Kriegel] Things to Discuss Who Needs a Data Warehouse? OLTP vs. Data Warehouse Business Intelligence Industrial Landscape Which Data Warehouse: Bill Inmon vs.

More information

Data Quality Assessment. Approach

Data Quality Assessment. Approach Approach Prepared By: Sanjay Seth Data Quality Assessment Approach-Review.doc Page 1 of 15 Introduction Data quality is crucial to the success of Business Intelligence initiatives. Unless data in source

More information

Data Testing on Business Intelligence & Data Warehouse Projects

Data Testing on Business Intelligence & Data Warehouse Projects Data Testing on Business Intelligence & Data Warehouse Projects Karen N. Johnson 1 Construct of a Data Warehouse A brief look at core components of a warehouse. From the left, these three boxes represent

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012 (70-463)

Implementing a Data Warehouse with Microsoft SQL Server 2012 (70-463) Implementing a Data Warehouse with Microsoft SQL Server 2012 (70-463) Course Description Data warehousing is a solution organizations use to centralize business data for reporting and analysis. This five-day

More information

Designing a Dimensional Model

Designing a Dimensional Model Designing a Dimensional Model Erik Veerman Atlanta MDF member SQL Server MVP, Microsoft MCT Mentor, Solid Quality Learning Definitions Data Warehousing A subject-oriented, integrated, time-variant, and

More information

JOURNAL OF OBJECT TECHNOLOGY

JOURNAL OF OBJECT TECHNOLOGY JOURNAL OF OBJECT TECHNOLOGY Online at www.jot.fm. Published by ETH Zurich, Chair of Software Engineering JOT, 2008 Vol. 7, No. 8, November-December 2008 What s Your Information Agenda? Mahesh H. Dodani,

More information

Master Data Management. Zahra Mansoori

Master Data Management. Zahra Mansoori Master Data Management Zahra Mansoori 1 1. Preference 2 A critical question arises How do you get from a thousand points of data entry to a single view of the business? We are going to answer this question

More information

Data Warehouse Snowflake Design and Performance Considerations in Business Analytics

Data Warehouse Snowflake Design and Performance Considerations in Business Analytics Journal of Advances in Information Technology Vol. 6, No. 4, November 2015 Data Warehouse Snowflake Design and Performance Considerations in Business Analytics Jiangping Wang and Janet L. Kourik Walker

More information

Course Outline: Course: Implementing a Data Warehouse with Microsoft SQL Server 2012 Learning Method: Instructor-led Classroom Learning

Course Outline: Course: Implementing a Data Warehouse with Microsoft SQL Server 2012 Learning Method: Instructor-led Classroom Learning Course Outline: Course: Implementing a Data with Microsoft SQL Server 2012 Learning Method: Instructor-led Classroom Learning Duration: 5.00 Day(s)/ 40 hrs Overview: This 5-day instructor-led course describes

More information

Data Warehousing Fundamentals for IT Professionals. 2nd Edition

Data Warehousing Fundamentals for IT Professionals. 2nd Edition Brochure More information from http://www.researchandmarkets.com/reports/2171973/ Data Warehousing Fundamentals for IT Professionals. 2nd Edition Description: Cutting-edge content and guidance from a data

More information

SENG 520, Experience with a high-level programming language. (304) 579-7726, Jeff.Edgell@comcast.net

SENG 520, Experience with a high-level programming language. (304) 579-7726, Jeff.Edgell@comcast.net Course : Semester : Course Format And Credit hours : Prerequisites : Data Warehousing and Business Intelligence Summer (Odd Years) online 3 hr Credit SENG 520, Experience with a high-level programming

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012 MOC 10777

Implementing a Data Warehouse with Microsoft SQL Server 2012 MOC 10777 Implementing a Data Warehouse with Microsoft SQL Server 2012 MOC 10777 Course Outline Module 1: Introduction to Data Warehousing This module provides an introduction to the key components of a data warehousing

More information

www.ijreat.org Published by: PIONEER RESEARCH & DEVELOPMENT GROUP (www.prdg.org) 28

www.ijreat.org Published by: PIONEER RESEARCH & DEVELOPMENT GROUP (www.prdg.org) 28 Data Warehousing - Essential Element To Support Decision- Making Process In Industries Ashima Bhasin 1, Mr Manoj Kumar 2 1 Computer Science Engineering Department, 2 Associate Professor, CSE Abstract SGT

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012

Implementing a Data Warehouse with Microsoft SQL Server 2012 Implementing a Data Warehouse with Microsoft SQL Server 2012 Module 1: Introduction to Data Warehousing Describe data warehouse concepts and architecture considerations Considerations for a Data Warehouse

More information

Mario Guarracino. Data warehousing

Mario Guarracino. Data warehousing Data warehousing Introduction Since the mid-nineties, it became clear that the databases for analysis and business intelligence need to be separate from operational. In this lecture we will review the

More information

POLAR IT SERVICES. Business Intelligence Project Methodology

POLAR IT SERVICES. Business Intelligence Project Methodology POLAR IT SERVICES Business Intelligence Project Methodology Table of Contents 1. Overview... 2 2. Visualize... 3 3. Planning and Architecture... 4 3.1 Define Requirements... 4 3.1.1 Define Attributes...

More information

Trends in Data Warehouse Data Modeling: Data Vault and Anchor Modeling

Trends in Data Warehouse Data Modeling: Data Vault and Anchor Modeling Trends in Data Warehouse Data Modeling: Data Vault and Anchor Modeling Thanks for Attending! Roland Bouman, Leiden the Netherlands MySQL AB, Sun, Strukton, Pentaho (1 nov) Web- and Business Intelligence

More information

A Case Study in Integrated Quality Assurance for Performance Management Systems

A Case Study in Integrated Quality Assurance for Performance Management Systems A Case Study in Integrated Quality Assurance for Performance Management Systems Liam Peyton, Bo Zhan, Bernard Stepien School of Information Technology and Engineering, University of Ottawa, 800 King Edward

More information

Microsoft. Course 20463C: Implementing a Data Warehouse with Microsoft SQL Server

Microsoft. Course 20463C: Implementing a Data Warehouse with Microsoft SQL Server Course 20463C: Implementing a Data Warehouse with Microsoft SQL Server Length : 5 Days Audience(s) : IT Professionals Level : 300 Technology : Microsoft SQL Server 2014 Delivery Method : Instructor-led

More information

Sterling Business Intelligence

Sterling Business Intelligence Sterling Business Intelligence Concepts Guide Release 9.0 March 2010 Copyright 2009 Sterling Commerce, Inc. All rights reserved. Additional copyright information is located on the documentation library:

More information

Data Warehousing: A Technology Review and Update Vernon Hoffner, Ph.D., CCP EntreSoft Resouces, Inc.

Data Warehousing: A Technology Review and Update Vernon Hoffner, Ph.D., CCP EntreSoft Resouces, Inc. Warehousing: A Technology Review and Update Vernon Hoffner, Ph.D., CCP EntreSoft Resouces, Inc. Introduction Abstract warehousing has been around for over a decade. Therefore, when you read the articles

More information

Data Warehousing and OLAP Technology for Knowledge Discovery

Data Warehousing and OLAP Technology for Knowledge Discovery 542 Data Warehousing and OLAP Technology for Knowledge Discovery Aparajita Suman Abstract Since time immemorial, libraries have been generating services using the knowledge stored in various repositories

More information

Deductive Data Warehouses and Aggregate (Derived) Tables

Deductive Data Warehouses and Aggregate (Derived) Tables Deductive Data Warehouses and Aggregate (Derived) Tables Kornelije Rabuzin, Mirko Malekovic, Mirko Cubrilo Faculty of Organization and Informatics University of Zagreb Varazdin, Croatia {kornelije.rabuzin,

More information

BUILDING OLAP TOOLS OVER LARGE DATABASES

BUILDING OLAP TOOLS OVER LARGE DATABASES BUILDING OLAP TOOLS OVER LARGE DATABASES Rui Oliveira, Jorge Bernardino ISEC Instituto Superior de Engenharia de Coimbra, Polytechnic Institute of Coimbra Quinta da Nora, Rua Pedro Nunes, P-3030-199 Coimbra,

More information

IST722 Data Warehousing

IST722 Data Warehousing IST722 Data Warehousing Components of the Data Warehouse Michael A. Fudge, Jr. Recall: Inmon s CIF The CIF is a reference architecture Understanding the Diagram The CIF is a reference architecture CIF

More information

Sizing Logical Data in a Data Warehouse A Consistent and Auditable Approach

Sizing Logical Data in a Data Warehouse A Consistent and Auditable Approach 2006 ISMA Conference 1 Sizing Logical Data in a Data Warehouse A Consistent and Auditable Approach Priya Lobo CFPS Satyam Computer Services Ltd. 69, Railway Parallel Road, Kumarapark West, Bangalore 560020,

More information

Data Integration and ETL Process

Data Integration and ETL Process Data Integration and ETL Process Krzysztof Dembczyński Intelligent Decision Support Systems Laboratory (IDSS) Poznań University of Technology, Poland Software Development Technologies Master studies, second

More information

QAD Business Intelligence Data Warehouse Demonstration Guide. May 2015 BI 3.11

QAD Business Intelligence Data Warehouse Demonstration Guide. May 2015 BI 3.11 QAD Business Intelligence Data Warehouse Demonstration Guide May 2015 BI 3.11 Overview This demonstration focuses on the foundation of QAD Business Intelligence the Data Warehouse and shows how this functionality

More information

Trivadis White Paper. Comparison of Data Modeling Methods for a Core Data Warehouse. Dani Schnider Adriano Martino Maren Eschermann

Trivadis White Paper. Comparison of Data Modeling Methods for a Core Data Warehouse. Dani Schnider Adriano Martino Maren Eschermann Trivadis White Paper Comparison of Data Modeling Methods for a Core Data Warehouse Dani Schnider Adriano Martino Maren Eschermann June 2014 Table of Contents 1. Introduction... 3 2. Aspects of Data Warehouse

More information

A Survey of Real-Time Data Warehouse and ETL

A Survey of Real-Time Data Warehouse and ETL Fahd Sabry Esmail Ali A Survey of Real-Time Data Warehouse and ETL Article Info: Received 09 July 2014 Accepted 24 August 2014 UDC 004.6 Recommended citation: Esmail Ali, F.S. (2014). A Survey of Real-

More information

Course 20463:Implementing a Data Warehouse with Microsoft SQL Server

Course 20463:Implementing a Data Warehouse with Microsoft SQL Server Course 20463:Implementing a Data Warehouse with Microsoft SQL Server Type:Course Audience(s):IT Professionals Technology:Microsoft SQL Server Level:300 This Revision:C Delivery method: Instructor-led (classroom)

More information

Microsoft Data Warehouse in Depth

Microsoft Data Warehouse in Depth Microsoft Data Warehouse in Depth 1 P a g e Duration What s new Why attend Who should attend Course format and prerequisites 4 days The course materials have been refreshed to align with the second edition

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server CÔNG TY CỔ PHẦN TRƯỜNG CNTT TÂN ĐỨC TAN DUC INFORMATION TECHNOLOGY SCHOOL JSC LEARN MORE WITH LESS! Course 20463 Implementing a Data Warehouse with Microsoft SQL Server Length: 5 Days Audience: IT Professionals

More information

CHAPTER 3. Data Warehouses and OLAP

CHAPTER 3. Data Warehouses and OLAP CHAPTER 3 Data Warehouses and OLAP 3.1 Data Warehouse 3.2 Differences between Operational Systems and Data Warehouses 3.3 A Multidimensional Data Model 3.4Stars, snowflakes and Fact Constellations: 3.5

More information

Dimensional Modeling 101. Presented by: Michael Davis CEO OmegaSoft,LLC

Dimensional Modeling 101. Presented by: Michael Davis CEO OmegaSoft,LLC Dimensional Modeling 101 Presented by: Michael Davis CEO OmegaSoft,LLC Agenda Brief history of Database Design Dimension Modeling Terminology Case study overview 4 step Dimensional Modeling Process Additional

More information

Implementing a Data Warehouse with Microsoft SQL Server

Implementing a Data Warehouse with Microsoft SQL Server Course Code: M20463 Vendor: Microsoft Course Overview Duration: 5 RRP: 2,025 Implementing a Data Warehouse with Microsoft SQL Server Overview This course describes how to implement a data warehouse platform

More information

Course Design Document. IS417: Data Warehousing and Business Analytics

Course Design Document. IS417: Data Warehousing and Business Analytics Course Design Document IS417: Data Warehousing and Business Analytics Version 2.1 20 June 2009 IS417 Data Warehousing and Business Analytics Page 1 Table of Contents 1. Versions History... 3 2. Overview

More information

Implementing a Data Warehouse with Microsoft SQL Server 2012

Implementing a Data Warehouse with Microsoft SQL Server 2012 Implementing a Data Warehouse with Microsoft SQL Server 2012 Course ID MSS300 Course Description Ace your preparation for Microsoft Certification Exam 70-463 with this course Maximize your performance

More information

Multidimensional Modeling - Stocks

Multidimensional Modeling - Stocks Bases de Dados e Data Warehouse 06 BDDW 2006/2007 Notice! Author " João Moura Pires (jmp@di.fct.unl.pt)! This material can be freely used for personal or academic purposes without any previous authorization

More information

Data Warehousing Systems: Foundations and Architectures

Data Warehousing Systems: Foundations and Architectures Data Warehousing Systems: Foundations and Architectures Il-Yeol Song Drexel University, http://www.ischool.drexel.edu/faculty/song/ SYNONYMS None DEFINITION A data warehouse (DW) is an integrated repository

More information

Business Intelligence

Business Intelligence 8 Business Intelligence Business intelligence has become a buzzword in recent years. The database tools found under the heading of business intelligence include data warehousing, online analytical processing

More information

CHAPTER SIX DATA. Business Intelligence. 2011 The McGraw-Hill Companies, All Rights Reserved

CHAPTER SIX DATA. Business Intelligence. 2011 The McGraw-Hill Companies, All Rights Reserved CHAPTER SIX DATA Business Intelligence 2011 The McGraw-Hill Companies, All Rights Reserved 2 CHAPTER OVERVIEW SECTION 6.1 Data, Information, Databases The Business Benefits of High-Quality Information

More information

Data Warehouse design

Data Warehouse design Data Warehouse design Design of Enterprise Systems University of Pavia 11/11/2013-1- Data Warehouse design DATA MODELLING - 2- Data Modelling Important premise Data warehouses typically reside on a RDBMS

More information

Emerging Technologies Shaping the Future of Data Warehouses & Business Intelligence

Emerging Technologies Shaping the Future of Data Warehouses & Business Intelligence Emerging Technologies Shaping the Future of Data Warehouses & Business Intelligence Appliances and DW Architectures John O Brien President and Executive Architect Zukeran Technologies 1 TDWI 1 Agenda What

More information

OLAP and OLTP. AMIT KUMAR BINDAL Associate Professor M M U MULLANA

OLAP and OLTP. AMIT KUMAR BINDAL Associate Professor M M U MULLANA OLAP and OLTP AMIT KUMAR BINDAL Associate Professor Databases Databases are developed on the IDEA that DATA is one of the critical materials of the Information Age Information, which is created by data,

More information

HYPERION MASTER DATA MANAGEMENT SOLUTIONS FOR IT

HYPERION MASTER DATA MANAGEMENT SOLUTIONS FOR IT HYPERION MASTER DATA MANAGEMENT SOLUTIONS FOR IT POINT-AND-SYNC MASTER DATA MANAGEMENT 04.2005 Hyperion s new master data management solution provides a centralized, transparent process for managing critical

More information

The Evolution of ETL

The Evolution of ETL The Evolution of ETL -From Hand-coded ETL to Tool-based ETL By Madhu Zode Data Warehousing & Business Intelligence Practice Page 1 of 13 ABSTRACT To build a data warehouse various tools are used like modeling

More information

Data warehouse and Business Intelligence Collateral

Data warehouse and Business Intelligence Collateral Data warehouse and Business Intelligence Collateral Page 1 of 12 DATA WAREHOUSE AND BUSINESS INTELLIGENCE COLLATERAL Brains for the corporate brawn: In the current scenario of the business world, the competition

More information

How to Enhance Traditional BI Architecture to Leverage Big Data

How to Enhance Traditional BI Architecture to Leverage Big Data B I G D ATA How to Enhance Traditional BI Architecture to Leverage Big Data Contents Executive Summary... 1 Traditional BI - DataStack 2.0 Architecture... 2 Benefits of Traditional BI - DataStack 2.0...

More information

STAT REVIEW/EXECUTIVE DASHBOARD TASK ORDER EXECUTIVE DASHBOARD SYNTHESIS OF BEST PRACTICES

STAT REVIEW/EXECUTIVE DASHBOARD TASK ORDER EXECUTIVE DASHBOARD SYNTHESIS OF BEST PRACTICES STAT REVIEW/EXECUTIVE DASHBOARD TASK ORDER EXECUTIVE DASHBOARD SYNTHESIS OF BEST PRACTICES October 26, 2011 This publication was produced for review by the United States Agency for International Development.

More information

Implementing a SQL Data Warehouse 2016

Implementing a SQL Data Warehouse 2016 Implementing a SQL Data Warehouse 2016 http://www.homnick.com marketing@homnick.com +1.561.988.0567 Boca Raton, Fl USA About this course This 4-day instructor led course describes how to implement a data

More information

Implementing a Data Warehouse with Microsoft SQL Server 2014

Implementing a Data Warehouse with Microsoft SQL Server 2014 Implementing a Data Warehouse with Microsoft SQL Server 2014 MOC 20463 Duración: 25 horas Introducción This course describes how to implement a data warehouse platform to support a BI solution. Students

More information

A Knowledge Management Framework Using Business Intelligence Solutions

A Knowledge Management Framework Using Business Intelligence Solutions www.ijcsi.org 102 A Knowledge Management Framework Using Business Intelligence Solutions Marwa Gadu 1 and Prof. Dr. Nashaat El-Khameesy 2 1 Computer and Information Systems Department, Sadat Academy For

More information

Data warehouses. Data Mining. Abraham Otero. Data Mining. Agenda

Data warehouses. Data Mining. Abraham Otero. Data Mining. Agenda Data warehouses 1/36 Agenda Why do I need a data warehouse? ETL systems Real-Time Data Warehousing Open problems 2/36 1 Why do I need a data warehouse? Why do I need a data warehouse? Maybe you do not

More information

Republic Polytechnic School of Information and Communications Technology C355 Business Intelligence. Module Curriculum

Republic Polytechnic School of Information and Communications Technology C355 Business Intelligence. Module Curriculum Republic Polytechnic School of Information and Communications Technology C355 Business Intelligence Module Curriculum This document addresses the content related abilities, with reference to the module.

More information

Near Real-time Data Warehousing with Multi-stage Trickle & Flip

Near Real-time Data Warehousing with Multi-stage Trickle & Flip Near Real-time Data Warehousing with Multi-stage Trickle & Flip Janis Zuters University of Latvia, 19 Raina blvd., LV-1586 Riga, Latvia janis.zuters@lu.lv Abstract. A data warehouse typically is a collection

More information

Data Ownership and Enterprise Data Management: Implementing a Data Management Strategy (Part 3)

Data Ownership and Enterprise Data Management: Implementing a Data Management Strategy (Part 3) A DataFlux White Paper Prepared by: Mike Ferguson Data Ownership and Enterprise Data Management: Implementing a Data Management Strategy (Part 3) Leader in Data Quality and Data Integration www.flux.com

More information

High-Volume Data Warehousing in Centerprise. Product Datasheet

High-Volume Data Warehousing in Centerprise. Product Datasheet High-Volume Data Warehousing in Centerprise Product Datasheet Table of Contents Overview 3 Data Complexity 3 Data Quality 3 Speed and Scalability 3 Centerprise Data Warehouse Features 4 ETL in a Unified

More information

SAS BI Course Content; Introduction to DWH / BI Concepts

SAS BI Course Content; Introduction to DWH / BI Concepts SAS BI Course Content; Introduction to DWH / BI Concepts SAS Web Report Studio 4.2 SAS EG 4.2 SAS Information Delivery Portal 4.2 SAS Data Integration Studio 4.2 SAS BI Dashboard 4.2 SAS Management Console

More information

LEARNING SOLUTIONS website milner.com/learning email training@milner.com phone 800 875 5042

LEARNING SOLUTIONS website milner.com/learning email training@milner.com phone 800 875 5042 Course 20467A: Designing Business Intelligence Solutions with Microsoft SQL Server 2012 Length: 5 Days Published: December 21, 2012 Language(s): English Audience(s): IT Professionals Overview Level: 300

More information

Dimensional Modeling for Data Warehouse

Dimensional Modeling for Data Warehouse Modeling for Data Warehouse Umashanker Sharma, Anjana Gosain GGS, Indraprastha University, Delhi Abstract Many surveys indicate that a significant percentage of DWs fail to meet business objectives or

More information

IS YOUR DATA WAREHOUSE SUCCESSFUL? Developing a Data Warehouse Process that responds to the needs of the Enterprise.

IS YOUR DATA WAREHOUSE SUCCESSFUL? Developing a Data Warehouse Process that responds to the needs of the Enterprise. IS YOUR DATA WAREHOUSE SUCCESSFUL? Developing a Data Warehouse Process that responds to the needs of the Enterprise. Peter R. Welbrock Smith-Hanley Consulting Group Philadelphia, PA ABSTRACT Developing

More information

Developing Business Intelligence and Data Visualization Applications with Web Maps

Developing Business Intelligence and Data Visualization Applications with Web Maps Developing Business Intelligence and Data Visualization Applications with Web Maps Introduction Business Intelligence (BI) means different things to different organizations and users. BI often refers to

More information

Business Intelligence: Effective Decision Making

Business Intelligence: Effective Decision Making Business Intelligence: Effective Decision Making Bellevue College Linda Rumans IT Instructor, Business Division Bellevue College lrumans@bellevuecollege.edu Current Status What do I do??? How do I increase

More information

The Power of Business Intelligence in the Revenue Cycle

The Power of Business Intelligence in the Revenue Cycle The Power of Business Intelligence in the Revenue Cycle Increasing Cash Flow with Actionable Information John Garcia August 4, 2011 Table of Contents Revenue Cycle Challenges... 3 The Goal of Business

More information

The Quality Data Warehouse: Solving Problems for the Enterprise

The Quality Data Warehouse: Solving Problems for the Enterprise The Quality Data Warehouse: Solving Problems for the Enterprise Bradley W. Klenz, SAS Institute Inc., Cary NC Donna O. Fulenwider, SAS Institute Inc., Cary NC ABSTRACT Enterprise quality improvement is

More information

Cúram Business Intelligence and Analytics Guide

Cúram Business Intelligence and Analytics Guide IBM Cúram Social Program Management Cúram Business Intelligence and Analytics Guide Version 6.0.4 Note Before using this information and the product it supports, read the information in Notices at the

More information

Challenges in developing a cost-effective data warehouse for a tertiary institution in a developing country

Challenges in developing a cost-effective data warehouse for a tertiary institution in a developing country Data Mining VII: Data, Text and Web Mining and their Business Applications 389 Challenges in developing a cost-effective data warehouse for a tertiary institution in a developing country A. Nazir & T.

More information

Beta: Implementing a Data Warehouse with Microsoft SQL Server 2012

Beta: Implementing a Data Warehouse with Microsoft SQL Server 2012 CÔNG TY CỔ PHẦN TRƯỜNG CNTT TÂN ĐỨC TAN DUC INFORMATION TECHNOLOGY SCHOOL JSC LEARN MORE WITH LESS! Course 10777: Beta: Implementing a Data Warehouse with Microsoft SQL Server 2012 Length: 5 Days Audience:

More information

ETL Process in Data Warehouse. G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT

ETL Process in Data Warehouse. G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT ETL Process in Data Warehouse G.Lakshmi Priya & Razia Sultana.A Assistant Professor/IT Outline ETL Extraction Transformation Loading ETL Overview Extraction Transformation Loading ETL To get data out of

More information

Presented by: Jose Chinchilla, MCITP

Presented by: Jose Chinchilla, MCITP Presented by: Jose Chinchilla, MCITP Jose Chinchilla MCITP: Database Administrator, SQL Server 2008 MCITP: Business Intelligence SQL Server 2008 Customers & Partners Current Positions: President, Agile

More information

MICHAEL SCHMITZ NOVEMBER 20-22, 2006 NOVEMBER 23-24, 2006 RESIDENZA DI RIPETTA - VIA DI RIPETTA, 231 ROME (ITALY)

MICHAEL SCHMITZ NOVEMBER 20-22, 2006 NOVEMBER 23-24, 2006 RESIDENZA DI RIPETTA - VIA DI RIPETTA, 231 ROME (ITALY) TECHNOLOGY TRANSFER PRESENTS MICHAEL SCHMITZ DATA WAREHOUSING Advanced Design and Implementation Issues ETL FOR THE DATA WAREHOUSE A Template-Driven Approach NOVEMBER 20-22, 2006 NOVEMBER 23-24, 2006 RESIDENZA

More information

European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project

European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project European Archival Records and Knowledge Preservation Database Archiving in the E-ARK Project Janet Delve, University of Portsmouth Kuldar Aas, National Archives of Estonia Rainer Schmidt, Austrian Institute

More information