Building a Data Warehouse Andrew Couch UK Access User Group. Andrew Couch 2016, All Rights Reserved

Size: px

Start display at page:

Terence Burke
7 years ago
Views:

Building a Data Warehouse Andrew Couch UK Access

2 Your experiences? How many of you have NOT Used SQL Server? Used SSIS (SQL Server Integration Services)? Built a DW (Data Warehouse)?

3 Presentation Contents Introduction Why build a Data Warehouse (DW)? DW Terminology and Layers SSIS Simple Demo Data Warehouse Layers Staging Layer (Demo) ETL (Extract Transform and Load) Framework Type II SCD Dimensions Enterprise Layer Facts & Kimball Methodology Data Mart Layer

4 Why build a Data Warehouse? Limitations of Existing Systems Business users find it difficult to piece together all the links in a relational database schema Objects such as tables and fields often don t represent the best business oriented names for data items RDBMS are great at showing operational state, but do not easily capture how data changes over time Operational systems may suffer performance issues when extensive analysis is undertaken against a live system Often need data from other systems/sources for full analysis

5 Data Warehouse Terminology Dimensional analysis Process of restructuring data into a form more suitable for a business analyst Fact or Measure (data) Contains measures, counts (transactional) and references dimensions Dimension (data) Contain attributes which can vary over time tied to the facts or possibly other dimensions Dimension types Some dimensions can have slowly changing data which captures changes over time and supports more accurate reporting Grain of a fact This could be a transaction, or other combination of key values; the grain is very important Star schema The aim is to construct a very simple data model, with a fact in the centre and relationships to several dimensions; it looks like a star Snowflake Schema Where we have dimensions linked to dimensions; to be avoided when possible

6 Physical Overview of a Data Warehouse Data Warehouse Live system Staging Enterprise or History Data Mart or Transformation SSIS and Packages

7 Data Warehouse Layers Staging Bring in the data to a set of tables which mirror the structure of the incoming data, data gets replenished with tables emptied and re-populated Enterprise or History Data accumulates and has minor restructuring; only accumulate data when it is new or changes, need to detect when data has changed Transformation or Data Mart Data presented in a simplified star schema for use by business analyst or to feed into cubes/ data mining for further analysis Some data marts accumulate data, others can be emptied and re-populated Not in 3NF; often highly de-normalised

8 Example of an OLTP System

Need Theory, Practice and Tools Donald Knuth Donald Ervin Knuth (born 10 January 1938) is an American computer scientist, Professor Emeritus at Stanford University, and winner of the 1974 Turing

9 Need Theory, Practice and Tools Donald Knuth Donald Ervin Knuth (born 10 January 1938) is an American computer scientist, Professor Emeritus at Stanford University, and winner of the 1974 Turing Award. If you find that you're spending almost all your time on theory, start turning some attention to practical things; it will improve your theories. If you find that you're spending almost all your time on practice, start turning some attention to theoretical things; it will improve your practice.

10 SSIS Demo (see how easy ETL is!) Steps Create a VS SSIS Project By default this creates an initial package Add a source connection Add a target connection Add a script task to empty the target table Add a data flow to populate the target table Strategies Use the GUI components Use your TSQL and stored procedures Write VB.NET or C# for non-standard file formats

11 My ETL Framework We have 3 layers in the DW, so create one Visual Studio SSIS Project for each Layer Each project has multiple packages Pre-processing package; get everything ready for the processing Dimension packages; process the dimensions Up to 8 or 10 processing sequences; each sequence processes a table Fact packages; process the facts Up to 8 or 10 processing sequences; each sequence processes a table Post-processing package; finish up and prepare next layer Everything is driven and recorded in the Framework system tables Framework supports Easy development, can be restarted when a sequence fails (big feature!) Robust error handling Easy extensibility

12 Figure 2: basic framework components Figure 3: worklist processing My Framework Concepts Figure 8: general layout of package structure

13 Key Features No Keys No Relationships Remove Identity Properties Staging Layer

14 Demo : Staging Data Using SSIS

15 Slowly Changing Dimensions: SCD Type II John works for the sales division in the Northwest, he then emigrates down south to the Southwest sales division All revenue before he moved need to be credited in the Northwest business unit, then after he moves to the Southwest business unit The data is slowly changing, because it changes slowly over time The data is called a dimension (think lookup) John needs two personal records, each with a different key that reflects the changes in his attributes over time We need a start and end date for when the dimension record is applicable to the related fact We need surrogate keys, because the dimensional data has several records for John which vary over time A fact (for example sales) need to be tied to the appropriate surrogate key Example :-

16 Dimension (SCD) Fact (Transactional) Id = 1, John In Northwest Sale 6 : Id = 1 Sale 7 : Id = 1 Id = 2, John In Southwest Sale 8 : Id = 2 Sale 9 : Id = 2 Id Name Region SCD_StartDate SCD_EndDate John NW 1/1/2015 1/1/ John SW 2/1/2016?????????????? : could be either NULL or a high date 31/12/9999 (slide shows UK dates)

17 Key Features Surrogate Keys New foreign keys SCD_StartDate SCD_EndDate Null Value Records Enterprise/ History Layer

18 Enterprise, SCD with SSIS SCD Data Flow, automatically created with a simple Wizard

19 Enterprise Layer Notes SCD can be implemented using several different techniques SSIS SCD Component If using built in SSIS SCD Component, then you can not have VARCHAR(MAX), NVARCHAR(MAX), TEXT VARBINARY(MAX) Stored Procedures; return counts and flags to SSIS Other SSIS Flows and Logic Can use Checksum calculations to determine when a row has changed and needs to be versioned In SQL Server, don t mix UNICODE and ASCII data types SSIS will not convert VARCHAR(10) in Staging implicitly to mapped NVARCHAR(10) in Enterprise; world of pain Keep your data with minimum calculated values and a close structure to the OLTP in this layer; in-case you change the rules for performing calculations

Facts: Paraphrasing Kimball (my interpretation) Some facts are pure transactional, for example in an accounting system we do not edit historical transactions, but make subsequent transactions to

20 Facts: Paraphrasing Kimball (my interpretation) Some facts are pure transactional, for example in an accounting system we do not edit historical transactions, but make subsequent transactions to correct a balance : Transactional Fact Some facts like a share price change on a periodic basis: Periodic Snapshot Fact Some facts can get changed/evolve. Maybe attributes need to move through a pipeline before becoming immutable, or maybe someone just edits them! Accumulating Snapshot Fact Delete and replace the fact? Update fact attributes? Other business specific strategies

21 Ralph Kimball (born 1944) is an author on the subject of data warehousing and business intelligence. He is one of the original architects of data warehousing and is known for long-term convictions that data warehouses must be designed to be understandable and fast. His methodology, also known as dimensional modeling or the Kimball methodology, has become the de facto standard in the area of decision support. A row in an accumulating snapshot fact table summarizes the measurement events occurring at predictable steps between the beginning and the end of a process. Pipeline or workflow processes, such as order fulfillment or claim processing, that have a defined start point, standard intermediate steps, and defined end point can be modeled with this type of fact table. There is a date foreign key in the fact table for each critical milestone in the process. An individual row in an accumulating snapshot fact table, corresponding for instance to a line on an order, is initially inserted when the order line is created. As pipeline progress occurs, the accumulating fact table row is revisited and updated. This consistent updating of accumulating snapshot fact rows is unique among the three types of fact tables. In addition to the date foreign keys associated with each critical process step, accumulating snapshot fact tables contain foreign keys for other dimensions and optionally contain degenerate dimensions. They often include numeric lag measurements consistent with the grain, along with milestone completion counters.

22 Key Features Star Schema Avoid Snowflakes Bring in only desired columns Can use views for the data No NULL Values for SSAS in fields Example DataMart

23 Sample Databases for my SSIS Framework Part I: Building a SSIS ETL Framework Concepts Part II: SSIS Staging Sample Framework Part III: SSIS Enterprise Sample Framework Part IV: SSIS Transformation Sample Framework Code can be used and modified for commercial use, no restrictions Some consultancy available

Presented by: Jose Chinchilla, MCITP

Presented by: Jose Chinchilla, MCITP Jose Chinchilla MCITP: Database Administrator, SQL Server 2008 MCITP: Business Intelligence SQL Server 2008 Customers & Partners Current Positions: President, Agile