Business Intelligence. 11. SSIS, ETL January 2014.

Transcription

1 Business Intelligence 11. SSIS, ETL January 2014.

2 SQL Server Integration Services Poslovna inteligencija

3 SQL Server Integration Services New project Integra on Services Project Data Sources new data source Define two connections source (RDB) and destination (DW) 3

4 Control Flow Control Flow defines structure and ordering of the tasks Basic elements are: Tasks (especially Data Flow Task) Containers (contains/groups other tasks) Precedence constraints (determines the order of execution): On success On completion On failure 4

5 Data Flow Data Flow elements extract, transform and load the data into the destination Basic elements are: DF sources DF transformations DF destinations 5

6 Example Employee dimension (1) 1. Control Flow: Add Data Flow Task 2. Data Flow: Add OLE DB Source that points to (via project's data source*) to the Employees table in the source (NorthWind) database * 6

7 Example Employee dimension (2) 3. Data Flow: Add the derived column transformation (changes existing attributes and adds new) Connect data flow output of the source element to the derived column transformation * Can also be achieved with CASE WHEN if SQL statement is used for the source 7

8 Example Employee dimension (3) 3. Data Flow: Add SCD transformation Define: 1. Destination dimension (NorthWindDW.dEmployee) 2. Business Key (EmployeeID) 8

9 Example Employee dimension (4)... define: 3. Attribute(change) types: Fixed should not change, causes errors Changing can be changed, updated i.e. SCD#1 Historical can be changed, insert new row i.e. SCD#2 9

10 Example Employee dimension (5)... define: 4. How do we record historical attributes; we use datefrom and dateto: (we'll set the redundant iscurrent flag later ) (deselect "inferred members support" at the next screen) 10

11 Example Employee dimension (6) Finally, the wizard generates the following (attention - every time it is (re)started, wizard will overwrite any manual changes): Add attribute ValidTo: (DT_DBDATE)(@[System::ContainerStartTime]) iscurrent is optional, you don t have to use it in your homework UPDATE [dbo].[demployee] SET [Address] =?,[Extension] =?,[HomePhone] =?,[Notes] =?,[ReportsTo] =?,[Title] =?,[TitleOfCourtesy] =? WHERE [EmployeeID] =? AND [validto] IS NULL UPDATE [dbo].[demployee] SET [validto] =?, iscurrent = 'No' WHERE [EmployeeID] =? AND [validto] IS NULL Map iscurrentyes to iscurrent 11

12 Example FT Lookup transformation Appropriate dimension surrogate keys have to be determined (looked up from the dimension tables) when the fact table is loaded E.g. dcustomer: Join on: business key And look up the surrogate key: SifCustomer 12

13 Example FT Orders Sequential surrogate key lookups All errors are "flowing" into the err log table 13

14 FT - Comment Analogously for the OrderDetails (not obligatory for the homework) We can lookup only NorthWindDW.cOrder: It does not have to re-do all the common dimension lookup (one query - faster) It can not be executed in parallel with corders, but only after corders When a large number of records is loaded it is recommended to remove the keys (indices) before loading and return (raise them) after Batch (fast) load: Faster loading What happens when one row in the batch fails to satisfy some constraint, e.g. FK? 14

15 Additional infrastructure CREATE TABLE etlrun( sifetlrun int IDENTITY(1,1) NOT NULL, datetimefrom datetime NOT NULL, datetimeto datetime NOT NULL, done char(1) NOT NULL, CONSTRAINT pketlrun PRIMARY KEY CLUSTERED(sifEtlRun) ) CREATE TABLE errcorder( errid int IDENTITY(1000,1) NOT NULL, OrderID int NOT NULL, errtimestamp datetime NOT NULL, errdesc NVARCHAR(100) NULL, errcode varchar(100) NULL, errcolumn varchar(255) NULL, CONSTRAINT pkerrcorder PRIMARY KEY CLUSTERED (errid) ) 15

16 CDC GenerateAuditTrail procedure Execute for the table we want to audit (all) Creates triggers and shadow table: 16

17 Procedure genrecords CREATE PROCEDURE [dbo].[genrecords] AS BEGIN /* A single execution of genrecords causes(generates): INSERT 0-1 records into the table Categories, Suppliers, Shippers, Employees, Customers INSERT 1-50 records into the table Products UPDATE 1 record from the Products (changes SupplierID) table UPDATE 1 record from the Products (changes CategoryID)table UPDATE 1 record from the Employees (changes City, Region, PostalCode, Country)table UPDATE 1 record from the Orders (changes OrderDate, RequiredDate, ShippedDate - fixes erroneous data from the previous run) UPDATE 1 record from the Customers table(changes City, Region, PostalCode, Country) INSERT records into the table Orders and 1-20 records into the table OrderDetails */ for each INSERT-irani Order. First INSERT-ed Order has erroneous data for OrderDate, RequiredDate, ShippedDate. They are fixed in the next run. int ; nvarchar(15) = 'CatName';... 17

18 ETL initialization, setting packet variables (1) INSERT INTO etlrun (datetimefrom, datetimeto, done) VALUES ( (SELECT CASE WHEN (MAX(dateTimeTo) IS NULL) THEN ' :00:00' ELSE MAX(dateTimeTo) END FROM etlrun WHERE done = 'D'), getdate(), 'N' ) SELECT MAX(sifEtlRun) FROM etlrun ResultSet = Single Row The result is assigned to the GLOBAL etlrunid variable visible throughout the package. 18

19 ETL initialization, setting packet variables (2) These variables will be used to select the new records on incremenal load ( BETWEEN ) 19

20 At the end of the ETL process UPDATE etlrun SET done = 'D' WHERE sifetlrun =? 20

21 All together Setup ETL run INITIAL dimensions XOR INCREMENTAL dimensions fact tables fact tables Update done flag Process OLAP structures It is not necessary to separate the logic to the initial and incremental load 21

22 Example selecting rows from corderdetails SELECT Orders_shadow.[OrderID], [ProductID], [UnitPrice], [Quantity], [Discount] FROM OrderDetails, Orders_shadow WHERE OrderDetails.OrderId = Orders_shadow.OrderId AND AuditDate between? AND? AND AuditAction = 'U' SELECT [OrderID], [ProductID], [UnitPrice], [Quantity], [Discount] FROM dbo.orderdetails_shadow WHERE AuditDate between? AND? AND AuditAction = 'I' redundant, always 'I' 22

23 7 th homework Define ETL for the part of the Northwind database - only one fact table (e.g. corders) and related dimensions At least one dimension must be SCD#2 Submit the solution (HW examination before teaching staff will be organized) Incomplete solutions will also be evaluated (e.g. only initial load) 7 points Deadline: at 9:00 23