Solving Business Pains with SQL Server Integration Services. SQL Server 2005 / 2008

Transcription

1 Solving Business Pains with SQL Server Integration Services SQL Server 2005 / 2008

2 Jason Strate Who Am I? SQL Server MVP Enterprise Consultant with Digineer Working with SQL Server since 1997 feeds2.feedburner.com/stratesql twitter.com/stratesql

3 Agenda 1. Discuss Goals 2. Business Scenarios 1. Describe 2. Background 3. Demo 4. Summary

4 Session Goals 1. Understand flexibility that SSIS features provide. 2. Demonstrate application of SSIS functionality to business needs. 3. Discuss current best practices in SSIS package design.

5 Business Scenarios

6 Business Scenario #1 Executing the same process across multiple servers: Consolidating data from multiple SQL Server instances Distributing data across single tenant databases

7 Business Scenario #1 Background Client needed to push core data to all instances of single tenant database. Manual tools were time consuming Changes were not pushed to all databases timely Some databases were missed at times Replication didn t fit since there were multiple databases per instance Client needed single source of reference information across enterprise. Hours were spent collecting information at the start of consolidation and migration projects This was a pre-sql Server 2008 solution (Management Data Warehouse) Provides back version compatibility

8 Business Scenario #1 Demo

9 Business Scenario #1 Summary Consolidate packages and deployment Use ForEach with ADO Enumerator for non-file looping Expressions on connections increase flexibility of package

10 Business Scenario #2 Executing the same process across multiple tables

11 Business Scenario #2 Background Client needed to migration n tables from production to archive server Tables archived shifted from depending on active services Existing process generated many errors that were not captured Code base for existing process was lengthy and difficult to maintain Individual table package maintenance was unbearable

12 Business Scenario #2 Demo

13 Business Scenario #2 Summary BULK INSERT provides imports without strict data typing BCP used to export data and format Architecture allowed for unlimited number of tables Outside scope of discussion but solution also had: Stored procedure used to create list of tables in foreign key order Dynamic SQL used to INSERT and UPDATE archive data Checks to validate source and destination were identical

14 Business Scenario #3 Performance of import package degrades over time Increased volumes in source system bottleneck package

15 Business Scenario #3 Background Proof of concept required performance improvements to SSIS package Performance was acceptable for initial deployment Over time performance degraded and data volume was found to be the issue SQL Server 2008 offers a number of alternatives to use of date/time stamps

16 Date/Time Stamps Obstacles What data has been inserted or updated? What data is now missing or deleted? Does every application properly log changes? Do ad-hoc users properly log changes? Were logging triggers disabled? Problems Forced to rely on create and modified dates from database Must bring back more data than necessary

17 New Options for Incremental Loading Change Data Capture Monitors DML changes Utilizes log reader All versions retained Change Tracking Monitors DML changes Lightweight monitoring Primary Key value retained

18 Business Scenario #3 Demo

19 Business Scenario #3 Summary Change Tracking and Change Data Tracking are new to SQL Server 2008 Technologies provide reliable method to load data incrementally Provides easy method to track all DML operations Space issues can arise if miss managed

20 Questions?

21 Solving Business Pains with SQL Server Integration Services SQL Server 2005 / 2008 jstrate@digineer.com