Solving Business Pains with SQL Server Integration Services SQL Server 2005 / 2008
Jason Strate Who Am I? SQL Server MVP Enterprise Consultant with Digineer Working with SQL Server since 1997 jstrate@digineer.com feeds2.feedburner.com/stratesql twitter.com/stratesql
Agenda 1. Discuss Goals 2. Business Scenarios 1. Describe 2. Background 3. Demo 4. Summary
Session Goals 1. Understand flexibility that SSIS features provide. 2. Demonstrate application of SSIS functionality to business needs. 3. Discuss current best practices in SSIS package design.
Business Scenarios
Business Scenario #1 Executing the same process across multiple servers: Consolidating data from multiple SQL Server instances Distributing data across single tenant databases
Business Scenario #1 Background Client needed to push core data to all instances of single tenant database. Manual tools were time consuming Changes were not pushed to all databases timely Some databases were missed at times Replication didn t fit since there were multiple databases per instance Client needed single source of reference information across enterprise. Hours were spent collecting information at the start of consolidation and migration projects This was a pre-sql Server 2008 solution (Management Data Warehouse) Provides back version compatibility
Business Scenario #1 Demo
Business Scenario #1 Summary Consolidate packages and deployment Use ForEach with ADO Enumerator for non-file looping Expressions on connections increase flexibility of package
Business Scenario #2 Executing the same process across multiple tables
Business Scenario #2 Background Client needed to migration n tables from production to archive server Tables archived shifted from 200-250 depending on active services Existing process generated many errors that were not captured Code base for existing process was lengthy and difficult to maintain Individual table package maintenance was unbearable
Business Scenario #2 Demo
Business Scenario #2 Summary BULK INSERT provides imports without strict data typing BCP used to export data and format Architecture allowed for unlimited number of tables Outside scope of discussion but solution also had: Stored procedure used to create list of tables in foreign key order Dynamic SQL used to INSERT and UPDATE archive data Checks to validate source and destination were identical
Business Scenario #3 Performance of import package degrades over time Increased volumes in source system bottleneck package
Business Scenario #3 Background Proof of concept required performance improvements to SSIS package Performance was acceptable for initial deployment Over time performance degraded and data volume was found to be the issue SQL Server 2008 offers a number of alternatives to use of date/time stamps
Date/Time Stamps Obstacles What data has been inserted or updated? What data is now missing or deleted? Does every application properly log changes? Do ad-hoc users properly log changes? Were logging triggers disabled? Problems Forced to rely on create and modified dates from database Must bring back more data than necessary
New Options for Incremental Loading Change Data Capture Monitors DML changes Utilizes log reader All versions retained Change Tracking Monitors DML changes Lightweight monitoring Primary Key value retained
Business Scenario #3 Demo
Business Scenario #3 Summary Change Tracking and Change Data Tracking are new to SQL Server 2008 Technologies provide reliable method to load data incrementally Provides easy method to track all DML operations Space issues can arise if miss managed
Questions?
Solving Business Pains with SQL Server Integration Services SQL Server 2005 / 2008 jstrate@digineer.com