SQL Server 2012 Integration Services Design Patterns Andy Leonard Matt Masson Tim Mitchell Jessica M. Moss Michelle Ufford
Contents J Foreword About the Authors About the Technical Reviewers Acknowledgments xviii xix xxi xxiii Chapter 1: Metadata Collection 1 Introducing SQL Server Data Tools 1 A Peek at the Final Product 1 SQL Server Metadata 3 Setting Up the Central Repository 4 The Iterative Framework 6 Metadata Collection 14 Summary 28 Chapter 2: Execution Patterns 29 Building the Demonstration SSIS Package 29 Debug Execution 30 Command-Line Execution 31 Execute Package Utility 31 The SQL Server 2012 Integration Services Service 32 Integration Server Catalogs 32 Integration Server Catalog Stored Procedures 33 vii
l CONTENTS Scheduling SSIS Package Execution 53 Scheduling an SSIS Package 53 Scheduling a File System Package 54 Running SQLAgent Jobs with the Custom Execution Framework 55 Running the Custom Execution Framework with SQL Agent 56 Metadata-Driven Execution 58 Execution from Managed Code 58 The Demo Application 58 ThefrmMain Form 59 Summary 69 Chapter 3: Scripting Patterns 71 The Toolset 71 Should I Use Script? 72 The Script Editor 72 Project Explorer 73 Full.NET Runtime 74 Compiler 74 The Script Task 75 The Script Component 77 Script Maintenance Patterns 78 Code Reuse 78 Copy/Paste 78 External Assemblies 78 Custom Tasks/Components 79 Source Control 79 Scripting Design Patterns 79 Connection Managers and Scripting 80 Using Connection Managers in the Script Task 81 Using Connection Managers in the Script Component 81 Variables 82
Variable Visibility 82 Variable Syntax in Code 83 Variable Data Types 84 Naming Patterns 85 Summary 85 Chapter 4: SQL Server Source Patterns 87 Setting up a Source 87 Selecting a SQL Server Connection Manager and Provider 88 AD0.NET 89 ODBC 90 OLE DB 91 Creating a SQL Server Source Component 92 Writing a SQL Server Source Component Query 94 ADO.NET Data Access 95 OLE DB Data Access 96 Waste Not, Want Not 97 Data Translations 97 Source Assistant 97 Summary 99 Chapter 5: Data Cleansing with Data Quality Services 101 Overview of Data Quality Services 101 Using the Data Quality Client 102 Using DQS with SSIS 107 Cleansing Data in the Data Flow 113 Handling the Output of the DQS Cleansing Transform 113 Performance Considerations 116 Approving and Importing Cleansing Rules 120 Summary 122 ix
Chapter 6: DB2 Source Patterns 123 DB2 Database Family 123 Selecting a DB2 Provider 124 Find Database Version 124 Pick Provider Vendor 125 Connecting to a DB2 Database 125 Querying the DB2 Database 128 DB2 Source Component Parameters 129 DB2 Source Component Dynamic Queries 130 Summary 132 Chapter 7: Flat File Source Patterns 133 Flat File Sources 133 Moving To SSIS! 134 Strong-Typing the Data 136 Introducing a Data-Staging Pattern 138 Variable-Length-Rows 141 Reading into a Data Flow 142 Splitting Record Types 143 Terminating the Streams 144 Header and Footer Rows 146 Consuming a Footer Row 146 Consuming a Header Row 149 Producing a Footer Row 151 Producing a Header Row 158 The Archive File Pattern 162 Summary 167 Chapter 8: Parallel Data Warehouse Patterns 169 PDW Architecture Overview 170 DWLoader vs. Integration Services 172 ETL vs. ELT 172 X
Installing the PDW Destination Adapter 174 The Data Source 175 The Data Flow 177 The Data Source The Data Transformation 178 179 The Data Destination 181 Multi-Threading 188 Summary 189 Chapter 9: XML Patterns 191 Using the XML Source 191 Dealing with Multiple Outputs 192 Making Things Easier with XSLT 198 Using a Script Component 201 Configuring the Script Component 201 Processing XML with XmlSerializer 206 Processing XML with XmlReader and LINQ to XML 207 Summary 209 Chapter 10: Expression Language Patterns 211 Getting to Know the Expression Language 211 What is Expression Language? 211 Why Use Expressions? 212 Language Essentials 213 Limitations 213 Putting Expression Language to Work 214 Package Expressions 214 Variable Expressions 215 Connection Managers 215 Project-level Connection Managers 217 Control Flow 217 Data Flow Expressions 220
2 CONTENTS Chapter 11: Data Warehouse Patterns 227 Incremental Loads 227 What Is an Incremental Load? 227 Why Incremental Loads? 227 The Slowly Changing Dimension 228 Incremental Loads of Fact Data 228 Incremental Loads in SSIS 228 Native SSIS Components 229 The Slowly Changing Dimension Wizard 232 The MERGE Statement 234 Change Data Capture (CDC) 237 Data Errors 242 Simple Errors 242 Missing Data 242 Coding to Allow Errors 245 Data Warehouse ETL Workflow 247 Dividing Up the Work 247 One Package=One Unit of Work 247 Summary 249 Chapter 12: Logging Patterns 251 Essentials of Logging 251 Why Logging? 251 Elements of Logging 252 Logging in SSIS 253 SSIS Catalog Logging 253 Package Logging 256 Custom Logging 258 What to Capture? 259 Logging Systems 260 Summary 260 xii
Chapter 13: Slowly Changing Dimensions 261 Slowly Changing Dimension Transform 261 Running the Wizard 262 Using the Transformations 266 Optimizing Performance 267 Third Party SCD Components 268 MERGE Pattern 269 Handling Type 1 Changes 270 Handling Type 2 Changes 271 Summary 271 Chapter 14: Loading the Cloud 273 Interacting with the Cloud 273 Incremental Loads to SQL Azure 274 Change Detection 274 New Rows (Only) 274 Building the Cloud Loader 275 Summary 278 Chapter 15: Logging and Reporting Patterns 279 Package Logging and Reporting 279 Setting Up Package Logging 280 Reporting on Package Logging 281 Design Pattern: Package Executions 281 Catalog Logging and Reporting 281 Setting Up Catalog Logging 281 Catalog Tables 283 Changing Logging Levels After the Fact 284 Design Patterns 285 Summary 289 xiii
Chapter 16: Parent-Child Patterns 291 Master Package Pattern 291 Assign the Child Package 292 Configure Parameter Binding 293 Dynamic Child Package Pattern 294 Child to Parent Variable Pattern 298 Summary 300 Chapter 17: Business Intelligence Markup Language 301 A Brief History of Business Intelligence Markup Language 301 Building Your First Biml File 302 Building a Basic Incremental Load SSIS Package 305 Adding Metadata 307 Specifying a Data Flow Task 308 Adding Transformations 308 Testing the Biml 314 Using Biml as an SSIS Design Patterns Engine 318 Time for a Test 324 Summary 326 Chapter 18: Configuration 327 Parameters 327 Configuring Your Package Using Parameters 329 Using the Parametrize Dialog 332 Creating Visual Studio Configurations 332 Specifying Entry Point Packages 334 Connection Managers 334 Parameter Configuration on the Server 335 Default Configuration 335 Server Environments 337 Default Parameter Values Using T-SQL 339 Package Execution through the SSIS Catalog 339 xiv
Parameters with DTEXEC 341 Projects on the File System 342 Projects in the SSIS Catalog 343 Dynamic Configurations 344 Configuring from a Database Table 344 Setting Values using a Script Task 348 Dynamic Package Executions 348 Summary 350 Chapter 19: Deployment 351 Project Deployment Model 351 SSIS Catalog 352 Deployment Methods 354 Deployment from the Command Line 355 Deployment Using Custom Code 356 Deployment Using PowerShell 357 Deployment Using SQL 358 Package Deployment Model 359 Summary 360 Chapter 20: Estimating ETL Projects 361 What is being measured? 361 Why estimate? 361 Challenges 362 It's difficult because it requires communication 362 It's difficult because it requires guesswork 363 It's difficult because it relies on technology 363 The secret to estimating ETL project timelines is 364 Don't forget the little things 364 Plan for the unexpected 365 XV
Know the personalities involved 366 Learn to do it right by doing it wrong 366 When the timeline slips, communicate early and often 366 Summary 366 Appendix A: Evolution of an SSIS Framework 367 Starting in the Middle 367 Introducing SSIS Applications A Note About Relationships 376 378 Retrieving SSIS Applications in T-SQL 381 Retrieving SSIS Applications in SSIS 384 Monitoring Execution 386 Building Application Instance Logging 387 Building Package Instance Logging 393 Building Error Logging 397 Reporting Execution Metrics 406 Summary 420 Index 423 xvi