Incremental Replication from SQL Server to Oracle. Kane McKenzie Liberty University

Transcription

1 Incremental Replication from SQL Server to Kane McKenzie Liberty University

2 Our Challenge Efficiently Replicate Data from SQL Server based CRM database to based Data Warehouse

3 Business Drivers Issues with Reporting against Production Impact to Production CRM Benefits of Data Integration Analyze CRM Data Alongside Other Replicated Data

4 Business Requirements Technology Sustainable Affordable Fresh

5 Environment Details Live Since February 2013 Source: SQL Server based CRM database Target: Data Warehouse Refreshed Every 30 Minutes (Takes 4 Minutes) 77 Tables Replicated Incrementally (30 GB) Busiest Table: 350K Changes/Day Largest Table: 61 Million Rows Takes approximately 1 hour to configure new table for replication.

6 Challenges Incremental Change Detection Heterogeneous Replication Available Tools Costs of Technology Extract Transform Load

7 ODBC High Level Architecture Source SQL Server Windows Gateway Server Target DB SQL Server Gateway

8 Aerial Architecture SQL Server CRM cron Source Table CDC Table Staging Table Note View Target Table Data Warehouse ODBC Gateway

9 Architecture Explored: SQL Server SQL Server CRM Source Table CDC Table View ODBC cron Source table identified for replication to Data Warehouse. Staging Table SQL Server change data capture (CDC) table used for incremental change detection in source table. Target Table Extraction view. Standard SQL Server view against underlying table which is used to handle necessary Gateway conversions due to differences between SQL Server and. Data Warehouse

10 Architecture Explored: Connectivity Linux cron tool used to schedule jobs which extract incremental SQL Server CRM changes every 30 minutes. Source Table generic ODBC Gateway utilized to enable to talk CDC Table to SQL Server and pull data. cron Staging Table View Standard ODBC driver enables Heterogeneous Services to connect to SQL Server. Target Table Data Warehouse ODBC Gateway

11 Architecture Explored: Standard Insert Select SQL copies incremental changes for SQL Server CRM each table to staging table in Data Warehouse. Source Table cron Staging Table captures and CDC Table interprets changes applied to change table, converting each change row into Viewan UPDATE, INSERT, or DELETE operation. applies Logical Change Records (LCR), created ODBC from updates to staging table, to target table. Gateway Target Table Data Warehouse

12 Details: Change Data Capture cron SQL Server CRM Source Table CDC Table View EXEC sys.sp_cdc_enable_db GO Staging Table EXEC = = N = NULL; GO Target Table Data Warehouse ODBC Gateway

13 Details: Heterogeneous Services Heterogeneous Services consists of an Database Server cron Home, a running listener, and one ore more Gateways. We use only the free, included, default ODBC Gateway. (Free per SR ) SQL Server CRM Source Table 1 init file CDC Table View 2 tnsnames entry ODBC Gateway Staging Table initcrmprod.ora HS_FDS_CONNECT_INFO = crmprod HS_FDS_TRACE_LEVEL = off tnsnames.ora SID_LIST_LISTENER Target Table = (SID_LIST = (SID_DESC Data Warehouse = (PROGRAM = dg4odbc) (SID_NAME = crmprod) (ORACLE_HOME = <path to home>) ) Key to the ODBC connection name

14 Details: Changes from SQL Server INSERTED into staging table. cron SQL Server CRM Source Table Handler converts INSERT operations to the appropriate type CDC based Table on what actually happened in the SQL Server world. View CAPTURE APPLY Staging Table Target Table Data Warehouse ODBC Gateway

15 Handler FUNCTION crm_handler(evt IN SYS.ANYDATA) RETURN SYS.ANYDATA IS... BEGIN Set LCR Command Type based on "operation" column -- value from MSSQL CDC Columns -- DELETE Operation if oper = 1 then -- Change Command Type in LCR to a DELETE lcr.set_command_type('delete');

16 Details: Miscellaneous cron SQL Server CRM DECLARE Source Table max_tsn VARCHAR2(38); Staging Table BEGIN CDC Table SELECT MAX(START_LSN) INTO max_tsn FROM CRMSTAGE.LEADBASE; INSERT ViewINTO CRMSTAGE.LEADBASE SELECT "dbo"."ludw_leadbase_ct".* FROM WHERE OPERATION IN (1,2,4) AND START_LSN > max_tsn; END; ODBC Gateway Target Table Data Warehouse

17 Aerial Architecture Review SQL Server CRM cron Source Table Staging Table CDC Table View Target Table Data Warehouse ODBC Gateway

18 Relevant Documents and Notes Document Name Part Number Concepts and Administration E Replication Administrator's Guide E Database Gateway Installation and Configuration Guide E Database Heterogeneous Connectivity User s Guide E Database Gateway for ODBC User s Guide E Support Note Note Number Master Note for Gateway Products Gateway Configuration ODBC Gateway License

19 Questions & Contact Kane McKenzie Liberty University (Google Voice)