SAP Data Services Hacks Auto Generating Data Migration Jobs Shobhit Acharya Session# 3507

Transcription

1 SAP Data Services Hacks Auto Generating Data Migration Jobs Shobhit Acharya Session# 3507

2 Learning Points Improve data migration efficiency using SAP Data Services and implementing a few custom approaches that will speed up the extraction and load of source data. These approaches deal with Programmatically generating data migration jobs to replace labor intensive and monotonous job design Using xml import mechanism to create job templates Using datastore configurations to ingest multiple instances of identical source databases

3 Agenda Introductions & Overview Data Migration & SAP EIM tools Data Services: Designer & Workbench Information Steward Review a baseline framework The use case for DS Workbench Alternative solutions for efficiency Understanding the DS Job xml structure Developing job generation programs for automation Datastore configurations When to use these programs Demo

4 Data Migration Assess Extract Consolidate Cleanse Load Reconcile Data migrations at an enterprise scale need a focus on migrating data efficiently, quickly, repeatedly and assuredly. The goal is not just to move and convert data; its to ensure that the data is of high quality and supports the business processes and target system s operational needs.

5 Data Migration 80% of organizations will underestimate the costs related to the data acquisition tasks by an average of 50 percent. Gartner

6 SAP Data Services & Information Steward Overview

7 SAP Data Services One Solution that Provides Data Integration Data Quality Text Data Processing One server to execute all capabilities One design environment to manage all development One administration console to monitor all functions

8 SAP Information Steward Business Measure and compare against Information governance rules and standards IT Share data quality metrics and problems with business

9 Case Study Review a baseline Data Migration Framework 1 2 Ingestion Extraction into Stage 4 Transform to common structure 7 Apply Relevancy Rules 10 Consolidate cleansed data 12 Fix Reference Data issues SAP BODS Source Legacy / Staging Transformed 4 7 Target Relevant 10 Load Target System 16 Staging Area 3 Initial Data Profiling 5 6 SAP IS 8 BoA 13 Pre-load sign off 5 Initial Health Check IVM 9 BoA Staging 14 Load to Target System 15 Reconciliation 6 Auto De-dup & Cleanse 8Secondary Health 9 Facilitated Check Cleansing 11 Reference Data validation 13 BOBJ (Reporting) Post load sign off

10 Case Study Ingestion Scope 25+ distinct source systems Multiple source product versions Need separate job streams Need separate job control Over 400 databases to ingest Hadoop as staging Multiple waves of migration 1 Source Extraction 1 SAP BODS 2 3 Legacy / Staging 2 Ingestion 3 Initial Data into Profiling Stage

11 The use case for DS Workbench Quick to build data replication projects Data flow design Additional customizations in DS Designer Progressively additional functionality added each release

12 The use case for DS Workbench Data Replication Design

13 The use case for DS Workbench Data flows

14 The use case for DS Workbench Goodies Monitor performance View data

15 The use case for DS Workbench And where it falls short No big data sources or targets Little or no workflow customizations A small list of supported transforms Additional work could be required in DS Designer for job control and customizations

16 Alternative solutions Generate your own jobs in XML Dataflow= Source Tables -> Query Transforms -> Target (including HDFS) Workflows Custom script stages Jobs Datastores and Configurations Flat file formats Import generated XML as DS Designer Jobs, workflows, dataflow Configure datastore for multiple deployments of the same source product database

17 Data Services job export in XML Understanding the structure of a simple job Example : 1 Dataflow in a job Export

18 Data Services job export in XML Understanding the structure of a simple job Example : 1 Dataflow in a job 350 lines of xml Lets look closer

19 Data Services job export in XML Understanding the structure of a simple job DIDatabaseDatastore DIAttributes DSConfigurations *Variables* Repeat for each --- <odbc_data_source>*datastore*</odbc_data_source> DITable DIProperties DIColumn DIDataflow DITransforms DIAttributes DIDatabaseTableSource DIOutputView DIFileTarget DIAttributes DIQuery DIAttribute DISchema DIElement DISelect DIProjection DIExpression DIFrom DIFlatFileDatastore DISchema DIElement DIAttributes DIUIOptions --- name="*table_name*" owner="*dbowner* datastore="*datastore --- List all columns and column properties, *COLUMN_NAME* --- name= *DATAFLOW*" --- array sizes, static parameters --- datastorename="*datastore* tablename="*table_name*" --- name="*table_name*" --- formatname="*datastore*_*table_name*" filename="*table_name* --- HDFS File location/path + static parameters --- name="*table_name*" value="*table_name*" --- List all columns and column properties, *COLUMN_NAME* --- column="*column_name*" --- Name="*DATASTORE*_*TABLE_NAME*" --- List all columns and column properties, *COLUMN_NAME* --- Datastore input and output file store attributes, 1 per job --- name="*datastore*_*table_name*" value="*table_name*"

20 Generating the xml programmatically Understanding what you need Programmers for your code Sql programming skills Or on java/python/.net Data Services Sandbox Repository Source database table and column definitions Oracle : all_tab_columns, Sql Server : information_schema, Progress DB: sysprogress.syscolumns_full. DB schema for code and column definitions + True Grit

21 Generating the xml programmatically Applying the understanding for complex designs Auto generated Imported xml This example: 1 Source Datastore 20+ Configurations 400+ Tables 400+ HDFS formats 400+ Dataflow 400+ Workflows 1 Job

22 Generating the xml programmatically Applying the understanding for complex designs Workflow Auto generated Scripts Dataflow Datastores

23 Data Services Datastore Configurations Contains alternate connection parameters for the datastore Typically used for promotions to new environments Could be leveraged for using template jobs on multiple databases with identical schemas (e.g. QAD Progress databases)

24 Best practices Always isolate any custom xml imports into a sandbox repository Use datastore configurations to maximum effect Pre-import, export (to xml) the intended source and destination datastore without any tables included Post-import, re-import these datastores to override the generated datastores

25 Generating the xml programmatically When could you need this?

26 Return on Investment Initial assessment at JCI for developing the custom programs needed (Codename : ATLGEN) Effort invested in ATLGEN development: 1 pers. week Typical pre-use efforts: 2 weeks per source Potential post use efficiency : 10x per source # of sources : distinct sources

27 Live demo

28 STAY INFORMED Follow the ASUGNews team: Tom Chris Craig

29 SESSION CODE 3507