Oracle Data Integration: CON7920 Making the Move to Oracle Data Integrator

Transcription

1

2 Oracle Data Integration: CON7920 Making the Move to Oracle Data Integrator Julien Testut Principal Product Manager, Oracle Data Integration Holger Friedrich Chief Technical Officer, SumIT Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

4 Agenda Oracle Data Integration Solutions Overview and ETL Strategy Why is Oracle Data Integrator Unique? Migration Utility Overview Reuse and Migrate OWB Mappings Q&A Copyright 2014 Oracle and/or its affiliates. All rights reserved. 4

5 Oracle Data Integration Solutions and Proven Benefits 24 x 7 x 365 Analytic Data Integration Big Data Integration & Governance Data Warehouse Integration Business Intelligence Applications Enterprise Data Integration and Governance Enterprise Data Quality and Profiling Comprehensive, Heterogeneous Data Integration Business Glossary and Metadata Management Business Continuity Active-Active for Maximum Availability Zero Downtime Migrations Data Consolidation / Application Modernization Improve Agility Deploy Projects Faster Reliable Real-Time Reduce Risk Popular, Proven Tools Open, Not Proprietary Reduce Costs Better Productivity Eliminate ETL Servers Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

6 Comprehensive Data Integration & Governance Capabilities Real-Time Data Movement Low impact capture, stage in Hadoop Continuous data availability Data Transformation Bulk data movement Pushdown data processing Data Federation Virtualized Data Services Data Quality & Verification Fix quality at the source Verify data consistency Metadata Management Lineage and Impact Analysis Business Glossary Semantics Oracle GoldenGate (Movement) Data Governance Foundation Oracle Data Integrator (Transformation) ELT Processing on Hadoop or SQL Continuous Availability Enterprise Data Quality (Profile, Cleanse, Match and De-duplicate) Fast Load Data Service Integrator (Federation) GoldenGate Veridata (Online Data Verification) Enterprise Metadata Management & Business Glossary (Business Glossary, Data Lineage, Impact Analysis and Data Provenance) Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

7 Differentiated Technical Approach Dynamic Data Movement Real-time CDC is by default, not ETL Least invasive on sources Proven best performance Integrated Oracle capture/apply No ETL Engines Take the processing to the data; don t move the data to the process Leverage your data engines for the workloads (Hadoop or SQL) Most Heterogeneous Leverage open source Hadoop, not proprietary distributions Hadoop is the Hub, not ETL tools Open metadata standards Oracle GoldenGate (Movement) Data Governance Foundation Oracle Data Integrator (Transformation) ELT Processing on Hadoop or SQL Continuous Availability Enterprise Data Quality (Profile, Cleanse, Match and De-duplicate) Fast Load Data Service Integrator (Federation) GoldenGate Veridata (Online Data Verification) Enterprise Metadata Management & Business Glossary (Business Glossary, Data Lineage, Impact Analysis and Data Provenance) Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

8 Oracle ETL Roadmap Since January 2010 As Communicated to OWB and ODI Customers Worldwide Unified Team ODI-EE License OWB-EE 11gR2 ODI 11g ODI 11g (BI Apps) ODI Oct 2008 Jan 2009 Sep 2009 Jul 2010 Apr 2013 Oct 2013 Knowledge Modules In-Database ETL Features OWB-EETL Concepts Copyright 2014 Oracle and/or its affiliates. All rights reserved.

9 OWB Support Policies Lifetime Sustaining Support for OWB Customers, No Forced Migrations Release GA Date Premier Support Ends Extended Support Ends OWB 9.2 Jul 2003 Oct 2007 Not Available Oct 2010 OWB 10.1 Apr 2004 Jul 2007 Not Available Jul 2010 OWB 10.2 Jul 2005 Jul 2010 Jul 2013 Indefinite OWB 11.1 Aug 2007 Aug 2012 Aug 2015 Indefinite OWB 11.2 Sep 2009 Jan 2015 Jan 2018 Indefinite Customer Downloadable OWB Support Policies: For more Information about Support Levels: Sustaining Support Ends Copyright 2014 Oracle and/or its affiliates. All rights reserved.

11 Oracle Data Integrator E-LT: Bulk Data Processing and Fast Data Transformation Cloud Apps Database Big Data Oracle Data Integrator High Performance E-LT Declarative Design Extensible Knowledge Modules Data Services Structured & Unstructured Data Certified for leading technologies to deliver fast time to value High-performance, low cost of ownership E-LT architecture Lightweight deployment Flexible, easy to enrich functionality Copyright 2014 Oracle and/or its affiliates. All rights reserved.

12 Industry Leading Performance Extremely Fast Execution and Reduced Cost Conventional ETL Architecture Extract Load Transform Next Generation Architecture E-LT Extract Load E-LT provides a flexible architecture for optimized performance on any platform Benefits Leverages set-based transformations Improves performance for loading, no network hop Takes advantage of existing infrastructure: hardware and software Transform Transform Copyright 2014 Oracle and/or its affiliates. All rights reserved. 12

13 Declarative Design Improved Developer Productivity For Faster Time to Solution Easy and universal for designing simple to complex mappings Robust and Reusable: Create Mappings once for any physical design Quick to define and refactor maps, high degree of automation Extensible model for any data integration mechanism ODI Declarative Design 1 Define What You Want 2 Automatically Generate Dataflow Benefits Significantly reduces the learning curve Shortens implementation times Define How: Knowledge Modules Simpler maintenance Copyright 2014 Oracle and/or its affiliates. All rights reserved. 13

14 Knowledge Modules Simpler Physical Design and Shorter Implementation Time Pluggable Knowledge Modules Architecture Reverse Engineer Metadata Journalize (CDC) Load from Source to Staging Check Constraints Integrate, Transform Data Service Sample out-of-the-box Knowledge Modules Oracle Sqoop Hive HBase SAP BW Oracle Spatial Oracle Datapump Siebel Oracle DBLink ebusines Suite JMS Oracle Merge External Tables SAP ERP Teradata IBM DB2 Netezza SCD Key Architecture Benefits Faster development and simpler maintenance using templates Easy to extend and add new best practices Enforces predictability and reduces cost of ownership Copyright 2014 Oracle and/or its affiliates. All rights reserved. 14

15 Extended Connectivity Leverages Existing IT, Faster Implementation Big Data: Hive, HBase, Sqoop Best for Oracle: Merge, Spatial, Multi-Table Insert, Optimizer Hints and more Real-Time: CDC with GoldenGate Applications: E-Business Suite, Siebel, PeopleSoft, JD Edwards Enterprise One, JD Edwards World, SAP ERP and SAP BW Heterogeneous: Optimizations for all major RDBMS: IBM DB2, Microsoft SQL Server, Teradata, Netezza etc. Cloud Big Data Applications Databases Legacy Copyright 2014 Oracle and/or its affiliates. All rights reserved. 15

17 Best Practices for OWB to ODI Migrations ODI 12c supports a phased migration approach Execute and monitor existing OWB 11gR2 jobs from ODI 12c Use the migration utility to convert existing OWB 11gR2 objects into their ODI equivalents Start new developments using ODI 12c Understand that OWB and ODI are different ETL tools There is no free lunch when migrating a technical migration between different tools will never be 100% perfect or predictable Copyright 2014 Oracle and/or its affiliates. All rights reserved.

18 Invoking OWB 11.2 Processes from ODI 12c New OWB Runtime Repository technology available in Topology OWB processes can be invoked using the OdiStartOwbJob tool in Packages OWB audit information is available in Operator and Enterprise Manager Copyright 2014 Oracle and/or its affiliates. All rights reserved.

19 OWB to ODI Migration Utility Overview Versions supported OWB / to ODI / Design-time metadata migration only Data Objects Mappings Separate command line utility 3 different running modes: Fast Check/Dry Run/Run log and report files are available Copyright 2014 Oracle and/or its affiliates. All rights reserved.

20 Migration Scope Details Supported OWB Design Time Metadata Module and Location Data Objects Table (Columns, Keys, Indexes) View (Columns, Keys) Materialized View (Columns, Keys, Indexes) External Table (Columns) File (Records, Fields) Sequence Mappings Classic, Code Template, Pluggable Most Mapping Operators Aggregator, Constant, Deduplicator Expression, Filter Joiner, Key Lookup Mapping Input Parameter Pivot, Pre/Post Mapping Process, Sequence Set, Sorter, Splitter, Subquery Table Function, Transformation Unpivot Copyright 2014 Oracle and/or its affiliates. All rights reserved.

21 Migration Scope Details Unsupported OWB Design time metadata (sample) Process Flows Dimensional metadata Oracle Discoverer metadata and derived OBIEE metadata Data Quality components Custom PL/SQL OMB Plus scripts Experts Copyright 2014 Oracle and/or its affiliates. All rights reserved.

23 Making the Move from Oracle Warehouse Builder to Oracle Data Integrator Dr.-Ing. Holger Friedrich

24 Agenda Introduction Runtime integration Meta data migration Conclusion 2014 sumit AG 03/

25 sumit AG Consulting and implementation services in Switzerland Experts for Data Warehousing and Business Intelligence solutions Focussed on Oracle technology BI Foundation specialized partner Data Warehousing specialized partner Exalytics competence center with own server Our motto: Get Value From Data Visit our web site: (in German) sumit AG 03/

26 Holger Friedrich Computer Science diploma of Karlsruhe Institute of Technology (KIT) Ph.D. in Robotics and Machine Learning More than 16 years experience with Oracle technology Expert for Data Integration Data Warehousing, Data Mining and Business Intelligence Technical Director of sumit AG First Oracle ACE for DWH/BI in Switzerland sumit AG 03/

27 Our Old Friend: Oracle Warehouse Builder widely successful flow-based design performant code (EL-T) option of Oracle DB 10g & 11g PL/SQL code generation freemium licensing model end of lifetime certified with DB 12cR1 only in 'maintenance' mode lacking state-of-the-art DI doesn t support new DB features no Big Data etc sumit AG 03/

28 New Kid in Town: Oracle Data Integrator 12c best of both world now flow-based design performant code (EL-T) built-in code generation for ANSI & Oracle DBs Big Data eco system Knowledge Module architecture Cloud Control integration typical Fusion customer Middleware situation integration large investments in OWB OWB maps & flows run well new developments to do little resources for migration changes in existing DI implementations required 2014 sumit AG 03/

29 Customers Need to be Able to Keep benefitting from their past investments in OWB Keep running existing OWB code Port code if it adds value only Do new stuff with ODI12c further integrate DI tools (e.g. OGG) extend our DI to new technologies (e.g. Big Data, Hyperion, ) use the KM architecture for special jobs 2014 sumit AG 03/

31 Co-Existence - OWB Runtime Integration Execution of OWB objects in an ODI package mappings process flows Integration of OWB objects in the ODI infrastructure adding the OWB Runtime Repository to the ODI topology entire OWB auditing is integrated with ODI auditing Orchestration of steps where desired (OWB or ODI) OWB process flows ODI load plans 2014 sumit AG 03/

32 OWB RT-Integration - Technology Repository owner or user OWB Runtime Repository Technology 2014 sumit AG 03/

33 OWB RT-Integration - Logical Architecture OWB Repository made available via context & logical schema 2014 sumit AG 03/

34 OWB RT-Integration - Package Integration OWB processes & mappings represented by ODI operator 2014 sumit AG 03/

35 OWB RT-Integration - Exec Properties Mapping or process flow configuration as needed 2014 sumit AG 03/

36 OWB RT-Integration - Seamless! 2014 sumit AG 03/

38 You Gotta Move It - Meta Data Migration Automatic, configurable migration tool Simple handling Large, yet limited functionality Offers two migration modes Big bang complete migration of entire projects On demand migration of collections of objects for further development 2014 sumit AG 03/

39 Support of OWB Meta Data Objects Supported: Classic OWB Mappings Code Template Mappings Pluggable Mappings Mapping Operators: Aggregator, Constant, Deduplicator, Expression, External Table, Filter, Flat File, Joiner, Lookup, Input parameters, Materialized View, Pivot, Pluggable Mapping, Pre/Post Mapping, Sequence, Set, Sorter, Splitter, Subquery, Table, Table Function, Transform-Function, Unpivot, View Unsupported: Process Flows Dimensional metadata Oracle Discoverer metadata & derived OBIEE metadata Data Quality components Custom PL/SQL OMB Plus scripts Experts 2014 sumit AG 03/

40 Tool Architecture & Migration Process & Patch conf file log file report file 2014 sumit AG 03/2012 <OWB_HOME>/owb/bin/unix>./migration.sh <ODI_MASTER_PASSWORD> <ODI_USER_PASSWORD> <OWB_WORKSPACE_OWNER_PASSWORD> <CONFIGURATION_FILE> 40

41 Configuration Information connection credentials for OWB & ODI migration mode just check dry run run reporting- & logging setup selection of objects all or some operator handling (don't) split Ansi joins do or don t migrate unbound operators 2014 sumit AG 03/

42 Step-By-Step Migration Logfile 2014 sumit AG 03/

43 Summary in Migration Report 2014 sumit AG 03/

44 OWB MD-Migration - Easy! 2014 sumit AG 03/

46 Summary: Existing OWB DI applications can be used with ODI 12c, thanks to RT integration Automatic migration of OWB mappings with the OWB2ODI migration tool Customers can choose between Big Bang migration and incremental migration on demand over time Like always in life the 80/20 rule applies Summary & Conclusion Conclusion: Oracle provides a powerful, easy-to-use tool for migration support from OWB to the next generation data integration technology ODI12c Recommendation: start testing the migration tool right away develop migration/replacement strategy depending on results plan and execute migration project accordingly 2014 sumit AG 03/

47 Agenda Oracle Data Integration Solutions Overview and ETL Strategy Why is Oracle Data Integrator Unique? Migration Strategy Reuse and Migrate OWB Mappings Q&A Copyright 2014 Oracle and/or its affiliates. All rights reserved. 47

49 Oracle DIS OOW 14 Oracle Data Integrator MON 4:00PM CON7899 Oracle Data Integrator: Product Update and Future Strategy TUE 5:00PM CON7820 Making he Move from Oracle Warehouse Building to Oracle Data Integrator WED 3:30PM CON7934 Tapping into the Big Data Reserve with All Data 4:45PM CON7922 Tame Big Data with Oracle Data Integration THU 9:30AM CON7926 Oracle Data Integration: A Crucial Ingredient for Cloud Integration 10:45AM CON7923 Oracle Data Integration & Metadata Management for Seamless Enterprise 2:30PM CON7921 Insight into Action: Business Intelligence Applications and Oracle Data Integrator Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

50 Join the Community Connect with Oracle on Social Media #OOW14 #ODI12c #GoldenGate12c #EDQ12c OR connect via the web Oracle Data Integration blog blogs.oracle.com/dataintegration Oracle Data Integration Home Page oracle.com/goto/dataintegration Copyright 2014, Oracle and/or its affiliates. All rights reserved.

52