Best Practices for Implementing Oracle Data Integrator (ODI) July 21, 2011

Transcription

1 July 21, 2011 Lee Anne Spencer Founder & CEO Global View Analytics Cheryl McCormick Chief Architect Global View Analytics

2 Agenda Introduction Oracle Data Integrator ODI Components Best Practices Implementation / Migration Approach Summary / Q & A

3 Global View Analytics Recognized Industry Leader Certified Experts in EPM and BI Dashboards and Analytics Planning and Forecasting Financial Reporting and Compliance Enterprise Data Management National and International Consultants ACE Methodology Assess -> Configure -> Empower Based on Agile Deployment Techniques Personalized Client Success Plans Focus on Measuring ROI

4 Enterprise Performance Management Global View Analytics supports a complete solution Dashboards & Analytics Planning & Forecasting Financial Reporting and Compliance Enterprise Data Management

5 Oracle Data Integrator Oracle Data Integrator (ODI)

6 Introduction to ODI Formerly Sunopsis - acquired by Oracle Oct Broad integration - connect anything to anything Complete data movement and transformation Real-time solution Supports SOA (Service Oriented Architecture) Tool for IT users Delivered with System 11 Hyperion Knowledge Modules (KMs) Bundled with HFM and Planning - restricted use

7 Factors to Consider Business users vs IT users Number of different source / target systems Volumes of data Meta data management strategy t Financial auditing & compliance Breadth of application solutions Corporate technology standards Internal skill sets Current versions and release dates Software licensing costs

8 Loading Metadata & Data to EPM with ODI 1. Define/Create EPM application EPM Sources EPM ODI Build and Load 2. Identify source EPM sources 3. Create models with ODI Designer by reverse engineering 4. Create ODI Interfaces which maps/transforms from source to target ODI within ODI for metadata and data load 5. Test ODI Interfaces 6. Create ODI Package - connect all 6 interfaces to perform complete build Create Scenario based on package - compiled version 8. Invoke Scenario, using Metadata Navigator (web enabled tool), using event, command line, SOA, etc Create Scenario Execute Scenario 9. Monitor Scenario Execution with Metadata Navigator or ODI Operator Monitor Application Create Models Create Package Test Create interface

9 ODI Components

10 ODI with Hyperion Oracle Hyperion Planning Oracle Hyperion Financial Mgt Oracle Hyperion Essbase Oracle Hyperion Application Adapters Planning API Metadata Lineage HFM API Oracle Data Integration Suite Data Distribution & Delivery APIs Bulk/Trickle Loading Information CDC Assets Other Sources Changed Data Capture Master Data ODI Knowledge Module Framework Oracle EBS PeopleSoft SAP/R3 Essbase API Data Quality & Profiling Bulk and Real-Time Data Processing Data Message Warehouse Queues Metadata Discovery Hyperion Planning Hyperion Financial Management Hyperion Essbase & Model Creation Extract Data Extracts Dimension Members Use Essbase KM Use Essbase KM Loads Data Loads Dimension Members Other Features Cube Refresh Consolidate Calculate

11 RKM - Reverse Engineering Modules Knowledge Modules Generate logical source / target tables for mapping of data Available for HFM, Planning, Essbase IKM Integration Knowledge Modules Used to integrate metadata / data Available for HFM, Planning, Essbase LKM - Loading Knowledge Modules Used to load metadata / data Available for HFM & Essbase (can also be used for Planning) CKM Check Knowledge Modules Check constraints in Source and Targets for violations JKM Journaling Knowledge Modules Create a journal of data modifications (insert, update and delete) of the source databases to keep track of the changes SKM Service Knowledge Modules Generates the code required to create data services

12 HFM Best Practices for Implementing Versions Supported Backward compatible to version Support >= Oracle Hyperion Financial Management Hyperion Planning Essbase Backward compatible to version Support >= Oracle Hyperion Planning Backward compatible to version Support >=Oracle Hyperion Essbase Note: ODI can load directly to interface tables for EPMA

13 Primary ODI Components Topology Manager - Defines the working environment Designer - Where the integration work happens Operator - Monitoring your integrations Security Manager - Manage ODI-related security

14 This area is CRITICAL. Adequate DESIGN TIME is Essential! Topology Manager Defines the topology of the information systems Defines all Source and Target Objects

15 Designer Knowledge Modules (KMs) Added Here via IMPORT Design Workflow through a graphical workflow navigation interface Primary Activities in Designer include: Developing Models of the data using the Reverse Engineering KM Creating Projects Developing Interfaces Developing Scenarios Executing integrations Checking Data Quality

16 Operator Provides visibility ibilit into the step by step integrations during execution Status and error messages are clearly displayed Ability to run a Scenario from Operator View scheduling information Ability to restart sessions

17 Security Manager Defines Security Policy related to ODI Objects, Instances & Methods Creation of Generic & Non-Generic Profiles ODI user security defined by the user profile & the rights assigned to objects and instances Assignment of authorization rights by profile or by user Define password policy

18 Best Practices

19 Correct Use of Topology & Contexts Read the documentation In ODI, all developments, as well as executions, are performed on top of a Logical Architecture (Logical schemas, logical agent), that resolves, in a given Context to a Physical Architecture (real/physical source/targets data servers/schemas and ODI runtime agents). Contexts allow you to switch the execution of the artifacts from one environment (context) to another. When you think you understand - Read it again

20 Enforce Data Quality Data Quality is a basic requirement Garbage in / Garbage moved & transformed / Garbage out Use ODI Static Checks to enforce data quality on the Source Data Use ODI Flow Checks to enforce data quality before data is pushed to the Target

21 Context-Independent Design Typical mistake we all make we used qualified object names Example: staging.temp_table t t where staging is the schema name = context dependant design Consider using Substitution Methods [ODIRef API]. Ensures the qualified object name is determined according to the Context you are using to generate your code.

22 Choosing the Right KM Understand d the requirements for each of the KMs For example, some technology-specific LKMs are dependant on loaders. Make sure the loader is installed and configured correctly before choosing this LKM. To start use a more generic SQL KM Don t over-engineer - Performance is at risk For example, can you do a simple INSERT vs an Incremental Load? Test the KM using the Default options first and then tweak

23 Utilize the Standard KMs Before customizing or writing KMs, take the time to read and understand the various options available with the delivered KMs Customized KMs are not supported Customized KMs need to be manually maintained during upgrades and possibly patches.

24 Keep Out of the ODI Meta Data Repository Respect the ODI meta data repository Don t hack into the back-end dodi repository tables. There is a lot of complexity, and it is easy to cause unintended results. Hacked ODI meta data repository table = NO SUPPORT when you break it

25 Implementation Approach / Implementation Approach / Migration from HAL

26 Implementation Approach 1) Assess integration requirements 2) If migrating, evaluate current HAL routines 3) Build ODI foundation 4) Design and develop ODI objects 5) Test & reconcile Things to be aware of when migrating This is not a one-to-one conversion process ODI requires more IT involvement than HAL

27 1) Assess your Integration Requirements First Things First: Take a step back and define Integration Requirements (Not the HOW ) Source Target Data Cleansing Document Determine business owners and approvals

28 2) If migrating, evaluate your current HAL routines Document current HAL Projects / Integrations ti Concentrate on WHAT and not on HOW - For example, if you need to concatenate a prefix, document the requirement not how it is done in HAL The more detail the better Identify all required connections - Source and Target Identify reusable HAL code Is there any vanilla SQL code in HAL that can be used in ODI? Identify reusable RDBMS Objects Stored Procedures Functions

29 3) Build the ODI Foundation Topology Manager (This is Critical) Import your Physical & Logical Technologies Create your Physical & Logical Objects Create Physical & Logical Agents Define your Context Designer Import your Knowledge Modules (KMs) Develop your Models Reverse Engineer your Sources Reverse Engineer your Targets Create / modify Data Stores (if required)

30 4) Design and Develop ODI Objects Design integration work flow Identify ODI Objects to support work flow Select KMs & update the KM Options Determine how processes can be simplified Add Error Handling to work flow Add notifications to work flow Add automation

31 4) Design and Develop ODI Objects Packages -sequence of organized steps Interfaces - a set of rules that define the loading of a Datastore or a temporary target structure from one or more source Datastores Procedures - a reusable component that groups operations that do not fit in the Interface framework. Can be encrypted. Variables - a variable s value is stored in ODI & can be changed during execution Sequences - a variable automatically ti incremented when used User Functions - customized functions useable in Interfaces & Procedures Knowledge Modules - define methods related to a specific technology. can be encrypted Markers - Flags (markers) are used to group and/or identify methodology Scenarios - when a component is finished it is compiled into a scenario. A scenario is the execution unit for production and can be scheduled.

32 5) Test and Reconcile Involve business owners in system integration testing Identify single version of the truth for reconciliation Define criteria for user sign-off

33 ODI Interface Development Sample Define Source & Target Datastores t in the Interface CASE statements, substrings & concatenations can all be done here Filters on source data are done here Mapping Source to Target is done here (JOINS are defined here) Variables can be used throughout Flow Tab Select required KMs Update KM Options Values Controls Tab Select required CKM Update CKM Options Values

34 ODI Interface Development Sample Sample ODI Package: Execution Tab Define the Execution Options Test the interface Scenario Tab Once the object is complete and tested, generate a scenario to productionize the object

35 Summary

36 Summary ODI is Oracle s strategic product for data integration IT development skills are required HAL is dead - you need a migration strategy Migration involves redesign - it is not a 1-1 conversion Do not replicate what you have - leverage ODI s objects Building the foundation is the most critical step You also need a strategy for managing meta data Global View Analytics can help you