Using Oracle Data Integrator with Essbase, Planning and the Rest of the Oracle EPM Products

Transcription

1 Using Oracle Data Integrator with Essbase, Planning and the Rest of the Oracle EPM Products Edward Roske BLOG: LookSmarter.blogspot.com WEBSITE: TWITTER: ERoske

2 2

3

4 4 About interrel 2008 & 2009 Oracle Titan Award winner - EPM Solution of the year 2008 Oracle EPM Excellence Award 2009 Oracle EPM/BI Innovation Award One of the fastest growing companies in the world (Inc. Magazine, 08 & 09) Two of the three Hyperion Oracle ACE Directors in the world Founding Hyperion Platinum Partner; now Oracle Certified Partner Focused exclusively on Oracle Hyperion EPM software Consulting Training Infrastructure and Installation Support Software sales

5 5 Hyperion Books Available: Essbase (7): Complete Guide Essbase System 9: Complete Guide Essbase System 9: End User Guide Smart View 11: End User Guide Essbase 11: Admin Guide ebooks available on Amazon Kindle Just out! Hyperion Planning for End Users Coming Soon Hyperion Planning for Admins (days now ) Hyperion Financial Management (Q1 2010) To order, check out 5 Copyright 2007, Hyperion. All rights reserved.

6 Disclaimer These slides represent the work and opinions of the presenter and do not constitute official positions of Oracle or any other organization. This material has not been peer reviewed and is presented here with the permission of the presenter. This material should not be reproduced without the written permission of interrel Consulting.

7 Agenda What is ODI? A Real World Example ODI and Planning ODI Demo A glimpse into the concepts Tips Getting Started Questions and Answers

8 What is ODI? Oracle Data Integrator

9 Oracle Data Integrator (ODI) Overview Integration tool that replaces HAL and DIM Used to move and transform information across systems Processes are based on metadata of existing systems Only metadata is stored in ODI, not data Sometimes referred to as Sunopsis

10 How is ODI used with EPM applications? Interface metadata and data between source data systems Build dimensions Load data Transfer data across files and relational databases and the EPM applications All supported with Hyperion Planning, Financial Management and Essbase Not supported with EPMA

11 Oracle ODI to EPM Functionality Metadata discovery & model creation Load data Load Metadata Extract data Planning Essbase HFM P P P P P P P P P O P P Extract metadata O P P Other Refresh to Essbase Calc, post and pre MaxL scripts Consolidate

12 Marketing Message Three main points: Extract-Load & Transform (E-LT) architecture Removes ETL server from infrastructure USP = Native target database performs transformations Declarative Design Developer define the rules, ODI: Writes the data flows Manages the data flow complexity Writes the behind-the-scenes code Knowledge Modules Similar to HAL adapters Talks (and almost always writes) to almost everything

13 A Real World Integration with ODI GL SAP DW Oracle 1 Accounts / Entities Planning Fcst Data 6 2 Accounts / Entities HFM Consolidated Results 7 3 Accounts / Entities Essbase Sales by Customer DB 5 4 Customer Dimension Sales by Customer Data 8 MaxL Run Allocations, Aggregate Data, etc

14 ODI Demo

15 A Glimpse ODI Components and Concepts

16 ODI Components Source Systems ODI (ELT) Extract Load-Transform Target Systems ODI Components Topology Manager Configure Architecture / Infrastructure of source & target technologies Master Repository Work Repository Designer Design & execute metadata models and interfaces to move data across systems Security Manager Maintain ODI Security (Infrastructure) (Interfaces) Operator Monitor Sessions Review errors Agent Coordinates communication & scheduling Metadata Navigator Web access to mettadata repository

17 Components GUI Purpose Users Repository Designer Operator Topology Manager Security Manager Configuration Info Metadata OLTP RDBMS Sunopsis Infrastructure Reverse Engineer Develop Projects Release Scenarios Operate Production Monitor Sessions Physical and Logical Architecture of the infrastructure (servers, schemas and agents) User privileges Administrators Developers Operators Developers Metadata Administrators Production Operators Infrastructure Administrators Security Administrators

18 Deployment Typically have 1 Master Repository that talks to multiple Work Repository (Dev, Test & Prod). Can set the production work repository to execution only Topology Security Versioning Master Repository (Infrastructure) Work Repository \ \ (s) (Interfaces) Models Projects Execution

19 Topology Manager Representation of the Information System Connect to resources for integration purposes Contains: Physical Architecture Data Server Physical Schemas and Physical Agents Logical Architecture (logical schemas & logical agents) Contexts Links logical and physical architectures Repositories Master and attached work repositories Languages - different types of languages available

20 Logical Architecture A Sample Topology dbo.database MSSQL Rep.App1 Essbase Rep.App2 Essbase PlanSamp. Consol Planning FinOp.Plan 1 Planning C:\ODI\File File Physical Architecture Windows Machine X (Development Environment) Windows Machine Y (Production Environment) Windows Machine Z (Production Environment) DW Essbase Source Files dbo.databa Rep.App1 se C:\ODI\File Rep.App2 Planning PlanSamp. Consol FinOp.Plan 1 DW dbo.databa se Essbase Rep.App1 Rep.App2 Planning PlanSamp. Consol FinOp.Plan 1 Source Files C:\ODI\File Data Server Logical Schema Physical Schema Physical Agents Logical Agents

21 Languages Languages used to generate code for the specific technologies

22 Designer Designer includes: Models Metadata for your source and target systems Projects - Integrations that move data from your source and target systems Interfaces Packages Procedures Variables Knowledge Modules The Designer module stores this information in a work repository, references the topology and the security information defined in the master repository.

23 Designing an Interface Before you start building an interface you: Import Knowledge Modules into the project Create your Metadata Models Then in the interface you: Define your source and target Define the mapping and translations (business rules) Define the flow requirements Execute To automate execution, move on to: Packages Procedures Variables Sequencing

24 Knowledge Modules Knowledge Modules (KMs) are components with built in knowledge (functions and logic) that connect technologies to ODI. Similar to HAL Adapters, but many more connection options. KMs connect to the technology, extract data from it, transform the data, check it, integrate it, etc. You can find a complete list of KMs at ence.pdf

25 Knowledge Modules Interfaces LKM (Loading Knowledge Modules) are used to extract data from the source database tables and other systems (files, middleware, mainframe, etc.). IKM (Integration Knowledge Modules) are used to integrate (load) data to the target system. CKM (Check Knowledge Modules) are used to check that constraints on the sources and targets are not violated..

26 Knowledge Modules Models RKM (Reverse Knowledge Modules) are used to perform a customized reverse-engineering of data models for a specific technology. JKM (Journalizing Knowledge Modules) are used to create a journal of data modifications (insert, update and delete) of the source databases to keep track of the changes.

27 Knowledge Modules Others SKM (Service Knowledge Modules) are used to generate the code required for creating dataservices.

28 An Example: Hyperion Planning Model Dimensions Planning Metadata

29 An Example: Hyperion Essbase Model Dimensions Essbase Metadata

30 Interfaces An ODI object which loads one target data store with data from one or more sources, based on business rules implemented as mappings

31 Inside the Interface: Mappings

32 More on Mappings Execute On Target Column Expression Editor Update

33 Multiple Sources, Joins and Filters Joins (similar to Joins in RDMS) Links multiple data sources together Uses SQL Clauses Multiple sources in a single interface must be connected with a join. Create joins by dragging and dropping columns together Alternatively, you can use the expression editor to add them manually Advanced joins Connect models from different technologies or model / schemas. (Heterogeneous Join) Connect 2 or more datastores

34 Join Definition SQL Syntax Specify Join type Use ISO-92 Syntax, Join Order, Autocalculate Order

35 Filter Reduce amount of data processed by the interface by applying a filter. Funnel icon represents filter applied on the source Use the Expression Editor or manually type in the SQL. Test filter, right-click and select Data.

36 Expression Editor

37 Flow Tab

38 SUNOPSIS_MEMORY_ENGINE SUNOPSIS_MEMORY_ENGINE sits between source and target Temporary staging area that is created during interfaces to facilitate mapping and translation Not recommended for large data sets for performance reasons Can be difficult to debug We recommend configuring a separate staging area

39 Operator: Checking Status

40 Automation Packages - Group of steps to complete a workflow. Can include: Interfaces OS Commands Procedures Workflow can include error trapping and status notifications Scenarios can schedule as a scenario or call a scenario from a batch file Variables - can have variables defined to pass parameters (i.e. file name, current month, etc.)

41 Create Error Log Files in the Interface Enable logging in the interface in the files to obtain more details or send information to a text file for use in debugging.

42 Errors Errors are not always ODI errors, they are typically being sent from other technologies (Oracle EPM in our case) You will begin to see commonality of errors but they are not always descriptive or useful. Plan extra time for debugging interfaces

43 Real Life Tips

44 Tips General Information Good sources of information: interrel 3 day ODI and EPM training class (available this fall) OTN Forum: Get the release No patching required for Essbase, Planning, and HFM Available at: i/index.html

45 Design and Development Tips Whiteboard the Topology before configuring in ODI Standardize naming conventions for topology components Incorporate as many error logs and error trapping processes as possible. Consider using a staging area versus Sunopsis Memory Engine. This will allow performance optimization and error resolution. If the source is a relational database, perform as many translations in a view as possible.

46 Design and Development Tips Recommend against using comma as a file delimiter for source files. Use or!. Versus loading Essbase Data with ODI, write a MaxL to load data to Essbase and add it as a step in the ODI package. Plan extra development time, learning curve is steep and it is more complex than HAL.

47 Project Tasks: Getting Started Plan the scope, identify source and target Establish success criteria Identify the resources Define the Topology Define your Models Create Interfaces Unit Testing Automate with Packages Schedule with Scenarios Integration Testing Deploy

48 How long will take? No short answer, depends on the complexity of the interface. If this is the first time, allow ample time to debug issues. Minimize risk by: Working with infrastructure team for connectivity issues Keeping data transformations simple or nonexistent in the initial development (prototype) Then consider the opportunities to automate translations or business rules Plan out your topology on a whiteboard first! It is critical to understand what links to what to avoid silly mistakes. Use Relational Sources where possible.

49 OTHER

50 Security Manager Separate from Shared Services Manage security for all ODI modules Create Users & Profiles Assign access ODI Objects (data servers, projects, etc) Methods (Edit, Create, Delete) Servers

51 Metadata Navigator J2EE application that provides web access to the repositories View Flow maps, trace the source of all data and drill down to field level Possible to trigger and monitor processing jobs from a web browser using this tool By default, installed on Jakarta Tomcat Application Server

52 Oracle Data Profiling and Quality Integrator extends data quality features of Oracle Data Integrator to provide more advanced data governance capabilities. Data Cleansing Duplicates Parsing Data Enrichment User driven rules Integrated with ODI

53 ODI and EPM Training Coming Soon from interrel Training! Introduction to ODI for EPM Administrators This class will be useful for administrators or developers that will design, develop and / or deploy data integrations that update Essbase, Planning or HFM with ODI. Includes several hours of hands on workshops!!!

54 Thank You. Edward Roske BLOG: LookSmarter.blogspot.com WEBSITE: TWITTER: ERoske