Oracle Data Integrators for Beginners Presented by: Dip Jadawala Company: BizTech Session ID: 9950
Please silence your cell phones
Overview Introductions Architecture of ODI Topology Manager Designer Operator Security Manager Q & A
About Dip Jadawala Senior Technical Consultant with BizTech 6 Years of Data warehousing, ETL, Reporting experience Experience with different ETL tools like ODI, Informatica, Pentaho and OWB. Ad hoc Reporting using OBIEE experience in all phases of the system development and software development lifecycle.
About BizTech Leading Mid-Atlantic Oracle Platinum Partner and IT Services firm focused on Oracle Applications and Technology solutions Over 400 successful Oracle implementations over the past 15 years Based in King of Prussia, PA with offices in New Jersey, New York City and Washington DC Service Fortune 500 companies, organizations and government agencies 100 + Oracle certified and experienced consultants
Client-Centric Practice Areas Oracle Applications - Full Portfolio of Oracle Applications Solutions - Implementation, Upgrade, Migration - Since 1990 MPL6 to R12 Experience - Over 400 successful implementations to date Oracle Applications Oracle Technology BI/EPM Oracle Technology and Business Intelligence - End to end service offering in BI and EPM - Fully staffed team of Data Architects and DBAs - Solid experience in RAC, HA and HS designs - Understand full Oracle technology stack Managed Services and IT Outsource - Remote or Onsite services - Full portfolio of Oracle Applications and Technologies - World-Class Data Center with 24x7 Support - Instant capacity, operational focused business model Managed Services ITO Clients Oracle Software Provider Oracle Software Provider - Full Portfolio of Oracle License Resell - Help Clients Optimize License models - RapidApp BI Software for the agile enterprise - RapidApp Auditor to manage change and GRC
Prerequisites 1. The system should include the following installed products: Oracle Database 11g Oracle Data Integrator 11gR1 2. Creation of ODI Master Repositoryand Work Repository has been done.
Architecture of ODI
Architecture of ODI Oracle Data Integratoris built around a modular repository architecture and accessed in client/server or thin client mode by components that are developed entirely in Java. Repositories. ODI Studio. Weblogic/Application Server.
Architecture of ODI Repositories The central component of the architecture is the Oracle Data Integrator Repository. It stores configuration information about the IT infrastructure, metadata of all applications, projects, scenarios, and the execution logs. There are 2 types of repositories in ODI. 1. Master repository. 2. Work repository. Work repositories can be 1 or more. There should be only one master repository talking to all work repositories. According to best practices we should have only 1 master repository and multiple work repositories like DEV, TEST and PROD.
Architecture of ODI Master Repository Master Repository consists of 1. Security information including users, profiles and rights for the ODI platform. 2. Topology information including technologies, server definitions, schemas, contexts, languages and so forth. 3. Versioned and archived.
Architecture of ODI Work Repository A Work Repository consists of 1. Models, including schema definition, data stores structures and metadata, fields and columns definitions, data quality constraints, cross references, data lineage etc. 2. Projects, including business rules, packages, procedures, folders, Knowledge Modules, variables etc. 3. Scenario execution, including scenarios, scheduling information and logs.
Architecture of ODI ODI Studio Administrators, Developers and Operators use the Oracle Data Integrator Studio to access the repositories. This Fusion Client Platform (FCP) based UI is used for administering the infrastructure (security and topology), reverseengineering the metadata, developing projects, scheduling, operating and monitoring executions. ODI Studio provides four Navigators for managing the different aspects and steps of an ODI integration project
Architecture of ODI
Architecture of ODI Run-Time Agent At design time, developers generate scenarios from the business rules that they have designed. The code of these scenarios is then retrieved from the repository by the Run-Time Agent. This agent then connects to the data servers and orchestrates the code execution on these servers. 2 types of agents comes with ODI. 1. The Java EE. 2. Standalone Agent.
Topology Manager Topology Navigatoris used to manage the data describing the information system's physical and logical architecture. Through Topology Navigator you can manage the topology of your information system, the technologies and their data types, the data servers linked to these technologies and the schemas they contain, the contexts, the languages and the agents, as well as the repositories. The site, machine, and data server descriptions will enable Oracle Data Integrator to execute the same integration interfaces in different physical environments.
Topology Manager Connection to Master Repository and Topology Manager User snmp1 has been created on database already. jdbc:oracle:thin:@<host>:<port>:<sid> You can use any Driver depending on target Database.
Topology Manager Work repository connection to Master Repository
Topology Manager Work repository connection to Master Repository
Topology Manager Work repository connection to Master Repository
Topology Manager Agent Setup
Topology Manager Agent Setup
Topology Manager Start Agent Services Command Prompt agent -NAME=localagent for windows machine Command Prompt./agent.sh NAME=localagent for linux/unix machine
Topology Manager SCHEMA Schema where ODI inserts and updates the target tables. WORK_SCHEMA Schema where all the ODI Temporary table ( C$_,I$_ etc ) are created and dropped by ODI.
Topology Manager Contexts are bridge between Physical and Logical Architecture. The contexts in ODI are basically labels that you can create as needed. ODI comes with only one context called Global. Typical implementations would actually have three contexts:development, QA, Production.
Designer Designer Navigatoris used to design data integrity checks and to build transformations such as for example: Automatic reverse-engineering of existing applications or databases Graphical development and maintenance of transformation and integration interfaces Visualization of data flows in the interfaces Automatic documentation generation Customization of the generated code 3 things in Designer where development is required are Interface Package Scenario
Designer Main Tabs on Designer 1. Projects The Projects accordion contains the developments made with Designer Navigator. 2. Models The Models accordion contains the descriptions of the data and applications structures. 3. Others The Others accordion contains the Global User Functions, Variables, Markers, and Sequences. 4. Solutions The Solutions accordion contains the Solutions that have been created when working with version management.
Designer Knowledge Module Description Reverse-engineering KM Check KM Loading KM Integration KM Journalizing KM Retrieves metadata to the Oracle Data Integrator work repository Checks consistency of data against constraints Loads heterogeneous data to a staging area Integrates data from the staging area to a target Creates the Change Data Capture framework objects in the source staging area Usage Used in models to perform a customized reverse-engineering Used in models, sub models and datastores for data integrity audit Used in interfaces for flow control or static control Used in interfaces with heterogeneous sources Used in interfaces Used in models, sub models and datastores to create, start and stop journals and to register subscribers. Service KM Generates data manipulation web services Used in models and datastores
Designer Connecting to Designer
Designer Data store creation in Model
Designer You can see Flat File is also reverse engineered in right most snapshot.
Designer Interface creation
Designer
Designer
Designer
Designer Execution of Interface Right click on the interface and click Execute You can see interface running in Operator.
Designer 2 nd example
Designer CKM Use
Designer Package Creation
Designer Scenario Creation
Designer Parallel Processing of Interfaces
Operator The Operator Navigator has the following accordions: Session List The Session List accordion displays all sessions organized per date, physical agent, status, keywords, and so forth. Hierarchical Sessions The Hierarchical Sessions accordion displays the execution sessions organized in a hierarchy with their child sessions. Scheduling The Scheduling accordion displays the list of physical agents and schedules. Scenarios The Scenarios accordion displays the list of scenarios available Solutions The Solutions accordion contains the Solutions that have been created when working with version management.
Operator
Operator In Designer Navigator Select the TRG_CUSTOMER datastore. Right-click and selectcontrol>errors... 2records that did not satisfy the FK_CUST_CITY constraint (for example, the CITY_ID value does not exist in the table of cities SRC_CITY table). 7records that did not satisfy the business rule acting on customers under 21 (AGE > 21 constraint). The invalid records were saved into an error table and were not integrated into the target table. Correct the data and Re run it.
Operator Scheduling the Scenario You can schedule the scenario either in operator or in Designer
Operator Scheduling the Scenario
Operator Scheduling the Scenario
Operator Update Scheduling in Topology
Operator Update Scheduling in Topology
Operator View the scheduler in Topology
Security Manager Security Navigatoris the tool for managing the security information in Oracle Data Integrator. Through Security Navigator you can create users and profiles and assign user rights for methods (edit, delete, etc) on generic objects (data server, data types, etc), and fine-tune these rights on the object instances (Server 1, Server 2, and so forth)
Security Manager User Creation
Security Manager Manage Profiles
Security Manager Manage Profiles
Security Manager Open Designer and drag /drop folder.
Security Manager Allow/Deny to appropriate Methods that User can access
Security Manager
Summary Architecture of ODI is made of Java Programming. Core part is Repository. Topology Manager has source and Target connection details. Designer has interfaces, packages, scenarios, variables, KM, procedures and so on. All the development components. Operator is monitoring tool which keeps the logs of interfaces, packages and scenarios. Security Manager keep the Users and their respective profiles to access different modules of ODI Studio.
References http://docs.oracle.com http://www.oracle.com/technetwork/tutorials http://odiexperts.com http://blogs.oracle.com
Any Questions? Dip Jadawala Senior Technical Consultant DJadawala@Biztech.com Twitter: @BizTechOracle Booth #1252 on Main Street