1 Oracle Data Integrator Technical Overview An Oracle White Paper Updated December 2006
2 Oracle Data Integrator Technical Overview Introduction... 3 E-LT Architecture... 3 Traditional ETL... 3 E-LT... 4 Declarative Design... 6 Conventional ETL Design... 6 Declarative Design... 6 Knowledge Modules... 8 Knowledge Modules Type... 8 Knowledge Modules at Design and Runtime... 8 Flexibility and Extensibility... 9 Event-Oriented Integration Message-Oriented Integration Changed Data Capture Publish-and-Subscribe Model Processing Consistent Sets of Changed Data SOA-Enabled Data and Transformation Services Web Services Access Data Integrity Declarative Rules for Data Integrity Data Integrity Firewall in the Integration Process Enforcing the Rules Using Third-Party Name and Address Cleansing Tools Architecture User Interfaces Agent Repositories Metadata Navigator / Lightweight Designer Scenarios Data Warehouse / Business Intelligence Service-Oriented Integration Master Data Management Conclusion Oracle Data Integrator Technical Overview Page 2
3 Oracle Data Integrator Technical Overview Oracle Data Integrator features an active integration platform that includes all styles of data integration: data-based, eventbased, and service-based. Capable of transforming large volumes of data efficiently, processing events in real time through its advanced Changed Data Capture (CDC) capability, or providing data services to the Oracle SOA Suite, Oracle Data Integrator unifies silos of integration. INTRODUCTION Integrating data and applications throughout the enterprise and presenting them in a unified view is a complex proposition. Not only are there broad disparities in technologies, data structures, and application functionality, but there are also fundamental differences in integration architectures. Some integration needs are data oriented, especially those involving large data volumes. Other integration projects lend themselves to an event-driven architecture (EDA) or a serviceoriented architecture (SOA), for asynchronous or synchronous integration. Many organizations address these diverse needs with a broad palette of tools and technologies, resulting in disjointed integration projects with no leverage or unity between them. These tools do not meet all performance, flexibility, and modularity requirements. Oracle Data Integrator features an active integration platform that includes all styles of data integration: data-based, event-based, and service-based. Capable of transforming large volumes of data efficiently, processing events in real time through its advanced Changed Data Capture (CDC) capability, or providing data services to the Oracle SOA Suite, Oracle Data Integrator unifies silos of integration. It also provides robust data integrity control features, assuring the consistency and correctness of data. With powerful core differentiators heterogeneous E-LT, declarative design, and Knowledge Modules Oracle Data Integrator meets the performance, flexibility, productivity, modularity and hot-pluggability requirements of an integration platform. E-LT ARCHITECTURE Traditional ETL Traditional extract, transform, and load (ETL) tools operate by first extracting the data from various sources, transforming the data on a proprietary, middle-tier ETL engine, and then loading the transformed data onto the target data warehouse or integration server. Hence, the term ETL represents both the names and the order of the operations performed. The data transformation step of the ETL process is by far the most computeintensive and is performed entirely by the proprietary ETL engine on a dedicated Oracle Data Integrator Technical Overview Page 3
4 server. The ETL engine performs data transformations (and sometimes data quality checks) on a row-by-row basis, and hence, can easily become the bottleneck in the overall process. In addition, the data must be moved over the network twice once between the sources and the ETL server, and again between the ETL server and the target data warehouse. Moreover, if you want to ensure referential integrity by comparing data flow references against values from the target data warehouse, the referenced data must be downloaded from the target to the engine, thus further increasing network traffic, download time, and leading to additional performance issues. Consider, for example, how a traditional ETL job would look up values from the target database to enrich data coming from source systems. To perform such a job, a traditional ETL tool could be used in one of the following three ways: Load lookup tables into memory: The entire lookup table is retrieved from the target server and loaded into the engine s memory. Matching (or joining) this lookup data with source records is done in memory before the resulting transformed data is written back to the target server. If the lookup table is large, the operation will require a large amount of memory and a long time to download its data and re-index it in the engine. Perform row-by-row lookups on the fly : For every row, the ETL engine sends a query to the lookup table located on the target server. The query returns a single row that is matched (or joined) to the current row of the flow. If the lookup table contains, for example, 500,000 rows, the ETL engine will send 500,000 queries. This will dramatically slow down the data integration process and add significant overhead to the target system. The E-LT architecture incorporates the best aspects of both manual coding and ETL approaches in the same solution. The E-LT approach changes where and how data transformation takes place, and leverages the existing developer skills, RDBMS engines and server hardware to the greatest extent possible. Use manual coding within the ETL job: Use the ETL engine only for loading source data to the target relational database management system (RDBMS) and manually write SQL code to join this data to the target lookup table. This raises the question: Why buy an ETL tool that requires manual coding on the target server, knowing that you lose all the benefits of metadata management and development productivity by doing so? Unfortunately, this is what many users end up doing once they notice 10x degradation in the overall performance of the integration process (when compared to the same operations executed by manual code). E-LT The E-LT architecture incorporates the best aspects of both manual coding and ETL approaches in the same solution. The E-LT approach changes where and how data transformation takes place, and leverages the existing developer skills, RDBMS engines and server hardware to the greatest extent possible. In essence, E-LT relocates the data transformation step on the target RDBMS, changing the order of operations to: extract the data from the source tables, load the tables into the destination server, and then transform the data on the target RDBMS using native SQL operators. Oracle Data Integrator Technical Overview Page 4
5 Conventional ETL Architecture Extract Transform Load Next Generation Architecture E-LT Transform Extract Load Transform ETL vs. E-LT approach Throughput is limited only by the characteristics of the existing servers. Since no extra server, technology, or skill requirement comes into play, the E-LT architecture provides optimal performance and scalability and eases the management of the integration infrastructure. The E-LT architecture leverages the power of the RDBMS engine. Throughput is limited only by the characteristics of the existing servers. Since no extra server, technology, or skill requirement comes into play, the E-LT architecture provides optimal performance and scalability and eases the management of the integration infrastructure. Let us see what would be the impact on the same lookup as described earlier: There is no need to excessively move data out of the source and target servers to the ETL server. Relevant source data would be moved to the target, and the lookup would directly occur in the same target server. This method reduces network traffic to the necessary part only. The processing will not be performed row-by-row, but with a single query using the join capabilities of the target server. Using the processing capabilities of the RDBMS engines provides the best performances for such an operation. The performances are therefore at least equal to those of manual coding, except that all the benefits of a data integration platform in terms of metadata management and productivity are still there. With no separate ETL engine and no dedicated ETL server hardware required, the initial hardware and software capital costs are significantly lower, as are the ongoing software and hardware maintenance expenses. Since the E-LT architecture uses any existing RDBMS to execute the ETL jobs, the overall cost is dramatically reduced. The same servers hosting the data are used to integrate them. Oracle Data Integrator Technical Overview Page 5
6 DECLARATIVE DESIGN With declarative design, you just need to design what the process does, without describing how it will be done. Conventional ETL Design To design an integration process with conventional ETL systems, a developer needs to design each step of the process. Consider, for example, a common case in which sales figures must be summed over time for different customer age groups. The sales data comes from a sales management database, and age groups are described in an age distribution file. In order to combine these sources and then insert and update appropriate records in the customer statistics systems, you must design each step, which includes 1. Load the customer sales data in the engine 2. Load the age distribution file in the engine 3. Perform a lookup between the customer sales data and the age distribution data 4. Aggregate the customer sales grouped by age distribution 5. Load the target sales statistics data into the engine 6. Determine what needs to be inserted or updated by comparing aggregated information with the data from the statistics system 7. Insert new records into the target 8. Update existing records into the target This method requires specialized skills, depending on the steps that need to be designed. It also requires significant efforts in development, because even repetitive succession of tasks, such as managing inserts/updates in a target, need to be developed into each task. Finally, with this method, maintenance requires significant effort. Changing the integration process requires a clear understanding of what the process does as well as the knowledge of how it is done. With the conventional ETL method of design, the logical and technical aspects of the integration are intertwined. Declarative Design With declarative design, you just need to design what the process does, without describing how it will be done. In our example, what the process does is Relate the customer age from the sales application to the age groups from the statistical file Aggregate customer sales by age groups to load sales statistics Oracle Data Integrator Technical Overview Page 6
7 How this is done that is, the underlying technical aspects or technical strategies for performing this integration task, such as creating temporary data structures, calling loaders is clearly separated from the declarative rules. Conventional ETL Design ODI Declarative Design 1 2 Define Automatically What Generate You Want Dataflow Define How: Built-in Templates Conventional ETL design vs. Declarative Design Declarative design in Oracle Data Integrator uses the well-known relational paradigm to declare in the form of an interface the declarative rules for a data integration task, which includes designation of sources, targets, and transformations. In our example, the sources are the sales table and the age spreadsheet, and the target is the customer statistics system. The transformations are a join between the sales table and the age spreadsheet and an aggregation of the sales-per-age group. Oracle Data Integrator Technical Overview Page 7
8 Designing a dataflow with Data Integrator With declarative design, you focus on what really matters that is, the rules for your integration instead of focusing on the underlying technical aspect of the integration process. With declarative design, you focus on what really matters that is, the rules for your integration instead of focusing on the underlying technical aspect of the integration process. The technical aspects are described in Knowledge Modules. KNOWLEDGE MODULES Knowledge Modules Type Oracle Data Integrator s Knowledge Modules implement how the integration processes occur. Each Knowledge Module type refers to a specific integration task: Reverse-engineering metadata from the heterogeneous systems for Oracle Data Integrator Handling Changed Data Capture (CDC) on a given system Loading data from one system to another, using system-optimized methods Integrating data in a target system, using specific strategies (insert/update, slowly changing dimensions) Controlling data integrity on the data flow Exposing data in the form of services These Knowledge Modules cover a wide range of technologies and techniques. Knowledge Modules at Design and Runtime A Knowledge Module is a code template for a given integration task. This code is independent of the interface, which defines the sources, targets, and transformations that will be processed. At design time, a developer creates metadata Oracle Data Integrator Technical Overview Page 8
9 describing integration processes. This metadata is merged with the Knowledge Module to generate code ready for runtime. At runtime, Oracle Data Integrator sends this code for execution to the source and target systems it leverages for running the process. Flexibility and Extensibility Knowledge Modules provide additional flexibility by giving users access to the most-appropriate or finely tuned solution for a specific task in a given situation. For example, to transfer data from one DBMS to another, a developer can use any of several methods depending on the situation: The DBMS loaders (Oracle s SQL*Loader, Microsoft SQL Server s BCP, Teradata TPump) can dump data from the source engine to a file and then load this file to the target engine. The database link features (Oracle Database Links, Microsoft SQL Server s Linked Servers) can transfer data directly between servers. These technical strategies, among others, correspond to Knowledge Modules tuned to exploit native capabilities of given platforms. Knowledge Modules are also fully extensible. Their code is open and can be edited through a graphical user interface by technical experts willing to implement new integration methods or best practices (for example, for higher performance or to comply with regulations and corporate standards). Without having the skill of the technical experts, developers can use these custom knowledge modules in the integration processes. Oracle Data Integrator comes out of the box with more than 100 Knowledge Modules for the major database engines and application packages of the market. Pluggable Knowledge Modules Architecture Reverse Engineer Metadata Reverse Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformation Services WS WS WS Staging Tables CDC Sources Journalize Load Check Error Tables Integrate Target Tables Services Sample out-of-the-box Knowledge Modules SAP/R3 Log Miner SQL Server Triggers Oracle DBLink JMS Queues Check MS Excel TPump/ Multiload Oracle Merge Oracle Web Services Siebel DB2 Journals DB2 Exp/Imp Oracle SQL*Loader Check Sybase Type II SCD Siebel EIM Schema DB2 Web Services Oracle Data Integrator Technical Overview Page 9
10 EVENT-ORIENTED INTEGRATION Oracle Data Integrator includes technology to support message-based integration and that complies with the Java Message Services (JMS) standard. Message-Oriented Integration Capturing events from message-oriented middleware or an enterprise service bus such as Oracle Enterprise Service Bus (ESB) has become a common task in integrating applications in a real-time environment. Applications and business processes generate messages for several subscribers, or they consume messages from the messaging infrastructure. Oracle Data Integrator includes technology to support message-based integration and that complies with the Java Message Services (JMS) standard. For example, a transformation job within Oracle Data Integrator can subscribe and source messages from any message queue or topic. Messages are captured and transformed in real time and then written to the target systems. The JMS model supports transactions to ensure data integrity and message delivery from the source middleware to the target systems. Other uses of this type of integration might require capturing changes at the database level and publishing them to the messaging infrastructure. The Changed Data Capture capability of Oracle Data Integrator, described below, can be coupled with JMS-based integration, resulting in the highest flexibility for designing the required integration flows and mixing heterogeneous technologies. For example, incoming orders can be detected at the database level using Changed Data Capture. These new orders are enriched and transformed by Oracle Data Integrator before being posted to the appropriate message queue or topic. Other applications, such as Oracle BPEL or Oracle Business Activity Monitoring, can subscribe to these messages, and the incoming events will trigger the appropriate business processes. Changed Data Capture The conventional data integration approach involves extracting all data from the source system, and then integrating the entire set possibly incrementally in the target system. This approach may reveal itself inefficient when the integration process reflects the need for event-oriented integration. A typical example would be the need to propagate a new or updated contact file from a customer relationship management (CRM) application to another application. In this case, the amount of data involved thousands of contacts makes data integration impossible in the given timeframes. In-house designed solutions such as filtering records according to a timestamp column or changed flag are possible, but may require modifications in the applications and are usually not sufficient for ensuring that all changes are properly taken into account. The Changed Data Capture capability of Oracle Data Integrator identifies and captures inserted, updated, or deleted data from the source and makes it available Oracle Data Integrator Technical Overview Page 10
11 for integration processes. Changed Data Capture can be an alternative to messageoriented integration; it can also be used in conjunction with target-specific needs, minimizing development efforts. Publish-and-Subscribe Model Changed Data Capture uses a publish-and-subscribe model. An identified subscriber usually an integration process subscribes to changes that happen in a datastore. Changes in the datastore are captured by the Changed Data Capture framework and published for the subscriber, which can at any time process the tracked changes, and consume these events. Orders Order #5A32 Capture (Publish) CDC Subscribe Consume Integration Process Load ODS Integrate ODS New Order #5A32 Oracle Data Integrator framework for tracking changes Oracle Data Integrator provides two methods for tracking changes from source datastores to the Changed Data Capture framework: triggers and RDBMS log mining. The first method can be deployed on most RDBMSes that implement database triggers. This method is optimized to minimize overhead on the source systems. For example, changed data captured by the trigger is not duplicated, minimizing the number of input/output operations, which slow down source systems. The second method involves mining the RDBMS logs the internal change history of the database engine. This has little impact on the system s transactional performance and is supported for Oracle (through the Log Miner feature) and IBM DB2/400. The Changed Data Capture framework used to manage changes is generic and open, so the change-tracking method can be customized. Any third-party change provider can be used to load the framework with changes. Processing Consistent Sets of Changed Data Changes frequently involve several data sources at the same time. For example, when an order is created, updated, or deleted, both the orders table and the order lines table are involved. When processing a new order line, it is important that the new order, to which the line is related, is taken into account too. Oracle Data Integrator Technical Overview Page 11
12 Oracle Data Integrator provides a mode of change tracking called Consistent Set CDC. This mode allows for processing sets of changes for which data consistency is guaranteed. Oracle Data Integrator plugs into the Oracle SOA Suite through three key service access capabilities: data services, transformation services, and Web services access. SOA-ENABLED Oracle Data Integrator plugs into the Oracle SOA Suite through three key service access capabilities: data services, transformation services, and Web services access. Data and Transformation Services Data services are specialized Web services that provide access to data stored in database tables. Coupled with the Changed Data Capture capability, data services can also provide access to the changed records for a given subscriber. Data services are automatically generated by Oracle Data Integrator and deployed as Web services to a Web container, usually a Java application server. Oracle Data Integrator can also expose its transformation processes as Web services to enable applications to use them as integration services. For example, a LOAD_SALES batch process used to update the CRM application can be triggered as a Web service from any service-compliant application, such as Oracle BPEL, Oracle Enterprise Service Bus, or Oracle Business Activity Monitoring. Transformations developed using Oracle Data Integrator can therefore participate in the broader service-oriented architecture initiative. Data services ready for generation and deployment Oracle Data Integrator Technical Overview Page 12
13 Web Services Access Third-party Web services can be invoked as part of an Oracle Data Integrator workflow and used as part of the data integration processes. Requests are generated on the fly and responses processed through regular transformations. Suppose, for example, that your company subscribed to a third-party service that exposes daily currency exchange rates as a Web service. If you want this data to update your multiple currency data warehouse, Oracle Data Integrator automates this task with a minimum of effort. You would simply invoke the Web service from your data warehouse workflow and perform any appropriate transformation to the incoming data to make it fit a specific format. Using Web services in workflows DATA INTEGRITY Data integrity is the large subset of data quality, which includes the processes that can be automated to ensure the quality of the enterprise information assets. Declarative Rules for Data Integrity Oracle Data Integrator uses declarative data integrity rules defined in its centralized metadata repository. These rules are applied to application data to guarantee the integrity and consistency of enterprise information. The data integrity benefits add to the overall data quality initiative and facilitate integration with existing and future business processes addressing this particular need. Oracle Data Integrator automatically retrieves existing rules defined at the data level (such as database constraints) by a reverse-engineering process. Oracle Data Integrator also allows developers to define additional, user-defined declarative rules Oracle Data Integrator Technical Overview Page 13
14 that may be inferred from data discovery and profiling within Oracle Data Integrator, and immediately checked. Define a new rule in Designer Quickly validate the new rule against the data Define a new integrity rule in Designer and immediately check it against the data. Declarative rules for data integrity include uniqueness rules, validation rules that enforce consistency at the record level, and simple or complex reference rules possibly involving heterogeneous technologies. Extensive information on data integrity makes possible detailed analysis, and takes erroneous data into account according to IT strategies and best practices for improving the overall data quality. Data Integrity Firewall in the Integration Process The components involved in an integration process are the source applications, the processes that transform and integrate data, and the target applications. With Oracle Data Integrator, data integrity is managed in all these subsystems, creating a real firewall that can be activated at any point of the integration process. Data integrity in the integration process Oracle Data Integrator Technical Overview Page 14
15 Enforcing the Rules Oracle Data Integrator s customizable Check Knowledge Modules help developers to automatically generate data integrity audits of their applications based on integrity rules that have been defined in the repository. Audits provide statistics on the quality of application data. They also isolate data that is detected as erroneous by applying the declarative rules. When erroneous records have been identified and isolated in error tables, they can be accessed directly from Oracle Data Integrator or from any other front-end application. This extensive information on data integrity makes possible detailed analysis, and takes erroneous data into account according to IT strategies and best practices for improving the overall data quality. Reviewing erroneous data in Designer Using Third-Party Name and Address Cleansing Tools The Oracle Data Integrator open architecture can be seamlessly integrated with third-party data quality products, such as Trillium TS Quality to perform complex data quality and cleansing operations. ARCHITECTURE Oracle Data Integrator is organized around modular repositories. All runtime and design-time components are Java components that store their information in metadata repositories. The components of this architecture can be installed and run on any platform. Oracle Data Integrator Technical Overview Page 15
16 Development User Interfaces Administrators Designers ODI Design-Time Environment Topology/Security Design-time Metadata/Rules Repositories Code Execution Log Agent Data Flow Conductor Development Servers and Applications Execution CRM Data Warehouse Return Codes Legacy ERP ESB Files / XML Production Scenarios and Projects Releases User Interfaces Administrators Operators ODI Runtime Environment Topology/Security Code Runtime Execution Log Repository Execution Log Agent Data Flow Conductor Production Servers and Applications Execution CRM Data Return Codes Warehouse Legacy Thin Client Data Stewarts Metadata Lineage Metadata Navigator ESB Files / XML ERP Oracle Data Integrator detailed architecture User Interfaces The four Oracle Data Integrator graphical modules are Designer: In this interface, users can define declarative rules for data transformation and data integrity. Database and application metadata can be imported or defined. Designer uses metadata and rules to generate scenarios for production. All project development is performed through this interface, and it is the main user interface for developers and metadata administrators at design time. Operator: In this interface, users can manage and monitor Oracle Data Integrator jobs in production. It is designed for production operators and shows the execution logs with error counts, the number of rows processed, execution statistics, the actual code that is executed, and so on. At design time, developers can also use Operator for debugging purposes. It is the main user interface at runtime. Topology Manager: In this interface, users can define the physical and logical architecture of the infrastructure. Servers, schemas, and agents are registered in the Oracle Data Integrator master repository through this interface, which is primarily used by the administrators of the infrastructure or project. Security Manager: In this interface, administrators can manage user accounts and privileges. It can be used to give profiles and users access rights Oracle Data Integrator Technical Overview Page 16
17 Agent to Oracle Data Integrator objects and features. This interface is primarily used by security administrators. At runtime, the runtime component the Agent orchestrates the execution of scenarios. Execution may be launched from one of the user interfaces and triggered by the built-in scheduler or by a third-party scheduler. Thanks to Oracle Data Integrator s E-LT architecture, the agent rarely performs any transformation itself. Usually, it simply retrieves code from the execution repository and requests that database servers, operating systems, or scripting engines execute that code. When the execution is completed, the agent updates the execution logs in the repository and reports error messages and execution statistics. The execution logs can be viewed from the Operator user interface or a Web interface: Metadata Navigator. It is important to understand that although it can act as a transformation engine, the agent is rarely used this way. Agents are installed at tactical locations in the information system to orchestrate the integration processes and leverage existing systems. Agents are multithreaded, load-balanced, lightweight components in this distributed integration architecture. Repositories The Oracle Data Integrator repository is composed of a master repository and several work repositories. These repositories are pluggable in any relational database management system. All objects configured, developed, or used with Oracle Data Integrator modules are stored in one of these two types of repository. There is usually only one master repository, which contains security information (user data and privileges), topology information (definitions of technologies and servers), and versions of objects. The work repository is where projects are stored. Several work repositories may coexist in a single Oracle Data Integrator installation. This is useful for maintaining separate environments or to reflect a particular versioning lifecycle, such as development, user acceptance tests, and production environments. A work repository stores information related to Models, including datastores, columns, data integrity rules, cross references, and data lineage Projects, including interfaces, packages, procedures, folders, knowledge modules, and variables Runtime information, including scenarios, scheduling information, and logs Oracle Data Integrator Technical Overview Page 17
18 Business users, developers, operators, and administrators can use their Web browser to access Metadata Navigator or Lightweight Designer. Through these Web interfaces, they can see flow maps, trace the source of all data, and even drill down to the field level to understand the transformations used to build the data. They can also launch and monitor scenarios and edit data mappings though these Web interfaces. Metadata Navigator / Lightweight Designer Business users, developers, operators, and administrators can use their Web browser to access Metadata Navigator or Lightweight Designer. Through these Web interfaces, they can see flow maps, trace the source of all data, and even drill down to the field level to understand the transformations used to build the data. They can also launch and monitor scenarios and edit data mappings though these Web interfaces. SCENARIOS Oracle Data Integrator can be used for all integration tasks. The following examples demonstrate typical situations for which the platform features can be exploited. Data Warehouse / Business Intelligence Load Transform Capture Changes Incremental Update Data Integrity Aggregate Export Cube Operational Data Warehouse Cube Analytics Cube Metadata Scenario: E-LT for data warehouse In this common scenario, Oracle Data Integrator is used in batch mode to load the data warehouse from heterogeneous sources, using incremental updates or slowly changing dimensions (SCD type 2 or 3) strategies. Performance provided by the E-LT architecture enables narrow batch windows and leverages the power of the data warehouse server. Data integrity and consistency is ensured through the data integrity firewall. The Changed Data Capture feature updates in near real-time the data warehouse with changes occurring in operational data. Oracle Data Integrator also aggregates and exports data from the data warehouse to load the online analytical processing (OLAP) cubes and analytics applications. Thanks to the comprehensive metadata supporting the integration processes, users can track data lineage from end to end. Oracle Data Integrator Technical Overview Page 18
19 Service-Oriented Integration Generate Data Services Expose Transformation Services Deploy and reuse Services Services Business Processes Operational Data Access Transformation Others Metadata Scenario: SOA initiative In service-oriented development architectures, Oracle Data Integrator provides automatically generated services to access the data, as well as the transformations and integration processes it handles. Services available in the infrastructure can also be used in the transformation processes. Oracle Data Integrator Technical Overview Page 19
20 Master Data Management Change Data Capture Master Data Load Canonical Format Design Cleansing and Reconciliation Master Data Publishing CDC CDC Master Data CDC Metadata Scenario: Master data management Thanks to the intuitive Common Format Designer graphical feature, users can define the master data canonical format by assembling heterogeneous sources. Oracle Data Integrator then generates the scripts to create the master data hubs as well as the data flows to reconcile and publish this master data from and to the operational systems. Oracle Data Integrator can also be used in existing master data management initiatives. It allows the synchronization of operational source systems with master data management applications such Oracle Customer Data Hub and Oracle Product Information Management. Oracle Data Integrator can also be used in existing master data management initiatives. It allows the synchronization of operational source systems with master data management applications such Oracle Customer Data Hub and Oracle Product Information Management. Thanks to its Changed Data Capture framework, Oracle Data Integrator transforms and synchronizes data in real time from sources to master data management applications and from master data management applications to the subscribing operational applications. Data Integrity Firewall ensures master data quality and integrity Finally, the comprehensive and extensible metadata in the repository provides users with a solid foundation for their master data management initiative. CONCLUSION Oracle Data Integrator fulfills the needs of data-oriented, event-driven and serviceoriented integration, and is fully adaptable to batch, real-time or synchronous latency requirements. It provides key features to meet the prerequisites for efficient data integration: Heterogeneous ETL for unmatched performance even in heterogeneous environments, declarative design for optimal productivity when designing Oracle Data Integrator Technical Overview Page 20
21 transformations, Knowledge Modules providing the modularity and hot-plugability to any system. Oracle Data Integrator Technical Overview Page 21
22 Oracle Data Integrator Technical Overview December 2006 Author: FX Nicolas Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA U.S.A. Worldwide Inquiries: Phone: Fax: oracle.com Copyright 2007, Oracle. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
An Oracle White Paper February 2009 Real-time Data Warehousing with ODI-EE Changed Data Capture Executive Overview Today s integration project teams face the daunting challenge of deploying integrations
ORACLE DATA INTEGRATOR ENTERPRISE EDITION KEY FEATURES AND BENEFITS ORACLE DATA INTEGRATOR ENTERPRISE EDITION OFFERS LEADING PERFORMANCE, IMPROVED PRODUCTIVITY, FLEXIBILITY AND LOWEST TOTAL COST OF OWNERSHIP
An Oracle White Paper February 2014 Oracle Data Integrator 12c Introduction Oracle Data Integrator (ODI) 12c is built on several components all working together around a centralized metadata repository.
An Oracle White Paper March 2014 Best Practices for Real-Time Data Warehousing Executive Overview Today s integration project teams face the daunting challenge that, while data volumes are exponentially
ORACLE DATA INTEGRATOR ENTEPRISE EDITION FOR BUSINESS INTELLIGENCE KEY FEATURES AND BENEFITS (E-LT architecture delivers highest performance. Integrated metadata for alignment between Business Intelligence
An Oracle White Paper March 2012 Managing Metadata with Oracle Data Integrator Introduction Metadata information that describes data is the foundation of all information management initiatives aimed at
ORACLE DATA INTEGRATOR ENTERPRISE EDITION Oracle Data Integrator Enterprise Edition 12c delivers high-performance data movement and transformation among enterprise platforms with its open and integrated
Business Intelligence and Service Oriented Architectures An Oracle White Paper May 2007 Note: The following is intended to outline our general product direction. It is intended for information purposes
Oracle University Contact Us: Local: 1800 103 4775 Intl: +91 80 4108 4709 Oracle Data Integrator 11g: Integration and Administration Duration: 5 Days What you will learn Oracle Data Integrator is a comprehensive
Comprehensive Data Quality with Oracle Data Integrator An Oracle White Paper Updated December 2007 Comprehensive Data Quality with Oracle Data Integrator Oracle Data Integrator ensures that bad data is
OWB Users, Enter The New ODI World Kulvinder Hari Oracle Introduction Oracle Data Integrator (ODI) is a best-of-breed data integration platform focused on fast bulk data movement and handling complex data
Manage Oracle Database Users and Roles Centrally in Active Directory or Sun Directory Overview August 2008 Introduction... 3 Centralizing DataBase Account Management using Existing Directories with OVD...
Is ETL Becoming Obsolete? Why a Business-Rules-Driven E-LT Architecture is Better Sunopsis. All rights reserved. The information contained in this document does not constitute a contractual agreement with
ORACLE BUSINESS INTELLIGENCE SUITE ENTERPRISE EDITION PLUS PRODUCT FACTS & FEATURES KEY FEATURES Comprehensive, best-of-breed capabilities 100 percent thin client interface Intelligence across multiple
Oracle Primavera Gateway Disclaimer The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is
ORACLE SOA SUITE KEY FEATURES AND BENEFITS COMPLETE INTEGRATED OPEN BEST-OF-BREED FEATURES SOA Composite Editor Graphical editor to assemble heterogeneous components into composite applications Service
An Oracle White Paper June 2009 Integration Technologies for Primavera Solutions Introduction... 1 The Integration Challenge... 2 Integration Methods for Primavera Solutions... 2 Integration Application
An Oracle White Paper Using Oracle GoldenGate to Achieve Operational Reporting for Oracle Applications Executive Overview... 1 Introduction: Right Time For Reporting... 2 Common Solutions for Reporting...
An Oracle White Paper February 2014 Oracle Data Integrator Performance Guide Executive Overview... 2 INTRODUCTION... 3 UNDERSTANDING E-LT... 3 ORACLE DATA INTEGRATOR ARCHITECTURE AT RUN-TIME... 4 Sources,
An Oracle White Paper November 2009 Oracle Primavera P6 EPPM Integrations with Web Services and Events 1 INTRODUCTION Primavera Web Services is an integration technology that extends P6 functionality and
Oracle Identity Analytics Architecture An Oracle White Paper July 2010 Disclaimer The following is intended to outline our general product direction. It is intended for information purposes only, and may
Oracle Data Integrator 11g New Features & OBIEE Integration Presented by: Arun K. Chaturvedi Business Intelligence Consultant/Architect Agenda 01. Overview & The Architecture 02. New Features Productivity,
Get More from Microsoft SharePoint with Oracle Fusion Middleware An Oracle White Paper January 2008 NOTE The following is intended to outline our general product direction. It is intended for information
An Oracle White Paper November 2010 Leveraging Massively Parallel Processing in an Oracle Environment for Big Data Analytics 1 Introduction New applications such as web searches, recommendation engines,
Oracle Data Integrator: Administration and Development What you will learn: In this course you will get an overview of the Active Integration Platform Architecture, and a complete-walk through of the steps
An Oracle White Paper October 2013 Oracle Data Integrator 12c (ODI12c) - Powering Big Data and Real-Time Business Analytics Introduction: The value of analytics is so widely recognized today that all mid
Performance and Scalability Benchmark: Siebel CRM Release 8.0 Industry Applications on HP BL460c/BL680c Servers running Microsoft Windows Server 2008 Enterprise Edition and SQL Server 2008 (x64) An Oracle
CUSTOMER MASTER DATA MANAGEMENT PROCESS INTEGRATION PACK KEY BUSINESS BENEFITS Faster MDM Implementation Pre built MDM integration processes Pre built MDM Aware participating applications Pre built MDM
Oracle Role Manager An Oracle White Paper Updated June 2009 Oracle Role Manager Introduction... 3 Key Benefits... 3 Features... 5 Enterprise Role Lifecycle Management... 5 Organization and Relationship
Monitoring and Diagnosing Production Applications Using Oracle Application Diagnostics for Java An Oracle White Paper December 2007 Monitoring and Diagnosing Production Applications Using Oracle Application
First Published January 2010 Updated October 2013 Oracle Data Integrator and Oracle Warehouse Builder Statement of Direction Disclaimer This document in any form, software or printed matter, contains proprietary
The need for data integration tools exists in every company, small to large. Whether it is extracting data that exists in spreadsheets, packaged applications, databases, sensor networks or social media
An Oracle White Paper October 2013 Maximize the Benefits of Oracle SOA Suite 11g with Oracle Service Bus Maximize the Benefits of Oracle SOA Suite 11g with Oracle Service Bus Table of Contents Introduction...
Oracle Net Services for Oracle10g An Oracle White Paper May 2005 Oracle Net Services INTRODUCTION Oracle Database 10g is the first database designed for enterprise grid computing, the most flexible and
High-Volume Data Warehousing in Centerprise Product Datasheet Table of Contents Overview 3 Data Complexity 3 Data Quality 3 Speed and Scalability 3 Centerprise Data Warehouse Features 4 ETL in a Unified
An Oracle White Paper February 2013 Schneider National Implements Next - Generation IT Infrastructure Introduction Schneider National, Inc., a leading provider of truckload, logistics, and intermodal services,
Oracle Fusion Middleware Getting Started with Oracle Data Integrator 12c Virtual Machine Installation Guide December 2014 Oracle Fusion Middleware Getting Started with Oracle Data Integrator, 12c Copyright
Oracle Data Integrator Best Practices for a Data Warehouse Oracle Best Practices March 2008 Oracle Data Integrator Best Practices for a Data Warehouse PREFACE... 7 PURPOSE... 7 AUDIENCE... 7 ADDITIONAL
An Oracle White Paper June 2011 Oracle Application Development Framework Overview Introduction... 1 Oracle ADF Making Java EE Development Simpler... 2 THE ORACLE ADF ARCHITECTURE... 3 The Business Services
An Oracle White Paper November 2011 Upgrade Best Practices - Using the Oracle Upgrade Factory for Siebel Customer Relationship Management Executive Overview... 1 Introduction... 1 Standard Siebel CRM Upgrade
INFORMATION CONNECTED Cover Your Lending and Leasing Business from All Angles Oracle FLEXCUBE Lending and Leasing Define. Comply. Collect. Adapt. A Comprehensive Solution for All Your Lending Needs The
APPLICATION MANAGEMENT SUITE FOR SIEBEL APPLICATIONS USER EXPERIENCE MANAGEMENT SERVICE LEVEL OBJECTIVE REAL USER MONITORING SYNTHETIC USER MONITORING SERVICE TEST KEY PERFORMANCE INDICATOR PERFORMANCE
D83167 Oracle Data Integrator 12c: Integration and Administration Learn To: Use Oracle Data Integrator to perform transformation of data among various platforms. Design ODI Mappings, Procedures, and Packages
Introducing Oracle Data Integrator and Oracle GoldenGate Marco Ragogna EMEA Principal Sales Consultant Data integration Solutions IT Obstacles to Unifying Information What is it costing you to unify your
ORACLE RAPID PLANNING KEY FEATURES Fast event-driven material and capacity planning Fast incremental simulation of plan and source data edits Embedded analytics for scenario comparison, order comparison,
Using Oracle Data Integrator with Essbase, Planning and the Rest of the Oracle EPM Products Edward Roske firstname.lastname@example.org BLOG: LookSmarter.blogspot.com WEBSITE: www.interrel.com TWITTER: ERoske 2 4
An Oracle White Paper May 2013 Oracle Audit Vault and Database Firewall 12.1 Sizing Best Practices Introduction... 1 Component Overview... 2 Sizing Hardware Requirements... 3 Audit Vault Server Sizing...
Highmark Unifies Identity Data With Oracle Virtual Directory An Oracle White Paper January 2009 Highmark Unifies Identity Data With Oracle Virtual Directory Executive Summary... 3 The Challenge: A Single
Oracle Identity Management: Integration with Windows An Oracle White Paper December. 2004 Oracle Identity Management: Integration with Windows Introduction... 3 Goals for Windows Integration... 4 Directory
The Benefits of a Unified Enterprise Content Management Platform An Oracle White Paper June 2010 The Benefits of a Unified Enterprise Content Management Platform A unified enterprise content management
An Oracle White Paper May 2012 Oracle Database Cloud Service Executive Overview The Oracle Database Cloud Service provides a unique combination of the simplicity and ease of use promised by Cloud computing
Oracle Identity Management for SAP in Heterogeneous IT Environments An Oracle White Paper January 2007 Oracle Identity Management for SAP in Heterogeneous IT Environments Executive Overview... 3 Introduction...
Reduce Trial Costs While Increasing Study Speed and Data Quality with Oracle Siebel CTMS Cloud Service Comprehensive Enterprise Trial Management in the Cloud Oracle Siebel CTMS Cloud Service lets you effectively
Page 1 of 8 ABOUT THIS COURSE This 5 day course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse with Microsoft SQL Server
Oracle University Appelez-nous: +33 (0) 1 57 60 20 81 Data Integration and ETL with Oracle Warehouse Builder NEW Durée: 5 Jours Description In this 5-day hands-on course, students explore the concepts,
Page 1 of 7 Overview This course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse with Microsoft SQL 2014, implement ETL
An Oracle White Paper June, 2013 Enterprise Manager 12c Cloud Control Executive Overview... 2 Introduction... 2 Business Application Performance Monitoring... 3 Business Application... 4 User Experience
Database Systems Journal vol. III, no. 1/2012 41 SAS Enterprise Data Integration Server - A Complete Solution Designed To Meet the Full Spectrum of Enterprise Data Integration Needs 1 Silvia BOLOHAN, 2
KATTA Durga Reddy TATA Consultancy Services. Oracle SOA Suite: The Evaluation from 10g to 11g Introduction Oracle SOA Suite is an essential middleware layer of Oracle Fusion Middleware. It provides a complete
SOA Governance: What s Required To Govern And Manage A Service-Oriented Architecture An Oracle White Paper October 2006 SOA Governance: What s Required to Govern and Manage a Service-Oriented Architecture.
July 21, 2011 Lee Anne Spencer Founder & CEO Global View Analytics Cheryl McCormick Chief Architect Global View Analytics Agenda Introduction Oracle Data Integrator ODI Components Best Practices Implementation
Oracle Team Productivity Center Overview An Oracle White Paper September 2011 Oracle Team Productivity Center Overview Oracle Team Productivity Center Overview Introduction... 1 Installation... 2 Architecture...
ORACLE REAL-TIME DECISIONS KEY BUSINESS BENEFITS Improve business responsiveness. Optimize customer experiences with cross-channel real-time decisions at the point of interaction. Maximize the value of
Oracle Data Integration: CON7920 Making the Move to Oracle Data Integrator Julien Testut Principal Product Manager, Oracle Data Integration Holger Friedrich Chief Technical Officer, SumIT Copyright 2014
Oracle Financial Services Data Integration Hub Foundation Pack Extension for Data Relationship Management Interface User Manual 126.96.36.199.0 Page i Table of Contents TABLE OF CONTENTS PREFACE... 4 Audience...
ORACLE BUSINESS INTELLIGENCE, ORACLE DATABASE, AND EXADATA INTEGRATION EXECUTIVE SUMMARY Oracle business intelligence solutions are complete, open, and integrated. Key components of Oracle business intelligence
Course 20463C: Implementing a Data Warehouse with Microsoft SQL Server Length : 5 Days Audience(s) : IT Professionals Level : 300 Technology : Microsoft SQL Server 2014 Delivery Method : Instructor-led
Oracle Business Rules Business Whitepaper An Oracle White Paper September 2005 NOTE: The following is intended to outline our general product direction. It is intended for information purposes only, and
Oracle University Contact Us: + 38516306373 Data Integration and ETL with Oracle Warehouse Builder: Part 1 Duration: 3 Days What you will learn This Data Integration and ETL with Oracle Warehouse Builder:
This course describes how to implement a data warehouse platform to support a BI solution. Students will learn how to create a data warehouse 2014, implement ETL with SQL Server Integration Services, and
Migrating Non-Oracle Databases and their Applications to Oracle Database 12c O R A C L E W H I T E P A P E R D E C E M B E R 2 0 1 4 1. Introduction Oracle provides products that reduce the time, risk,
For Midsize Organizations Oracle Product Brief Edition One Why your organization needs a Business Intelligence (BI) solution A large and growing supply of highly valuable data when does this become a burden
WHITE PAPER Integrating data in the Information System An Open Source approach Table of Contents Most IT Deployments Require Integration... 3 Scenario 1: Data Migration... 4 Scenario 2: e-business Application
An Oracle White Paper March 2015 A Comprehensive Solution for API Management Executive Summary... 3 What is API Management?... 4 Defining an API Management Strategy... 5 API Management Solutions from Oracle...
New 11g Features in Oracle Developer Tools for Visual Studio An Oracle White Paper January 2008 New 11g Features in Oracle Developer Tools for Visual Studio Introduction... 3 Integration with Visual Studio
Oracle BI Publisher Enterprise Cluster Deployment An Oracle White Paper August 2007 Oracle BI Publisher Enterprise INTRODUCTION This paper covers Oracle BI Publisher cluster and high availability deployment.
Disclaimer: This document is for informational purposes. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development,
Course 10777 : Implementing a Data Warehouse with Microsoft SQL Server 2012 Page 1 of 8 Implementing a Data Warehouse with Microsoft SQL Server 2012 Course 10777: 4 days; Instructor-Led Introduction Data
Course: Analyzing, Designing, and Implementing a Data Warehouse with Microsoft SQL Server 2014 Elements of this syllabus may be change to cater to the participants background & knowledge. This course describes
Oracle Business Intelligence Applications Overview An Oracle White Paper March 2007 Note: The following is intended to outline our general product direction. It is intended for information purposes only,
Course 10777A: Implementing a Data Warehouse with Microsoft SQL Server 2012 OVERVIEW About this Course Data warehousing is a solution organizations use to centralize business data for reporting and analysis.
An Oracle White Paper July 2011 The Benefits of a Unified Enterprise Content Management Platform Executive Overview Documents, emails, spreadsheets and presentations are created faster than we can count
ORACLE PRODUCT DATA HUB THE SOURCE OF CLEAN PRODUCT DATA FOR YOUR ENTERPRISE. KEY FEATURES Out-of-the-box support for Enterprise Product Record Proven, scalable industry data models Integrated best-in-class