<Insert Picture Here> Oracle GoldenGate and Oracle Data Integrator

Transcription

1 <Insert Picture Here> Oracle GoldenGate and Oracle Data Integrator Milomir Vojvodic

2 Oracle Data Integration Solutions Oracle Applications Custom Applications MDM Applications Business Intelligence Activity Monitoring SOA Platforms Comprehensive Data Integration Solution SOA Abstraction Process Manager Service Bus Data Services Data Federation Oracle Data Integrator E-LT/ETL Oracle GoldenGate Real-Time Data Integration Oracle Data Quality Profiling Cleansing Parsing Storage Data Warehouse/ Data Mart OLTP System OLAP Cube Flat Files Web 2.0 Web and Event Services, SOA

3 DIS products vs. out of the box Oracle OOTB Streams Data Guard OWB DIS Oracle strategic products Oracle GoldenGate Active Data Guard Oracle Data Integrator Difference? - Heterogeneous - Scalability, performance, reliability - Best Productivity - Makes HA/DR investments work - Best MAA solution for 11g - Provides reporting benefits - Increased ROI - Broader support/heterogenity - Best productivity & scalability Maintenance Mode

4 Oracle Excellent Product Viability Weight -ing IBM Oracle Informatica iway SAP SAS Micro -soft Bulk data movement 55% Federated views 15% Message-oriented movement 10% Data replication and synchronization 20% Total Rating Weighted Rating Source: Gartner Critical Capabilities for Data Integration Tools: Common Data Delivery Styles, Dec 2010 (showing top 7 of 11 vendors in chart above)

5 Pre-Built Solutions SOA Suite ID Mgmt Business Intelligence Other Apps (+more) BAM & CEP ODI Embedded Pre-built Content BPEL & OSB ODI Integrated Identity Analytics ODI Embedded Pre-built Content BI Apps ODI Embedded Pre-built Content BIEE Suite ODI Integrated Agile PLM ODI Embedded Pre-built Content Retek ODI Embedded Pre-built Content Enterprise Performance Management and Financials HFM Planning Essbase MDM/DRM Spend Analyzer Governance & Risk Siebel CRM ODI Embedded ODI Embedded Pre-built Content ODI Integrated ODI Integrated ODI Embedded Pre-built Content ODI Embedded Pre-built Content ODI Integrated GoldenGate Integrated New Oracle Fusion ERP Applications Fusion Pillar Replication OGG Embedded New Fusion AIA ODI Embedded Fusion HCM ODI Embedded Fusion MDM ODI Embedded Fusion CRM ODI Embedded Fusion Financials ODI Embedded Fusion Migration ODI Embedded New Pre-built Content Pre-built Content Pre-built Content Pre-built Content Pre-built Content Pre-built Content Pre-built Content

6 <Insert Picture Here> Oracle Golden Gate Milomir Vojvodic

7 What is GG and how it works? Capture: Committed changes are captured (and can be filtered) as they occur by reading the transaction logs. Trail files: Stages and queues data for routing. Pump: Distribute data for routing to multiple targets. Route: Data is compressed, encrypted for routing to targets. Delivery: Applies data with transaction integrity, transforming the data as required. Capture Source Trail Pump Pump LAN / WAN / Internet (TCP/IP) Target Trail Delivery Source Database(s) Delivery Target Trail Bi-directional Source Trail Capture Target Database(s) 7

8 Expanded Heterogeneity Databases Oracle GoldenGate Capture: Oracle DB2 NEW for v 9.7 Microsoft SQL Server NEW for 2008 Sybase ASE Teradata Enscribe SQL/MP SQL/MX MySQL NEW JMS message queues NEW Oracle GoldenGate Delivery: All listed above, plus: NEW TimesTen, IBM i Series Netezza, Greenplum, and any ODBC compatible databases ETL products O/S and Platforms Lnux Sun Solaris Windows 2000, 2003, XP HP NonStop HP-UX HP TRU64 HP OpenVMS IBM AIX IBM z Series zlinux 8

9 If we can identify top 3 differentiators Performance Sub-second latency for data movement Low-impact capture, routing, transformation, and delivery of transactional data Flexible & Extendable Open, modular architecture Supports heterogeneous sources and targets Agile and adaptable to modern data integration architectures Reliable Maintains transactional integrity Resilient against interruptions and failures Fast Recovery after process interruptions.

10 Oracle GoldenGate Message Bus New Application/ OS/ Database/Hardware Zero Downtime Upgrade & Migration Query Offloading, Disaster Recovery Database Oracle GoldenGate Replica/ Standby Data Synchronization across the Enterprise Distributed Systems Real-time BI, Operational Reporting ODS/Data Warehouse Legacy Event Driven Architecture, Message Bus SOA

11 Popular use cases of GG App A1 1 Data Failure Protection 1 2 Zero DT Migration solution X db X app A app A 3 Real Time DW/BI 4 Offload DB Deployment 5 Active Active DB Deployment 6 Regional DBs BI 7 Data level solutions/apps integration DW regional part of

12 Popular use cases of GG 1 Data Failure Protection app A

13 Data Failure Protection Business Impacts Where it is happening Avoid revenue & employee productivity loss Prevent decrease in customer loyalty and brand equity Decrease the manpower needed for system maintenance and recovery Limit credits and penalties due to not meeting SLAs for availability HA/MAA Infrastructure for Cloud Databases Minimizes risks during data center consolidation or migration to better platforms government regulations In Telecommunications OSS/BSS systems without interrupting operations In Financial Services/Banking Ensure continuous operations for critical applications such as payment processing, fraud detection, ATM, core-banking, trading systems Ecommerce/Retail Continuous operations for customer portal and POS systems

14 Active Data Guard and Golden Gate Physical replication with Active Data Guard Best for Disaster Recovery for Oracle Applications Best for Disaster Recovery for Oracle Database 11g Logical, heterogeneous replication with GoldenGate Only solution for real-time replication from Teradata Best for regular maintenance in heterogeneous environment Active-Active bi-directional or multi-master replication with GoldenGate Best for custom and specialized applications

15 Popular use cases of GG App A1 1 1 Data Failure Protection 2 Zero DT Migration app A

16 Zero DT Migration Business Impacts Avoid revenue & employee productivity loss Prevent decrease in customer loyalty and brand equity Decrease the manpower needed for system maintenance and recovery Limit credits and penalties due to not meeting SLAs for availability SAP maintaince costs Oracle support costs Modernize hardware and move to the most cost effective one Where it is happening HA/MAA Infrastructure for Cloud Databases Minimizes risks during data center consolidation or migration to better platforms government regulations In Telecommunications OSS/BSS systems without interrupting operations In Financial Services/Banking Ensure continuous operations for critical applications such as payment processing, fraud detection, ATM, core-banking, trading systems Ecommerce/Retail Continuous operations for customer portal and POS systems Move to the clouds Customers that want to consolidate databases and data centers to reduce costs Oracle DB customers that need ZDT Migration from 8i or 9i SAP customers Siebel CRM and JDE customers

17 Oracle GoldenGate 11g for Oracle Applications Oracle GoldenGate for Application Upgrades & Migrations Siebel CRM (V6 V8 upgrading to V8.x) All platforms (Oracle, DB2(LUW & z/os) & MSSQL) Also available to support Siebel Remote and incremental application repository zero downtime releases. JDE (all versions) Low downtime (not zero) upgrade solution All platforms except iseries are supported. (iseries is on the roadmap) SAP (Oracle to Oracle Online Migrations aka Triple-O )

18 Popular use cases of GG App A1 1 1 Data Failure Protection 2 Zero DT Migration app A 3 Real Time DW/BI BI DW

19 Real Time DW/BI Business Impacts Agility to prevent costs/looses and achive revenue Performance of the production system (productivity) Decreasing Manpower costs for batch loading of DW Where it is happening Customers want faster Apps (less invasive reporting), esspecially Oracle Apps BI customers that are looking to Use realtime or near real-time data from heterogeneous sources to heterogeneous targets in a mixed platform environment. BI customers that require transaction integrity for accurate and granular analysis ETL users that want to reduce overhead on source systems via log-based change data capture Also other BI customers that need stability of the BI environment BI customers where it t it takes too much time to load/refresh the Data Warehouse

20 Popular use cases of GG App A1 1 1 Data Failure Protection 2 Zero DT Migration app A 3 Real Time DW/BI 4 Offload DB Deployment BI DW

21 Oracle GoldenGate 11g for Oracle Applications Oracle GoldenGate is Certified for Operational Reporting for: Oracle E-Business Suite v12.x and higher Oracle JD Edwards v9.x and higher Oracle PeopleSoft Enterprise v8.51 and higher Oracle Siebel CRM v8.x and higher

22 Popular use cases of GG App A1 1 1 Data Failure Protection 2 Zero DT Migration app A app A 3 Real Time DW/BI 4 Offload DB Deployment 5 Active Active DB Deployment BI DW

23 Active Active DB Deployments Business Impacts Increased availabilty and performance but at the same time protect from data failure Better use of existing server assets Telco Where it is happening

24 Popular use cases of GG App A1 1 1 Data Failure Protection 2 Zero DT Migration app A app A 3 Real Time DW/BI 4 Offload DB Deployment 5 Active Active DB Deployment 6 Regional DBs BI regional part of DW

25 Popular use cases of GG App A1 1 Data Failure Protection 1 2 Zero DT Migration solution X db X app A app A 3 Real Time DW/BI 4 Offload DB Deployment 5 Active Active DB Deployment 6 Regional DBs BI 7 Data level solutions/apps integration DW regional part of

26 <Insert Picture Here> Oracle Data Integrator Milomir Vojvodic

27 What is ODI and how it works? ODI Data Services Data Access Service Transformation Service Batch Data Service ODI Declarative Design ODI Agent Data Flow Conductor 1 2 Define What You Want Automatically Generate Dataflow ODI Knowledge Modules Reverse Engineer Metadata Reverse CDC Sources Journalize Read from CDC Source Journal ize Sample out-of-the-box Knowledge Modules Load From Sources to Staging Load Check Constraints before Load Staging Tables Check Integrate Transform and Move to Targets Integrate Error Tables Target Tables Service Expose Data and Transformati on Services W W S W S S Services Define How: Built-in Templates ODI E-LT Transform Transform Extract Load SAP/R3 Siebel Log Miner DB2 Journals SQL Server Triggers DB2 Exp/Imp Oracle DBLink JMS Queues Oracle SQL*Load er Check Sybase Check MS Excel Type II SCD TPump/ Multiload Oracle Merge Siebel EIM Schema Oracle Web Services DB2 Web Services Benefits 27 27

28 If we can identify top 3 differentiators Faster Simpler Saves Money (Lower TCO, Higher ROI) Fastest E-LT bulk/batch performance Real-Time integration (sub-sec trickle) with CDC, replication, and SOA infrastructure Faster project setup, design and delivery Simpler to setup, configure, manage, & monitor Map using declarative SQL interfaces Deploy with fewer hardware devices Extend with knowledge module & code templates Less HW & Energy Costs with E-LT Architecture Less Time Wasted on Unnecessary ETL Mappings, Scripting, and Complex Training Less Integration Overhead Integrating with Applications, SOA, and Management Software

29 Capital Cost Reduction Reduce or Eliminate Unneeded Hardware and Human Capital ODI-based Oracle BI Apps Number of Setup Steps 7 Number of Servers 1 Number of connections 3 INFA-based Oracle BI Apps Number of Setup Steps 10 Number of Servers 3 Number of connections 7

30 Popular use cases of GG App A1 1 Data Failure Protection 1 2 Zero DT Migration solution X db X app A app A 3 Real Time DW/BI 4 Offload DB Deployment 5 Active Active DB Deployment 6 Regional DBs BI 7 Data level solutions/apps integration DW regional part of

31 Which of them require data transformation and bulk data movement? App A1 1 solution X app A 2 Zero DT Migration 3 Real Time DW/BI db X BI DW 7 Data level solutions/apps integration

32 Popular use cases of ODI X, Y, A, B solution X solution Y app B app A db XYAB db X db Y db B 1 Loading DW Files Excel XML 2 Data level solutions/apps integration 3 Data migrations&conversions in modernization/consolidation projects BI DW

33 Not only DB E-LT and E-T-L any OLTP to EDW/Mart Any to OLTP DB any any App/API to EDW/Mart Any to File(s) Any to App/API any File/WS/JMS to EDW/Mart Any to OLAP

34 Popular use cases of ODI solution X solution Y app B app A db X db Y db B 1 Loading DW Files Excel XML BI DW

35 Loading DW Business Impacts Comparing to Manual Coding : Productivity (number of langauges to use, prebuilt code (no need to write SQL)) Maintaince (hard to change smt, every script contains special rules in several machines)and Peformance and availabilty and Costs (E-LT) Comparing to other ETL : Costs (E- LT) and Productivity (still manual coding and SQL tuning) Where it is happening Customers with recent M&A activities (looking to reduce capital costs) Customer has dedicated cluster of ETL servers running at/near capacity and is looking to add capacity Large teams of ETL development staff are fully engaged with maintenance activity and not able to take on new projects Siebel, PeopleSoft, JDEdwards applications customers since they are more likely to have heterogeneous environments and to value heterogeneous database support. Oracle BI Applications, Oracle EPM, Oracle BI EE Plus

36 Popular use cases of ODI solution X solution Y app B app A db X db Y db B 1 Loading DW Files Excel XML 2 Data level solutions/apps integration BI DW

37 Data Level Solutions/Apps Integration Business Impacts One architecture for all integration tasks No more performance issues No more maintenance complexity (easy change a transformation rule) get a project done in a couple of weeks than a couple of months Where it is happening Customers with heterogeneous sources or targets in a mixed platform environment and with advanced knowledge of database technologies and understanding of database power For the data like Customer, Product, Supplier, Financials to create custom Data Hubs and Master Data Mgmt Mergers and acquisitions SOA customers Customers want to standradize on Java

38 Data Hubs in Key Industries Financial Services Single view of high quality customer data drives accurate customer insight and improved marketing effectiveness Supports compliance and reporting KYC requirements Retail Harmonizes customer data from multiple channels to improve sales and marketing effectiveness Enhances opportunity and customer lifetime value identification Telco Improves customer insight for revenue optimization and targeted customer retention Effective compliance and risk mitigation for next generation services Energy & Utilities Utilities Expands understanding of network assets and customer delivery points Improves management of regulatory compliance and reporting requirements Healthcare Government Delivers a comprehensive view of patient for care and billing Manages patient, epidemiology, diagnosis and treatment data quality across systems and organizations Single view of citizen for better internal information sharing, service delivery, licensing, provision of child care, and fraud detection Reduce costs through system rationalisation 38

39 Understanding Performance Choices Less than 10MB XML File DB Depends on whether an intermediary XML format is useful for other processing (use ESB), or if joining File data to tabular RDB data is required (use ODI) XML ESB ESB ESB File ESB ESB depends DB ESB depends ODI Between 10-50MB XML File DB Depends on ho much cross-referencing among the data values and rows is required during transformation the more there is, the faster ODI will perform relative to ESB XML ESB depends ODI File depends ODI ODI DB ODI ODI ODI Greater than 50MB XML File DB If the source and target are both XML, and there is no cross-referencing of data among rows, then a streaming-type or parallel-engine-type approach might scale XML depends ODI ODI File ODI ODI ODI DB ODI ODI ODI *caveat always benchmark if you are unsure and require best possible results

40 Popular use cases of ODI X, Y, A, B solution X solution Y app B app A db XYAB db X db Y db B 1 Loading DW Files Excel XML 2 Data level solutions/apps integration 3 Data migrations&conversions in modernization/consolidation projects BI DW

41 Data migrations&conversions in modernization/consolidation projects Business Impacts Get double-duty from database investment by using it for transformations Reduced migration time and expense Where it is happening Virtualization Cloud Exadata (ODI for mixed workload use cases and handle both loading and querying workloads simultaneously.) Application Migrations require heavy data transformation mainframe application to open systems (major schema changes) new generation of DW and BI solutions

42 ODI Outperforms Informatica for Less Run ODI Directly on Exadata 7.5 TB /hr $0 ETL H/W E L 7tb/hr ODI T Complex Data Transformations Linear ETL Scalability Fully Leverages DBFS/Infiniband, Smart Storage, and Advanced Compression 4:1 advantage* $5m 3yr savings** 1 Exadata X2-8 Informatica 1.8 TB /hr $5m ETL H/W * TPC-H data sets with transformations ** Production hardware savings (not including Dev + Test environments, management costs or software savings) E T 1.8 tb/hr L HP Superdome 64 + XP24000 w/ Flash Copyright 2011, Oracle Corporation and/or its affiliates

43 <Insert Picture Here> DIS Packaging, References and Competitive Milomir Vojvodic

44 Data Integration & Availability Pillar Products $17,500 (Src & Trgt DB CPUs) $100,000 (Src & Trgt DB CPUs) $23,000 (Target DB CPUs) $70,000 (Target DB CPUs) $10,000 (per CPU) Oracle GoldenGate For Oracle (includes Active Data Guard) For Non-Oracle Database (Includes Teradata) Oracle GoldenGate for Mainframe Includes HP Non-stop and DB2 Mainframe ODI Enterprise Edition Oracle Data Integrator Oracle Warehouse Builder Enterprise ETL ODI Suite Oracle Data Integrator Enterprise Edition Oracle Data Integrator Management Pack WebLogic (App Server) Suite (restricted) Oracle Coherence (restricted) Oracle Service Bus (restricted) Oracle BPEL Process Manager (restricted) Active Data Guard Requires Oracle Database Enterprise Edition Use Case: HA / Real-time DI Options: Management Pack ($3,500 per CPU) Application Adapters ($20,000 per CPU) Veridata ($30,000 per mid-tier CPU) Use Cases: HA / Real-time DI Change Capture for Data Warehouses Use Case: Bulk Data Transformation Options: Application Adapters ($2300 per CPU) Use Cases: ODI-EE with Clustering & ODI-EE for SOA Use Cases Use Cases: Disaster recovery & data protection for Oracle DB

45 Oracle Data Integration Solutions Platform Specific (logical HA) Sybase Replication Server Quest Shareplex Microsoft SQL Replication Not suitable for heterogeneous, enterprise deployments High Availability & Replication Data Integration Block-Level Solutions (physical HA) EMC Symantec / Veritas Hitachi Data Systems IBM PPRC HP StorageWorks no transaction management, no transformation, homogenous ETL Solutions IBM Infosphere (Ascential) Informatica SAP / Business Objects Ab Initio Invasive to source, Not real time, Requires Mid-Tier, Expensive & Monolithic EAI Solutions Tibco Progress Software IBM Websphere MQ Lower throughput, more overhead, not direct to database 45

46 <Insert Picture Here> Oracle ETL strategy Milomir Vojvodic

47 Evolving Data Integration Roadmap Roadmap 2000 to 2007 Roadmap 2007 to OWB-EETL 2007 ODI OWB OWB (free DB features) OWB-EETL OWB (free DB features) Roadmap 2009 to 2010 Roadmap from Jan 2010 to Current KMs New ETL ODI KMs OWB (free DB features) OWB (free DB features)

48 OWB Support Policies Customer Downloadable OWB Support Policies: Release GA Date Premier Support Ends Extended Support Ends OWB 9.2 Jul 2003 Oct 2007 Not Available Oct 2010 OWB 10.1 Apr 2004 Jul 2007 Not Available Jul 2010 OWB 10.2 Jul 2005 Jul 2010 Jul 2013 Indefinite OWB 11.1 Aug 2007 Aug 2012 Aug 2015 Indefinite OWB 11.2 Sep 2009 Current Dates: Jan 2015 Current Dates: Jan 2018 Sustaining Support Ends Indefinite Future Dates: Same as DB12 Future Dates: Same as DB12 For more Information about Support Levels:

49 OWB License Migrations Existing Product License Migrating to ODI EE license Notes OWB ETL features (included in Database licenses) No migration credit applies. Pay full ODI EE price. OWB Enterprise ETL option by CPU Ratio 2:1 Using list-to-list migration. OWB Enterprise ETL option by NUP Ratio 100:1 Using list-to-list. If customers have less than 100 NUP, use a percentage credit. i.e. 25 NUP would = 25% credit toward purchase of 1 processor. Customer pays the remaining 75% of the list price, minus standard and approved discounts. OWB included in internet Developer Suite (9i) or OWB 2.x versions No migration credit applies. All 9i features became free with OWB Note that OWB SAP Connector migrations are covered in the esource Migration Listing OWB DQ and DP No migration credit applies. In some cases customers may buy more Licenses (if they already own some of the same type) For additional License Migration queries, contact [email protected] Additional discounts or price promotions may apply, contact [email protected]

50 Oracle ETL Product Bundles Features DB (free ETL) ODI-EE ODI Suite Basic features for Oracle data loading Standard features for enterprise ETL use cases Advanced features for enterprise information management Oracle DB Targets Y Y Y Non-Oracle Sources Oracle Gateways Y Y Slowly Changing Dimensions (type 1) Y Y Y OLAP Targets (Oracle + Essbase) Oracle OLAP only Y Y Built-in Scheduler Oracle DBMS only Y Y Non-Oracle DB Targets - Y Y Changed Data Capture (including OGG integration) - Y Y Web Services (API & Src/Trgt) & XML Targets - Y Y Data Lineage / Impact Analysis - Y Y Declarative Design Environment - Y Y Slowly Changing Dimensions (2 & 3) - Y Y Load Plans, Parallel Execution and Restartability - Y Y JEE Deployment, Clustering and High Availability - Y Y Java based SDK for full Programmatic Control - Y Y Unified Management and Administration with OEM - Y Y Pre-built Integration with OBI-EE, Hyperion, SOA/AIA, etc. - Y Y Human Workflow (BPEL PM) - - Y Process Orchestration & Service Bus - - Y Data Relationship Management / MDM - - Y WebLogic Suite (Coherence Data Grid, B2B, Biz Rules) - - Y

51 OWB Enterprise ETL Accessing sources and targets Native support for heterogeneous databases, including the definition of new platform types, JDBC connectivity, knowledge modules, and code templates Change Data Capture mappings in ETL Publishing and consuming web services, including integrating with process orchestration technologies such as Oracle BPEL Process Manager Importing Cobol copybooks using the built-in wizard Targeting XML files Designing and executing ETL jobs Right-time data warehousing utilizing queues and real-time mappings Reusing and embedding mapping code with pluggable mappings Specifying the load order of multiple targets in a single mapping Partitioning and parallelizing updates to tables by utilizing the data chunking feature Scheduling ETL jobs using the built-in Schedule interface <<-- note: use of other schedulers with OWB is supported and does not require the ODI-EE license Using advanced process flows features (activity templates, use of the following activities: Assign, Set Status, For Loop, While Loop, Route, Notification, EJB, Java Class, and Webservice) Designing mappings to handle complex types using the following operators: varray iterator, construct object, and expand object Rapidly moving large amounts of data with transportable modules Using spatial transformations and streams transformations

52 OWB Enterprise ETL Working with dimensional objects Creating slowly changing dimensions, types 2 and 3 Storing data in OLAP cube-organized materialized views Defining orphan management policies Integrating with other products Deploying metadata directly to Oracle Discoverer End User Layer Deriving and deploying metadata to Oracle Business Intelligence Enterprise Edition Administering Warehouse Builder and its metadata repository Managing environments, including promoting from development to test to production, using the multiple configurations functionality Interactively analyzing metadata lineage and impact and automatically propagating metadata change using the Metadata Dependency Manager Extending the metadata repository using the following: user-defined objects, user-defined associations, userdefined modules, creating icon sets and custom icons Hosting the OWB design repository in an Oracle Real Application Clusters (Oracle RAC) environment (executing in an Oracle RAC environment does not require this license)