Oracle Data Integration Solutions

Transcription

1

3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. Copyright 2014 Oracle and/or its affiliates. All rights reserved. 3

4 Agenda Oracle Data Integration Solutions (DIS) Oracle Data Integrator (ODI) Oracle GoldenGate (OGG) Oracle Enterprise Data Quality (OEDQ) Oracle Enterprise Metadata Management (OEMM) Oracle Data Service Integrator (ODSI) Oracle Active Data Guard (OADG) Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

6 Oracle Data Integration Solutions 12c Deliver real-time data integration for Cloud and Big Data Cloud Apps Database Big Data Oracle Data Integrator Oracle GoldenGate Oracle Enterprise Data Quality Oracle Metadata Management Oracle Data Services Integrator Oracle Active Data Guard Real-time data replication; optimized for Database 12c and Oracle Exadata End-to-end integrated with simplified deployment Unified tooling for both structured data sources and Hadoop / NoSQL Flexible deployment on-premise or in the Cloud for heterogeneous systems Expanded support for 3rd party systems and Oracle Applications in real-time data integration and continuous availability solutions Copyright 2014 Oracle and/or its affiliates. All rights reserved.

7 Oracle Data Integrator E-LT: Bulk Data Processing and Fast Data Transformation Cloud Apps Database Big Data Oracle Data Integrator High Performance E-LT Declarative Design Extensible Knowledge Modules Data Services Structured & Unstructured Data Certified for leading technologies to deliver fast time to value High-performance, low cost of ownership E-LT architecture Lightweight deployment Flexible, easy to enrich functionality Copyright 2014 Oracle and/or its affiliates. All rights reserved.

8 Industry Leading Performance Extremely Fast Execution and Reduced Cost Conventional ETL Architecture Extract Load Transform E-LT provides a flexible architecture for optimized performance on any platform Benefits Leverages set-based transformations Next Generation Architecture E-LT Extract Load Improves performance for loading, no network hop Takes advantage of existing infrastructure: hardware and software Transform Transform Copyright 2014 Oracle and/or its affiliates. All rights reserved. 8

9 Declarative Design Improved Developer Productivity For Faster Time to Solution Easy and universal for designing simple to complex mappings Robust and Reusable: Create Mappings once for any physical design Quick to define and refactor maps, high degree of automation Extensible model for any data integration mechanism Benefits Significantly reduces the learning curve Shortens implementation times Simpler maintenance ODI Declarative Design 1 Define What You Want 2 Automatically Generate Dataflow Define How: Knowledge Modules Copyright 2014 Oracle and/or its affiliates. All rights reserved. 9

10 Knowledge Modules Simpler Physical Design and Shorter Implementation Time Pluggable Knowledge Modules Architecture Reverse Engineer Metadata Journalize (CDC) Load from Source to Staging Check Constraints Integrate, Transform Data Service Sample out-of-the-box Knowledge Modules: Oracle Sqoop Hive HBase SAP BW Oracle Spatial Oracle Datapump Siebel Oracle DBLink ebusines Suite JMS Oracle Merge External Tables SAP ERP Teradata IBM DB2 Netezza SCD Key Architecture Benefits: Faster development and simpler maintenance using templates Easy to extend and add new best practices Enforces predictability and reduces cost of ownership Copyright 2014 Oracle and/or its affiliates. All rights reserved. 10

11 Extended Connectivity Leverages Existing IT, Faster Implementation Big Data: Hive, HBase, Sqoop Best for Oracle: Merge, Spatial, Multi-Table Insert, Optimizer Hints and more Real-Time: CDC with GoldenGate Applications: E-Business Suite, Siebel, PeopleSoft, JD Edwards Enterprise One, JD Edwards World, SAP ERP and SAP BW Heterogeneous: Optimizations for all major RDBMS: IBM DB2, Microsoft SQL Server, Teradata, Netezza, etc. Cloud Big Data Applications Databases Legacy Copyright 2014 Oracle and/or its affiliates. All rights reserved.

12 How ODI Works: E-LT Architecture High Performance, Flexible, Lightweight Architecture Application & SOA ODI Agent Business Intelligence & Data Warehouse ODI Agent may be deployed in any part of the architecture JKM A B LKM C$_0 IKM D File C RKM LKM C$_1 I$ E$ (Errors) IKM CKM Extract-Load Transform Check-Load Key Architecture Benefits: 100% Java, Open APIs, fast E-LT Copyright 2014 Oracle and/or its affiliates. All rights reserved. 12

13 Integration Between GoldenGate and ODI 12c Enable GoldenGate journalizing on interfaces without restructuring. Manage GoldenGate Capture/Delivery instances in ODI Topology Bulk Load Scenario Capture Real Time Scenario Delivery Set common GoldenGate parameters graphically Deploy GoldenGate processes via ODI Support real-time and bulk from the same interface Copyright 2014 Oracle and/or its affiliates. All rights reserved.

14 Oracle GoldenGate Cloud Apps Database Big Data Heterogeneous Data Replication Oracle GoldenGate Log-based Change Data Capture and Delivery Active-Active Replication Heterogeneous Source and Targets Reliability and Transaction Integrity Big Data/Java/FlatFile Integration High-performance, lowimpact real-time data integration and replication Timely data for improved business insight Continuous availability for 24/7 operation Copyright 2014 Oracle and/or its affiliates. All rights reserved.

16 Oracle Enterprise Data Quality Data Quality for Customer and Product Data Cloud Apps Database Big Data Oracle Enterprise Data Quality Profile, Explore, Audit Parse, Standardize, Cleanse Match, Merge, Enrich Governance Case Management Improves data accuracy, usability and fitness for purpose Unified interface for ease-of-use, lower TCO Depth of capability in multiple data domains reduces project risk Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

17 Data Quality Issues A simple Customer Table Matching Records Non Standard formats Name Address City State Zip Phone Bob Williams 36 Jones Avenue Newton MA [email protected] Robert Williams 36 Jones Av. MA Burkes, Mike and Ilda 38 Jones av. Nweton MA [email protected] Jason Bourne, Bourne & Cie. 76 East 51 st Newton MA Multiple Names Typos Mis-fielded data Mixed business and contact names Missing Data Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 17

18 Profile, Cleanse and Govern Business Data Solution Profile business data to find the bad data and assess quality over time Cleanse, match and merge data before it gets loaded into the Data Mart or Data Warehouse Repair data during batch processing flows Benefits Stop the garbage-in-garbage-out cycle! Improve the trust of data within business marts and data warehouses Enable IT to deliver value to the lines of business during every data integration flow Improve business efficiency with match and de-duplicate redundant records Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 18

19 Profiling Understand the data first Interactive exploration of data, identifying distribution and outlying values with drilldowns Identify and quantify issues in data Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 19

20 Oracle Enterprise Data Quality Standardization Name: Dr Ellen Van Der Heijde Name: Mr RJ & Mrs FB MacDonald Name: Jalila Abdul-Alim (Do Not Call) Title: Dr First: Ellen Last: Van Der Heijde Gender: Female Title: Mr First: R Middle: J Last: MacDonald Gender: Male Title: Mrs First: F Middle: B Last: MacDonald Gender: Female First: Jalila Last: Abdul-Alim Gender: Female Note: Do Not Call Standardize, Transform and Parse Split names and name elements Identify individuals and businesses Derive additional attributes Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 20

21 Oracle Enterprise Data Quality - Matching Title: Mr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Phone: Address: 9405 Main St Fairfax Virginia First: Bob Last: Fulmar Gender: Male [email protected] Title: Dr First: R Last: Fulmer DoB: 01/01/ [email protected] Address: 9407 Main Street Fairfax VA Title: Dr First: Robert Last: Fulmar Gender: Male DoB: 12/05/ [email protected] Phone: Address: 9407 Main St Fairfax VA Match & Merge data from disparate sources Create best record based on survivorship rules Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 21

22 Oracle Enterprise Metadata Management Trust your data Cloud Apps Database Big Data Oracle Enterprise Metadata Management Report to Source Lineage Impact Analysis, Model Versioning Annotations and Tagging Supports Metadata Standards Big Data Enabled Exploring Metadata Sarbanes-Oxley Act (SOX) Compliance Stewardship Collaboration Lifecycle Change Management Data Standardization & Data Governance Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

23 Metadata Management Metadata Management involves managing data about other data, whereby this other data is generally referred to as content. - wikipedia Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 23

24 Value of Enterprise Metadata Management Solves a significant pain point for a wide variety of business consumers and technical staff Which reports use this customer data? What will happen if I change this table? ETL Developer Executive Application User App GG ETL CDC ETL BI Dashboards How was sales figure calculated? What reports use the mainframe data? Sys Admin ETL Can I trust the sources of this customer data? Data Steward Hadoop Data Lake Data Scientist BI Developer I want to design an experiment to measure the success of a signup page. What data do I have? Where did this data come from? Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 24

25 Impact Analysis Oracle Metadata Management Logical Architecture Actionable Information Data Discovery Analytics Data Lineage Business Glossary Semantic Linking ETL Data Quality Data Integration Data Streams Integration Data Enrichment Data Sources Metadata Repository File Based Cloud Big Data Relational Applications Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential 25

26 Detailed Feature List Exploring Metadata Text Search and Metadata Browser, Model/Diagram Visualizers, Birds-eye View, Metadata Reporting Data Flow and Lineage Analyzer Sarbanes-Oxley Act (SOX) Compliance Versioning and Model Compare URI Based Call back from BI reports or Business applications Stewardship Collaboration Feedback Comments and Review Board Metadata Tagging with Labels Multimedia attachments (documents, videos, presentation, code) URL link references for , blogging, social networking Lifecycle Change Management Metadata Harvesting from any DM, DW, ETL/DI, BI Tooling and Big Data/Hadoop Enterprise Architecture Management (Metadata Configuration Management) Impact of Change Data Standardization & Data Governance Supports Variety of Metadata Standards Annotations and Tagging Business Glossary with Semantic Lineage Oracle Integrated Oracle Database Oracle Data Integrator, Oracle GoldenGate Oracle Business Intelligence Enterprise Edition Oracle Business Intelligence Applications (ODI and older Informatica based OBIA) Copyright 2014 Oracle and/or its affiliates. All rights reserved.

27 Oracle Data Services Integrator Data Federation and Virtualization Cloud Apps Database Big Data Oracle Data Services Integrator Federated Data Services Data Virtualization Integrates with Oracle SOA Bi-directional Reads & Writes Integrates with Oracle Business Analytics Easily develop data services and ad-hoc queries without infrastructure changes Open and standards based SOA, Web Services, Java, Service Data Object Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

28 Oracle Service Oriented Architecture (SOA) Product Mapping Oracle Reference Architecture, SOA Infrastructure, Release 3.2, , February 2012 Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

29 Different Data Access Techniques Consolidation: Move data from distributed sources to central hub; consuming applications query hub database Data Virtualization / Federation: Query and update distributed sources in-place and provide combined result Hybrid Approach: Federation and Consolidation are combined based on best approach for individual use cases Consumer Consumer Consumer Consolidated Hub ODSI ODSI ODI/OGG Source A Source B Consolidated Hub Source A Source B Data Virtualization only ODI/OGG Consolidation only Source A Source B Source C Hybrid Consolidation & Data Virtualization Copyright 2014 Oracle and/or its affiliates. All rights reserved. 29

30 When to Use Which Integration Technology Data Virtualization Data cannot be replicated (security, ownership) Consolidation is not efficient for access pattern Real-time access is required and replication is not efficient Transactional updates Need to change model frequently Consolidation Performance is priority Data needs to be offloaded from original source Replication is used for redundancy Data Virtualization Combine & normalize data access services (Create, Read, Update, Delete) Service Bus / Business Process Orchestrate services to business processes Service Routing / Mediation Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

31 Data Virtualization Use Cases Data Services Layer Provide a clean data abstraction for SOA and application developers Shield consumers from changes in data topology Augmenting Business Intelligence Extend access to sources outside the data warehouse Rapid Prototyping of Data Integration Quick creation of an interface to integrated data without building a consolidated store Cloud Data Integration Combine onsite and offsite data resources in a unified interface Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

32 ODSI Core Strengths Native SOA Data Integration Native XML type system allows processing and passing of hierarchical & SOA data. Based on open standards (XML, WS, JDBC/SQL, XQuery) Tight integration in Oracle SOA: Fast-path with OSB, Governance with OER Normalization and Reuse through Data Services Multi-layer approach to normalizing and transforming source data Highly Scalable, Industrial-Strength Performance Linear scalability through WLS Clusters Performance through caching, streaming and query optimization Fine-Grained Security and Auditing Row and column-based data security Comprehensive Update Capabilities Automated CRUD operations on data services Distributed XA transactions through WLS Transaction Manager Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

33 Oracle Active Data Guard High Availability of the Database Cloud Apps Database Big Data Oracle Active Data Guard Snapshot Database Open for Read-Only Operations Using for GoldenGate Extract Continuous Recovery Automatic Block Repair Main Component of Maximum Availability Architecture (MAA) Role Transition between Source and Standby Database in Case of Failure Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

34 Use Cases: Data Guard and Active Data Guard Real-Time Data Protection and Availability for Oracle Database Data Guard & Active Data Guard Exact copy of primary Exact copy of primary Snapshot Standby Disaster Recovery Manual or Automatic Failover Query & Report Offload Open Read-Only Convert to Test Database (open read-write) Single Command Refresh SYNC or ASYNC New DB Version Exact copy of primary Standby First Patching, Database Rolling Maintenance Offload RMAN Backups Note: A single DR copy may be multipurposed for different combinations of the use cases described Exact copy of primary Exact copy of primary Redo Source for thin snaps/clones Extract offload, source for GoldenGate ALO mode Far Sync, GoldenGate Downstream Copyright 2014 Oracle and/or its affiliates. All rights reserved.

35 Data Guard Standby Database: Failover Target Read-write Workload Real-time Reporting Fast Incremental Backups Production Database Continuous redo shipping, validation & apply Physical Standby Database Copyright 2014 Oracle and/or its affiliates. All rights reserved.

36 Active Data Guard Standby Database: Offload Production + Failover Target Read-write Workload Real-time Reporting Fast Incremental Backups Continuous redo shipping, validation & apply Production Database Active Standby Database (physical standby open read-only) Copyright 2014 Oracle and/or its affiliates. All rights reserved.

37 Active Data Guard Best Data Protection, Auto-Repair, and High ROI Primary Data Center DR Data Center Offload read-only workload to open standby database Zero data loss at any distance Automatic Block Repair Enterprise Manager Cloud Control Offload Fast Incremental Backups Copyright 2014 Oracle and/or its affiliates. All rights reserved. 37

38 Data Guard and Active Data Guard Feature Sets Data Guard (included with EE) Data Protection with HA Zero or near-zero data loss protection Transparent all datatypes, workloads Continuous data validation Detect silent corruption Automatic database failover Dual-purpose DR as test system Simple migrations and upgrades Oracle Enterprise Manager integration Active Data Guard (Option) Advanced Protection with High ROI Zero data loss at any distance Automatic corruption repair Auto-replay of inflight transactions Offload transport compression* Offload read-only workload Offload read-mostly workload Offload fast incremental backups Automation for rolling upgrades Copyright 2014 Oracle and/or its affiliates. All rights reserved. * Also requires Advanced Compression Option

39 Oracle Replication Solutions for Business Continuity Active Data Guard: Disaster Recovery and Data Protection High-performance, simple, drop-in physical replication for HA and DR Zero data loss, unique data corruption protection, automatic failover, offload read-only workloads Disaster recovery for all data types, applications and workloads GoldenGate: Flexible HA, Maintenance and Migrations, Real-Time Data Integration Logical replication for advanced replication requirements, target open read-write Enable minimal/zero downtime upgrades and application migration using 2 nd database copy Heterogeneous platforms, bi-directional replication, transformations, subsetting, multiple topologies Copyright 2014 Oracle and/or its affiliates. All rights reserved. 39

40 Oracle Products for Data Movement Comparing Tradeoffs and Benefits Driven by Different Architectures Active Data Guard Not invasive on source Fastest read and write Oracle Database only No transformation; no partial sets Unidirectional GoldenGate Not Invasive on source Very fast read and write Heterogeneous or homogeneous Basic transformation and partial sets Uni- and Bi-Directional Data Integrator Optimized Set-based transformation Highest latency for real-time feeds Heterogeneous Supports highly complex transformation Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

41 Resources Oracle Data Integrator Oracle GoldenGate Oracle Enterprise Data Quality Oracle Enterprise Metadata Management Oracle Data Services Integrator Oracle Data Integration Oracle Data Integration ORCL DataIntegration blogs.oracle.com/dataint OracleGoldenGate egration Data Integration Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle OpenWorld

44