Oracle Data Integrator 12c (ODI12c) - Powering Big Data and Real-Time Business Analytics. An Oracle White Paper October 2013

Transcription

1 An Oracle White Paper October 2013 Oracle Data Integrator 12c (ODI12c) - Powering Big Data and Real-Time Business Analytics

2 Introduction: The value of analytics is so widely recognized today that all mid to large scale enterprises have deployed some level of Business Intelligence (BI) tools and technologies. In spite of this widespread implementation of analytics, the promise of operational effectiveness is not being fully realized. According to a recent survey conducted by independent analyst firm, BI Scorecard, adoption of Business Intelligence (BI) remains surprisingly low reaching only 25 percent of potential users. For many, the Business Intelligence (BI) tools available are difficult to use and slow to respond and the content they deliver is of little relevance. As a result, business users default to making decisions based on incomplete information or gut feeling. The same survey also noted that while 26 percent of respondents reported that their Business Intelligence deployments were successful, the vast majority of Business Intelligence (BI) programs are falling short of their potential to increase revenues, improve customer service, reduce costs, and use strategic insights to create innovative new products or services Is Data Integration Limiting Business Analytics? There are a number of factors why organizations have not realized their full potential in analytics. These are three key factors where Data Integration technologies can help play a key role in solving: Business Analytics systems are running on stale and incorrect data. Without real-time good quality data in the data warehouses, data loses its relevance. This negatively impacts operational efficiencies and jeopardizes the carefully crafted customer experiences. Business Analytics systems are not designed for Big and Fast Data. Future-proofing business analytics and data warehousing system is becoming a growing concern to address the increasing volume, velocity and variety of big data. Business Analytics systems are disconnected. When Business Intelligence (BI) systems are fragmented from various planning systems, transactional systems and packaged applications (i.e. ERP, CRM), facts and assumptions are misaligned. The pitfalls include cost overruns, inventory shortage and surplus, unnecessary risks, and delayed responses to threats and opportunities. Each of these obstacles can be overcome by utilizing key best practices in data integration as the core foundation to business analytics and data warehousing. These strategies include integrating data in realtime, effectively tapping the value by integrating big data, and finally, integrating data end-to-end with business analytics systems. In this paper we shall discuss these requirements and how the latest release of Oracle Data Integrator 12c (ODI12c) along with supporting Oracle Technologies addresses this growing need of powering Business Analytics for organizations. 2

3 The Enterprise Data Warehouse Feeding Business Analytics Even in today s world of Big Data, the enterprise data warehouse plays a critical role in feeding Business Intelligence applications for businesses. Most organizations have a traditional Data warehouse that needs to be kept current and integrated with their reporting structures to make sure that relevant data is being surfaced to line of business operational reports and management dashboards. These are increasingly being integrated with new analytic data warehouses and Big Data clusters and real time data feeds to provide a comprehensive data store to the Analytics technologies. This population of the data store requires a Data Integration technology that Handles bulk loads of traditional and Big Data in and out of the sources and data warehouses Incorporates Change Data Capture technologies to enable Real Time data movement. Should be part of a Fast Data ecosystem to handle Big Data, and most importantly Should be tightly integrated with an Enterprise Business Analytics Solution. Analyzing Big and Fast Data Businesses today need to analyze huge amounts of data, including social media activity, web logs, warranty claims, call center activity, movement of assets with RFID tags, and sensor data. In most cases, organizations need to integrate unstructured data with data that resides in multiple data sources such as data warehouses, enterprise business applications, and online analytic processing (OLAP) cubes. This big data is having a tremendous impact on the solution space for business analytics, data warehousing and data integration. This has to do primarily with the fact that big data poses new questions for the best ways to process volumes and varieties of data at higher speeds and at faster velocity. One recent new standard that has emerged, Fast Data, specifically deals especially with this velocity component of big data and it s increasingly seen as an important element that ties together concepts in real-time and in-memory processing with big data and analytics. 3

4 Fig1: Big Data and Enterprise Data should be used in conjunction for best results. Loading and Transforming Big Data using Oracle Data Integrator Oracle Data Integrator is the flagship Oracle Data Integration product. Oracle Data Integrator is critical to big data projects in providing much needed connectivity to load and transform data from Hadoop sources and to support structured and relational sources. Oracle Data Integrator leverages unified tooling for both big data and enterprise data which translates into a faster learning curve as well as seamless usability so that the data scientist or data analyst can focus on integration versus usability. Flow based declarative designs in the new release of Oracle Data Integrator, ODI12c helps build complex expressions that are easy to maintain and support. Two representations of the same ELT mappings, the logical representation and the physical representation, provide customized working environments for business analysts and data scientists. Oracle Data Integrator Application Adapter for Hadoop simplifies data loading and movement between Hadoop and an Oracle Database through Oracle Data Integrator 12c (ODI12c) s easy to use rebuilt interfaces. 4

5 Fig 2: By providing efficient connectivity between Oracle Database and Hadoop, Oracle Big Data Connectors enables analysis of all data, both structured and unstructured, in enterprise data warehouses. The Need for Speed - Real-time Data Warehousing and Analytics: Real time data is absolutely required for efficient operational decision-making. As latency increases, data becomes less relevant and the ability to act on it decreases, robbing a company s opportunity to improve operational efficiencies and increase revenue. Real-time data integration ensures that the data used in decision-making is as current as possible and in some cases, virtually in sync with operational information. Traditional Business Intelligence and Data Warehousing solutions give users an excellent view of past events and entities by accessing historical data. These solutions typically monitor slowly moving trends, such as product defects or sales performance. Users gain insight into these activities by querying a data warehouse, which is updated periodically. Historical analysis is important, but it mainly targets strategic, long-term decision-making by analyzing trends, and it doesn t let people respond to operational events happening in the organization. As data latency increases, its relevance to operational events diminishes. BI solutions for day-to-day operations need to leverage the most current data to enable employees to take action right away. Data integration solutions that leverage heterogeneous Change Data Capture (CDC) technology can establish real-time links to production data sources in addition to performing bulk updates. These solutions work efficiently by moving only the changed data to the decision support system. 5

6 Real-time Business Analytics - Oracle Data Integrator with Oracle GoldenGate Real time reporting is best delivered when coupled with effective Change Data Capture (CDC) technology. Oracle s strategy for enabling real time is realized through Oracle GoldenGate. To enable real-time data acquisition, Oracle GoldenGate uses log-based, real-time CDC capabilities to provide continuous capture and delivery of the most recently changed data between OLTP systems and the data warehouse. The application offers transactional, real-time data capture, routing, transformations, and delivery, using the push approach. Real time changes to the data are moved to the data warehouse where it can drive enhanced, strategic, and operational BI capabilities. Oracle GoldenGate can perform basic, row-level transformations at the point of capture or at the time of delivery. For heavy transformation requirements Oracle GoldenGate delivers transactional data to a staging area in the data warehouse for in-database transformations with Oracle Data Integrator to support an E-LT architecture. Oracle Data Integrator utilizes the native database or execution platform to ensure that performance gains are not just limited to within the Oracle technology but also in the platform investments that forms a large part of the Data Integration landscape. Because of this E-LT architecture, query performances are faster and consequently transformations are faster and more efficient. Fig 3: Oracle GoldenGate and Oracle Data Integration combine to provide real time Data warehousing. 6

7 Integrated BI Applications Oracle BI Powered by Oracle Data Integrator Oracle s latest Business Intelligence Applications (OBIA) uses Oracle Data Integrator (ODI) for its data integration requirements. Built into the OBIA solution, Oracle Data Integrator provides load plans, pre built Knowledge Modules and user roles to populate and maintain fact and dimension tables for OBIA. Load plans, are executables that comprises of multiple dependent and parallel ETL steps that can be configured to load fact tables from sources. A load plan generator plug is provided that helps in creating load plans. For more information on understanding how ODI works with OBIA click here. Previously, for OBIA versions before , this ETL part was performed via 3 rd party tools. With the integration of ODI into the solution monitoring and user management is seamless with the ODI console acting as a monitoring and restarting during failovers and or job statuses. Dedicated BIApps Knowledge Modules can be used for updating Facts, Slowly Changing Dimensions, targets and periods. The use of Knowledge Modules eases development cycles in integrating data into source and target table definitions and allows your whole Business Intelligence solutions to be maintained as a single unit, delivering a single point of support, following a logical product lifecycle and lowering TCO. Oracle Data Integrator enables organizations using BI Applications to increase IT efficiency and reduce costs with a comprehensive data integration platform that covers all data integration requirements including big data, application integration, as well as BI / data warehousing. Fig 4: Complete integrated Data Integration components for multiple use scenario 7

8 Summary Only Oracle provides a complete and best-of-breed platform for integrated business analytics solutions to handle increasing volume, velocity & variety of big data. The new releases of Oracle Data Integrator 12c (ODI12c) and Oracle GoldenGate 12c has cemented Oracle s commitment to generate increased value for customers looking to implement real time analytics use cases, better tap big data sources. The components of this platform are engineered to work together. In addition, each component is based on open standards and interfaces that allow customers to leverage their existing enterprise standard components in areas where investments may have already been made. Oracle is the only vendor today that can add even more increased value in analytics with unmatched performance, an eye to the future trends and lower TCO. To learn more about Oracle Data Integration: Oracle Data Integration To learn more about Oracle Business Analytics: Oracle BI 8

9 White Paper Title October 2013 Author: Madhu R Nair Contributing Authors: Dain C Hansen Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA U.S.A. Worldwide Inquiries: Phone: Fax: oracle.com Copyright 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0113