ETL Implementation for Extreme Performance. Presented By: Mrs. Catherine Boeving Mr. Greg Wade

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "ETL Implementation for Extreme Performance. Presented By: Mrs. Catherine Boeving Mr. Greg Wade"

Transcription

1 1

2 ETL Implementation for Extreme Performance Presented By: Mrs. Catherine Boeving Mr. Greg Wade 2

3 Topics About Us Tips and tricks for high performance mapping design Pipeline techniques to improve throughput Stacked pipelines to achieve extreme throughput Ensuring data integrity in an extreme environment Q&A 3

4 About Us Who we are Catherine Boeving, Software Developer Greg Wade, Information Systems Architect What we do Build large scale active data warehouses with near real time data loads and high availability for Department of Defense (DoD) customers Where do we work Lockheed Martin Global Systems and Solutions; A leading federal services and information technology contractor 4

5 Our Environment EDW Teradata ETL SPARC Enterprise M GB RAM Oracle Solaris 10 OS Informatica PowerCenter and DataTransformation V9.1.0 HotFix 3 Staging Oracle Data Acquisition Input Excess ETL server capacity needed to achieve extreme throughput. 5

6 Performance vs. Throughput Performance The execution time for one run of a workflows/mappings Throughput The volume of data that can be ETL ed in a specified period of time Performance is needed to achieve throughput High performance workflows/mappings are not always enough to meet demanding service level agreements Demanding SLAs require both high performance and throughput 6

7 Tips and Tricks Lookups Requirements Reference Data Lookups Validate the source data Expand the source data Stage Data Lookups Previous source data Processing of partial transactions Improve integration Carefully implemented lookups can improve mapping performance 7

8 Tips and Tricks Lookups Approaches Cached Only one DB request Best used when referenced data does not change Better for small tables May be able to compensate for slow/overloaded DB Un-Cached Many DB requests Required when referenced data changes Better for large tables Most performance improvement with fast DB Selecting the correct lookup type is key to performance 8

9 Tips and Tricks Lookups Calculations Part Number Example 1000 part numbers and descriptions 20 character part numbers with 50 character descriptions Simple calculations provide some insight but testing is needed in your environment 9

10 Tips and Tricks Lookups Calculations DB Transfer Entire Table = # rows * (characters per row) = 1000 * ( ) = 70K Bytes A good estimate but ignores DB speed, network, etc. One Un-Cached Lookup = SQL request + SQL response = ( ) = 170 Bytes Break even point = 70K / 170 = about 411 lookups per mapping execution 10

11 Tips and Tricks Lookups Implementation Historical Data Load Cached large table and process large amounts of data one time 25 files/10k rows of data against 25M cached lookup Average 50 minute workflow execution times and 6 hours total load time Peer review or inspection checklists should include validating lookup type selection 11

12 Tips and Tricks Stored Procedures Reduces Round Trips to the Database Combine Several Lookups with Sequence Logic Simplifies Complex Database Insert Logic Potential Loss of Data Lineage Controlling network chatter between ETL and the DB is essential for high performance 12

13 Tips and Tricks Stored Procedures Implementation Eliminate redundant stored procedure calls 13

14 Tips and Tricks Stored Procedures Implementation Ensure Matching Port and Parameter Sizes Mismatched parameter sizes will send extra bytes to database Occurs when database and mapping development done in parallel Verify the Import of Stored Procedures 14

15 Tips and Tricks Mapping Design Goals Rapid Development Reusable components and patterns Understandability Onboard new staff with unique ETL approach Maintenance Source system updates Future performance tuning 15

16 Tips and Tricks Mapping Design Mapplet Execution Used to handle similar code from different sources Smaller risk in onetime changes 16

17 Tips and Tricks Mapping Design Worklet Execution Isolates performance tuning and minimizes regression testing Implements standards for new developers 17

18 Tips and Tricks Mapping Design Stage Table Implementation Improves critical path completion Simplifies complex data 18

19 Pipeline Processing Standard Approach No Pipeline 2 Files Processed in 4 Minutes 19

20 Pipeline Processing Pipeline Approach 2 Files Processed in 3 Minutes 20

21 Load Transform Extract Pipeline Processing Pre-Process Sort transaction types Split large files Added Complexity requires standards and review Transform in Multiple Steps DataTransformation (DT) Break sources in multiple logical parts Smart Loading Decouple DB loading from transformations Use of external loaders 21

22 Pipeline Processing Implementing the Pipeline Use Flat Files Between Pipeline Steps High demands on ETL server s file system Requires highly tuned cluster file systems Pipeline Steps Have Similar Run Times Simple Three Step Pipeline DT File, Workflow Output File, External DB Loader Flat File Movement Adds Complexity And Must Be Monitored 22

23 Pipeline Processing Implementing the Pipeline Batch File Processing 23

24 Pipeline Processing Stacked Pipelines Threaded 3x Files applied to the DB in non-deterministic order 6 Files Processed in 3 Minutes 24

25 Pipeline Processing Pipeline Threading Implementation Manipulate XML Replicate Parameter Files 25

26 Pipeline Processing Example Calculations F = files to process = 100 files T = time to ETL and load a file = 5 minutes P = number of pipeline steps = 3 steps S = number of stacked pipelines = 4 pipelines Assume All pipeline steps take the same amount of time Ignore any overhead for intermediate files Estimate with your workload to see the possibilities 26

27 Pipeline Processing Example Calculations Summary Standard Processing (No Pipeline) F * T = 100 files * 5 minutes = 500 minutes Pipeline (P + (F 1)) * (T/P) = (3 + (100-1)) * (5/3) = 170 minutes 294% speedup over standard processing Stacked Pipelines ((P + (F 1)) * (T/P)) / S = 170 / 4 = % speedup over single pipeline 1162% speedup over standard processing Theoretical speedup shown. Actual speedup depends on your environment 27

28 Pipeline Processing Pipeline Metrics 2 Steps Standard Processing 387 sec for one file batch Pipeline Processing Step #1 222 sec for one file batch Step #2 240 sec for one file batch 100 File Batches Calculation Standard = 100 * 387 = sec Pipeline = (2 + (100 1))* (240/2) = Speedup = 313 % 28

29 Pipeline Processing Threaded Workflows Metrics -- 4 Threads Non-Threaded Workflows Threaded Workflows TOTAL RUNTIME (SECONDS) TOTAL INPUT ROWS TOTAL WEIGHTED AVERAGE RUNTIME TOTAL ROWS/ TOTAL WEIGHTED AVERAGE RUNTIME PERCENT DIFFERENCE 1,836, ,148, , % 5,503, ,555, , % Full speedup not realized. Consider data volume when threading. We ran out of files to process! 29

30 Data Integrity Customer Satisfaction, Trust, and Growth Is the data accurate? How complete is the picture? Finding the Bottleneck? Building the System of Record 30

31 Load Transform Extract Data Integrity File Monitoring System exchange Reaching pre-processing phase Data Monitoring Check for data validity Track session execution times Output File Monitoring Output files load time Follow invalid output files 31

32 Data Integrity Event Tracking E T L E Pre-Process Event E Bulk Process Event E Output File Event E Exchange Event Database File Event E 32

33 Data Integrity Monitor the Data - Alert Maintain Timelines for Files and Data Define expectations for data Empower the System System operators resolution Track to data quality issues 33

34 Data Integrity Proven Metrics Checking the Box on the SLA Quantifiable numbers Build and Track Future Growth Handle errors and invalid data Review of metrics may require redesign. 34

35 Key Points Tips and Tricks Stored Procedures, Lookups, Mapping Design Pipelines Pipeline ETL Processing Stacked Pipelines and Threaded Workflows Data Integrity Events for ETL, Alerting, Proven Metrics 35

36 36

Informatica Online Training

Informatica Online Training WWW.ARANICONSULTING.COM Informatica Online Training Arani Consulting 2014 A R A N I C O N S U L T I N G, H Y D E R A B A D, I N D I A Informatica Online Training Highlights Introduction and Architecture

More information

Performance Tuning Guidelines for Relational Database Mappings

Performance Tuning Guidelines for Relational Database Mappings Performance Tuning Guidelines for Relational Database Mappings 1993-2016 Informatica LLC. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying,

More information

Data Integrator Performance Optimization Guide

Data Integrator Performance Optimization Guide Data Integrator Performance Optimization Guide Data Integrator 11.7.2 for Windows and UNIX Patents Trademarks Copyright Third-party contributors Business Objects owns the following

More information

INFORMATICA POWERCENTER 8.6 ETL DEVELOPER COURSE

INFORMATICA POWERCENTER 8.6 ETL DEVELOPER COURSE INFORMATICA POWERCENTER 8.6 ETL DEVELOPER COURSE Informatica PowerCenter 8.6 ETL Developer (IPED-001) Informatica PowerCenter is a single, unified enterprise data integration platform that allows companies

More information

SAP Data Services 4.X. An Enterprise Information management Solution

SAP Data Services 4.X. An Enterprise Information management Solution SAP Data Services 4.X An Enterprise Information management Solution Table of Contents I. SAP Data Services 4.X... 3 Highlights Training Objectives Audience Pre Requisites Keys to Success Certification

More information

Memory-Centric Database Acceleration

Memory-Centric Database Acceleration Memory-Centric Database Acceleration Achieving an Order of Magnitude Increase in Database Performance A FedCentric Technologies White Paper September 2007 Executive Summary Businesses are facing daunting

More information

IBM WebSphere DataStage Online training from Yes-M Systems

IBM WebSphere DataStage Online training from Yes-M Systems Yes-M Systems offers the unique opportunity to aspiring fresher s and experienced professionals to get real time experience in ETL Data warehouse tool IBM DataStage. Course Description With this training

More information

Maximize MicroStrategy Speed and Throughput with High Performance Tuning

Maximize MicroStrategy Speed and Throughput with High Performance Tuning Maximize MicroStrategy Speed and Throughput with High Performance Tuning Jochen Demuth, Director Partner Engineering Maximize MicroStrategy Speed and Throughput with High Performance Tuning Agenda 1. Introduction

More information

A Scalable Data Transformation Framework using the Hadoop Ecosystem

A Scalable Data Transformation Framework using the Hadoop Ecosystem A Scalable Data Transformation Framework using the Hadoop Ecosystem Raj Nair Director Data Platform Kiru Pakkirisamy CTO AGENDA About Penton and Serendio Inc Data Processing at Penton PoC Use Case Functional

More information

INFORMATICA POWERCENTER TRAINING

INFORMATICA POWERCENTER TRAINING INFORMATICA POWERCENTER 9.6.1 TRAINING POWERCENTER 9.6.1 DURATION 35hrs AVAILABLE BATCHES WEEKDAYS (7.30AM TO 8.30AM) & WEEKENDS (10AM TO 1PM) MODE OF TRAINING AVAILABLE ONLINE INSTRUCTOR LED CLASSROOM

More information

High-Volume Data Warehousing in Centerprise. Product Datasheet

High-Volume Data Warehousing in Centerprise. Product Datasheet High-Volume Data Warehousing in Centerprise Product Datasheet Table of Contents Overview 3 Data Complexity 3 Data Quality 3 Speed and Scalability 3 Centerprise Data Warehouse Features 4 ETL in a Unified

More information

SSIS Scaling and Performance

SSIS Scaling and Performance SSIS Scaling and Performance Erik Veerman Atlanta MDF member SQL Server MVP, Microsoft MCT Mentor, Solid Quality Learning Agenda Buffers Transformation Types, Execution Trees General Optimization Techniques

More information

The Evolution of ETL

The Evolution of ETL The Evolution of ETL -From Hand-coded ETL to Tool-based ETL By Madhu Zode Data Warehousing & Business Intelligence Practice Page 1 of 13 ABSTRACT To build a data warehouse various tools are used like modeling

More information

Testing Big data is one of the biggest

Testing Big data is one of the biggest Infosys Labs Briefings VOL 11 NO 1 2013 Big Data: Testing Approach to Overcome Quality Challenges By Mahesh Gudipati, Shanthi Rao, Naju D. Mohan and Naveen Kumar Gajja Validate data quality by employing

More information

PUBLIC Performance Optimization Guide

PUBLIC Performance Optimization Guide SAP Data Services Document Version: 4.2 Support Package 6 (14.2.6.0) 2015-11-20 PUBLIC Content 1 Welcome to SAP Data Services....6 1.1 Welcome.... 6 1.2 Documentation set for SAP Data Services....6 1.3

More information

Monitoring and Diagnosing Oracle RAC Performance with Oracle Enterprise Manager. Kai Yu, Orlando Gallegos Dell Oracle Solutions Engineering

Monitoring and Diagnosing Oracle RAC Performance with Oracle Enterprise Manager. Kai Yu, Orlando Gallegos Dell Oracle Solutions Engineering Monitoring and Diagnosing Oracle RAC Performance with Oracle Enterprise Manager Kai Yu, Orlando Gallegos Dell Oracle Solutions Engineering About Author Kai Yu Senior System Engineer, Dell Oracle Solutions

More information

Managing Third Party Databases and Building Your Data Warehouse

Managing Third Party Databases and Building Your Data Warehouse Managing Third Party Databases and Building Your Data Warehouse By Gary Smith Software Consultant Embarcadero Technologies Tech Note INTRODUCTION It s a recurring theme. Companies are continually faced

More information

PowerCenter Developer: Tips & Tricks for Mapping Designer

PowerCenter Developer: Tips & Tricks for Mapping Designer PowerCenter Developer: Tips & Tricks for Mapping Designer Lingaraju Ramasamy (Raju), Technical Architecture Manager Informatica Professional Services 2 Agenda Introduction Architecture Best Practices Mapping

More information

Monitoring and Diagnosing Oracle RAC Performance with Oracle Enterprise Manager

Monitoring and Diagnosing Oracle RAC Performance with Oracle Enterprise Manager Monitoring and Diagnosing Oracle RAC Performance with Oracle Enterprise Manager Kai Yu, Orlando Gallegos Dell Oracle Solutions Engineering Oracle OpenWorld 2010, Session S316263 3:00-4:00pm, Thursday 23-Sep-2010

More information

ETL Overview. Extract, Transform, Load (ETL) Refreshment Workflow. The ETL Process. General ETL issues. MS Integration Services

ETL Overview. Extract, Transform, Load (ETL) Refreshment Workflow. The ETL Process. General ETL issues. MS Integration Services ETL Overview Extract, Transform, Load (ETL) General ETL issues ETL/DW refreshment process Building dimensions Building fact tables Extract Transformations/cleansing Load MS Integration Services Original

More information

Real time information -Philips case

Real time information -Philips case Real time information -Philips case Leyla Akgez-Laakso Lead architect for Information Platforms 11 maart 2014 Enterprise information architecture All structured information that is relevant to Philips

More information

Oracle BI Application: Demonstrating the Functionality & Ease of use. Geoffrey Francis Naailah Gora

Oracle BI Application: Demonstrating the Functionality & Ease of use. Geoffrey Francis Naailah Gora Oracle BI Application: Demonstrating the Functionality & Ease of use Geoffrey Francis Naailah Gora Agenda Oracle BI & BI Apps Overview Demo: Procurement & Spend Analytics Creating a ad-hoc report Copyright

More information

Virtuoso and Database Scalability

Virtuoso and Database Scalability Virtuoso and Database Scalability By Orri Erling Table of Contents Abstract Metrics Results Transaction Throughput Initializing 40 warehouses Serial Read Test Conditions Analysis Working Set Effect of

More information

INFORMATICA POWERCENTER AND DATA QUALITY ON ORACLE EXADATA

INFORMATICA POWERCENTER AND DATA QUALITY ON ORACLE EXADATA INFORMATICA POWERCENTER AND DATA QUALITY ON ORACLE EXADATA 2 3 Challenges The quality and timeliness of business insights on high performance database platforms like Oracle Exadata Database Machine is

More information

SQL Server PDW. Artur Vieira Premier Field Engineer

SQL Server PDW. Artur Vieira Premier Field Engineer SQL Server PDW Artur Vieira Premier Field Engineer Agenda 1 Introduction to MPP and PDW 2 PDW Architecture and Components 3 Data Structures 4 PDW Tools Data Load / Data Output / Administrative Console

More information

Data Warehouse and Business Intelligence Testing: Challenges, Best Practices & the Solution

Data Warehouse and Business Intelligence Testing: Challenges, Best Practices & the Solution Warehouse and Business Intelligence : Challenges, Best Practices & the Solution Prepared by datagaps http://www.datagaps.com http://www.youtube.com/datagaps http://www.twitter.com/datagaps Contact contact@datagaps.com

More information

Performance Tuning Guidelines for PowerExchange for Microsoft Dynamics CRM

Performance Tuning Guidelines for PowerExchange for Microsoft Dynamics CRM Performance Tuning Guidelines for PowerExchange for Microsoft Dynamics CRM 1993-2016 Informatica LLC. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying,

More information

Oracle BI Applications 7.9: Develop a Data Warehouse

Oracle BI Applications 7.9: Develop a Data Warehouse Oracle University Contact Us: 0180 2000 526 / +49 89 14301200 Oracle BI Applications 7.9: Develop a Data Warehouse Duration: 5 Days What you will learn This Oracle BI Applications 7.9: Develop a Data Warehouse

More information

Customer Use Cases: Proactive Monitoring for PowerCenter Operations and Development Governance

Customer Use Cases: Proactive Monitoring for PowerCenter Operations and Development Governance Customer Use Cases: Proactive Monitoring for PowerCenter Operations and Development Governance Prasad Sunkara Assistant Director, Illinois State University Pankaj Mittal Manger, NBC Universal 1 Implementing

More information

Application of Predictive Analytics for Better Alignment of Business and IT

Application of Predictive Analytics for Better Alignment of Business and IT Application of Predictive Analytics for Better Alignment of Business and IT Boris Zibitsker, PhD bzibitsker@beznext.com July 25, 2014 Big Data Summit - Riga, Latvia About the Presenter Boris Zibitsker

More information

SOLUTION BRIEF. JUST THE FAQs: Moving Big Data with Bulk Load. www.datadirect.com

SOLUTION BRIEF. JUST THE FAQs: Moving Big Data with Bulk Load. www.datadirect.com SOLUTION BRIEF JUST THE FAQs: Moving Big Data with Bulk Load 2 INTRODUCTION As the data and information used by businesses grow exponentially, IT organizations face a daunting challenge moving what is

More information

Oracle Data Integrator 11g New Features & OBIEE Integration. Presented by: Arun K. Chaturvedi Business Intelligence Consultant/Architect

Oracle Data Integrator 11g New Features & OBIEE Integration. Presented by: Arun K. Chaturvedi Business Intelligence Consultant/Architect Oracle Data Integrator 11g New Features & OBIEE Integration Presented by: Arun K. Chaturvedi Business Intelligence Consultant/Architect Agenda 01. Overview & The Architecture 02. New Features Productivity,

More information

Running a Workflow on a PowerCenter Grid

Running a Workflow on a PowerCenter Grid Running a Workflow on a PowerCenter Grid 2010-2014 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise)

More information

LearnFromGuru Polish your knowledge

LearnFromGuru Polish your knowledge SQL SERVER 2008 R2 /2012 (TSQL/SSIS/ SSRS/ SSAS BI Developer TRAINING) Module: I T-SQL Programming and Database Design An Overview of SQL Server 2008 R2 / 2012 Available Features and Tools New Capabilities

More information

<Insert Picture Here> Oracle Premier Support Il Supporto di Oracle sulla Tecnologia e sulle Applicazioni

<Insert Picture Here> Oracle Premier Support Il Supporto di Oracle sulla Tecnologia e sulle Applicazioni Oracle Premier Support Il Supporto di Oracle sulla Tecnologia e sulle Applicazioni Gianfranco Dragone Premier Support Senior Sales Manager Oracle Corporation Scale $24.2B in TTM revenue

More information

Exploring Oracle BI Apps: How it Works and What I Get NZOUG. March 2013

Exploring Oracle BI Apps: How it Works and What I Get NZOUG. March 2013 Exploring Oracle BI Apps: How it Works and What I Get NZOUG March 2013 Copyright This document is the property of James & Monroe Pty Ltd. Distribution of this document is limited to authorised personnel.

More information

Oracle Database 11g Comparison Chart

Oracle Database 11g Comparison Chart Key Feature Summary Express 10g Standard One Standard Enterprise Maximum 1 CPU 2 Sockets 4 Sockets No Limit RAM 1GB OS Max OS Max OS Max Database Size 4GB No Limit No Limit No Limit Windows Linux Unix

More information

Agenda. SSIS - enterprise ready ETL

Agenda. SSIS - enterprise ready ETL SSIS - enterprise ready ETL By: Oz Levi BI Solution architect Matrix BI Agenda SSIS Best Practices What s New in SSIS 2012? High Data Quality Using SQL Server 2012 Data Quality Services SSIS advanced topics

More information

Oracle Warehouse Builder 10g

Oracle Warehouse Builder 10g Oracle Warehouse Builder 10g Architectural White paper February 2004 Table of contents INTRODUCTION... 3 OVERVIEW... 4 THE DESIGN COMPONENT... 4 THE RUNTIME COMPONENT... 5 THE DESIGN ARCHITECTURE... 6

More information

Monitor and Manage Your MicroStrategy BI Environment Using Enterprise Manager and Health Center

Monitor and Manage Your MicroStrategy BI Environment Using Enterprise Manager and Health Center Monitor and Manage Your MicroStrategy BI Environment Using Enterprise Manager and Health Center Presented by: Dennis Liao Sales Engineer Zach Rea Sales Engineer January 27 th, 2015 Session 4 This Session

More information

Is ETL Becoming Obsolete?

Is ETL Becoming Obsolete? Is ETL Becoming Obsolete? Why a Business-Rules-Driven E-LT Architecture is Better Sunopsis. All rights reserved. The information contained in this document does not constitute a contractual agreement with

More information

An Oracle White Paper February 2014. Oracle Data Integrator Performance Guide

An Oracle White Paper February 2014. Oracle Data Integrator Performance Guide An Oracle White Paper February 2014 Oracle Data Integrator Performance Guide Executive Overview... 2 INTRODUCTION... 3 UNDERSTANDING E-LT... 3 ORACLE DATA INTEGRATOR ARCHITECTURE AT RUN-TIME... 4 Sources,

More information

Data processing goes big

Data processing goes big Test report: Integration Big Data Edition Data processing goes big Dr. Götz Güttich Integration is a powerful set of tools to access, transform, move and synchronize data. With more than 450 connectors,

More information

SQL Server 2005 Features Comparison

SQL Server 2005 Features Comparison Page 1 of 10 Quick Links Home Worldwide Search Microsoft.com for: Go : Home Product Information How to Buy Editions Learning Downloads Support Partners Technologies Solutions Community Previous Versions

More information

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database

An Oracle White Paper June 2012. High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database An Oracle White Paper June 2012 High Performance Connectors for Load and Access of Data from Hadoop to Oracle Database Executive Overview... 1 Introduction... 1 Oracle Loader for Hadoop... 2 Oracle Direct

More information

Amadeus SAS Specialists Prove Fusion iomemory a Superior Analysis Accelerator

Amadeus SAS Specialists Prove Fusion iomemory a Superior Analysis Accelerator WHITE PAPER Amadeus SAS Specialists Prove Fusion iomemory a Superior Analysis Accelerator 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com SAS 9 Preferred Implementation Partner tests a single Fusion

More information

dbspeak DBs peak when we speak

dbspeak DBs peak when we speak Data Profiling: A Practitioner s approach using Dataflux [Data profiling] employs analytic methods for looking at data for the purpose of developing a thorough understanding of the content, structure,

More information

Extraction Transformation Loading ETL Get data out of sources and load into the DW

Extraction Transformation Loading ETL Get data out of sources and load into the DW Lection 5 ETL Definition Extraction Transformation Loading ETL Get data out of sources and load into the DW Data is extracted from OLTP database, transformed to match the DW schema and loaded into the

More information

THE DEVELOPER GUIDE TO BUILDING STREAMING DATA APPLICATIONS

THE DEVELOPER GUIDE TO BUILDING STREAMING DATA APPLICATIONS THE DEVELOPER GUIDE TO BUILDING STREAMING DATA APPLICATIONS WHITE PAPER Successfully writing Fast Data applications to manage data generated from mobile, smart devices and social interactions, and the

More information

Getting it Right: How to Find the Right BI Package for the Right Situation Norma Waugh. RMOUG Training Days February 15-17, 2011

Getting it Right: How to Find the Right BI Package for the Right Situation Norma Waugh. RMOUG Training Days February 15-17, 2011 Delivering Oracle Success Getting it Right: How to Find the Right BI Package for the Right Situation Norma Waugh RMOUG Training Days February 15-17, 2011 About DBAK Oracle solution provider Co-founded

More information

The Data Warehouse ETL Toolkit

The Data Warehouse ETL Toolkit 2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. The Data Warehouse ETL Toolkit Practical Techniques for Extracting,

More information

EII - ETL - EAI What, Why, and How!

EII - ETL - EAI What, Why, and How! IBM Software Group EII - ETL - EAI What, Why, and How! Tom Wu 巫 介 唐, wuct@tw.ibm.com Information Integrator Advocate Software Group IBM Taiwan 2005 IBM Corporation Agenda Data Integration Challenges and

More information

Optimizing the Performance of the Oracle BI Applications using Oracle Datawarehousing Features and Oracle DAC 10.1.3.4.1

Optimizing the Performance of the Oracle BI Applications using Oracle Datawarehousing Features and Oracle DAC 10.1.3.4.1 Optimizing the Performance of the Oracle BI Applications using Oracle Datawarehousing Features and Oracle DAC 10.1.3.4.1 Mark Rittman, Director, Rittman Mead Consulting for Collaborate 09, Florida, USA,

More information

High performance ETL Benchmark

High performance ETL Benchmark High performance ETL Benchmark Author: Dhananjay Patil Organization: Evaltech, Inc. Evaltech Research Group, Data Warehousing Practice. Date: 07/02/04 Email: erg@evaltech.com Abstract: The IBM server iseries

More information

White Paper February 2010. IBM InfoSphere DataStage Performance and Scalability Benchmark Whitepaper Data Warehousing Scenario

White Paper February 2010. IBM InfoSphere DataStage Performance and Scalability Benchmark Whitepaper Data Warehousing Scenario White Paper February 2010 IBM InfoSphere DataStage Performance and Scalability Benchmark Whitepaper Data Warehousing Scenario 2 Contents 5 Overview of InfoSphere DataStage 7 Benchmark Scenario Main Workload

More information

An Architectural Review Of Integrating MicroStrategy With SAP BW

An Architectural Review Of Integrating MicroStrategy With SAP BW An Architectural Review Of Integrating MicroStrategy With SAP BW Manish Jindal MicroStrategy Principal HCL Objectives To understand how MicroStrategy integrates with SAP BW Discuss various Design Options

More information

Exadata High Volume Testing - Findings from a Customer POC. Hans-Dieter Zapf Oracle Solution Center for SAP Competency

Exadata High Volume Testing - Findings from a Customer POC. Hans-Dieter Zapf Oracle Solution Center for SAP Competency Exadata High Volume Testing - Findings from a Customer POC Hans-Dieter Zapf Oracle Solution Center for SAP Competency Customer POC: High Volume Testing on Exadata Customer Case Decision

More information

Data Warehousing. Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de. Winter 2015/16. Jens Teubner Data Warehousing Winter 2015/16 1

Data Warehousing. Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de. Winter 2015/16. Jens Teubner Data Warehousing Winter 2015/16 1 Jens Teubner Data Warehousing Winter 2015/16 1 Data Warehousing Jens Teubner, TU Dortmund jens.teubner@cs.tu-dortmund.de Winter 2015/16 Jens Teubner Data Warehousing Winter 2015/16 13 Part II Overview

More information

THE DATA WAREHOUSE ETL TOOLKIT CDT803 Three Days

THE DATA WAREHOUSE ETL TOOLKIT CDT803 Three Days Three Days Prerequisites Students should have at least some experience with any relational database management system. Who Should Attend This course is targeted at technical staff, team leaders and project

More information

Relational Databases for the Business Analyst

Relational Databases for the Business Analyst Relational Databases for the Business Analyst Mark Kurtz Sr. Systems Consulting Quest Software, Inc. mark.kurtz@quest.com 2010 Quest Software, Inc. ALL RIGHTS RESERVED Agenda The RDBMS and its role in

More information

Applying Operational Profiles to Demonstrate Production Readiness Of an Oracle to SQL Server Database Port using Web Services.

Applying Operational Profiles to Demonstrate Production Readiness Of an Oracle to SQL Server Database Port using Web Services. Applying Operational Profiles to Demonstrate Production Readiness Of an Oracle to SQL Server Database Port using Web Services James Cusick, Imran Riaz, Hubert Huang, Allen Tse, Murugan Gnanavel {james.cusick;

More information

Tips and Tricks for Using Oracle TimesTen In-Memory Database in the Application Tier

Tips and Tricks for Using Oracle TimesTen In-Memory Database in the Application Tier Tips and Tricks for Using Oracle TimesTen In-Memory Database in the Application Tier Simon Law TimesTen Product Manager, Oracle Meet The Experts: Andy Yao TimesTen Product Manager, Oracle Gagan Singh Senior

More information

End to End Solution to Accelerate Data Warehouse Optimization. Franco Flore Alliance Sales Director - APJ

End to End Solution to Accelerate Data Warehouse Optimization. Franco Flore Alliance Sales Director - APJ End to End Solution to Accelerate Data Warehouse Optimization Franco Flore Alliance Sales Director - APJ Big Data Is Driving Key Business Initiatives Increase profitability, innovation, customer satisfaction,

More information

What s New with Informatica Data Services & PowerCenter Data Virtualization Edition

What s New with Informatica Data Services & PowerCenter Data Virtualization Edition 1 What s New with Informatica Data Services & PowerCenter Data Virtualization Edition Kevin Brady, Integration Team Lead Bonneville Power Wei Zheng, Product Management Informatica Ash Parikh, Product Marketing

More information

2009 Oracle Corporation 1

2009 Oracle Corporation 1 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material,

More information

OLTP Meets Bigdata, Challenges, Options, and Future Saibabu Devabhaktuni

OLTP Meets Bigdata, Challenges, Options, and Future Saibabu Devabhaktuni OLTP Meets Bigdata, Challenges, Options, and Future Saibabu Devabhaktuni Agenda Database trends for the past 10 years Era of Big Data and Cloud Challenges and Options Upcoming database trends Q&A Scope

More information

BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB

BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB Planet Size Data!? Gartner s 10 key IT trends for 2012 unstructured data will grow some 80% over the course of the next

More information

Trusted, Enterprise QlikViewreporting. data Integration and data Quality (It s all about data)

Trusted, Enterprise QlikViewreporting. data Integration and data Quality (It s all about data) Trusted, Enterprise QlikViewreporting with Informatica data Integration and data Quality (It s all about data) Arjan Hijstek senior sales consultant Informatica Nederland bv ahijstek@informatica.com 06-22.454.327

More information

Starbucks Enterprise Data Warehouse (EDW) Backup and Recovery Tuning. Greg Green Senior Database Administrator September 22, 2010

Starbucks Enterprise Data Warehouse (EDW) Backup and Recovery Tuning. Greg Green Senior Database Administrator September 22, 2010 Starbucks Enterprise Data Warehouse (EDW) Backup and Recovery Tuning Greg Green Senior Database Administrator September 22, 2010 1 Starbucks Enterprise Data Warehouse (EDW) Backup and Recovery Tuning Greg

More information

Informatica ETL Development:

Informatica ETL Development: Informatica ETL Development: Informatica Architectural Overview - Review informatica role in Data warehouse - Informatica system service Architecture - How Informatica services interact with server - Underlying

More information

Implementing and Maintaining Microsoft SQL Server 2008 Integration Services

Implementing and Maintaining Microsoft SQL Server 2008 Integration Services Course 6234A: Implementing and Maintaining Microsoft SQL Server 2008 Integration Services Length: 3 Days Language(s): English Audience(s): IT Professionals Level: 200 Technology: Microsoft SQL Server 2008

More information

GRIDS IN DATA WAREHOUSING

GRIDS IN DATA WAREHOUSING GRIDS IN DATA WAREHOUSING By Madhu Zode Oct 2008 Page 1 of 6 ABSTRACT The main characteristic of any data warehouse is its ability to hold huge volume of data while still offering the good query performance.

More information

PostgreSQL Business Intelligence & Performance Simon Riggs CTO, 2ndQuadrant PostgreSQL Major Contributor

PostgreSQL Business Intelligence & Performance Simon Riggs CTO, 2ndQuadrant PostgreSQL Major Contributor PostgreSQL Business Intelligence & Performance Simon Riggs CTO, 2ndQuadrant PostgreSQL Major Contributor The research leading to these results has received funding from the European Union's Seventh Framework

More information

MS SQL Performance (Tuning) Best Practices:

MS SQL Performance (Tuning) Best Practices: MS SQL Performance (Tuning) Best Practices: 1. Don t share the SQL server hardware with other services If other workloads are running on the same server where SQL Server is running, memory and other hardware

More information

An Overview of SAP BW Powered by HANA. Al Weedman

An Overview of SAP BW Powered by HANA. Al Weedman An Overview of SAP BW Powered by HANA Al Weedman About BICP SAP HANA, BOBJ, and BW Implementations The BICP is a focused SAP Business Intelligence consulting services organization focused specifically

More information

Performance Workload Design

Performance Workload Design Performance Workload Design The goal of this paper is to show the basic principles involved in designing a workload for performance and scalability testing. We will understand how to achieve these principles

More information

SQL Server Administrator Introduction - 3 Days Objectives

SQL Server Administrator Introduction - 3 Days Objectives SQL Server Administrator Introduction - 3 Days INTRODUCTION TO MICROSOFT SQL SERVER Exploring the components of SQL Server Identifying SQL Server administration tasks INSTALLING SQL SERVER Identifying

More information

Enterprise Information Integration (EII) A Technical Ally of EAI and ETL Author Bipin Chandra Joshi Integration Architect Infosys Technologies Ltd

Enterprise Information Integration (EII) A Technical Ally of EAI and ETL Author Bipin Chandra Joshi Integration Architect Infosys Technologies Ltd Enterprise Information Integration (EII) A Technical Ally of EAI and ETL Author Bipin Chandra Joshi Integration Architect Infosys Technologies Ltd Page 1 of 8 TU1UT TUENTERPRISE TU2UT TUREFERENCESUT TABLE

More information

Oracle9i Data Warehouse Review. Robert F. Edwards Dulcian, Inc.

Oracle9i Data Warehouse Review. Robert F. Edwards Dulcian, Inc. Oracle9i Data Warehouse Review Robert F. Edwards Dulcian, Inc. Agenda Oracle9i Server OLAP Server Analytical SQL Data Mining ETL Warehouse Builder 3i Oracle 9i Server Overview 9i Server = Data Warehouse

More information

<Insert Picture Here> Oracle Premier Support Get Ahead. Stay Ahead.

<Insert Picture Here> Oracle Premier Support Get Ahead. Stay Ahead. Oracle Premier Support Get Ahead. Stay Ahead. Emilio Salvadori Services Renewal Sales Senior Manager Oracle Support Get Ahead. Stay Ahead. Industry leadership in customer services

More information

International Journal of Advancements in Research & Technology, Volume 3, Issue 2, February-2014 10 ISSN 2278-7763

International Journal of Advancements in Research & Technology, Volume 3, Issue 2, February-2014 10 ISSN 2278-7763 International Journal of Advancements in Research & Technology, Volume 3, Issue 2, February-2014 10 A Discussion on Testing Hadoop Applications Sevuga Perumal Chidambaram ABSTRACT The purpose of analysing

More information

Large Scale High Performance OpenLDAP

Large Scale High Performance OpenLDAP Large Scale High Performance OpenLDAP A real production world experience Wolfgang Hummel Solution Architect October 10 th 2011 1 2010 Hewlett-Packard Development Company, L.P. The information contained

More information

PERFORMANCE TIPS FOR BATCH JOBS

PERFORMANCE TIPS FOR BATCH JOBS PERFORMANCE TIPS FOR BATCH JOBS Here is a list of effective ways to improve performance of batch jobs. This is probably the most common performance lapse I see. The point is to avoid looping through millions

More information

An Oracle White Paper October 2013. Oracle Data Integrator 12c New Features Overview

An Oracle White Paper October 2013. Oracle Data Integrator 12c New Features Overview An Oracle White Paper October 2013 Oracle Data Integrator 12c Disclaimer This document is for informational purposes. It is not a commitment to deliver any material, code, or functionality, and should

More information

Condusiv s V-locity Server Boosts Performance of SQL Server 2012 by 55%

Condusiv s V-locity Server Boosts Performance of SQL Server 2012 by 55% openbench Labs Executive Briefing: April 19, 2013 Condusiv s Server Boosts Performance of SQL Server 2012 by 55% Optimizing I/O for Increased Throughput and Reduced Latency on Physical Servers 01 Executive

More information

Big Data With Hadoop

Big Data With Hadoop With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials

More information

Oracle BI Applications (BI Apps) is a prebuilt business intelligence solution.

Oracle BI Applications (BI Apps) is a prebuilt business intelligence solution. 1 2 Oracle BI Applications (BI Apps) is a prebuilt business intelligence solution. BI Apps supports Oracle sources, such as Oracle E-Business Suite Applications, Oracle's Siebel Applications, Oracle's

More information

In Memory Accelerator for MongoDB

In Memory Accelerator for MongoDB In Memory Accelerator for MongoDB Yakov Zhdanov, Director R&D GridGain Systems GridGain: In Memory Computing Leader 5 years in production 100s of customers & users Starts every 10 secs worldwide Over 15,000,000

More information

InfoSphere CDC To DataStage Integration Options IBM Corporation

InfoSphere CDC To DataStage Integration Options IBM Corporation InfoSphere To DataStage Integration Options 00 IBM Corporation Business Challenges Driving Real-Time Data Integration Dynamic Warehousing & Business Intelligence and Reporting Yesterday s data inadequate

More information

Automated Data Ingestion. Bernhard Disselhoff Enterprise Sales Engineer

Automated Data Ingestion. Bernhard Disselhoff Enterprise Sales Engineer Automated Data Ingestion Bernhard Disselhoff Enterprise Sales Engineer Agenda Pentaho Overview Templated dynamic ETL workflows Pentaho Data Integration (PDI) Use Cases Pentaho Overview Overview What we

More information

PowerExchange 101. Basics of PowerExchange. Presented by Andy Bristow, Product Specialist May 2016

PowerExchange 101. Basics of PowerExchange. Presented by Andy Bristow, Product Specialist May 2016 PowerExchange 101 Basics of PowerExchange Presented by Andy Bristow, Product Specialist May 2016 Agenda Introduction Who am I and why are we here? What is PowerExchange? How can it help you? Let me show

More information

HP ProLiant Gen8 vs Gen9 Server Blades on Data Warehouse Workloads

HP ProLiant Gen8 vs Gen9 Server Blades on Data Warehouse Workloads HP ProLiant Gen8 vs Gen9 Server Blades on Data Warehouse Workloads Gen9 Servers give more performance per dollar for your investment. Executive Summary Information Technology (IT) organizations face increasing

More information

<Insert Picture Here> Oracle BI Standard Edition One The Right BI Foundation for the Emerging Enterprise

<Insert Picture Here> Oracle BI Standard Edition One The Right BI Foundation for the Emerging Enterprise Oracle BI Standard Edition One The Right BI Foundation for the Emerging Enterprise Business Intelligence is the #1 Priority the most important technology in 2007 is business intelligence

More information

Oracle Architecture, Concepts & Facilities

Oracle Architecture, Concepts & Facilities COURSE CODE: COURSE TITLE: CURRENCY: AUDIENCE: ORAACF Oracle Architecture, Concepts & Facilities 10g & 11g Database administrators, system administrators and developers PREREQUISITES: At least 1 year of

More information

Irish SQL Academy Level 300. Bob Duffy

Irish SQL Academy Level 300. Bob Duffy Irish SQL Academy 2008. Level 300 Bob Duffy DTS 2000 SSIS 2005 1.75 Developers *Figures are only approximations and should not be referenced or quoted Optimize and Stabilize the basics Minimize staging

More information

Report and Dashboard Template 9.5.1 User Guide

Report and Dashboard Template 9.5.1 User Guide Report and Dashboard Template 9.5.1 User Guide Introduction The Informatica Data Quality Reporting and Dashboard Template for Informatica Data Quality 9.5.1, is designed to provide you a framework to capture

More information

ENZO UNIFIED SOLVES THE CHALLENGES OF REAL-TIME DATA INTEGRATION

ENZO UNIFIED SOLVES THE CHALLENGES OF REAL-TIME DATA INTEGRATION ENZO UNIFIED SOLVES THE CHALLENGES OF REAL-TIME DATA INTEGRATION Enzo Unified Solves Real-Time Data Integration Challenges that Increase Business Agility and Reduce Operational Complexities CHALLENGES

More information

Performance Tuning using Upsert and SCD. Written By: Chris Price

Performance Tuning using Upsert and SCD. Written By: Chris Price Performance Tuning using Upsert and SCD Written By: Chris Price cprice@pragmaticworks.com Contents Upserts 3 Upserts with SSIS 3 Upsert with MERGE 6 Upsert with Task Factory Upsert Destination 7 Upsert

More information

SSIS Training: Introduction to SQL Server Integration Services Duration: 3 days

SSIS Training: Introduction to SQL Server Integration Services Duration: 3 days SSIS Training: Introduction to SQL Server Integration Services Duration: 3 days SSIS Training Prerequisites All SSIS training attendees should have prior experience working with SQL Server. Hands-on/Lecture

More information

Data Integration Checklist

Data Integration Checklist The need for data integration tools exists in every company, small to large. Whether it is extracting data that exists in spreadsheets, packaged applications, databases, sensor networks or social media

More information