Ongoing ETL with Exalytics

Transcription

1 Ongoing ETL with Exalytics Borkur Steingrimsson, Rittman Mead Consulting UKOUG, Birmingham 2012

2 About Rittman Mead Oracle BI and DW platinum partner - see us at Stand 30 in the Exhibition Hall World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI Approximately 50 consultants worldwide All expert in Oracle BI and DW Offices in US (Atlanta), Europe, Australia and India Skills in broad range of supporting Oracle tools: OBIEE OBIA ODIEE Essbase, Oracle OLAP GoldenGate Exadata Endeca

3 Rittman Mead and Oracle Exalytics First UK Oracle Partner (and only?) to take delivery of an Oracle Exalytics server (April 2012) Centre of excellence for Exalytics deployments, infrastructure, migrations and implementations Hosted in our London datacentre Available for Customer PoCs, training and evaluation

4 Agenda 1. Exalytics Bits and Bobs 2. Aggregate Awareness 3. In-Memory Awesomeness 4. How to get it working? 5. Now What? 6. Conclusion

5 Bits and Bobs What is Exalytics? Is it Plug n Play? What is TimesTen? Why talk about ETL?

6 Ongoing ETL with Exalytics Oracle Business Intelligence Appliance

7 What is Oracle Business Intelligence? Oracle BI provides a set of web-based tools for reporting and analyzing data Based around dashboards, built around graphs, tables, pivots etc Multiple server architecture that splits data access / presentation elements Connects to multiple data sources Oracle DB & Exadata SQL Server, DB/2, Teradata Essbase, Oracle OLAP, MSAS ADF View Objects etc

8 Oracle BI within the Oracle BI/DW Technology Stack BI provides the query and presentation element for BI/DW Catalog of reports, analyses, alerts, KPIs Additional calculation and data federation capabilities Web Server Aggregate navigation Security, scoping of data ERP/Apps Oracle BI DW Essbase SAN / NAS

9 How Oracle Business Intelligence 11g Works Data sources are mapped into an internal dimensional model called the Oracle BI Repository Users request data through analyses, which translate into logical SQL requests The Oracle BI Server converts these logical SQL requests into one or more physical SQL, MDX or other queries These are then processed by the underlying data sources, using their optimizations Data is then combined and returned to users in the form of dashboards

10 In a BI/DW System, Where Does the Time and Heavy Lifting Go? Ideal is to return data to users with consistent sub-second response times But large data volumes and unpredictable queries can make this tough When analyzing query response times, most of the times goes in The database returning the query results The storage tier returning data to the database The network connecting storage and the database Oracle BI Tuning, efficient queries and Exadata can help with this But queries can still sometimes take >2 mins or so And query times can be erratic - Some are instant - Some take 2-3 minutes - Depends on load on the database ERP/Apps DW SAN / NAS

11 BI System Performance Objectives Every query returns data immediately No performance penalty for drilling-into and investigating the data Response time is consistent Response-time stays constant even when you scale up to 100 s of users

12 Potential Solutions to Query Performance Issues Buy an Exadata box (or several Exadata boxes...) Add materialized views and use Query Rewrite Add more indexes, in-database OLAP to the DW Move data into an OLAP server such as Essbase But... Can be expensive Adds to DBA complexity Requires new skills (Essbase etc) Scaling issues for OLAP servers

13 Another Solution - In-Memory Databases Move some, or all, of the data being queried into RAM Sub-second response times Consistent response times A solution being applied elsewhere for similar problems Oracle BI In-Memory DB/Cache ERP/Apps DW SAN / NAS

14 Oracle Exalytics In-Memory Machine X2-4 BI Appliance that combines hardware + BI software Sun hardware, Four Intel Xeon E series processors 1TB of RAM, made up of 64 16GB DDR3 ECC registered DIMMs QDR (40Gb/second) InfiniBand connectivity to Exadata & Exalogic 10Gb and 1Gb Ethernet interfaces Optimized version of Oracle BI EE In-memory version of Oracle Essbase 3.6TB of disk, for internal use Integrated Lights-Out Mgmt (ILOM)

15 Oracle Exalytics In-Memory Machine X2-4 BI Appliance that combines hardware + BI software Sun hardware, Four Intel Xeon E series processors 1TB of RAM, made up of 64 16GB DDR3 ECC registered DIMMs QDR (40Gb/second) InfiniBand connectivity to Exadata & Exalogic 10Gb and 1Gb Ethernet interfaces Optimized version of Oracle BI EE In-memory version of Oracle Essbase 3.6TB of disk, for internal use Integrated Lights-Out Mgmt (ILOM) Sold as an appliance/engineered system Software only available in combination with hardware

16 Exalytics as the Exa-Machine for BI Runs the BI layer on a high-performance, multi-core, 1TB server In-memory cache used to accelerate the BI part of the stack If Exadata addresses 80% of the query performance, Exalytics addresses the remaining 20% Consistent response times for queries In-memory caching of aggregates 40 cores for high concurrency Re-engineered BI and OLAP software that assumes 40 cores and 1TB RAM Oracle BI In-Memory DB/Cache ERP/Apps DW

17 Ongoing ETL with Exalytics Aggregate Awareness

18 Aggregate Awareness What is Aggregate Awareness? Where do we configure this? What should we aggregate on? Aggregate Summary Advisor Usage Tracking Analysis User complaints / Gut-feeling By Design - No Surprises

19 What is Aggregate Awareness We can design our RPD repository so that the same information can be found in more than one physical table Detailed transactional sales data Daily aggregates per city per product Daily aggregates per region per product category Monthly aggregate per store per product category per customer segment The BI server is thus aware of these aggregates and can choose the best/quickest option to retrieve the data Using dimensional hierarchies Number of members on each level (?) Logical Table Priority Groups Number of dimensions involved in the logical request Data warehouse team responsible to ensure that data is correct and refreshed in each aggregate table As opposed to Materialized Views and staleness

20 In-Memory Awesomeness Memory is fast Memory is cheap In-Memory Capacity Increased In-Memory Cost Decreased In-Memory is Fast MB 2002 $0.2/MB HDD 5ms response GB 2012 $0.009/MB ns response

21 In-Memory Awesomeness Memory is fast Memory is cheap In-Memory Data Caching TimesTen has In-Memory Columnar Storage In-Memory Capacity Increased In-Memory Cost Decreased In-Memory is Fast In-Memory Row and Column Compression In-Memory Indexes In-Memory Predictability Optimizer MB 16GB $0.2/MB $0.009/MB HDD ms response 100ns response Non-Uniform Memory Access (NUMA) Support (Scale Up) Parallel Query Processing Analytic Functions In-Memory Unstructured Data High Performance Writes/Updates Data Persistence on Disk Transactional Integrity Multiversion Concurrency - Read Consistency

22 In-Memory Awesomeness Memory is fast Memory is cheap In-Memory Capacity Increased MB 16GB In-Memory Cost Decreased $0.2/MB $0.009/MB Still playing catch-up with Oracle RDBMS But it s getting there HDD 2012 In-Memory is Fast 5ms response 100ns response TimesTen has In-Memory Data Caching In-Memory Columnar Storage In-Memory Row and Column Compression In-Memory Indexes In-Memory Predictability Optimizer Non-Uniform Memory Access (NUMA) Support (Scale Up) Parallel Query Processing Analytic Functions In-Memory Unstructured Data High Performance Writes/Updates Data Persistence on Disk Transactional Integrity Multiversion Concurrency - Read Consistency

23 How To Get It Working Summary Advisor Wizard Analyzes Usage Tracking data to suggest best use of TimesTen aggregates - BIDomain.BIInstance.ServerConfiguration MBean to configure this Builds scripts to - Change the RPD to add Aggregate Navigation objects - Prefixes and iconizes new objects - Build physical tables to store new aggregates - Populate the new physical tables one-off Exalytics only - You are not allowed to use this wizard on your regular OBIEE installation!

24 What Does the Summary Advisor Create? Aggregate tables in the TimesTen database One per aggregate fact table One per shared dimension Physical tables in the Oracle BI Repository physical layer Logical Table Sources in the BMM layer All mapped in and active

25 What Does the Summary Advisor Create? Aggregate tables in the TimesTen database One per aggregate fact table One per shared dimension Physical tables in the Oracle BI Repository physical layer Logical Table Sources in the BMM layer All mapped in and active But there is a Wizard changing my RPD! Objects are marked as Is System Generated

26 How To Get It Working - Manually Decide which aggregates to create and where to store them In-Memory TimesTen database In-Memory Essbase cube Essbase or other OLAP Regular RDBMS datamarts Model this in your RPD Physical tables Logical Table Sources (LTS) Set dimensionality Set potential Fragmentation clauses Reroute requests for data, that doesn t fit in memory, to disk Complete control of all moving parts - Nothing really new!

27 Ongoing ETL with Exalytics OK, That Was Easy Enough

28 ... and then what? Our data warehouse keeps changing New sales figure loaded daily New product codes introduced weekly Newcomers/Leavers from HR This process is already handled and automated using some form of ETL Scripts OWB mappings ODI interfaces...

29 ... and then what? Our data warehouse keeps changing New sales figure loaded daily New product codes introduced weekly Newcomers/Leavers from HR This process is already handled and automated using some form of ETL Scripts OWB mappings ODI interfaces... Q: Natural extension of the ETL paradigm to include Exalytics TimesTen as a target?

30 Answer 1: Re-running the Summary Advisor / Aggregate Persistence Wizard Summary Advisor will advice Create new aggregates based on new usage patterns Modify existing aggregates Drop existing aggregates (to make space for new ones) Aggregate Persistence Wizard is the dude that builds scripts to Drops existing aggregates from the RPD Builds new aggregates Builds the scripts to drop and rebuild TimesTen aggregates

31 Answer 1: Re-running the Summary Advisor / Aggregate Persistence Wizard Summary Advisor will advice Create new aggregates based on new usage patterns Modify existing aggregates Drop existing aggregates (to make space for new ones) Aggregate Persistence Wizard is the dude that builds scripts to Drops existing aggregates from the RPD Builds new aggregates Builds the scripts to drop and rebuild TimesTen aggregates This is the Oracle Way Suggests Exalytics as an appliance that you just plug in Reacts with new trends in user behavior Easy to do - requires very little coding ERP/Apps Oracle BI In-Memory DB/Cache DW SAN / NAS

32 Re-running the Summary Advisor / Aggregate Persistence Wizard Summary Advisor will take into account new end-user behavior Other areas of interest Newly popular dashboards But there is a limit to how much we can fit in to the In-Memory TimesTen DB Won t this be a round-robin of kicking out today what we brought in yesterday? Conclusion RPD will have to be altered - online! Release procedures might strictly forbid that! Drop and Create - No incremental refresh of data Scripts will have to be executed - risk of failure Errors can be tricky to debug

33 Re-running the Summary Advisor / Aggregate Persistence Wizard Summary Advisor will take into account new end-user behavior Other areas of interest Newly popular dashboards c:\oraclebi\orahome But there is a limit to how much we can fit in to the In-Memory TimesTen DB Won t this be a round-robin of kicking out today what we brought in yesterday? Conclusion RPD will have to be altered - online! Release procedures might strictly forbid that! Drop and Create - No incremental refresh of data Scripts will have to be executed - risk of failure Errors can be tricky to debug \bifoundation\server\bin\nqcmd.exe -d exalytics -u biadmin -p welcome1 -s c:\summary_advisor_scripts\agg_wiz.sql

34 Re-running the Summary Advisor / Aggregate Persistence Wizard Summary Advisor will take into account new end-user behavior Other areas of interest Newly popular dashboards c:\oraclebi\orahome But there is a limit to how much we can fit in to the In-Memory TimesTen DB Won t this be a round-robin of kicking out today what we brought in yesterday? \bifoundation\server\bin\nqcmd.exe -d exalytics -u biadmin -p welcome1 -s c:\summary_advisor_scripts\agg_wiz.sql Conclusion RPD will have to be altered - online! Release procedures might strictly forbid that! Drop and Create - No incremental refresh of data Scripts will have to be executed - risk of failure Errors can be tricky to debug [10058][State: S1000] [NQODBC] [SQL_STATE: S1000] [nqserror: 10058] A general error has occurred [nqserror: 43113] Message returned from OBIS. [nqserror: 43119] Query Failed: [nqserror: 84008] Statement preparation failed

35 Re-running the Summary Advisor / Aggregate Persistence Wizard Summary Advisor will take into account new end-user behavior Other areas of interest Newly popular dashboards c:\oraclebi\orahome But there is a limit to how much we can fit in to the In-Memory TimesTen DB Won t this be a round-robin of kicking out today what we brought in yesterday? \bifoundation\server\bin\nqcmd.exe -d exalytics -u biadmin -p welcome1 -s c:\summary_advisor_scripts\agg_wiz.sql Conclusion RPD will have to be altered - online! Release procedures might strictly forbid that! Drop and Create - No incremental refresh of data Scripts will have to be executed - risk of failure Errors can be tricky to debug [10058][State: S1000] [NQODBC] [SQL_STATE: S1000] [nqserror: 10058] A general error has occurred [nqserror: 43113] Message returned from OBIS. [nqserror: 43119] Query Failed: [nqserror: 84008] Statement preparation failed Model Checker Utility will help minimizing the risk of errors occurring

36 Answer 2: Oracle Data Integrator Install the TimesTen JDBC Driver in the ODI Drivers directory Install the TimesTen Client utilities on the Agent host Create Physical and Logical Schema as per usual Create a TimesTen Model Create from scratch Reverse Engineer using JDBC driver - Tables - Views - Columns - Keys andforeign Keys Reverse Engineer using RKM

37 Answer 2: Oracle Data Integrator Install the TimesTen JDBC Driver in the ODI Drivers directory Install the TimesTen Client utilities on the Agent host Create Physical and Logical Schema as per usual Create a TimesTen Model Create from scratch Reverse Engineer using JDBC driver - Tables - Views - Columns - Keys andforeign Keys Reverse Engineer using RKM Set up Data Quality (optionally) to check data integrity against constraints on TimesTen tables

38 Knowledge Modules CKM SQL IKM SQL Control Append IKM TimesTen Incremental Update (MERGE) LKM SQL to TimesTen LKM File to TimesTen (ttbulkcp) RKM SQL (Jython)

39 Knowledge Modules CKM SQL IKM SQL Control Append IKM TimesTen Incremental Update (MERGE) Integrates data from staging area into a TimesTen target table using TimesTen JDBC driver in incremental update mode. For example, inexistent rows are inserted; already existing rows are updated. LKM SQL to TimesTen Loads data from an ANSI SQL-92 source to a TimesTen staging table using the TimesTen JDBC driver. LKM File to TimesTen (ttbulkcp) Loads data from a file to a TimesTen staging table using ttbulkcp utility. RKM SQL (Jython)

40 Less Impact Approach Minimizing the impact on existing DW Use Oracle GoldenGate as CDC to stage changes Use ODI to populate TimesTen Sources CDC Staging TimesTen GoldenGate ODI IKM

41 Answer 3: Command line tools and ODBC and other ETL dudes OK, so you are not on the ODI train yet? ttloadfromoracle is new in TimesTen v Perhaps worth writing your own ODI KM ttbulkload Loads/Unloads data from flat files Pretty much what the Aggregate Persistence wizard does Fast and Furious ttsql SQL*Plus of TT. Script your way to freedom! TRUNCATE TABLE INSERT INTO... SELECT FROM

42 TimesTen SQL CreateAndLoadfromOraQuery T1 Create table T1 AS SELECT... ttloadfromoracle(tblowner, tblname, Query, numthreads) call ttloadfromoracle ('TT','DIM_PRODUCT','select prod_id,prod_name,prod_cat from dwh.dim_products ); tttableschemafromoraqueryget(tblowner, tblname, Query) call tttableschemafromoraqueryget('hr','employees', 'SELECT first_name, FROM hr.employees');

43 TimesTen SQL CreateAndLoadfromOraQuery T1 Create table T1 AS SELECT... ttloadfromoracle(tblowner, tblname, Query, numthreads) call ttloadfromoracle ('TT','DIM_PRODUCT','select prod_id,prod_name,prod_cat from dwh.dim_products ); tttableschemafromoraqueryget(tblowner, tblname, Query) call tttableschemafromoraqueryget('hr','employees', 'SELECT first_name, FROM hr.employees'); < CREATE TABLE "HR"."EMPLOYEES" ( "FIRST_NAME" varchar2(20 byte), " " varchar2(25 byte) NOT NULL ) > 1 row found.

44 Ongoing ETL with Exalytics That was just as easy

45 Other Points Of Interest High Availability Clustering and Failover Global 24/7 Service Tiny Refresh Windows TT refresh probably added at the end of the existing ETL window Fast Refresh Depending on what needs to be done, JDBC might not be as fast as advertised Consider using native TimesTen client tool for full-loads Have seen some very slow refreshes - Time to Tune TimesTen Infiniband for Exadata connection - Bigger Pipes Double Buffering

46 HA and Double Buffering 1TB of RAM sounds like a lot But we need same amount of TEMP space as we need Storage space We need RAM for the OS and applications We might use some space for RAM disks to speed up the BI cache Depending on our options for data refresh policy Full refresh every time Partial refresh Trickle feed We might have to use two instances of TimesTen

47 HA and Double Buffering 1TB of RAM sounds like a lot But we need same amount of TEMP space as we need Storage space We need RAM for the OS and applications We might use some space for RAM disks to speed up the BI cache Depending on our options for data refresh policy Full refresh every time Partial refresh Trickle feed We might have to use two instances of TimesTen 1TB of RAM -> ±200GB active data storage :-( Get more boxes! Larry will love you more

48 HA and Double Buffering Create two sets of equally sized data targets Target1 and Target2 Set both data targets up as physical sources in OBIEE with their own connection pool Target1. ConnectionPool1 and Target2. ConnectionPool2 As we refresh Target1 we use Target2 as the Active schema After refresh of Target1 we swap them back and refresh Target2 Target1 and Target2 can not (automatically) be used for Load Balancing

49 HA and Double Buffering Create two sets of equally sized data targets Target1 and Target2 Set both data targets up as physical sources in OBIEE with their own connection pool Target1. ConnectionPool1 and Target2. ConnectionPool2 As we refresh Target1 we use Target2 as the Active schema After refresh of Target1 we swap them back and refresh Target2 Target1 and Target2 can not (automatically) be used for Load Balancing Script this or use the Aggregate Persistence Scripts to build the aggregates

50 delete aggregates "Target1".."Clone1"."SalesFactAggr"; set variable LOGLEVEL=7, INACTIVE_SCHEMAS='"Target2".."Clone2"': create aggregate "SalesFactAggr" for "FACT_SALES"("AMOUNT") at levels ("DB"."DIM_PRODUCTdim"."DIM_PRODUCTdim Detail") using connection pool "Target1"."ConnectionPool1" in "Target1".."Clone1"; delete aggregates "Target1".."Clone1"."SalesFactAggr"; set variable LOGLEVEL=7: create aggregate "SalesFactAggr" for "FACT_SALES"("AMOUNT") at levels ("DB"."DIM_PRODUCTdim"."DIM_PRODUCTdim Detail") using connection pool "Target2"."ConnectionPool2" in "Target2".."Clone2";

51 Conclusion

52 Conclusion OBIEE is Aggregate Aware

53 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM

54 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis

55 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates

56 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change

57 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built

58 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory

59 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory Good starting point

60 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory Good starting point More control might be needed

61 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory Good starting point More control might be needed Use your own ETL procedures

62 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory Good starting point More control might be needed Use your own ETL procedures Controlled process

63 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory Good starting point More control might be needed Use your own ETL procedures Controlled process Design for usage

64 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory Good starting point More control might be needed Use your own ETL procedures Controlled process Design for usage ETL load performance is not always fab

65 Conclusion OBIEE is Aggregate Aware Exalytics is fast and has lots of RAM TimesTen in-memory DBMS is used as an Aggregate store to speed up analysis Aggregates can be automatically built using measured Usage Tracking to suggest aggregates RPD change Physical objects built Physical tables populated in-memory Good starting point More control might be needed Use your own ETL procedures Controlled process Design for usage ETL load performance is not always fab HA and Clustering is always a challenge

66 More Information Thank you for attending this presentation, and more information can be found at Contact us at or Look out for our book, Oracle Business Intelligence Developers Guide out now! Follow-us on Twitter or Facebook (facebook.com/rittmanmead) Come see talk to us in the Exhibition hall - Stand 30 Win a free copy of the book!

67 More Rittman Mead Talks at UKOUG 2012 Borkur Steingrimsson : Exalytics and TimesTen: Ongoing Data Refresh and ETL (Mon at 12.45pm, Hall 7a) Mark Rittman : Deploying OBIEE 11g in the Enterprise (Monday at 2.25pm, Hall 9) Mark Rittman : Exalytics Roundtable: Lime or Lemon? (Monday at 3.20pm, Hall 1 Foyer L5 Jordan Meyer : Designing Eye-Catching and Effective Dashboards (Tuesday at 9.15am, Hall 10a Michael Rainey : OWB Making the Move to Oracle Data Integrator (Tuesday at 3.35pm, Hall 7b Robin Moffatt : Exalytics One Size Fits All? (Tuesday 5.20pm, Hall 7a)

68 Thanks For All The Fish

69 Ongoing ETL with Exalytics Borkur Steingrimsson, Rittman Mead Consulting UKOUG, Birmingham 2012