Performance Tuning Oracle s BI Applications

Transcription

1 Performance Tuning Oracle s BI Applications Jeff McQuigg Senior BI Architect April 24, 2013 Start Here KPI Partners Inc. Contact Us

2 Agenda 2

3 Agenda 1Introducing The Performance Layer 2Building The Performance Layer 3Mapping Into Oracle BI 4Implementation Considerations 5Q&A 3

4 Introducing The Performance Layer 4

5 Introducing The Performance Layer Targeted At Organizations Who Have: Large Data Volumes Highly Customized Tables Questionable Design Extensions Aggressive Performance Targets Slow Hardware Only a Data Warehouse 5

6 Introducing The Performance Layer These performance concepts are applicable to any BI system BI Apps (OBIA) Stars Custom Stars in OBIA Custom Built Warehouses SQL Server 6

7 Introducing The Performance Layer Wide tables carry more data than you need Dashboards only need a few fields Smaller is faster 7

8 Introducing The Performance Layer Eliminate other and conflicting priorities Singular focus on performance Performance starts with clear design goals 8

9 Introducing The Performance Layer Great performance requires perfect design for how it is used A Dashboarding environment is an Application Use a top-down design approach to support that application Specialized design for specialized usage 9

10 Introducing The Performance Layer Pre-built logic Clean star models Reduced data weight Tables which match usage by Oracle BI Top-Down design yields 10

11 Introducing The Performance Layer >> Oracle s View On Data Warehouse Architecture 11

12 Introducing The Performance Layer Keep the BI Apps/DW model mostly as-is & add a Performance Layer 1 Build a new data model 2 Copy data from BI Apps/DW tables 3 Bring only what you need 4 Denormalize & pre-calculate *Optimize To Usage* BI Apps Data Fridge ETL Performance Mini-Fridge 12

13 Introducing The Performance Layer - Takeaways 1 The Performance Layer is an industry standard architecture 2 Design is driven only by report performance improvement 3 Travel light 4 No need to alter BI Apps or DW 13

14 Building The Performance Layer 14

15 Building The Performance Layer 1. Identify a priority area (select a Fact table) 2. Identify common use cases (reports w/ prompts & data security) 3. Analyze resulting physical SQL 4. Try to tune the BI Apps model first! (Indexes, etc) 15

16 Building The Performance Layer 5. Prototype a new data model to match those needs (SQL handcrafting of new tables) 6. Adjust SQL & benchmark (SQL handcrafting needed) 7. Map into Oracle BI & test (Unit & Regression) 8. Benchmark the Oracle BI report using prototyped tables (Reports have many SQLsn) 16

17 Building The Performance Layer 9. Build the tables using INFA & DAC - Complete Oracle BI RPD mapping 10. Formal Regression Test 11. Deploy 12. Enjoy praise from users 17

18 Building The Performance Layer Reduce I/O with extreme prejudice Tune the BI Apps model first! It may work for you with low effort Employ techniques to eliminate I/O wherever possible Partition Elimination, Compression, Indexes, Aggregates, Star Transformations Let the Performance Layer do the work, not the report query Follow the KISS principle: Use a simple and clean Star. No Snowflakes! Ensure OBI is mapped properly and uses correct tables with perfect SQL Favor a general approach as opposed to a case-by-case approach A rising tide lifts all boats 18

19 Building The Performance Layer There are 4 kinds of tables in the Performance Layer: 1. Skinny Dimension and Fact tables 2. New Dimension tables 3. Mini-Dimension tables 4. Fact Aggregate tables Built directly from the base BI Apps or DW tables BI Apps or DW Perf. Layer Goal: use these tables in as many reports as possible Guiding principles and performance influences: 1. Application use cases drive the layer s design 2. Use minimal data for the job at hand 3. Aggregate Fact data when needed 4. Denormalize dimensions to eliminate extra joins 5. Pre-Build calculations to eliminate extra joins 6. Pre-Split data sets based on logical usage 19

20 Building The Performance Layer Table Partitioning Query Indexing (4 Types) For all Fact tables of a reasonable size (e.g., > 5M rows) Usually partition on Month (Range or Interval) The Database can easily eliminate the majority of the table Allows for smaller, local indexes Before Beginning: Tune the OOTB Model 1 Single column, local bitmap indexes on all Fact table FKs (_WIDs) and filter fields (DELETE_FLG) 2 Single column bitmap indexes on all dimensional fields used in any sort of prompt or report filter 3 Special composite B-Tree indexes to assist Snowflaked areas 4 Composite B-Tree indexes on large dimensions for join backs (for list reports) 20

21 Building The Performance Layer Skinny Tables are highly selective versions of the BI Apps or DW tables Horizontal Aggregation E.g., 10 columns vs. 100 in the base table Both Dimensions and Facts Very easy to build and use Goal: Reduce Avg. Row Length to 1/5 th - 1/20 th original size Include only the columns you will need for top-down reporting analysis If you don t need Customer Address, don t include it Ignore Meta Data columns (e.g., INTEGRATION_ID, etc.) Row sets are identical (1:1) with the base tables For Dimensions use the same ROW_WIDs - can be used with existing fact tables easily 21

22 Building The Performance Layer Build using: 1. Create Table as Select (CTAS) 2. Insert /*+ APPEND */ 3. Materialized Views Compress the table Use Parallel hints & options Don t forget partitions Enhance the tables with calculation logic Database is very fast at these operations expect only a few minutes for 100M rows CREATE TABLE WC_ACCT_BUDGET_SF COMPRESS NOLOGGING PARALLEL (DEGREE 8) PARTITION BY RANGE(PERIOD_END_DT_WID) INTERVAL(NUMTOYMINTERVAL(1, 'MONTH')) (PARTITION Part_01 VALUES LESS THAN ( )) AS SELECT /*+ PARALLEL(F,8) */ F.PERIOD_END_DT_WID, F.X_PERIOD_END_DT_WID, F.COMPANY_ORG_WID, F.GL_ACCOUNT_WID, F.X_POSTED_TOTAL_AMT, case when GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES, FROM W_ACCT_BUDGET_F F, W_GL_ACCOUNT_D GL_D WHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID; 22

23 Building The Performance Layer Enhance _SF tables with logic Identify CASE WHEN statements which require other dimensions Potential great benefit if the join can be eliminated Don t over do it table will get less skinny with each column Identify any data set splitting from the RPD HR Workforce Events table has both Events and Snapshots records but they are always used separately in the RPD Usage drives design: Split them out! Huge benefit for Event counting metrics (~10% of table) case when GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES, FROM W_ACCT_BUDGET_F F, W_GL_ACCOUNT_D GL_D WHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID; Create table WC_WRKFC_EVT_EVENTS_SF WHERE SNAPSHOT_IND = 0 Create table WC_WRKFC_EVT_MONTH_SNP_SF WHERE SNAPSHOT_IND = 1 23

24 Building The Performance Layer Typical to get a 10X to 20X and even 50X I/O benefit in the _SF vs. the base _F in size Reduced AVG_ROW_LEN COMPRESSION Record Set Splitting All without any aggregation Skinny Dimensions also have benefits: Real Examples Sub Ledger (custom) Workforce Snap Workforce Events GL Balance Acct Budget I/O Benefit 24X 11X 56X 21X 32X 1. All I/O is a killer and slows down the entire system 2. De-normalize into a Star (eliminate snowflakes & outer joins) 3. Real World Ex. #1: 2 wide dims 2 skinny dims: 6X query improvement 4. Real World Ex. #2: One query going from 11s to 4s with one skinny dim 5. Real World Ex. #3: GL Account Dimension: 37X I/O benefit 24

25 Building The Performance Layer Pre-build major pieces of commonly used but complex logic into the Data Model Over-relying on the RPD or Reports for logic can harm performance Let the ETL for the Performance Layer do the work not the query Example #1: Large binning and bucketing CASE WHEN FACT.ORDER_AMT BETWEEN 0 and 100 THEN ELSE CASE WHEN FACT.ORDER_AMT BETWEEN 101 and 200 THEN END Build a new dimension table to hold these values WC_CUST_ORDER_QTY_BAND_D Example #2: Date format conversions dynamically building a new column with a string concatenation statement: substring(t66755."per_name_month", 1, 4), '') + '-' + isnull(right(t66755."per_name_month", 2), '') Build a new column in the W_DAY_D table & index it 25

26 Building The Performance Layer Simply a higher level or levels of a larger dimension A combination of several Kimball concepts Granularities will be mixed Make a new table from the large, base dimension Contains distinct combinations Use only commonly used fields Get cues from dashboard prompts, column selectors, report filters Create a new ROW_WID Compress and index as normal, Parallel if needed for creation Easy to build and map Create table WC_EMPLOYEE_MD COMPRESS as select ROWNUM AS ROW_WID, W_ETHNIC_GRP_DESC, WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE, W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME from ( select distinct W_ETHNIC_GRP_DESC, WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE, W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME from W_EMPLOYEE_D); This real world example created 5,400 records from a W_EMPLOYEE_D of 9+ Million rows. 26

27 Building The Performance Layer _MD tables are used in the Performance Layer in two places: 1. Link into Skinny Facts Use a separate FK in addition to the base _WID Fact table has both EMPLOYEE_WID and EMPLOYEE_MD_WID Thus the _SF can join to all of the following: New Mini Dimension (_MD) (~1% rows, some columns) New Skinny Dimension (_SD) (100% rows, some columns) Base BI Apps/DW Dimension (_D) (100% rows, 100% columns) _MD Conceptual Size Differences The OBI RPD can select which one is best for each query _SD _D Benefits of linking into the _SF 1. The same set of fact rows are selected no benefit 2. Reduced dimension I/O, CPU and buffer space 3. Faster join-back on list reports 4. Very fast prompts, especially when constrained 27

28 Building The Performance Layer 2. Use them for very high level Fact Aggregates Build a Fact aggregate at the Mini-Dimension level Allows greater field inclusion with excellent aggregation ratios Multiple fields are available - not just one A good Mini Dimension and Skinny Fact/Aggregate can serve a large % of dashboard queries MD s & Fact Aggregates offer extreme performance: 1. Real World Ex #1: From time-out after 10 minutes to 4 seconds 2. Real World Ex #2: GL Account MD: 131X I/O benefit 28

29 Building The Performance Layer Aggregates are used when summary reports exist Pre-aggregate the dataset to make it smaller & faster Sometimes they are the only solution Typically a minimum of a 10:1 ratio is used Use all database tools as with any fact table Partitioning, Indexing, Star Transformations, Compression For extreme needs, consider merging facts together Ex: Monthly Actuals and Budgets Be mindful of gaps in datasets and non-conformed dimensions Advanced implementations use partition management to build only changed data for faster load times 29

30 Building The Performance Layer- Takeaways 1 Building the underlying tables is relatively simple 2 But strong SQL & Tuning expertise is needed 3 Take cues from dashboard, report & RPD configuration 4 Any savings in I/O helps the overall system 5 Use all of the available database performance tools 30

31 Mapping Into Oracle BI 31

32 Mapping Into Oracle BI Link as much as possible to allow for the best performance across all scenarios Best Performance Layer Better BI Apps Base 32

33 Mapping Into Oracle BI The 3 Fact tables are mapped like any aggregate The Skinny Fact (_SF) will have fewer dimensions and fewer metrics mapped to it Along the Employee dimension however it is the same as the base _F OBI will prefer to use the _SF over the _F Uses regular aggregate navigation concepts Uses the _F when needed as a backup plan _A _SF _F 33

34 Mapping Into Oracle BI Raise the priority group on the base _D to have OBI prefer the _SD As both LTS grains are identical, OBI needs more info to make a choice _D _SD 34

35 Mapping Into Oracle BI Create a dummy hierarchy level and map the LTSs for the Mini Dimension and Fact Aggregate to it The grain of the Mini Dimension is arbitrary As long as OBI knows it is higher than the other LTSs it will be preferred (Priority groups not needed) _MD _A 35

36 Mapping Into Oracle BI - Takeaways Table Mapping The mapping of tables is straightforward Link Tables As Much As Possible Let Oracle BI make the best choice 36

37 Implementation Considerations 37

38 Implementation Considerations The whole prototyping process can be done on a simple star in roughly two weeks Allow for more time if you have: Large data volumes Difficult performance targets More complex models and logic Many disparate report patterns or lots of reports to consider More stars are needed (e.g., Actuals and Budgets together) Development effort depends on # new objects Typically only another two weeks needed (ETL & OBI RPD) A few more for regression test and deployment 38

39 Implementation Considerations Use Production data volumes for accurate analysis Use Production DDL, ETL code, OBI RPD and OBI Webcat Quiet, unused machine for accurate benchmarking Use hardware that is as similar to Prod as possible KPI uses a database benchmarking tool to compare environments 39

40 Implementation Considerations Additional ETL & RPD Development Use SQL scripts instead of Informatica mappings (less effort, faster execution) Additional Testing Regression test is easy Additional ETL Run Time may be critical Additional Database size - minor Customization Propagation / Impact Analysis True of any aggregate Complex logic will be more difficult E.g. #1: Financial Analytics uses snowflakes with multiple segment hierarchies E.g. #2: HR Workforce Event & Snapshot logic uses effective dates for future dated events 40

41 Q & A 41

42 Transform Data Into Insight Strategic Consulting Systems Implementation Training Staff built from Oracle/Siebel/Hyperion engineering teams On-site, off-shore and blended shore delivery models Exclusive pre-built solutions for Oracle BI & E-Business Suite Depot Repair Analytics Fixed Asset Analytics Manufacturing Analytics Student Info Analytics Subledger (SLA) Analytics and more Oracle BI Hyperion Endeca Exalytics Salesforce.com Analytics The Leader In Oracle BI & EPM 42

43 Contact Us Web: kpipartners.com/contact KPI World Headquarters Balentine Drive Suite #375 Newark, CA Phone: (510) North America Offices New York, NY Minneapolis, MN Chicago, IL San Diego, CA Boston, MA Greensboro, NC Global Offices Bangalore, India Hyderabad, India The Leader In Oracle BI & EPM 43

44 44