Data Aggregation in Today's Data Warehouse

Transcription

1 Data Aggregation in Today's Data Warehouse New England Business Objects User Group Yossi Matias CTO HyperRoll

2 Recap BI made easy 2 1/24/2006

3 Reports made easy 3 1/24/2006

4 and wide spread.. 4 1/24/2006

5 Business Objects XI Platform Capabilities High performance Scalability Reliability Service-oriented architecture But what about the underlying data warehouse? 5 1/24/2006

6 An example application 6 1/24/2006

7 BO Universe 7 1/24/2006

8 Aggregate queries 8 1/24/2006

9 Typical Data Warehouse A schema Methodology Lots of summary tables Table management challenges Numbers of tables Complex configurations Table refresh Redundant storage 9 1/24/2006

10 What s s wrong with this picture? Multiple Views of summary tables and complex universe 10 1/24/2006

11 Performance Fundamentals: Aggregations Pre-calculated summaries of data Intersections of levels from each dimension Tradeoff between processing and query times Time Query Query Time Time Processing Time 11 1/24/2006 Number of Aggregations

12 The Summary Table Dilemma At some point summary table maintenance becomes unbearable Unbearable Query Performance A few queries have acceptable Performance.but the majority of queries, especially ad-hoc requests, perform poorly and system adoption suffers ROLAP engines require a steady diet of summary tables to perform Maintenance Burden Simple 12 1/24/2006 # of Summary Tables

13 Typical Data Warehouse Environments Low NEED FOR REAL TIME INFORMATION High Budgeting & Planning $ $ Multidimensional Data Stores DSS Ad-hoc Query Summary Tables Operational CPM BAM Real-Time BI Dashboards Poor Query Performance & Poor User Concurrency Bursted Reports Cached Reports! Data Alerts Extra Hardware Memory, CPUs Data Warehouse ODS Data marts Longer Batch Window ETL ETL Layer Layer Applications EAI/EDI Databases Flat Files Mainframe 13 1/24/2006

14 On the limitation of RDBMS In fact, relational DBMS were never intended to provide the very powerful functions for data synthesis, analysis, and consolidation that is being defined as multi-dimensional data analysis. These types of functions were always intended to be provided by separate, end-user tools that were outside and complementary to the relational DBMS products. 14 1/24/2006 E.F. Codd, S.B. Codd and C.T. Salley Providing OLAP to User-Analysts: An IT Mandate

15 The Catch-22 of data aggregation in DW We want a Data Warehouse that performs data aggregations effectively The Data Warehouse should ideally consist of relational databases Relational databases are not set to support effectively data aggregation 15 1/24/2006

16 The HyperRoll approach Build an effective non-relational data aggregation server Have the data aggregation server provide aggregation services to a relational database As a result, have a HyperRoll enabled relational database that effectively supports aggregations 16 1/24/2006

17 HyperRoll for Relational Benefit Access Up to 100x faster queries, and end users continue to use familiar applications Storage HyperRoll Engine Gateway DBMS View DBMS FACT TABLE Benefit Summary table storage & maintenance reduced or eliminated Data is loaded into HR in order to build aggregates ETL Benefit Up to 90% reduction in batch window compared to existing aggregation strategies Loading 17 1/24/2006 DB2 CLI ODBC Oracle OCI ASCII

18 HyperRoll-enabled enabled Data Warehouse Star Schema ROLAP Queries (SQL) View Aggregates 10x 100x performance improvement HyperRoll Replace or Complement Summary Tables (but does NOT Build or stores Summary tables) Data Warehouse or Mart 18 1/24/2006

19 A DW implementation MV3 MV9 Fact1 MV1 MV4 Query Tools 400 Millions MV10 HyperRoll MV5 MV6 MV11 Fact2 MV2 MV7 36 Millions MV12 MV8 19 1/24/2006

20 Typical Data Warehouse A schema Methodology Lots of summary tables Table management challenges Numbers of tables Complex configurations Table refresh Redundant storage 20 1/24/2006

21 Data Warehouse with HyperRoll Same methodology Same schema Now only one summary table Represents all aggregations Simplifies management 21 1/24/2006

22 Data Warehouse with HyperRoll Same methodology Same schema Now only one summary table Represents all aggregations Simplifies management 22 1/24/2006

23 What s s wrong with this picture? Multiple Views of summary tables and complex universe 23 1/24/2006

24 Data Warehouse with HyperRoll 24 1/24/2006 One View of All Possible Tables

25 Query to the HyperRoll View HyperRoll View --- Simple Query Few Seconds - query response time!!! 25 1/24/2006

26 Significant Performance Enhancement Number of Seconds to Complete Query Business Objects + Oracle Less than 1 second 1M 5M 10M 15M 20M Millions of Records Business Objects + Oracle + HyperRoll 26 1/24/2006

27 Typical Data Warehouse Environments Low NEED FOR REAL TIME INFORMATION High Budgeting & Planning $ $ Multidimensional Data Stores DSS Ad-hoc Query Summary Tables Operational CPM BAM Real-Time BI Dashboards Poor Query Performance & Poor User Concurrency Bursted Reports Cached Reports! Data Alerts Extra Hardware Memory, CPUs Data Warehouse ODS Data marts Longer Batch Window ETL ETL Layer Layer Applications EAI/EDI Databases Flat Files Mainframe 27 1/24/2006

28 Typical Data Warehouse Environments Low NEED FOR REAL TIME INFORMATION High Budgeting & Planning $ $ Multidimensional Data Stores DSS Ad-hoc Query Summary Tables Operational BI CPM BAM Real-Time Dashboards Poor Query Performance & Poor User Concurrency Bursted Reports Cached Reports! Data Alerts Extra Hardware Memory, CPUs HyperRoll Engine Data Warehouse ODS Longer Batch Window ETL ETL Layer Layer Data marts HyperRoll Engine Applications EAI/EDI Databases Flat Files Mainframe 28 1/24/2006

29 The best of both world Relational Unlimited scope of data Variety of client tools High maintenance Complex table joins and aggregations slows down queries Complex analysis difficult RDBMS OLAP OLAP Fast Queries Complex Analysis Limited scope Long cube builds Limited client tools HyperRoll offers the best of both worlds Transparent integration with both Relational and Multidimensional databases Seamless to the existing client tools Fast build process (dramatically faster then OLAP) Fast queries without having to design, build and maintain multiple summary tables Broader scope of analysis (dimensions and data) Eliminates complex Joins and GroupBy 29 1/24/2006

30 The HyperRoll aggregation server What s the magic with the HyperRoll aggregation server Does it compute all possible aggregates? How come it can perform so much better than OLAP cubes 30 1/24/2006

31 Multidimensional Cube Problems: Sparsity Irregularity Legend 31 1/24/2006 Theoretical scope of data Leaf level data Aggregated Data

32 Typical OLAP Problems Data Explosion Number of Aggregations (4 levels 32 in 1/24/2006 each dimension) 0 Data Explosion Syndrome Number of Dimensions 65536

33 What is the HyperRoll? An intelligent Aggregation Server Software engine based on proprietary algorithms for data aggregation Pre-computes a small-footprint data store Enables quick computation of aggregate values Highly-efficient I/O The logical equivalent of OLAP for relational without the limitations Patented Architecture for standalone data aggregation Integrated into existing relational databases and Business Intelligence systems 33 1/24/2006

34 What about Hardware-based solutions Will better H/W make the aggregation problem go away? The good news: Better h/w platforms improve performance The bad news The problem will just get worse over time 34 1/24/2006

35 InfoGlut is Only Getting Worse Multiple Volume of Corporate Data 3 Linear Processing Capability (Moore s s Law) Mo. 18 Mo. Time Consequences Limited Analysis Flexibility Out of Date Information Longer Processing Times Soaring Costs 35 1/24/2006

36 Financial Institution Financial Services Company Application: Expense Management Primary Business Issue: Analyst Productivity Before Expense process required 4 reports to run sequentially Total time to complete task taking 4 hours Queries ran from 11 to 14 minutes Query Performance Increase: 37 to 90X Process now completed in minutes Queries run in 2 to 18 seconds Projected manpower saving: >$500K Oracle, Business Objects 36 1/24/2006

37 Customer Test Results Query Name Oracle Timing (MV) HR Timing Improvement Bill to 7 min 51 sec 14 sec 31 X Territory 7 min 15 sec 1 sec 427 X Region 11 min 4 sec 1 sec 422 X Sales Force 12 min 12 sec 1 sec 438 X All Sales Force 16 min 37 sec 1 sec 541 X 37 1/24/2006

38 38 1/24/2006 Customers

39 Setting up Queries to work with HyperRoll Step1 Analyze the Business Objects Universe, reports, queries and schema 39 1/24/2006 Step1 Analyze Reports A n a l y z e Analyze Semantic Layer t h e B u s i n e s s Select Measures Select Dimensions Select Hierarchies Obtain Design Validation Look for hidden requirements

40 Setting up Queries to work with HyperRoll Step2 Design the HyperRoll metadata structure using HyperRoll HDF Builder 40 1/24/2006

41 Setting up Queries to work with HyperRoll Step3 HyperRoll is loaded with source data: RDBMS Flat files HyperRoll Source data is read HyperRoll aggregation engine is loaded and calculated Hierarchies are developed 41 1/24/2006

42 Setting up Queries to work with HyperRoll Step4 Create the Database view and link it to HyperRoll HyperRoll Define an ODBC System DSN for HyperRoll Create a DBlink for the DSN Create View as Select * from HyperRoll@DBlink 42 1/24/2006

43 Setting up Business Objects to work with HyperRoll Step5 Modify the Business Objects Universe by adding the Database View that points to HyperRoll Add the View to the Universe Enable the Aggregate Aware Function 43 1/24/2006

44 Setting up Queries to work with HyperRoll Step6 Execute queries against Database as they normally would RDBMS HyperRoll Transparent redirection between detail and aggregate data No user training Dramatically improved query response Dramatically improved manageability 44 1/24/2006

45 Add the View to Business Objects Here the new Database view has been added to the current BO Universe The view comprises the aggregated data for the existing schema Database View Accessing HyperRoll data 45 1/24/2006

46 Enable Aggregate Aware Function In the Aggregate Aware function place the matching column from the view as the first parameter, and the column from the fact table as the second SH. SALES.AMOUNT_SOLD) 46 1/24/2006

47 Setting up Queries to work with HyperRoll Business Objects End User Layer Query Request Business Objects SQL Generation N Is it a summary request? Y SQL Request FACT TABLE DETAILED RDBMS VIEW SUMMARIZED Gateway HyperRoll Instance 47 1/24/2006

48 HyperRoll Value Propositions Improved query performance Reduced batch window to load data Lower maintenance and support costs Enables operational BI Complimentary to existing BI, DB and DW infrastructures 48 1/24/2006

49 How to learn more On algorithms for massive data sets On HyperRoll Talk to me over the break Talk to Kathleen (845) Take a webinar /24/2006

50 Yossi Matias, CTO HyperRoll NEBOUG January 19, /24/2006 Realizing the Potential of Business Intelligence