IBM DB2 Database and MicroStrategy 10: A functional overview including recommendations for performance optimization MicroStrategy World 2016
DB2 Is A Large Family of Relational Database Engines IBM DB2 for Linux, UNIX and Windows is a next generation data platform for transactional and analytical operations This session talks specifically about IBM DB2 for Linux, UNIX and Windows Also applies to the latest cloud based offerings, such as dashdb IBM DB2 is a commonly used by MicroStrategy customers MicroStrategy and DB2 team have been working collaboratively on the technical integration for 20 years
MicroStrategy Data Access Workflows There are numerous ways for MicroStrategy to interact with IBM DB2 Adhoc Schema For Analysts familiar with data in database Schema is created automatically on the fly Optimal time-to-value Live Connect User actions trigger analytical queries against DB2 Good for frequently changing data Modeled Schema BI Architect creates logical model of data in MicroStrategy Analyst or Consumers use model objects (attributes and metrics) to express their analytical needs MicroStrategy generates translated to multipass SQL to database In-Memory Dataset Data is imported from database into Multidimensional In-Memory cache Can improve performance and user scale
MicroStrategy Pushes Analytical Workloads to IBM DB2 Analytical queries have specific technical characteristics A typical analytical query Accesses a large amount of data (up to terabytes) Processes large amount of data MicroStrategy customers like to have interactive experience Challenge is to achieve fast response times MicroStrategy and IBM DB2 work together to tackle this challenge MicroStrategy formulates good queries DB2 executes queries well
MicroStrategy Provides Unique Optimizations for IBM DB2 DB2-optimized SQL syntax DB2 Analytical functions (OLAP functions) CASE expressions Full outer joins Set operators Subqueries Setting Isolation levels Seamless support for key DB2 features DB2 BLU Acceleration DB2 Intrapartition Parallelism DB2 Compression DB2 Workload Manager DB2 function library Multi-pass SQL for analytical sophistication Use of common table expressions Use of declared global temporary tables Control over distribution keys for intermediate results MicroStrategy Parallel SQL Execution Extensions to DB2 functionality Aggregate awareness with physical summary tables Middle-tier computation of calculations not available in IBM DB2 Middle-tier caching via Intelligent Cubes Report caching Application-level partitioning
DB2-Optimized SQL Syntax MicroStrategy integrates with IBM DB2 s broad list of database functions and SQL functionality to improve analytical performance DB2-specific SQL DB2 version-specific MicroStrategy database connector Common table expressions, case expressions, full outer joins, set operators, subqueries, etc. Data type support Function push-down MicroStrategy provides a library of mathematical, statistical, and financial functions Function push down to the database leverages the power of IBM DB2 s database server MicroStrategy s SQL Engine pushes over 100 analytical functions to DB2
MicroStrategy Provides Analytical Sophistication Multi-pass SQL optimized for IBM DB2 database MicroStrategy provides analytical richness using multi-pass SQL Result of a SELECT statement used as input for the following Intermediate result is truly temporary in nature SELECT FROM WHERE MicroStrategy can handle intermediate data results in multiple ways Common Table Expression or CTE (Default) True Temporary Tables Some MicroStrategy functionality cannot use CTE syntax Middle-tier processing of data (Analytical Engine) MicroStrategy Partitioning feature SELECT FROM WHERE
MicroStrategy Switches Intermediate Table Syntax Dynamically Depending on report design a different approach provides optimal query performance In general, CTE syntax provides the best execution plan In case of too many passes or when joining too many tables declared global temp tables (DGTTs) are the better option Two settings control whether to generate CTE or DGTT syntax VLDB Category VLDB Property Setting Default Value Tables Maximum SQL Passes Before Fallback 0 Tables Maximum Tables in From Clause Before Fallback 0
MicroStrategy Switches Intermediate Table Syntax Dynamically Common Table Expression (CTE) Declared Global Temp Tables (DGTTs)
Leveraging the power of DB2 BLU with MicroStrategy IBM BLU Acceleration is designed for analytical workloads BLU speeds up access to data Columnar storage Compression Data skipping BLU speeds up data processing CPU optimized processing Next generation in-memory computing For optimal support in MicroStrategy Use the MicroStrategy database object IBM DB2 Version 10.5 for Linux, UNIX and Windows Update the settings below for optimal intermediate table syntax VLDB Category VLDB Property Setting Value Tables Intermediate Table Type Permanent Table Indexing Intermediate Table Index Don t create an index Pre/Post Statements Insert Post Statement 1 CALL SYSPROC.ADMIN_CMD ('RUNSTATS ON TABLE??? TABLESAMPLE SYSTEM(20) )
Example: SQL Generation Configured for IBM BLU
No BLU? Other Key DB2 Features to Optimize Performance Other DB2 features to improve MicroStrategy analytic workload performance IBM DB2 Intrapartition Parallelism Allows MSTR to throttle the degree of parallelism depending on the type of query being submitted MicroStrategy recommends the use of parallelism for analytical workloads such as MicroStrategy analytical queries IBM Table Partitioning Feature Table row data is distributed according to the partitioning key Improves query performance when performing join operations in analytical workloads IBM DB2 s Adaptive Compression Adaptive compression is automatic with column tables For row-organized tables, Adaptive Row Compression offers the best storage savings MicroStrategy fully supports the use of Adaptive Compression on warehouse base tables Materialized Query Tables (MQT) Store precomputed results which can improve performance Not referenced in SQL but chosen by the IBM optimizer at runtime if the MQT can improve performance Can be row-organized or column-organized
What is Workload Management? Why is it Important? Workload Management (WLM) is necessary to optimize access to resources for concurrently executing jobs or queries. The goals of a functional workload management are Optimally leverage available (hardware) resources for performance and throughput Prioritize access for high priority jobs Assure resource availability by avoiding system lock-up by any small set of jobs
Workload Management In MicroStrategy Threads Have Priority Levels to Ensure High Priority Jobs Execute Quickly Report requests are submitted Jobs are routed by Priority to their corresponding database Connection High Priority Threads Medium Priority Threads Low Priority Threads MicroStrategy 14
Workload Management in MicroStrategy Job Priorities Can be Set on Different Application Parameters 15
MicroStrategy Can Integrate with IBM DB2 WLM IBM DB2 Workload Manager (WLM) enables administrators to easily monitor and control active work in the system In a mixed workload environment (transactional and analytics) you need a way to prioritize queries. DB2 WLM provides this functionality With DB2 with BLU acceleration or dashdb, there is built-in WLM Use WLM service classes to set different priorities for each of your MicroStrategy workloads: MicroStrategy Metadata Requests MicroStrategy Element Requests MicroStrategy Report Requests A report pre statement tags the MicroStrategy job VLDB Category VLDB Property Setting Value Pre/Post Statements Report Pre Statement CALL SYSPROC.WLM_SET_CLIENT_INFO( 'MSTRUser=!u', NULL, 'Project=!p', 'Report=!o', NULL)
Managing MicroStrategy Workloads in IBM DB2 Create different WLM service classes for your various MicroStrategy workloads Please refer to the Workload Management with MicroStrategy Software and IBM DB2 paper for more implementation details
Summary MicroStrategy and IBM DB2 work together to tackle challenges of analytics Product defaults are optimized for typical analytical scenarios MicroStrategy offers flexibility to quickly adjust to your environment Details in the integration paper on MicroStrategy Community: TN45823 Please email your feedback to jdemuth@microstrategy.com
Steps to leverage BLU with MicroStrategy IBM BLU Acceleration is the single most important feature for improving analytical workloads MicroStrategy Recommendations Set the DB2 database default table orientation to column-organized (DB2_WORKLOAD=ANALYTICS ) Convert existing row-organized tables into column-organized tables (db2convert) Use the MicroStrategy database object IBM DB2 Version 10.5 for Linux, UNIX and Windows Update the MicroStrategy VLDB properties for proper intermediate table syntax