Integrating MicroStrategy With Netezza

Transcription

1 MicroStrategy, Inc. Integrating MicroStrategy With Netezza This document provides an overview of Netezza and how MicroStrategy integrates with Netezza MicroStrategy Incorporated, 2013 Page 0 of 36

2 Contents Introduction... 4 About MicroStrategy... 4 About Netezza... 4 Query Performance... 4 DDL Considerations... 4 Data Distribution... 4 Data distribution for intermediate tables... 5 Physical Data Organization... 7 Maintaining Table Statistics... 8 Work Load Management... 8 Workload Management in MicroStrategy... 9 Monitoring of MicroStrategy Workload Monitoring of Netezza Workload WLM in Netezza High Availability Failover support Security User Authentication in Netezza MicroStrategy Database Authentication MicroStrategy Passthrough execution Analytic Functions: Functions for OLAP Ranking Custom Group Banding via the CASE statement Database Write-back Recommended Tuning Netezza Certification Status Major Netezza 7.0 Features Small Query Performance Improvements Workload Management Support for Netezza Analytics Functions Database and SQL Language Changes MicroStrategy Incorporated, 2013 Page 1 of 36

3 Netezza Certification Status Major Netezza 6.0 Features Guaranteed Resource Allocation (GRA) Netezza Certification Status Major Netezza 5.0 Features SPU (Snippet Processing Unit) Table Cache Netezza Certification Status Major Netezza 4.6 Features Stored Procedures Unicode Object Names VLDB Reference Overview of MicroStrategy Architecture MicroStrategy Technology Philosophy: Leverage the Power of the RDBMS Model-based Dynamic SQL Generation Schema Abstraction Aggregate Awareness Multi-pass SQL Default VLDB Settings for Netezza Detailed Discussion of VLDB Settings Intermediate Table Types Drop Temp Table Method Table Creation Type Other controls over distribution keys Sub Query Type Set Operator Optimization SQL Global Optimization Parallel SQL Execution Additional VLDB Settings Data Type Mapping MicroStrategy Incorporated, 2013 Page 2 of 36

4 Supported data types: Unsupported data types: Support of MicroStrategy Function Library Supported Functions Basic Functions Date & Time Functions Internal Functions Mathematical Functions Null & Zero Functions OLAP Functions Statistical Functions String Functions MicroStrategy Incorporated, 2013 Page 3 of 36

5 Introduction This paper provides a brief overview of the MicroStrategy architecture and explains how this architecture takes advantage of the technology advances and business intelligence functionality of Netezza. MicroStrategy offers an unsurpassed range of analytical functionality, which is further optimized when integrated with Netezza s broad list of database features that improve data warehousing performance. About MicroStrategy MicroStrategy is a full featured BI platform, delivering all 5 styles of BI in a fully integrated architecture. The MicroStrategy platform provides VLDB drivers for all supported RDBMS platforms to generate optimized SQL that takes advantage of database specific functionality. The full set of VLDB properties is documented in the MicroStrategy System Administration Guide. Settings that are most relevant to Netezza are discussed below. About Netezza Netezza designs and markets high-performance data warehouse appliances and advanced analytics applications for uses including enterprise data warehousing, business intelligence, predictive analytics and business continuity planning. Query Performance DDL Considerations MicroStrategy differentiates two sets of tables as part of its analytical workload. The first are made up of a permanent set of base tables that are created and populated by a DBA. These generally consist of fact and lookup tables as required to implement the desired logical data model for the analytical application. The second is a set of intermediate tables that is generated by MicroStrategy s Multi-pass SQL and is required to answer analytical questions that cannot be answered with a single SQL query block. MicroStrategy offers significant flexibility in how these intermediate tables are defined to optimally leverage all features of the RDBMS. Data Distribution Data Distribution is an important concept in Netezza. It allows breaking down query requirements for efficient parallel processing across all hardware nodes. When creating any table in Netezza, the distribution key is used to determine the node on which each row of data is stored. That is, any table will be distributed across multiple nodes based upon the distribution key. An important factor in Netezza database design is selecting good distribution keys for fact tables. The data distribution is defined by the optional distribution key field in the create table DDL. For each row, the value of this key determines the node on which the data for that row will physically reside. The goal is to have data evenly distributed across the nodes for a table, and where feasible, to have MicroStrategy Incorporated, 2013 Page 4 of 36

6 tables that join together use the same distribution key. When two tables are joined on their respective distribution keys, matching rows from both tables will reside on the same node. In this way, choosing join keys as distribution keys for related tables that are frequently joined minimizes the need to move data between compute nodes at run time. If there is not a good candidate for the distribution key, then Netezza recommends against defining an explicit distribution key, and instead relying on the default distribution mechanism, RANDOM. The RANDOM distribution places data round-robin on each node to ensure each node has the same amount of data. Distribution keys can be defined for all Netezza tables, whether permanent or temporary. Each table can have up to 1 key, and a key can consist of up to 4 columns. MicroStrategy, like any SQL application, transparently takes advantage of distribution keys defined on base tables. MicroStrategy recommends following Netezza recommended best practices when implementing the physical schema of the base tables. For more details on defining a distribution key, refer to the Netezza System Administrator s Guide under the section called Creating Distribution Keys. Data distribution for intermediate tables Data distribution across nodes can also be relevant in MicroStrategy when creating temporary tables. Depending on the reporting requirements, intermediate tables can only contain a small set of data or a very large set of data. The default settings assume intermediate tables to contain a small data set. The following sections will discuss different ways to adjust DDL generated by the MicroStrategy SQL Engine for optimal performance in Netezza. By default the MicroStrategy SQL Engine uses Temporary Tables and Implicit Table Creation for Netezza. Example 1 shows the syntax for a typical query. With Implicit Table Creation (i.e. Create Table As Select), MicroStrategy does not specify a distribution key. In this case, Netezza determines the distribution key, based on the distribution key of the underlying table, the join keys of the select statement and the group by of the select statement (i.e. it does not simply default to the first column in the table.) In general, the data distribution of the intermediate table is defined by the final node in the access plan for the corresponding SELECT statement. If the final node has a valid distribution, the system assigns that distribution to the intermediate table. Only if the final plan node has no distribution does the system default to the table s first column. More information about how the distribution key is generated and how other types of distribution work, please refer to the Netezza Database User s Guide in the Netezza SQL Command Reference for CREATE TABLE. Usually, MicroStrategy intermediate tables contain multiple tables joined together and a GROUP BY clause. In this case, the distribution keys are inherited from the GROUP BY keys. In Example 1, the distribution key would be (YEAR_ID, SUBCAT_ID). CREATE TEMP TABLE ZZSP00 as select a13.year_id YEAR_ID, a12.subcat_id SUBCAT_ID, sum(a11.tot_unit_sales) WJXBFS1 from ITEM_MNTH_SLS a11 join LU_ITEM a12 MicroStrategy Incorporated, 2013 Page 5 of 36

7 on (a11.item_id = a12.item_id) join LU_MONTH a13 on (a11.month_id = a13.month_id) group by a13.year_id, a12.subcat_id Example 1: By default the distribution key is defined by the columns in the GROUP BY clause Distributing intermediate tables in this manner is generally a good approach because intermediate tables are typically joined to other tables using a join key that happens to be the same as the distribution key. In other cases, e.g. when using the MicroStrategy Multi-Source option, the query may benefit from distributing intermediate tables in different ways. If Table Creation Type is changed to Explicit, the SQL Engine can specify specific distribution keys on intermediate tables defined by the value for the Intermediate Table Index setting. The Intermediate Table Index setting is only applied when Table Creation Type is set to Explicit. If no other settings are changed, the default distribution key specified is round-robin as can be seen in Example 2. VLDB Category VLDB Property Setting Value Tables Table Creation Type Explicit Table CREATE TEMP TABLE ZZSP00( YEAR_ID NUMERIC(38, 0), SUBCAT_ID NUMERIC(38, 0), WJXBFS1 FLOAT) DISTRIBUTE ON RANDOM insert into ZZSP00 select a13.year_id YEAR_ID, a12.subcat_id SUBCAT_ID, sum(a11.tot_unit_sales) WJXBFS1 from ITEM_MNTH_SLS a11 join LU_ITEM a12 on (a11.item_id = a12.item_id) join LU_MONTH a13 on (a11.month_id = a13.month_id) group by a13.year_id, a12.subcat_id Example 2: Explicitly defined data distribution for intermediate tables The MicroStrategy SQL Engine may also be configured to specify a distribution key on all attribute ID columns that are used in the temp table. The following combination of VLDB settings may be used to generate the SQL in Example 3. VLDB Category VLDB Property Setting Value Tables Intermediate Table Type Permanent table Tables Table Creation Type Explicit Tables Table Qualifier TEMP Tables Create Post String DISTRIBUTE ON (!a) Indexing Intermediate Table Index Don t create an index MicroStrategy Incorporated, 2013 Page 6 of 36

8 Note that!a is a wildcard for all ID columns corresponding to attributes in the intermediate table. create TEMP table ZZSP00 ( YEAR_ID NUMERIC(38, 0), SUBCAT_ID NUMERIC(38, 0), WJXBFS1 FLOAT) DISTRIBUTE ON (YEAR_ID, SUBCAT_ID) insert into ZZSP00 select a13.year_id YEAR_ID, a12.subcat_id SUBCAT_ID, sum(a11.tot_unit_sales) WJXBFS1 from ITEM_MNTH_SLS a11 join LU_ITEM a12 on (a11.item_id = a12.item_id) join LU_MONTH a13 on (a11.month_id = a13.month_id) group by a13.year_id, a12.subcat_id Example 3: Explicit data distribution for intermediate tables across all attribute columns This syntax specifies a distribution key on all attribute ID s in the temp table. As mentioned in the discussion of implicit table creation above, usually temp tables will group by those same keys, so the distribution key ends up being the same as if the temp table had been created with implicit table creation. In the (relatively rare) cases where implicit table creation does not result in distribution on all attribute ID s, this explicit syntax can guarantee it. See Additional VLDB Settings below for more detail on the settings that can be customized when using Permanent Tables. Physical Data Organization Netezza does not automatically reclaim and reuse space that is available when you delete or update rows in tables. When a table is updated, the new data is added while the old data is just marked as outdated and remains on disk. Over time this can have a severe impact on Netezza resource utilization and performance. While the maintenance of the physical data organization is generally transparent to MicroStrategy, potential performance impacts can be noticed in MicroStrategy. For this reason MicroStrategy recommends following Netezza guidelines for regular grooming of base tables. Because of their temporary nature grooming of intermediate tables is not necessary. As of Netezza 6.0, a new feature has been introduced to reclaim space called GROOM. This command can be run via nzsql with the following syntax: GROOM TABLE <table_name> RECORDS ALL; Example 4: Groom syntax Once this has completed, a message indicating the number of records purged and size change of the table will appear. While the GROOM TABLE command handles individual tables, reclaiming space for an entire database or all databases on the entire system can be handled via a command called nzreclaim. MicroStrategy Incorporated, 2013 Page 7 of 36

9 nzreclaim u <user> -pw <password> -host <hostname> -db <database> [-alldbs] RECORDS ALL; Example 5: nzreclaim syntax Both of these commands will allow you to reclaim lost space. For more information regarding the groom and the nzreclaim command, refer to Chapter 9: Managing User Content on the Netezza Appliance from the IBM Netezza System Administrator s Guide. Maintaining Table Statistics Table statistics are vitally important to the query planner to generate optimal query execution plans. In Netezza, statistics are collected using the GENERATE STATISTICS statement and can be set to generate statistics for an entire database, a specific table, or an individual column. Within these statistics, information primarily around cardinality and distribution of values within columns are generated and collected. In the context of a MicroStrategy deployment it is important that all queries submitted by MicroStrategy are supported with current statistics. There are two key aspects to this. First, there needs to be a strategy to maintain current statistics on all tables that implement the data model used by MicroStrategy. The second aspect involves statistics collections for intermediate tables created by MicroStrategy as part of multi-pass SQL functionality. Statistics collection ( GENERATE STATISTICS ) in Netezza is extremely low resource intensive. Netezza automatically creates statistics for temporary tables and tables created using implicit table creation syntax as well as any time an INSERT or UPDATE and GROOM TABLE is run. Since MicroStrategy uses implicit table creation and temporary tables, Netezza would automatically analyze all tables created by MicroStrategy. If the report designer chooses explicit tables creation it is important to manually instruct the MicroStrategy SQL Engine to generate statistics on intermediate tables. Enabling enter VLDB setting here allows to do so as shown in Example 3. Work Load Management In general, a typical MicroStrategy workload consists of a mix of queries with significantly varying complexity. These queries originate from different MicroStrategy jobs, e.g. element requests, simple grids (select * from), complex analytical multi-pass reports, Dashboards that rely on multiple (simple or complex) queries, database write-back from Transaction Services, and Cube reports designed to fill MicroStrategy In-Memory Cubes. Additionally, it needs to be assumed that other workloads are being simultaneously submitted to Netezza from other sources. Workload Management (WLM) is necessary to optimize access to database resources for concurrently executing queries. The goals of a functional workload management are to - Optimally leverage available (hardware) resources for performance and throughput - Prioritize access for high priority jobs MicroStrategy Incorporated, 2013 Page 8 of 36

10 - Assure resource availability by avoiding system lock-up by any small set of jobs Effective workload management starts with comprehensive monitoring, allowing identifying bottleneck conditions, and then leveraging the available platform tools to implement a workload management strategy that meets all of the above goals. Workload Management in MicroStrategy MicroStrategy internally breaks down every user request into one or more jobs that are processed independently. Each job advances through a series of processing steps, one of which might be the submission of multi-pass queries to a database. MicroStrategy is capable of processing and submitting multiple jobs in parallel to a database. By default MicroStrategy opens multiple connections called Database Threads to any data source. This is illustrated in the diagram below. A typical BI environment encompasses queries ranging from very quick and urgent to long-running and low priority. To avoid the scenario where a small number of expensive queries can block all access to database resources it all database threads are assigned priority classes (High, Medium and Low). When report jobs are submitted, jobs are assigned a priority based on a list of different application parameters User groups, Application type, Project, Request type and Cost. The MicroStrategy work load management routes each job according to their priority to their corresponding database threads. When no database threads are available jobs will be queued until a database thread of the appropriate class becomes available. MicroStrategy Incorporated, 2013 Page 9 of 36

11 For more information on how to set these priorities refer to technical note TN5401 (TN ): How to set Group Prioritization in MicroStrategy Intelligence Server 8.x x? Administrators can for each priority class and depending on the available database resources specify the number of warehouse connections that are required for efficient job processing. - Best practices for MSTR WLM. Point to best practice paper. - Monitor in EM for queue time. This indicates insufficient resources for a given priority class. While the intuitive instinct at this point is to increase the number of database threads care must be taken not to overload the database as this will at least lead to suboptimal performance, and at worst to unstable BI environments. The optimal number of connections is dependent on several factors, however the main criterion to consider when setting the number of connections is the number of concurrent queries the warehouse can support, and in the case of Netezza it should be limited to less than 32 queries. For more information on Job prioritization and connection mapping please refer to technical note TN8486 (TN X-0405): What is Job Prioritization and Connection Management in MicroStrategy Intelligence Server 8.x and 9.x? Monitoring of MicroStrategy Workload The primary goals when monitoring work load is to ensure that the work load is utilizing the available hardware resources efficiently, that sufficient hardware resources are available to handle the given workload, and, in case the first conditions are not met, that it will provide the insights to identify issues and areas for improvement. A good starting point for monitoring the work load of a MicroStrategy implementation is MicroStrategy Enterprise Manager. Enterprise Manager easily provides an overview of the BI workload, the consumed resources, such as time spent and CPU cycles, and allows identifying the time spent querying the database both from an aggregated view down to individual user requests. MicroStrategy Incorporated, 2013 Page 10 of 36

12 Enterprise Manager can help achieve detailed analysis in several categories: In operation analysis one can monitor concurrency, queue and response time trends by hour/minute and insights on scheduling, caching, prioritization, clustering. This can help summarize the usage, breakup of the load and the performance of the system. Below is an example of an out of the box Operational Analysis Report (Weekly Summary of Project Activity) MicroStrategy Incorporated, 2013 Page 11 of 36

13 In some cases the time spent on the database will make up for a significant portion of the overall processing time for a user request. At this point any further analysis needs to take place on the database layer. Monitoring of Netezza Workload WLM in Netezza Netezza is made up of a 2 tier system which uses the best features of both an SMP and MPP architecture. Netezza has an SMP host which connects to all of the AMPP drives of the system which contain the Snippet Processing Units (SPU). The SPUs handle processing early on in the data flow as possible to eliminate any potential delays. There are 4 different features that are used by Netezza to manage resource allocations. The 4 features are: 1) Short query bias 2) Guaranteed resource allocation (GRA) 3) Prioritized query execution (PQE) 4) Gate keeper The gate keeper option is the only one out of the 4 features that are not enabled automatically. The other 3 are enabled by default and are taken advantage of by MicroStrategy without any additional intervention. For more information about the resource allocation features, refer to Chapter 12: Managing Workloads on the Netezza Appliance from the Netezza System Administrator s Guide. Workload management can be monitored within the Netezza Administrator tool via reports and graphs or by querying system views. All of the features above ultimately go back to the GRA and the information can be found through these reports through multiple levels of detail. The levels of reports are listed below: Level Summary History Graph Description Displays performance status for the last 60 minutes. Displays all available table information from summary data captured in 10 minute intervals. Displays resource allocation for specific days. Usage trends, usage performance, and # of jobs are listed. The other option to gather workload management details is to query them from the system views. Listed below are the different views that can be used: View _v_sched_gra_ext _v_sched_sn_ext Description Displays information about how busy the system is and how GRA resources are being allocated. Displays information about how busy the system is and how snippet resources are MicroStrategy Incorporated, 2013 Page 12 of 36

14 being allocated. _v_gra_sched_ext_latest Contains previous 10 minutes of scheduling information in the GRA scheduler. _v_sched_sn_ext_latest Contains previous 10 minutes of scheduling information in the snippet scheduler. _v_plan_resource Resource usage over time, keeps at least 2000 records of query plans. _v_system_util Shows system utilization for host, SPU CPU, disk, memory resources. Updated every 60 seconds. Besides using the Netezza Server Administrator to monitor activity on the server, the system views listed above can be used in a MicroStrategy project. This can be done like any other table by adding them through the Warehouse Catalog and have objects created off of them. High Availability All versions of Netezza come with a High Availability (HA) solution borrowed from the underlying Linux operating system. Netezza 6.0 and newer uses a combination of Linux-HA and Distributed replicated block device (DRBD**) in synchronous mode. Netezza versions prior to 6.0 use the Red Hat Cluster Manager to manage HA. High Availability in a Netezza system is managed by 2 SMP Linux servers. On these servers, the daemon Heartbeat, watches the hosts, and manages communication and status checks of all services on the system. When conditions showing potential host failure or loss of service, failover can be initiated automatically by Heartbeat to the standby host. DRBD** mirrors the content of block devices such as hard drive, partitions, logical volumes. The main part of the system that is closely monitored is /nz/ and /export/home/ partition. ** For more details about DRBD please see the documentation available at Failover support Failovers in Netezza are an automated event that gets triggered when certain conditions indicate a failure in a particular node. The following events could trigger a failover operation switching from the troubled node to a standby node: - Network links are lost - Heartbeat is manually stopped - Clean shutdown such as issuing shutdown h or unclean shutdowns such as power failures - Resource groups or services on the node does not start properly When creating the DSN that MicroStrategy will use in the Database Instance, the IP address or host name that is used, will direct the connection to one of the 2 available hosts. As long as one of the two Netezza hosts is still running, there will be no interruption seen on the MicroStrategy side and the report or document will complete as expected. In an event where both hosts are down, the report or document will fail. Once Netezza has become available again the job will need to be re-submitted. In this case MicroStrategy will automatically and transparently to the end user reconnect to Netezza. For more MicroStrategy Incorporated, 2013 Page 13 of 36

15 information regarding the HA features and details regarding failover operations in Netezza, please refer to Chapter 4: Managing Netezza HA Systems from the Netezza Database User s Guide Security To access the Netezza database, users must have an account with sufficient privileges created within the database. MicroStrategy will connect to the Netezza database via ODBC and will need proper privileges granted in order to work properly. To determine the correct privileges a user should have on the Netezza user, please refer to Chapter 8: Establishing Security and Access Control from the Netezza Database User s Guide. User Authentication in Netezza There are 2 types of authentication methods that are supported by Netezza, local and LDAP. This is set at a system level where LDAP is enabled or disabled, but local authentication can take place in either situation. By default, the CREATE USER syntax will create a user that utilizes local authentication unless the AUTH parameter is specified otherwise. Local authentication utilizes the user created by the Netezza administrator directly on the database server while LDAP authentication utilizes LDAP username and password from the LDAP server to authenticate. Both of these types of authentication can be used with MicroStrategy and can be specified while creating the Database Instance and configuring the project source. For basic information on how to configure a MicroStrategy database instance, refer to the following technical note: TN How to create a database instance for project warehouse in MicroStrategy Desktop 8.x and 9.x? SSL can also be used in conjunction with LDAP to manage the communication between the database server and the LDAP server more securely. Detailed steps on how to enable this can be found here: Chapter 8: Establishing Security and Access Control from the Netezza Database User s Guide. MicroStrategy Database Authentication Users created on the database can be also used to log into MicroStrategy projects via Desktop and Web. Instead of validating user credentials stored in the MicroStrategy metadata, user credentials are passed through directly to the database for authentication. For more information and instructions on how to setup database authentication, refer to the MicroStrategy System Administration Guide:3. Identifying Users: Authentication along with the following technical note: TN31832 How to use Database Authentication in MicroStrategy 8.1.x and 9.x MicroStrategy Passthrough execution MicroStrategy can take advantage of user security implemented in Netezza by submitting queries for MicroStrategy users with a specific Netezza login. This feature is called Passthrough execution. This allows you to link a MicroStrategy user to a Netezza login, and will be used when browsing attribute elements, or executing reports/documents. Benefits of using passthrough execution is to allow easy auditing to track which users are running which queries and also to apply built in security that may already be configured for the Netezza login. MicroStrategy Incorporated, 2013 Page 14 of 36

16 Passthrough execution needs to be configured through the both the User Editor and the Project Configuration via MicroStrategy Desktop. For more information on Passthrough execution, refer to the MicroStrategy System Administration Guide: Section 2. Setting up User Security under the section called Linking database users and MicroStrategy users: Passthrough execution and also TN14031 Analytic Functions: Functions for OLAP OLAP functions are a powerful family of functions for business intelligence and data warehousing applications. These functions can provide significant analytical capabilities for many business intelligence queries. <analytic function> ::= <function_name> OVER ([<partition clause>] [<order by clause> [<windowing clause>]]) Example 6: SQL Syntax for OLAP Functions MicroStrategy s OLAP functions will be calculated via SQL in Netezza rather than in the MicroStrategy analytical engine. There is no VLDB setting associated with this option. select a12.region_id REGION_ID, max(a13.region_name) REGION_NAME, a11.day_date DAY_DATE, sum(a11.tot_dollar_sales) TOT_DOLLAR_SALES, avg(sum(a11.tot_dollar_sales)) over(partition by a12.region_id order by sum(a11.tot_dollar_sales) asc rows 2 preceding) MOVING_AVG from DAY_CTR_SLS a11 join LU_CALL_CTR a12 on (a11.call_ctr_id = a12.call_ctr_id) join LU_REGION a13 on (a12.region_id = a13.region_id) group by a12.region_id, a11.day_date Example 7: Moving average example Ranking The RANK function is closely related to other OLAP functions. Key distinctions of the RANK function are that it requires an order by clause and does not support a window clause. The MicroStrategy Engine is aware that Netezza supports the Rank function, so the ranking will be done by the database rather than the MicroStrategy Analytical Engine. select a13.region_id REGION_ID, max(a14.region_name) REGION_NAME, a11.emp_id EMP_ID, max(a12.emp_last_name) EMP_LAST_NAME, max(a12.emp_first_name) EMP_FIRST_NAME, sum(a11.tot_dollar_sales) TOT_DOLLAR_SALES, rank () over(partition by a13.region_id order by sum(a11.tot_dollar_sales) asc nulls last) SALES_RANK from ITEM_EMP_SLS a11 join LU_EMPLOYEE a12 on (a11.emp_id = a12.emp_id) join LU_CALL_CTR a13 on (a12.call_ctr_id = a13.call_ctr_id) join LU_REGION a14 MicroStrategy Incorporated, 2013 Page 15 of 36

17 on (a13.region_id = a14.region_id) group by a13.region_id, a11.emp_id Example 8: Rank example Custom Group Banding via the CASE statement Custom Group Banding is a powerful technique for generating histograms and other complex analyses. The Engine uses Netezza s CASE Statement to implement custom group banding reports. CREATE TEMP TABLE ZZMQ00 as select a11.customer_id CUSTOMER_ID, Case when (a11.tot_dollar_sales= and = *floor(( )/100.0) ) then floor(1.0*(a11.tot_dollar_sales- 1.0)/100.0) when (a11.tot_dollar_sales between 1.0 and ) then floor(1.0*(a11.tot_dollar_sales- 1.0)/100.0) +1 else 0 end DA62 from CUSTOMER_SLS a11 where (a11.tot_dollar_sales between 1.0 and ) select a12.da62 DA62, sum(a11.tot_unit_sales) WJXBFS1 from CUSTOMER_SLS a11 join ZZMQ00 a12 on (a11.customer_id = a12.customer_id) join LU_CUSTOMER a13 on (a11.customer_id = a13.customer_id) join LU_CUST_CITY a14 on (a13.cust_city_id = a14.cust_city_id) where a14.cust_state_id in (5) group by a12.da62 Example 9: Custom Group Banding Database Write-back MicroStrategy provides the ability to write back data into Netezza. This functionality is implemented using the optional MicroStrategy Transaction Services. Transaction Services allows concurrent writeback of data from mobile devices or over the web using database transactions. The syntax used to implement write-back is defined using a Transaction Report. By default Netezza performs a serialization dependency check and will prevent non-serializable executions. If two concurrent transactions in this mode attempt to modify the same data, the system automatically fails and rolls back the youngest transaction. In most Transaction Services environments this will result in the following error message: Could not serialize - transaction aborted. As a best practice it is recommended to switch Netezza into multi-versioning (a snapshot isolation) mode by adding the set serializable=0 statement to the transaction SQL. In this mode each transaction sees a consistent state that is isolated from other transactions that have not been committed. The Netezza hardware ensures that the system can quickly provide the correct view to each transaction. set serializable=0; Begin Transaction <update statement...> End Transaction Example 10: Sample Syntax for a Transaction Report MicroStrategy Incorporated, 2013 Page 16 of 36

18 Other notes that are helpful when designing transaction SQL statements: o The syntax to change the isolation level in Netezza from serializable to repeatable read, read uncommitted, or read committed is supported but not implemented. The only supported isolation level in Netezza is serializable. (Netezza Database User s Guide, p03-36) o Statements in stored procedures are executed as a single transaction o Maximum of 31 concurrent read/write transactions o Netezza does not allow explicit locking on tables; this is controlled by the DBMS Recommended Tuning There are two session level settings that can be adjusted for optimal performance. You will need to experiment with to find the best results as they may differ from table to table. These values can be sent to the Netezza server through the use of the following report/project level MicroStrategy VLDB properties. VLDB Category VLDB Property Setting Value Pre/Post Statements Report Pre Statement 1 set enable_factrel_planner = true Pre/Post Statements Report Pre Statement 1 set factrel_size_threshold = n (where n is the number of rows) set enable_factrel_planner = true; o This tells the Netezza optimizer to force the Netezza engine to consider tables with more rows than the threshold set by factrel_size_threshold as a fact table and not to redistribute across the SPUs. set factrel_size_threshold = ; o This sets the engine s size for a defining fact tables. This value should be set higher than the total number of rows in the largest dimension table. Any table with more rows than this value will be considered a fact table by the Netezza engine and be subject to result redistribution which can affect performance. Netezza 7.0 Certification Status OS platform Recommended ODBC Driver Status Windows 32-bit 32-bit Netezza Driver 7.0.x Certified: WH Windows 64-bit 32-bit Netezza Driver 7.0.x Certified: WH AIX (64-bit) 32-bit Netezza Driver 7.0.x Certified: WH Solaris (64-bit) 32-bit Netezza Driver 7.0.x Certified: WH RedHat & SUSE Linux (64-bit) 32-bit Netezza Driver 7.0.x Certified: WH MicroStrategy Incorporated, 2013 Page 17 of 36

19 HP-UX (64-bit Itanium) 64-bit Netezza Driver 7.0.x Certified: WH Major Netezza 7.0 Features Small Query Performance Improvements Architectural and processing changes have been made to improve the performance of small queries allowing throughput to be increased. Zone maps have been reduced in size, where before each zone map was 3MB while the newer Page Granular Zone Maps have been reduced to 128KB to reduce I/O overhead. If upgrading from an older version, to take advantage of this, statistics will need to be generated and can be done by running GENERATE STATISTICS on a table or an entire database. Refer to the earlier section Maintaining Table Statistics to ensure that statistics are current. Workload Management Support for Netezza Analytics Functions - User-defined functions (UDF) and analytic executables (AE) are now monitored in workload management system views. This allows you to get a deeper level of detail when querying off of those views to understand system usage. Database and SQL Language Changes - Maximum object IDs generated are now up to 2.14 billion versus 1 billion. - Length(x) returns character value for length when x=catalog text types - Query plan includes include more information about the snippet resource consumption as well as the plan resource consumption Netezza 6.0 Certification Status OS platform Recommended ODBC Driver Status Windows 32-bit 32-bit Netezza Driver 6.0.x Certified: WH Windows 64-bit 32-bit Netezza Driver 6.0.x Certified: WH AIX (64-bit) 32-bit Netezza Driver 6.0.x Certified: WH Solaris (64-bit) 32-bit Netezza Driver 6.0.x Certified: WH RedHat & SUSE Linux (64-bit) 32-bit Netezza Driver 6.0.x Certified: WH HP-UX (64-bit Itanium) 64-bit Netezza Driver 6.0.x Certified: WH Major Netezza 6.0 Features Guaranteed Resource Allocation (GRA) Previously, this feature only allowed you to specify the minimum percentage of resources that can be allocated per group when the system is busy. Under normal circumstances, system resources will be allocated evenly among the active groups. In Netezza 6.0, there is now an option to specify both the minimum and maximum percentage of resources per group that can be used so that groups can be prioritized and appropriate limits can be set. MicroStrategy Incorporated, 2013 Page 18 of 36

20 In the most common setup MicroStrategy submits all queries to Netezza using a single Netezza user account to access a particular database. To guarantee a minimum amount of resources, GRA can be used to allocate resources to a user or group dedicated to MicroStrategy. An example of this would be management versus daily operations. The daily operations MicroStrategy project would have their own login that accesses the database, and can be allocated a minimum amount of resources so that it will not take away from another group such as Management. This can allow the Management user to take as many resources as it needs to complete its workload. For more information on how to use this feature, please refer to Chapter 12: Managing Workloads on the Netezza Appliance from the Netezza Database User s Guide. Netezza 5.0 Certification Status OS platform Recommended ODBC Driver Status Windows 32-bit 32-bit Netezza Driver 5.0.x Certified: WH Windows 64-bit 32-bit Netezza Driver 5.0.x Certified: WH AIX (64-bit) 32-bit Netezza Driver 5.0.x Certified: WH Solaris (64-bit) 32-bit Netezza Driver 5.0.x Certified: WH RedHat & SUSE Linux (64-bit) 32-bit Netezza Driver 5.0.x Certified: WH HP-UX (64-bit Itanium) 64-bit Netezza Driver 5.0.x Certified: WH Major Netezza 5.0 Features SPU (Snippet Processing Unit) Table Cache This feature caches table data in memory for joins with common dimension tables, and temp tables. By default MicroStrategy benefits from this feature for all reports that create temporary tables. The SPU table cache is a built in feature that is enabled automatically on the Netezza appliance. Netezza 4.6 Certification Status OS platform Recommended ODBC Driver Status Windows 32-bit 32-bit Netezza Driver 4.6.x Supported: WH Windows 64-bit 32-bit Netezza Driver 4.6.x Supported: WH AIX (64-bit) 32-bit Netezza Driver 4.6.x Supported: WH Solaris (64-bit) 32-bit Netezza Driver 4.6.x Supported: WH RedHat & SUSE Linux (64-bit) 32-bit Netezza Driver 4.6.x Supported: WH HP-UX (64-bit Itanium) 64-bit Netezza Driver 4.6.x Supported: WH MicroStrategy Incorporated, 2013 Page 19 of 36

21 Major Netezza 4.6 Features Stored procedures Unicode Object Names Support for DISTINCT window aggregates Support for NULLS FIRST or NULLS LAST in an ORDER BY clause Stored Procedures In this version of Netezza, stored procedure support is introduced. Stored procedures have the benefit to manipulate and query database information along with the flexibility of programming to handle more complex tasks. These new stored procedures can then be integrated with MicroStrategy via pre/post statements in a report, called from freeform SQL reports, or anywhere else that custom SQL is allowed. Unicode Object Names Support for Unicode object names is introduced where objects such as tables, columns, users, groups and databases can be named with local languages. There are 3 supported encoding forms of Unicode, and they are UTF-8, UTF-16, and UTF-32. VLDB Reference Overview of MicroStrategy Architecture MicroStrategy Technology Philosophy: Leverage the Power of the RDBMS The MicroStrategy architecture has its roots in the principles of Relational OLAP (ROLAP). A ROLAP architecture provides OLAP functionality to the end user (e.g. multidimensional framework, slice-anddice interaction, drilling, etc.), but uses a relational database to resolve queries and perform calculations, rather than using a specialized proprietary multidimensional database. While the virtues of ROLAP are fully extolled elsewhere, this is an important distinction for this article for two reasons. First, MicroStrategy applications have been able to achieve industry-leading data scale because of the reliance on the processing power and data management capabilities of the underlying RDBMS. The system design principle of partnering with the RDBMS, through features discussed in this document, is the cornerstone of achieving such scalability. Second, this focus on integration with the RDBMS allows a business intelligence system to tap into a powerful dynamic over time: technical advances in RDBMS technology are seamlessly accessible to a MicroStrategy-based BI system. The net benefit to the customer is a BI system that increases in value when either MicroStrategy or Netezza adds features to their products. Model-based Dynamic SQL Generation Within the MicroStrategy Intelligence Server lies the MicroStrategy SQL Engine, this is responsible for generating SQL for all requests to the data warehouse. Hence, to the RDBMS, a MicroStrategy application is an SQL-based application, in many ways like any other SQL application accessing Netezza. MicroStrategy Incorporated, 2013 Page 20 of 36

22 Schema Abstraction The SQL Engine performs its work based on a metadata model defined to the system. Note that the MicroStrategy metadata is not used to store joins or schema-type information, such as star or snowflake. Instead, the metadata model stores content information for each table indicating that it contains a set of particular facts and a set of particular attributes. When a report request is submitted, the Engine breaks the report down into the individual components (i.e. attributes and facts), then begins searching the model to determine which combination of tables will be necessary and efficient in resolving the request. Schema abstraction of the database columns (into MicroStrategy attributes and facts) provides the flexibility necessary to allow applications to be created quickly without having to change the structure of the data model. MicroStrategy is able to support virtually any type of star, snowflake, or hybrid physical design, including transactional schemas. The business model defined in MicroStrategy is easily able to span multiple stars/snowflakes in a single application and even a single query. MicroStrategy supports dimensional models well, but does not require a dimensional model. Aggregate Awareness Query performance in many data warehouses is enhanced through the use of aggregate tables. Aggregate tables, which are also called summary tables, store pre-computed results of data allowing users to query from a summarized set of data rather than the detail level data that would be stored in the fact table. In many cases, use of aggregate tables will improve query performance by orders of magnitude. MicroStrategy provides an aggregate-aware Engine. MicroStrategy has allowed transparent navigation of aggregate tables, directing queries to summary tables when they exist without the user having to specify to use the table. The MicroStrategy SQL engine determines use of aggregate tables transparently at query time. Multi-pass SQL One of the key elements to providing analytical sophistication in business intelligence applications is MicroStrategy s ability to generate multi-pass SQL. Multi-pass SQL is required to answer analytical questions that cannot be answered with a single SQL query block. Examples of questions / scenarios that require multi-pass SQL include: Set qualification: Show me sales by region over the last six months, but only for customers who purchased one of the 5 most popular products. Split metrics: query returns sales data from a sales star schema and inventory data from an inventory star schema Metrics calculated at different levels of aggregation Metrics calculated with different filtering criteria Simulating outer joins on RDBMS platforms that do not support them natively Querying multiple tables due to application-level partitioning Support for these scenarios, especially when combined together, provide a framework for significant analytic questions and value to the users of the system. One of the main optimizations the MicroStrategy SQL Engine makes is to generate SQL that performs these multi-pass queries as efficiently as possible. MicroStrategy Incorporated, 2013 Page 21 of 36

23 Default VLDB Settings for Netezza 6.0 VLDB Category VLDB Property Setting Value Tables Intermediate Table Type True temporary table Tables Drop Temp Table Method Do nothing Tables Table Creation Type Implicit Table Indexing Intermediate Table Index Create primary index/partition key Indexing Primary Index Control Create primary index/partition key on temp table Query Optimizations Sub Query Type Where (col1, col2) in (Select s1.col1, s1.col2 ) Indexing Max Columns in Index 4 Joins Full Outer Join Support Supported Select/Insert Distinct/Group By option Use Group By (when no aggregation and not table key) Query Optimizations SQL Global Optimization Level 4: Level 2 + Merge All Passes with Different Where Query Optimizations Set Operator Optimization Enable set operator optimization Detailed Discussion of VLDB Settings Intermediate Table Types The ability to generate multi-pass SQL is a key feature of the MicroStrategy SQL Engine. Netezza supports a number of different ways to implement multi-pass SQL. True temporary table One way to implement multi-pass SQL is to execute each pass (i.e. each query block) in a separate table. By default, the VLDB property Intermediate Table Type is set to True Temporary Table, which results in each passes results stored in a temporary table. Temporary tables in Netezza incur less overhead than permanent tables. CREATE LOCAL TEMPORARY TABLE ZZMD00( WJXBFS1 FLOAT) insert into ZZMD00 select sum(a11.tot_dollar_sales) WJXBFS1 from "YR_CATEGORY_SLS" a11 select a12.category_id CATEGORY_ID, a12.category_desc CATEGORY_DESC, pa11.wjxbfs1 WJXBFS1 from ZZMD00 pa11 cross join "LU_CATEGORY" a12 drop table ZZMD00 Example 11: True temporary table syntax Derived Tables Rather than implement each pass in a separate table, Derived Table syntax allows the SQL Engine to issue additional passes as query blocks in the FROM clause. Instead of issuing multiple SQL statements that create intermediate tables, the SQL engine generates a single large SQL statement. This can allow MicroStrategy Incorporated, 2013 Page 22 of 36

24 queries to run faster since there are no CREATE TABLE or DROP TABLE statements to catalog, no corresponding locks on the system tables, and no logging of records inserted into a physical table. In theory, derived table syntax should result in query performance at least as good as when using temporary tables. Customers may want to experiment with this option to determine if it is beneficial in their specific environment. select pa11.year_id YEAR_ID, pa11.subcat_id SUBCAT_ID, a13.subcat_desc SUBCAT_DESC, pa11.wjxbfs1 WJXBFS1, pa12.wjxbfs1 WJXBFS2 from (select a13.year_id YEAR_ID, a12.subcat_id SUBCAT_ID, sum(a11.tot_unit_sales) WJXBFS1 from ITEM_MNTH_SLS a11 join LU_ITEM a12 on (a11.item_id = a12.item_id) join LU_MONTH a13 on (a11.month_id = a13.month_id) group by a13.year_id, a12.subcat_id ) pa11 join (select a13.year_id YEAR_ID, a12.subcat_id SUBCAT_ID, sum(a11.units_received) WJXBFS1 from INVENTORY_ORDERS a11 join LU_ITEM a12 on (a11.item_id = a12.item_id) join LU_MONTH a13 on (a11.month_id = a13.month_id) group by a13.year_id, a12.subcat_id ) pa12 on (pa11.subcat_id = pa12.subcat_id and pa11.year_id = pa12.year_id) join LU_SUBCATEG a13 on (pa11.subcat_id = a13.subcat_id) Example 12: Derived table syntax Note that not all reports are able to use derived tables. There are two primary scenarios in which temporary tables must be used instead of derived tables: When a report uses a function supported in the MicroStrategy analytical engine that is not supported in Netezza (e.g. many of the functions in the financial and statistical function packages). If these functions are used in intermediate calculations, the MicroStrategy analytical engine will perform calculations and then insert records back into the RDBMS for further processing. Inserting records back into Netezza requires a temporary table. When a report uses the MicroStrategy partitioning feature. When using partitioning, the SQL Engine executes a portion of the query in order to determine which partitions to use. The results are then used to construct the rest of the query. Because the full structure of the query is not known prior to execution, the SQL engine must use temporary tables to execute the query in multiple steps. MicroStrategy Incorporated, 2013 Page 23 of 36

25 These situations do not cover 100% of the cases in which temporary tables must be used. The rest of the cases are relatively obscure combinations of VLDB settings, such as certain combinations of Sub Query Type plus outer join settings on metrics plus non-aggregatable metrics. If the Intermediate Table Type is set to Derived tables, then True temporary tables should be specified as the fallback table type should the specific report requirements not support the use of derived table expressions. For each report, the SQL Engine will follow an all or nothing policy in determining whether to use derived table syntax. If the entire report cannot be resolved in a single statement with derived tables, the SQL Engine will automatically revert to the Fallback Table Type syntax and not use derived table syntax at all for the report. Note: When using derived tables, the UNION Multiple INSERT setting should be setting to Use Union. This allows the engine to use UNION statements within derived tables instead of multiple INSERT INTO statements. This setting is relevant for reports that use partitioning and consolidations. Permanent Tables When using True Temporary Tables as the Intermediate Table Type, intermediate tables are created with the TEMP keyword. If Intermediate Table Type is set to Permanent Table, MicroStrategy will generate a plain vanilla CREATE TABLE statement for intermediate tables. This can be useful because other VLDB settings such as Table Qualifier, Table Space, etc. can be used to customize this standard CREATE TABLE syntax. See Additional VLDB Settings below for more detail on the settings that can be customized when using Permanent Tables. create table ZZSP00 as select a13.year_id YEAR_ID, a12.subcat_id SUBCAT_ID, sum(a11.tot_unit_sales) WJXBFS1 from ITEM_MNTH_SLS a11 join LU_ITEM a12 on (a11.item_id = a12.item_id) join LU_MONTH a13 on (a11.month_id = a13.month_id) group by a13.year_id, a12.subcat_id create table ZZSP01 as select a13.year_id YEAR_ID, a12.subcat_id SUBCAT_ID, sum(a11.units_received) WJXBFS1 from INVENTORY_ORDERS a11 join LU_ITEM a12 on (a11.item_id = a12.item_id) join LU_MONTH a13 on (a11.month_id = a13.month_id) group by a13.year_id, a12.subcat_id select pa11.subcat_id SUBCAT_ID, a13.subcat_desc SUBCAT_DESC, pa11.year_id YEAR_ID, pa11.wjxbfs1 WJXBFS1, pa12.wjxbfs1 WJXBFS2 from ZZSP00 pa11 join ZZSP01 pa12 on (pa11.subcat_id = pa12.subcat_id and pa11.year_id = pa12.year_id) join LU_SUBCATEG a13 MicroStrategy Incorporated, 2013 Page 24 of 36