Functionality & Performance Testing Analysis Services 2005 with Teradata v12

Functionality & Performance Testing Analysis Services 2005 with Teradata v12 Rupal Shah Teradata Corporation James Basilico Microsoft Corporation Rev. 1.0 July, 2008

Copyright The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.

Table of Contents Scope... 4 Introduction... 5 Lab Environment Design and Configuration... 7 Functionality Testing of Teradata v12.0 with Analysis Services 2005... 10 Performance Results Against Teradata Database v2r6.2.1.3... 13 100 User Load Test Results... 13 200 User Load Test Results... 14 500 User Load Test Results... 15 1000 User Load Test Results... 16 Comparison of User Load Test Results using Teradata Database v2r6.2.1.3... 17 Performance Results Against Teradata Database v12.0... 18 100 User Load Test Results... 18 200 User Load Test Results... 19 500 User Load Test Results... 20 1000 User Load Test Results... 21 Comparison of User Load Test Results using Teradata Database v12.0... 22 Conclusions... 23 Follow-On Performance Testing Efforts... 24 Appendix A: Applicable Links... 25 Appendix B: Analysis Services Stress Testing Toolset... 26 Query Generation through ASQueryGen... 26 Client Simulation through ASLoadSim... 28 Stress Test Reports and Counters Used... 29 For more information:... 30 2008 Microsoft Corporation. All rights reserved Page 3 of 30

Scope This paper describes the results from functionality and basic performance testing of SQL Server 2005 Analysis Services leveraging Teradata v12. This paper is targeted at Business Intelligence (BI) administrators, system integrators and database administrators. Readers are expected to have a basic understanding of the features of the Teradata Database Aggregate Join Index and Microsoft SQL Server 2005 Analysis Services products. 2008 Microsoft Corporation. All rights reserved Page 4 of 30

Introduction The goal of this paper is to document lab environment efforts identifying the key architectural design strategies of a combined Teradata Relational Database and Analysis Services solution in a ROLAP environment. The objective of the lab environment tests are to benchmark and understand the.net Data Provider for Teradata v1.02 and Analysis Services 2005 components with respect to functionality, throughput requests and response time. Lab environment exercises that were performed focused on preliminary benchmark tests using a 100GB database, and then tuned the system to get optimal results based both on a Teradata Database v2r6.2.1.3 and Teradata v12.0 backend using the.net Data Provider for Teradata v12.0 in both cases. Also part of this testing was to compare these results to the previous benchmark results conducted by the Microsoft Technology Center (MTC) in July 2007, which used Teradata Database v2r6.2.1.3 and.net Data Provider for Teradata v1.2 with Analysis Services 2005. As mentioned already, this effort leveraged the Teradata Database v2r6.2.1.3 environment that was constructed for the original MTC testing that was completed in July 2007. The original testing focused on optimizing the Project REAL (http://www.microsoft.com/sql/solutions/bi/projectreal.mspx) warehouse with a goal of identifying the optimal hardware configuration and best practices for a ROLAP environment. The scope of that effort was as follows: Database preparation o o o o o Migration of the Project REAL sample data kit to a Teradata system provided by Teradata. Identify an appropriate database schema for Project REAL analytics based on the POC objectives. Redesign the Project REAL solution into a Snow-Flake schema to more effectively take advantages of Teradata optimization capabilities. Grow the data to 100GB. Tune the Teradata system for the user load. For our effort, we focused on the following: Functionality testing of Analysis Services running against Teradata Database v12.0 and.net Data Provider for Teradata v12.0. Test script preparation leveraging Visual Studio Test Suite for Testers. Benchmark and optimize the Project REAL solution for user load against Teradata Database v2r6.2.1.3 using the new.net Data Provider for Teradata v12.0. Database migration from Teradata Database v2r6.2.1.3 to Teradata Database v12.0 Benchmark and optimize the Project REAL solution for user load against Teradata Database v12.0 using the new.net Data Provider for Teradata v12.0. 2008 Microsoft Corporation. All rights reserved Page 5 of 30

The successful execution of the engagement resulted in the following benchmark results (which was based on SQL Server 2005 Analysis Services SP2): Ran various user loads up to 1000 users with response times less than 1.2 second for tests against the Teradata Database v2r6.2.1.3 engine using the.net Data Provider for Teradata v12.0. Ran various user loads up to 1000 users with response times less than 0.93 second for tests against the Teradata Database v12.0 engine using the.net Data Provider for Teradata v12.0. The hardware configuration is as follows: Analysis Services System: DELL two socket, quad-core system (eight total processor cores) with 1.86 GHz Intel Xeon and 4GB RAM 1 TB drive - The Analysis Services System does not require disk storage space for a ROLAP storage deployment. In a Teradata scenario, a best practice is to store the dimensions as MOLAP as a result an insignificant amount of disk space was used. Windows 2003 Server X64 SQL Server 2005 x64 SP2 (patch level 3175) Teradata System: 4-Node Teradata Database 5500 system running v2r6.2.1.3, which was also then upgraded to v12.0. Disks: RAID1 with 96 x 73GB disks Each node has: 12 AMPs and 2 PE's 4 CPU's @ 2.66Ghz 8GB RAM Linux/SUSE9 (64-bit) 2008 Microsoft Corporation. All rights reserved Page 6 of 30

Lab Environment Design and Configuration For the tests described within this paper, the lab environment leveraged the great work that was completed in July 2007 by the Chicago MTC and Teradata personnel. Instead of needing to setup the Teradata environment or collecting representative queries to benchmark against, the lab exercises could focus on functionality testing of v12.0, and benchmarking results of the new.net Data Provider for Teradata v12.0 (against v2r6.2.1.3 and v12.0). The following is a summary of the activities performed during the course of the original efforts in July 2007. The exact order and level of parallel effort may not be directly expressed in this summary. Database Preparation: Migrated sample data from the Project REAL kit into a Teradata system provided by Teradata. This was accomplished at the Microsoft Technology Center in Chicago through remote access to a Teradata system that was located at the Rancho Bernardo Center in San Diego. Used Teradata Administration tools to migrate the schema and SQL Server Integration Services to load the data. Identified a set of Reporting Services reports for this activity and with these reports, identified the tables and columns that were being queried. Migrated the Project REAL Star Schema into a Snow-Flake and tuned it for the Teradata System. Star schemas would perform well, but would depend on volume of data, cardinality and size of Teradata Database system. As well as, customer objectives and Aggregate Join Index (AJI) 1 design for analytics. For our activities, performed the following database tuning and considerations mentioned below: o o o o o o Created additional tables to support snowflake dimensions Ensured all dimension table primary keys are defined as unique utilizing the UNIQUE constraint, or the primary key is defined as a UNIQUE PRIMARY INDEX. Ensured all UNIQUE PRIMARY INDEX (UPI) columns are defined with NOT NULL Ensured (where possible) all primary and foreign keys are on the ID and not Name or Description columns. This will result in a smaller AJI, which means faster data access. Ensured single level dimensions have supporting reference/lookup/dimension table for optimal performance. Populated the snow-flake dimension tables with data (i.e. insert/select) 1 An AJI is an aggregated result set saved as an index in the database. It is transparent to an end-user and BI Administrators and will be used automatically by the Teradata optimizer when a query plan contains frequently made like columns and aggregates. Refer to the whitepaper in Appendix A. 2008 Microsoft Corporation. All rights reserved Page 7 of 30

Ensured Fact Table design is wide 2 (i.e., columns for each dimension and measure) and modified Primary Index (PI) as a composite key of all nonmeasure columns Collected statistics on all primary key/foreign key relationship columns. Implemented Referential Integrity (RI) on the primary key/foreign key columns. RI can be defined with the check (a.k.a., hard RI) or no check option (a.k.a., soft RI). A Snow-Flake schema with the tuning mentioned above will result in smaller AJI which can produce faster query response versus a Star schema which may result in less database tuning activities, but will require all columns of interest to be included in the AJI definition. Customers should do their own due diligence to determine the best schema design and database tuning based on their objectives. Redesigned the Project REAL Analysis Services Solution on the Snow-Flake schema while maintaining a consistent matching database structure to facilitate the query requirements of the Reporting Services reports. Increased the size of the fact table to 100GB. The Project REAL sample kit contained a sample set of rows spanning 1 year and it was replicated for additional years to grow the fact data. The following dimension map identifies the Analysis Services and Reporting Services reporting requirement for our AJI strategy for performance. TIME STORE ITEM DEPARTMENT BUYER REPLEN MEASURE STRATEGY Year_Desc Division Product Purchase Department Buyer Strategy Sales_Qty Vendor Index Alpha Type Qtr_Desc Region Subject Purchase Buyer Strategy Sale_Amt Vendor Name Month_Desc District Category Title Type. Week_Desc City Sub. Category Date_Value Store_ Desc Item_Desc. Built 1 broad 3 AJI to support/cover the above reporting requirement (in yellow): CREATE JOIN INDEX ms.aji_store_sales,no FALLBACK,CHECKSUM = DEFAULT AS SELECT COUNT(*)(FLOAT, NAMED CountStar ), ms.f.buyer_alpha, ms.a.sk_dept_id, ms.e.strategy_ind, ms.b.purch_vendor_num, ms.b.dept, ms.b.subcategory_code, ms.c.city, ms.d.calendar_year_desc, 2 A wide Fact table design has the measure and dimension columns in it with the dimension columns being used in the Composite Primary Key of the fact table. Refer to the whitepaper in Appendix A.. 3 The broad AJI constitutes as an index which covers all dimensions and levels of interest based on the aforementioned physical database stipulations to effectively address multi level dimensional hierarchy/model. Refer to the whitepaper mentioned earlier in this paper for more details. 2008 Microsoft Corporation. All rights reserved Page 8 of 30

FROM ms.d.calendar_qtr_desc, ms.d.calendar_month_desc, ms.d.calendar_week_desc, SUM(ms.a.Sales_Qty )(FLOAT, NAMED Sales_Qty ), SUM(ms.a.Sale_Amt )(FLOAT, NAMED Sale_Amt ) ms.vtbl_fact_store_sales a, ms.vtbl_dim_item b, ms.vtbl_dim_store c, ms.vtbl_dim_date d, ms.vtbl_dim_model_strategy e, ms.vtbl_dim_buyer f WHERE ((((ms.a.sk_item_id = ms.b.sk_item_id ) AND (ms.a.sk_store_id = ms.c.sk_store_id )) AND (ms.a.sk_date_id = ms.d.sk_date_id )) AND (ms.a.sk_model_strategy_id = ms.e.sk_model_strategy_id )) AND (ms.a.sk_buyer_id = ms.f.sk_buyer_id ) GROUP BY ms.f.buyer_alpha, ms.a.sk_dept_id, ms.e.strategy_ind, ms.b.purch_vendor_num, ms.b.dept, ms.b.subcategory_code, ms.c.city, ms.d.calendar_year_desc, ms.d.calendar_qtr_desc, ms.d.calendar_month_desc, ms.d.calendar_week_desc PRIMARY INDEX (Buyer_Alpha,SK_Dept_ID,Strategy_Ind,Purch_Vendor_Num, Dept,Subcategory_Code,City,Calendar_Year_Desc,Calendar_Qtr_Desc, Calendar_Month_Desc,Calendar_Week_Desc ); Test Scripts preparation: Captured the MDX from the Project REAL reporting solution. Used a random query generation tool (ASLoadSim) to create a large set of queries and each query contained random parameter values. Additional lab design and configuration for our latest tests included the following. Database Preparation: Once the benchmarking against the Teradata Database v2r6.2.1.3 environment using the.net Data Provider for Teradata v12.0 was completed, the database environment was upgraded to Teradata Database v12.0. Benchmarking: Created a Visual Studio.NET 2005 Team Suite test project (using ASLoadSim) to simulate various Analysis Services OLAP user loads in a stepped fashion, with each load running for 3 minutes once the maximum user load was reached. Ran increasing user loads up to 1000 users, first against Teradata Database v2r6.2.1.3 environment and then against the v12.0 environment. Optimized the benchmark testing methods to apply stress to the.net Data Provider for Teradata as much as possible. 2008 Microsoft Corporation. All rights reserved Page 9 of 30

Functionality Testing of Teradata v12.0 with Analysis Services 2005 The functionality tests successfully completed a series of Analysis Services development functions using a remote Analysis Services 2005 SP2 (build 3175) server connected to Teradata v12.0. These tests were conducted for initial functionality testing prior to the actual benchmarking runs. The functionality tests included the following areas: Functional Test 1 Ability to create an Analysis Services Data Source View (DSV) against Teradata Database v12.0 using the.net Data Provider for Teradata v12.0. Functional Test 2 Ability to process Analysis Services dimensions and cubes which were sourced from Teradata Database v12.0 using the.net Data Provider for Teradata v12.0. 2008 Microsoft Corporation. All rights reserved Page 10 of 30

Functional Test 3 Ability to query Analysis Services ROLAP cubes that are created against Teradata Database v12.0 using the.net Data Provider for Teradata v12.0. Functional Test 4 Ability to execute reports from Reporting Services 2005 which have their data source defined against Analysis Services cubes that have Teradata Database v12.0 data using the.net Data Provider for Teradata v12.0. 2008 Microsoft Corporation. All rights reserved Page 11 of 30

Functional Test 5 Ability to execute reports from Reporting Services Report Builder 2005, whereby the Report Model was generated from an Analysis Services 2005 ROLAP cube that has its data source connected to a Teradata Database v12.0 using the.net Data Provider for Teradata v12.0. 2008 Microsoft Corporation. All rights reserved Page 12 of 30

Performance Results Against Teradata Database v2r6.2.1.3 The following results are from the user stress tests against Teradata Database v2r6.2.1.3 using the.net Data Provider for Teradata v12.0. 100 User Load Test Results Figure 1 100 User Load Test Results from within Visual Studio.NET 2005 User Load 100 Warm-Up Period and Step Intervals Think Time 10 seconds for both 2 seconds Test Duration (step time plus constant time) 50 seconds +3 minutes Avg % Processor Time 91 Avg Response Time 0.1 seconds Queries Executing on TD 655 2008 Microsoft Corporation. All rights reserved Page 13 of 30

At a 100 concurrent users with a Think Time of 2 seconds, a total of 812 tests were executed out of which 655 queries executed on the Teradata server. The Analysis Services machine was running at 91 Avg % Processor Time and the Avg Query Response Time was well under a second (0.10 sec). 200 User Load Test Results Figure 2 200 User Load Test Results from within Visual Studio.NET 2005 User Load 200 Warm-Up Period and Step Intervals 10 seconds for both Think Time 2 seconds Test Duration (step time plus constant time) 100 seconds +3 minutes Avg % Processor Time 88 Avg Response Time 0.59 seconds Queries Executing on TD 1076 At 200 users, the number of total tests executed was 958 with 1076 tests resulting in queries executing at the Teradata server. As more tests were being executed, 2008 Microsoft Corporation. All rights reserved Page 14 of 30

Analysis Services Cache gets built up and most of the requests were being answered from the Cache. The Avg % CPU Time was 88 with an Avg Response Time of 0.59 seconds. 500 User Load Test Results Figure 3 500 User Load Test Results from within Visual Studio.NET 2005 User Load 500 Warm-Up Period and Step Intervals 10 seconds for both Think Time 2 seconds Test Duration (step time plus constant time) 250 seconds +3 minutes Avg % Processor Time 88 Avg Response Time 0.72 seconds Queries Executing on TD 1722 At 500 users, the number of total tests executed was 913 with 1722 queries executing at the Teradata server. The CPU reached its threshold during various points of the test and was at 88%. Avg Response Time of queries increased considerably from around 0.10 seconds for the 100 user test to 0.72 seconds. 2008 Microsoft Corporation. All rights reserved Page 15 of 30

1000 User Load Test Results Figure 4 1000 User Load Test Results from within Visual Studio.NET 2005 User Load 1000 Warm-Up Period and Step Intervals Think Time 10 seconds for both 2 seconds Test Duration (step time plus constant time) 500 seconds +3 minutes Avg % Processor Time 92 Avg Response Time Queries Executing on TD 4088 1.2 seconds At 1000 users, the number of total tests executed was 961 with 4088 queries executing at the Teradata server. The CPU reached its threshold during various points of the test and was at 92%. Avg Response Time of queries increased considerably from around 0.10 seconds for the 100 user test to 1.2 seconds. With increased workload of 1000 users, the CPU reaches threshold violations at 90%. 2008 Microsoft Corporation. All rights reserved Page 16 of 30

Comparison of User Load Test Results using Teradata Database v2r6.2.1.3 Figure 5 Comparison of Teradata Database v2r6.2.1.3 User Load Test Results 2008 Microsoft Corporation. All rights reserved Page 17 of 30

Performance Results Against Teradata Database v12.0 The following results are from the user stress tests against Teradata Database v12.0 using the.net Data Provider for Teradata v12.0. 100 User Load Test Results Figure 6 100 User Load Test Results from within Visual Studio.NET 2005 User Load 100 Warm-Up Period and Step Intervals Think Time 10 seconds for both 2 seconds Test Duration (step time plus constant time) 70 seconds +3 minutes Avg % Processor Time 92 Avg Response Time 0.10 seconds Queries Executing on TD 649 At a 100 concurrent users with a Think Time of 2 seconds, a total of 676 tests were executed out of which 649 queries executed on the Teradata server. The Analysis 2008 Microsoft Corporation. All rights reserved Page 18 of 30

Services machine was running at 92 Avg % Processor Time and the Avg Query Response Time was well under a second (0.10 sec). 200 User Load Test Results Figure 7 200 User Load Test Results from within Visual Studio.NET 2005 User Load 200 Warm-Up Period and Step Intervals 10 seconds for both Think Time 2 seconds Test Duration (step time plus constant time) 100 seconds +3 minutes Avg % Processor Time 92.67 Avg Response Time 0.26 seconds Queries Executing on TD 1091 At 200 users, the number of total tests executed was 719 with 1091 queries executing at the Teradata server. As more tests were being executed, Analysis Services Cache gets built up and most of the requests were being answered from the Cache. The Avg % CPU Time was 92.67 with an Avg Response Time of 0.26 seconds. 2008 Microsoft Corporation. All rights reserved Page 19 of 30

500 User Load Test Results Figure 8 500 User Load Test Results from within Visual Studio.NET 2005 User Load 500 Warm-Up Period and Step Intervals Think Time 10 seconds for both 2 seconds Test Duration (step time plus constant time) 250 seconds +3 minutes Avg % Processor Time 90.33 Avg Response Time Queries Executing on TD 1825 0.67 seconds At 500 users, the number of total tests executed was 700 with 1825 queries executing at the Teradata server. The CPU reached its threshold during various points of the test and was at 90.33%. Avg Response Time of queries increased considerably from around 0.26 seconds to 0.67 seconds. 2008 Microsoft Corporation. All rights reserved Page 20 of 30

1000 User Load Test Results Figure 9 1000 User Load Test Results from within Visual Studio.NET 2005 User Load 1000 Warm-Up Period and Step Intervals Think Time 10 seconds for both 2 seconds Test Duration (step time plus constant time) 500 seconds +3 minutes Avg % Processor Time 90 Avg Response Time Queries Executing on TD 3791 0.925 seconds With increased workload of 1000 users, the CPU reached many threshold violations at 90%, and three tests failed. 2008 Microsoft Corporation. All rights reserved Page 21 of 30

Comparison of User Load Test Results using Teradata Database v12.0 Figure 10 Comparison of Teradata Database v12.0 User Load Test Results 2008 Microsoft Corporation. All rights reserved Page 22 of 30

Conclusions The following are the drawn conclusions from this functional/performance test: 1. Analysis Services ROLAP cube to Teradata Database v12.0 outperformed all tests as compared to Teradata Database v2r6.2.1.3 with the.net Data Provider for Teradata v12.0. 2. Except for the outlier result of the v2r6.2.1.3 result for 200 users, the Teradata Database v12.0 RDBMS platform outperformed all other results. (note also that the original v2.6.2.1.3 test may be need to be rerun do to unknown simulation settings). 3. As the number of users reaches 500, the average response time increases from just around.10 second for all test configurations (at 100 users) to.90 seconds in the worst case for 500 users. Therefore, a sweet-spot for concurrent user threshold (based upon average response time) would be around 500 users for consistent sub-second response time using same or similar hardware. 4. It also noteworthy to mention that during the tests (against all environments), Analysis Services began answering queries via its cache at some point and thus did not query the Teradata database after that point. This is a good occurrence for query performance usually, but with limited memory and fully utilized CPU resources during our tests, additional user load and/or improved average response times may have been observed otherwise. 5. For practical reasons, a more precise benchmarking test environment would have separated the SQL Server RDBMS which was used to record test results from the Analysis Services service. Add to that the actual testing tool also ran from this same server in many instances it was noticed that between the SQL Server relational database and the testing tool, CPU resources would exceed 75%, leaving only 25% to Analysis Services. 2008 Microsoft Corporation. All rights reserved Page 23 of 30

Figure 11 Comparison of Teradata Database v2r6.2.1.3 to v12.0 User Load Test Results Follow-On Performance Testing Efforts As this paper describes the results from functionality and basic performance testing of SQL Server 2005 Analysis Services leveraging Teradata sources, there are additional high performance stress tests that would provide greater insight into throughput potential of the.net Data Provider for Teradata and Analysis Services. A future lab environment effort would use the above documented key learnings and optimizations to execute high performance stress tests. No date and/or team have been set yet for this potential phase 2 round of tests, but is expected within six months of this paper s publish date. 2008 Microsoft Corporation. All rights reserved Page 24 of 30

Appendix A: Applicable Links The following links are provided as additional references to materials which are relevant to the discussions. Analysis Services section of the SQL Server Developer Center on MSDN http://msdn2.microsoft.com/en-us/sql/aa336310.aspx Whitepaper/Code SQL Server Analysis Services 2005 Load Testing Best Practices This paper describes how you can leverage Visual Studio.NET to create stress tests against Microsoft SQL Server Analysis Services using the CodePlex open source ASLoadSim tool. The link below will direct you to the ASLoadSim tools and accompanying white paper on its use. http://www.codeplex.com/sqlsrvanalysissrvcs Whitepaper - Improve your OLAP Environment with Microsoft and Teradata This paper describes how you can leverage Microsoft SQL Server Analysis Services and Teradata technologies to improve your analytical OLAP application environment, specifically relational (ROLAP) type solution. http://www.microsoft.com/downloads/details.aspx?familyid=8f38fafb-a7c3-4cc1- A059-7A7A732058A2&displaylang=en SQL Server 2005 Books Online Download an updated version of Books Online for Microsoft SQL Server 2005, the primary documentation for SQL Server 2005. http://technet.microsoft.com/en-us/sqlserver/bb428874.aspx 2008 Microsoft Corporation. All rights reserved Page 25 of 30

Appendix B: Analysis Services Stress Testing Toolset Microsoft Visual Studio 2005 Team Edition for Software Testers is an easy-touse yet comprehensive testing technology for web and performance testers. Through the use of the built-in performance testing tools of the software, the following benefits are delivered: Developers and testers can simulate production loads and diagnose performance issues quickly in their testing labs and pre-production environments. Prescriptive guidance increases tester productivity through pre-packaged counters, pre-set thresholds, and built-in knowledge of server behavior. Leverage pre-built views of load statistics and their effect on the servers under test. Enable your entire team to identify bottlenecks early in the software development lifecycle. Flexible load-agent licensing helps more team members get access to testing tools. Through the use of this architecture from the VS.NET 2005 Team Edition Software Tester offering, a new performance test template was created by the Microsoft SQL Server Product Team to track query performance of Analysis Services through a simulated concurrent user workload. The add-in template actually is comprised of two parts, one of which lives outside the VS.NET 2005 Team Test environment. The two parts of this simulation tool are: Analysis Services Query Generator (ASQueryGen for short, and is the portion of the tool that lives outside of Visual Studio), and Analysis Services Load Simulator (ASLoadSim for short). The following two sections describes these two VS.NET 2005 Team Edition tester template tools, how they are configured, and how they were used for this Teradata-SSAS end-user query stress test. Query Generation through ASQueryGen The ASQueryGen tool is a command line program that accepts an XML configuration file (partially shown below), that when executed generates one to how ever many client query files. Each query file will have the same number of MDX queries generated, but if/when the token values are used with the query tag statements, the AS QueryGen tool will replace the token with randomized dimension values as specified by the token, thus creating slightly different MDX queries to run through SSAS. <Queries> <Query>/* 1 : 1 */ WITH MEMBER [Measures].[cPct of Sbj Qty per Vndr] AS '([Measures].[Sale Amt])/([Measures].[Sale Amt], [Item].[Purchase Vendor].[All])' MEMBER [Measures].[cPct of Subj Qty per Curr Vendor] AS '([Measures].[Sale Amt])/([Measures].[Sale Amt], [Item].[Purchase Vendor].currentmember, [Item].[By Dept].[All])' SELECT NON EMPTY { [Measures].[Sale Amt], [Measures].[cPct of Sbj Qty per Vndr], [Measures].[cPct of Subj Qty per Curr Vendor] } ON COLUMNS, NON EMPTY { ([Item].[By Purchase Vendor].[Purchase Vendor].ALLMEMBERS * [Item].[By Dept].[Dept].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]) ) ON COLUMNS FROM ( SELECT ( { [Item].[By Dept].[Dept].&[JUVENILE], [Item].[By Dept].[Dept].&[HARDCOVER], [Item].[By 2008 Microsoft Corporation. All rights reserved Page 26 of 30

Dept].[Dept].&[MASS MARKET], [Item].[By Dept].[Dept].&[TRADE PAPERBACK] } ) ON COLUMNS FROM [REAL Warehouse])) WHERE ( IIF( ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]).Count = 1, ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]), [Time].[Fiscal].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> <Query>/* 1 : 2 */ SELECT NON EMPTY { [Measures].[cAvg Retail] } ON COLUMNS, NON EMPTY { ([Time].[Calendar].[Calendar Month].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Store].[District].&[2.8E1]) ) ON COLUMNS FROM ( SELECT ( ([Time].[Calendar].[Calendar Year].&[2008]) ) ON COLUMNS FROM [REAL Warehouse])) WHERE ( IIF( ([Store].[District].&[2.8E1]).Count = 1, ([Store].[District].&[2.8E1]), [Store].[District].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> <Query>/* 1 : 3 */ SELECT NON EMPTY { [Measures].[Sales Qty], [Measures].[cAvg Retail] } ON COLUMNS, NON EMPTY { ([Item].[Item].[Item].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Time].[Calendar].[Calendar Month].&[6]&[2008]) ) ON COLUMNS FROM ( SELECT ( ([Item].[Category].&[174D]) ) ON COLUMNS FROM ( SELECT ( ([Store].[District].&[2.8E1]) ) ON COLUMNS FROM [REAL Warehouse]))) WHERE ( IIF( ([Store].[District].&[2.8E1]).Count = 1, ([Store].[District].&[2.8E1]), [Store].[District].currentmember ), IIF( ([Item].[Category].&[174D]).Count = 1, ([Item].[Category].&[174D]), [Item].[Category].currentmember ), IIF( ([Time].[Calendar].[Calendar Month].&[6]&[2008]).Count = 1, ([Time].[Calendar].[Calendar Month].&[6]&[2008]), [Time].[Calendar].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> <Query>/* 1 : 4 */ WITH MEMBER [Measures].[cPct of Sbj Qty per Vndr] AS '([Measures].[Sale Amt])/([Measures].[Sale Amt], [Item].[Purchase Vendor].[All])' SELECT NON EMPTY { [Measures].[Sale Amt], [Measures].[cPct of Sbj Qty per Vndr] } ON COLUMNS, NON EMPTY { ([Item].[By Purchase Vendor].[Purchase Vendor].ALLMEMBERS * [Item].[By Dept].[Dept].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]) ) ON COLUMNS FROM ( SELECT ( { [Item].[By Dept].[Dept].&[JUVENILE], [Item].[By Dept].[Dept].&[HARDCOVER], [Item].[By Dept].[Dept].&[MASS MARKET], [Item].[By Dept].[Dept].&[TRADE PAPERBACK] } ) ON COLUMNS FROM [REAL Warehouse])) WHERE ( IIF( ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]).Count = 1, ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]), [Time].[Fiscal].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> <Query>/* 1 : 5 */ WITH MEMBER [Measures].[Pct of Qty Vendor] AS '[Measures].[Sales Qty]/([Measures].[Sales Qty], [Item].[Purchase Vendor].[All])' SELECT NON EMPTY { [Measures].[Sale Amt], [Measures].[Pct of Qty Vendor], [Measures].[Sales Qty] } ON COLUMNS, NON EMPTY { ([Item].[By Purchase Vendor].[Purchase Vendor].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Time].[Fiscal].[Fiscal Week].&[20]&[2008]) ) ON COLUMNS FROM ( SELECT ( ([Store].[District].&[2.8E1]) ) ON COLUMNS FROM [REAL Warehouse])) WHERE ( IIF( ([Store].[District].&[2.8E1]).Count = 1, ([Store].[District].&[2.8E1]), [Store].[District].currentmember ), IIF( ([Time].[Fiscal].[Fiscal Week].&[20]&[2008]).Count = 1, ([Time].[Fiscal].[Fiscal Week].&[20]&[2008]), [Time].[Fiscal].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> <Query>/* 1 : 6 */ WITH MEMBER [Measures].[Pct of Qty Vendor] AS '[Measures].[Sales Qty]/([Measures].[Sales Qty], [Item].[All])' SELECT NON EMPTY { [Measures].[Sale Amt], [Measures].[Pct of Qty Vendor], [Measures].[Sales Qty] } ON COLUMNS, NON EMPTY { ([Item].[By Category].[Subject].ALLMEMBERS * [Item].[By Purchase Vendor].[Purchase Vendor].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Time].[Fiscal].[Fiscal Week].&[20]&[2008]) ) ON COLUMNS FROM ( SELECT ( ([Store].[District].&[2.8E1]) ) ON COLUMNS FROM [REAL Warehouse])) WHERE ( IIF( ([Store].[District].&[2.8E1]).Count = 1, ([Store].[District].&[2.8E1]), [Store].[Geography].currentmember ), IIF( ([Time].[Fiscal].[Fiscal Week].&[20]&[2008]).Count = 1, ([Time].[Fiscal].[Fiscal Week].&[20]&[2008]), [Time].[Fiscal].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> <Query>/* 1 : 7 */ SELECT NON EMPTY { [Measures].[Sale Amt] } ON COLUMNS, NON EMPTY { ([Item].[By Category].[Subject].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Item].[Dept].&[HARDCOVER]) ) ON COLUMNS FROM ( SELECT ( ([Store].[District].&[2.8E1]) ) ON COLUMNS FROM ( SELECT ( ([Time].[Fiscal].[Fiscal Qtr].&[2]&[2008]) ) ON COLUMNS FROM [REAL Warehouse]))) WHERE ( IIF( ([Time].[Fiscal].[Fiscal 2008 Microsoft Corporation. All rights reserved Page 27 of 30

Qtr].&[2]&[2008]).Count = 1, ([Time].[Fiscal].[Fiscal Qtr].&[2]&[2008]), [Time].[Fiscal].currentmember ), IIF( ([Store].[District].&[2.8E1]).Count = 1, ([Store].[District].&[2.8E1]), [Store].[District].currentmember ), IIF( ([Item].[Dept].&[HARDCOVER]).Count = 1, ([Item].[Dept].&[HARDCOVER]), [Item].[Dept].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> <Query>/* 1 : 8 */ SELECT NON EMPTY { [Measures].[Sales Qty] } ON COLUMNS, NON EMPTY { ([Item].[Subject].[Subject].ALLMEMBERS ) } DIMENSION PROPERTIES MEMBER_CAPTION, MEMBER_UNIQUE_NAME ON ROWS FROM ( SELECT ( ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]) ) ON COLUMNS FROM ( SELECT ( ([Store].[District].&[2.8E1]) ) ON COLUMNS FROM [REAL Warehouse])) WHERE ( IIF( ([Store].[District].&[2.8E1]).Count = 1, ([Store].[District].&[2.8E1]), [Store].[District].currentmember ), IIF( ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]).Count = 1, ([Time].[Fiscal].[Fiscal Period].&[5]&[2008]), [Time].[Fiscal].currentmember ) ) CELL PROPERTIES VALUE, BACK_COLOR, FORE_COLOR, FORMATTED_VALUE, FORMAT_STRING, FONT_NAME, FONT_SIZE, FONT_FLAGS</Query> </Queries> In our creation of the XML configuration file, we set the number of client files to be created at 1000, which corresponds with the maximum number of clients used in a single SSAS database baseline cube query stress run. Client Simulation through ASLoadSim Once the client files are created, a Load Simulation project within Visual Studio 2005 Team Edition can be established. Since the AS LoadSim tool leverages the Software Tester environment, most of the steps to create the load simulation test are driven directly off the new test wizard. For example, the wizard helped to create the test scenario, the load sets (aka number of simultaneous clients), duration of the test, and warm-up period. Depicted below is the loadtest project after the wizard information was completed. Figure 12. VS.NET 2005 Team Edition ASLoadSim Project 2008 Microsoft Corporation. All rights reserved Page 28 of 30

One manual alteration to the ASLoadSim project has to be made at this point, and this is to add the specific location for where the project is to pick up its XML configuration file. This XML configuration file for the AS LoadSim project contains important information such as which SSAS database to connect to, which SQL Server RDBMS to store the test results into, what client files to execute, how many of them, what think time mix/maximums should be used, and whether or not to randomize the MDX queries that are executed against the SSAS database. Below is representation of the AS LoadSim XML configuration file which was used. <Configuration> <Server>localhost</Server> <Database>REAL Warehouse Sample TD</Database> <ClientFileName>C:\REAL_TD_ROLAP_TESTS\QueryGenData\Queries\Query.xml</C lientfilename> <ClientFileStartingNumber>1</ClientFileStartingNumber> <NumOfClientFiles>200</NumOfClientFiles> <ThinkTimeMin>2</ThinkTimeMin> <ThinkTimeMax>2</ThinkTimeMax> <DataAccessMethod>Random</DataAccessMethod> <LogToServer>localhost</LogToServer> <LogToDB>LoadTest</LogToDB> </Configuration> Figure 13. XML Configuration File for AS LoadSim Project Stress Test Reports and Counters Used While the stress tests were being run, several key counters were being recorded via Windows System Monitor (Perfmon). These included such counters as processor utilization, total memory consumption, memory usage by Analysis Services, Analysis Services connections, etc. A key part of the ASLoadSim toolset is that all of the characteristics of each stress load, and more importantly, the results of those stress load tests are recorded into a SQL Server 2005 relational database for future analysis and reporting. Included with ASLoadSim are several SQL Server Reporting Services reports that help identify lead test runs, and then key result statistics for specific runs, such as average response time, number of queries executed, and errors (fatal queries/connections). 2008 Microsoft Corporation. All rights reserved Page 29 of 30

For more information: http://www.microsoft.com/isv/teradata 2008 Microsoft Corporation. All rights reserved Page 30 of 30