IBM Software Business Analytics Cognos Business Intelligence IBM Cognos 10: Enhancing query processing performance for IBM Netezza appliances
2 IBM Cognos 10: Enhancing query processing performance for IBM Netezza appliances Contents 2 Introduction 2 About the tests 2 Scalability 3 Performance 7 Conclusion Introduction With the release of IBM Cognos Business Intelligence V10.1.1, Dynamic Query mode extends its support to relational databases including IBM Netezza, IBM DB2, Teradata, Oracle and Microsoft SQL Server. Cognos Business Intelligence Dynamic Query is an optional mode included with the Cognos Business Intelligence query service. It enables 64-bit, in-memory calculations and intelligent query processing to enhance query performance against supported OLAP and relational data sources. IBM Netezza solutions are data warehouse appliances that transform the data warehouse and analytics landscape with a platform built for extreme, industry-leading performance with appliance simplicity. Built for high speed analytics, its strength comes not from the most powerful and expensive components but from having the right components assembled and working together. Massively parallel processing streams combine multi-core microprocessors with Netezza Field Programmable Gate Arrays and Accelerated Streaming Technology engines for performance that in many cases exceeds expectations. Because it is an appliance, you can see phenomenal results almost immediately, with no indexing or tuning required. Appliance simplicity extends to application development, enabling your organization to innovate rapidly and bring high performance analytics to the widest range of users and processes. This paper describes the performance and scalability testing results of combining Cognos Business Intelligence V10.1.1 and a Netezza appliance. About the tests The architecture and configuration of Cognos Business Intelligence V10.1.1 for the tests included: One web gateway One report server One content manager Reporting packages using Dynamic Query Mode (DQM) Dedicated 64 GB Java Virtual Machine for the Dynamic Query Engine The Cognos Framework Manager modeler used the Cognos dimensionally modeled relational (DMR) modeling technique that presents relational data sources as OLAP cubes. All reports were built using IBM Cognos Business Insight Advanced and Cognos Reporting capabilities with no further query tuning. The architecture and configuration for the Netezza data warehouse included a test database based on the TPC-DS test database with sales-fact table sizes of 800,000,000 and 2,800,000,000 and 8,600,000,000 rows of fact data. The database used standard Netezza tuning. The data was partitioned by date, zone maps were set up and the data was dimensionally organized by item, store and date. Scalability Cognos Business Intelligence has demonstrated an enterpriselevel scalability that meets the diverse and complex business intelligence needs of global organizations. In the performance and scalability tests, Cognos Business Intelligence V10.1.1 with Netezza demonstrated improved user scalability in comparison
IBM Software Group 3 to previous versions of Cognos Business Intelligence, which did not feature the optimizations that take advantage of Netezza performance capabilities. In particular, user load testing showed a reduced number of data source connections and workload on the Netezza warehouse. In fact, as the number of users increased, the load on the Netezza server tended to plateau and decrease because of increased utilization and the sharing of cached results (Figure 1). In a typical Cognos Business Intelligence environment, the likelihood of users asking similar questions increases as the number of users rises. With the advanced in-memory cache introduced by Cognos Dynamic Query, Cognos Business Intelligence V10.1.1 puts less strain on your data warehouse, leaving it free to serve other business applications. Performance The performance testing for Cognos Business Intelligence V10.1.1 with Netezza demonstrated improvements in the performance of OLAP databases, advanced analytics applications, analysis solutions, reporting and dashboards using Cognos Dynamic Query mode over previous versions of Cognos Business Intelligence using Compatible Query mode. Faster performance with relational database sources Using DMR, the Cognos Dynamic Query mode provides significant performance improvements because the Cognos Dynamic Query cache can be used for the same or similar queries from different users. That means if a query result is available fully or partially in the memory cache, the query does not have to go back to the database and that means faster 14000 12000 10000 8000 6000 4000 50 Users Prev Version 50 Users V10.1.1 25 Users Prev Version 25 Users V10.1.1 2000 0 <1 sec 1-2 sec 2-3 sec 3-5 sec Figure 1: Query response time in a multiuser Cognos Business Intelligence environment
4 IBM Cognos 10: Enhancing query processing performance for IBM Netezza appliances response times. The Cognos Dynamic Query cache also can be used in an analysis scenario. For example, to drill down into a report, an already queried aggregate does not need to be calculated again or calculations based on data already in the cache can occur in memory (for example, percentage of total calculation). Faster analytics To illustrate analytics performance, IBM tested a set of typical business scenarios often seen in the retail industry but also common in other industries. In the retail industry, measures are typically viewed by relative time periods with calculations such as week to date (WTD), year to date (YTD) and previous year, week or day. Table 1 shows sales by products and various time periods. When IBM measured the performance of reports that contain similar analytics functions, the results exceeded expectations. For test purposes, IBM ran reports that featured a WTD and a YTD calculation on fact tables with 800,000,000 rows and 2,800,000,000 rows. Figures 2 and 3 show that Cognos Dynamic Query was more than twice as fast in the report execution and also at least 60 percent faster in the SQL execution on the Netezza system compared to when Compatible Query mode was used. As data sizes grow from 800,000,000 fact rows to 2,800,000,000 fact rows, Cognos Dynamic Query mode scales linearly. Table 1: Sales by product categories and different time periods Sales Prev. Week Sunday Monday WTD YTD % of Total 5364 Books 18.17K 2.50K 2.66K 5.16K 444.73K 0.26% 18.38K Unknown 688.33K 97.50K 97.01K 194.51K 16,641.98K 9.79% 683.50K Men 689.69K 99.21K 97.48K 196.69K 16.748.93K 9.94% 694.04K Children 706.21K 102.01K 99.99K 202.00K 17,104.33K 10.13% 707.70K Shoes 698.83K 99.19K 99.69K 198.88K 16,887.06K 10.01% 698.96K Music 697.87K 101.02K 99.36K 200.38K 16,847.53K 9.99% 697.98K Jewelry 702.69K 99.43K 99.43K 199.67K 17,005.66K 10.08% 703.79K Home 701.92K 99.29K 100.24K 197.40K 16,930.32K 9.97% 696.48K Sports 702.75K 99.96K 98.11K 199.90K 16,963.73K 10.06% 702.61K Books 701.58K 100.70K 99.95K 199.97K 16,945.87K 10.03% 700.15K Electronics 677.78K 96.70K 99.27K 193.83K 16,432.57K 9.73% 679.73K All categories 6,985.83K 997.50K 990.90K 1,998.40K 168,952.71K 6,983.34K
IBM Software Group 5 60 WTD Response time in sec 50 40 30 20 10 0 800M Report 2.8B Report 800M SQL 2.8B SQL V10.1.1 9 14 8 8 Prev. Version 22 50 14 42 Figure 2: Cognos Dynamic Query performance for sample analytics with a WTD calculation compared to Compatible Query mode 60 YTD Response times in sec 50 40 30 20 10 0 800M Report 2.8B Report 800M SQL 2.8B SQL V10.1.1 17 27 12 23 Prev. Version 22 50 14 41 Figure 3: Cognos Dynamic Query performance for sample analytics with a YTD calculation compared to Compatible Query mode
6 IBM Cognos 10: Enhancing query processing performance for IBM Netezza appliances Faster analysis Analysis is the process of analyzing data, and it typically helps users understand why a measure is on or off target. Analysis most often uses a dimensional model so that users can drill into data and calculate cross-dimensionally. Cognos Business Intelligence V10.1.1 with Netezza is faster for analysis because it takes less time to drill up and drill down. Already queried aggregates can be fetched from the cache and analytical calculations that the users add can occur in memory if the underlying data is completely or partially in cache. Reports with analysis functions run at least 40 percent faster with the Cognos Business Intelligence V10.1.1 optimizations, and drilling down is more than twice as fast (Figure 4). Cognos Query Manager caches the aggregate so that it does not have to be recalculated afterwards. Drilling up is instantaneous because typically all the data is already in the cache. Adding analytical functions is also significantly faster (Figure 5). For example, if you start with a report that features Sunday, Monday and Tuesday on the column and you want to add a WTD calculation, all the data for that calculation is already in memory and the calculation can be completed in memory. Faster reporting and dashboards Users creating reports and dashboards with the combination of Cognos Business Intelligence V10.1.1 and Netezza benefit from the fact that data elements often overlap dashboard objects and reports. Cognos Dynamic Query provides a shared % Performance Improvement 400% 300% 200% 100% 0% -100% Query 1 Query 1 Query 2 Query 2 Query 2 YTD Query 2 YTD Query 3 Query 3 Report -50% -17% 127% 271% 41% 160% 160% 350% SQL -60% 0% 63% 333% 0% 86% 100% 250% Figure 4: Analysis performance improvement with Dynamic Query mode
IBM Software Group 7 user cache. Depending on the security settings of that cache, any report can take advantage of complete or partial result sets that are available. If the database uses row-level security, however, the cache will not be shared with multiple users, so settings are critical. Conclusion Cognos Business Intelligence V10.1.1 scales linearly and performs faster on Netezza because its query engine optimization pushes analytical functions down to the Netezza database enabling it to take full advantage of Netezza performance capabilities. The result is faster-performing queries because less data needs to be returned from the Netezza appliance. The combination of Cognos Business Intelligence V10.1.1 and Netezza appliances creates a new frontier in advanced analytics, with the ability to carry out monumental processing challenges with blazing speed, without barriers or compromises. For your organization, it means the best intelligence for all who need it even as demands for information escalate. 30 25 20 15 10 5 0 Report SQL Report SQL Report SQL Report SQL Report SQL Report SQL Report SQL Report SQL Query 1 Query 1 drill Music Query 2 WTD Query 2 WTD Drill Men Query 2 YTD Query 2 YTD Drill Men Query 3 WTD % Total Query 3 WTD % Total drill Men V10.1.1 10 5 6 3 11 8 7 3 17 14 10 7 10 7 6 4 Prev Version 5 2 5 3 25 13 26 13 24 14 26 13 26 14 27 14 Figure 5: Analytical function performance with Dynamic Query mode compared to Compatible Query mode
About Netezza Netezza, an IBM Company, is a global leader in data warehouse, analytic and monitoring appliances that dramatically simplify high-performance analytics across an extended enterprise. Netezza technology enables organizations to process enormous amounts of captured data at exceptional speed, providing a significant competitive and operational advantage in today s data-intensive industries, including digital media, energy, financial services, government, health and life sciences, retail and telecommunications. For more information about Netezza, visit ibm.com/facts About IBM Business Analytics IBM Business Analytics software delivers actionable insights decision-makers need to achieve better business performance. IBM offers a comprehensive, unified portfolio of business intelligence, predictive and advanced analytics, financial performance and strategy management, governance, risk and compliance and analytic applications. With IBM software, companies can spot trends, patterns and anomalies, compare what if scenarios, predict potential threats and opportunities, identify and manage key business risks and plan, budget and forecast resources. With these deep analytic capabilities, our customers around the world can better understand, anticipate and shape business outcomes. Copyright IBM Corporation 2011 IBM Software Group Route 100 Somers, NY 10589 U.S.A. Produced in the United States of America October 2011 All Rights Reserved IBM, the IBM logo, ibm.com, Cognos, and DB2 are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/ copytrade.shtml. Netezza is a trademark or registered trademark of Netezza Corporation, an IBM Company. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Please Recycle For more information For further information or to reach a representative please visit ibm.com/analytics. Request a call To request a call or to ask a question, go to ibm.com/businessanalytics/contactus. An IBM representative will respond to your inquiry within two business days. YTW03202-CAEN-02