NetIQ and LECCOTECH: Integrated Solutions for Optimal SQL Server Performance October 2003

Transcription

1 NetIQ and LECCOTECH: Integrated Solutions for Optimal SQL Server Performance October 2003 Contents Introduction... 1 Traditional Methods of Monitoring and Tuning... 1 The NetIQ and LECCOTECH Solution... 1 Conclusion... 9 Many SQL Server performance issues are directly related to application performance, which means that the efficiency of the SQL statements being run needs to be scrutinized. NetIQ and LECCOTECH provide an integrated solution to help database administrators (DBAs) and developers scrutinize and resolve performance issues. Together, NetIQ s DiagnosticManager for SQL Server and LECCOTECH s LECCO SQL Expert solution deliver a comprehensive solution enabling rapid diagnosis of performance problems and automating the process of SQL optimization. DBAs can work with DiagnosticManager to locate problematic SQL statements and then employ LECCO SQL Expert to provide SQL alternatives that will overcome the deficiencies of currently used statements. This white paper outlines the step-by-step approach DBAs and application developers can use to identify bottlenecks and improve performance in a SQL Server environment using DiagnosticManager and LECCO SQL Expert.

2 THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT ARE FURNISHED UNDER AND ARE SUBJECT TO THE TERMS OF A LICENSE AGREEMENT OR A NON-DISCLOSURE AGREEMENT. EXCEPT AS EXPRESSLY SET FORTH IN SUCH LICENSE AGREEMENT OR NON-DISCLOSURE AGREEMENT, NETIQ CORPORATION PROVIDES THIS DOCUMENT AND THE SOFTWARE DESCRIBED IN THIS DOCUMENT AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. SOME STATES DO NOT ALLOW DISCLAIMERS OF EXPRESS OR IMPLIED WARRANTIES IN CERTAIN TRANSACTIONS; THEREFORE, THIS STATEMENT MAY NOT APPLY TO YOU. This document and the software described in this document may not be lent, sold, or given away without the prior written permission of NetIQ Corporation, except as otherwise permitted by law. Except as expressly set forth in such license agreement or non-disclosure agreement, no part of this document or the software described in this document may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, or otherwise, without the prior written consent of NetIQ Corporation. Some companies, names, and data in this document are used for illustration purposes and may not represent real companies, individuals, or data. This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time NetIQ Corporation, all rights reserved. U.S. Government Restricted Rights: If the software and documentation are being acquired by or on behalf of the U.S. Government or by a U.S. Government prime contractor or subcontractor (at any tier), in accordance with 48 C.F.R (for Department of Defense (DOD) acquisitions) and 48 C.F.R and (for non-dod acquisitions), the government s rights in the software and documentation, including its rights to use, modify, reproduce, release, perform, display or disclose the software or documentation, will be subject in all respects to the commercial license rights and restrictions provided in the license agreement. Check Point, FireWall-1, and Provider-1 are trademarks or registered trademarks of Check Point Software Technologies Ltd. ActiveAgent, ActiveAnalytics, ActiveAudit, ActiveKnowledge, ActiveReporting, ADcheck, AppAnalyzer, Application Scanner, AppManager, AuditTrack, AutoSync, Chariot, ClusterTrends, CommerceTrends, Configuration Assessor, the cube logo design, DBTrends, DiagnosticManager, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, End2End, Exchange Administrator, Exchange Migrator, Extended Management Pack, FastTrends, File Security Administrator, Firewall Appliance Analyzer, Firewall Reporting Center, Firewall Suite, Ganymede, the Ganymede logo, Ganymede Software, Group Policy Administrator, Intergreat, Knowledge Scripts, Migrate.Monitor.Manage, Mission Critical Software, Mission Critical Software for E-Business, the Mission Critical Software logo, MP3check, NetIQ, the NetIQ logo, the NetIQ Partner Network design, NetWare Migrator, OnePoint, the OnePoint logo, Operations Manager, PentaSafe, PSAudit, PSDetect, PSPasswordManager, PSSecure, Qcheck, Security Analyzer, Security Manager, Server Consolidator, SQLcheck, VigilEnt, Visitor Mean Business, Vivinet, W logo, WebTrends, WebTrends Analysis Suite, WebTrends for Content Management Systems, WebTrends Intelligence Suite, WebTrends Live, WebTrends Log Analyzer, WebTrends Network, WebTrends OLAP Manager, WebTrends Report Designer, WebTrends Reporting Center, WebTrends Warehouse, Work Smarter, WWWorld, and XMP are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States and other jurisdictions. All other company and product names mentioned are used only for identification purposes and may be trademarks or registered trademarks of their respective companies.

3 Introduction Monitoring and locating performance issues in a Microsoft SQL Server environment are tasks that every administrator must perform to satisfy end users and adhere to corporate requirements. Many performance issues are directly related to application performance and the efficiency of the application s SQL statements needs to be scrutinized. SQL Server offers limited capabilities in this area. Relying on the Microsoft SQL Server tools, database structure changes, and even developer skill sets will not solve short-term issues with database and application performance and only add to long term and potential problems. In addition, the many hours that are lost due to performance issues can greatly affect the bottom line. Performance tuning focuses on the analysis of SQL statements that are either embedded within, or called by applications. This uncovers the root cause of performance issues the SQL statements that require excessive system resources and time to run. Through identification and tuning of these SQL statements, system performance can be improved enough to eliminate the need for more expensive hardware and SQL Server tuning experts. SQL Server offers several choices to perform an identification and analysis of the statements that could be causing performance issues, but each does have limits that should be considered. Traditional Methods of Monitoring and Tuning The Enterprise Manager and SQL Server Profiler tools provided by Microsoft offer limited functionality that expert DBAs can use to assist in identifying performance problems and aid in identifying problem SQL statements in a given environment. However, there are no graphical tools that will actually display the overall health of SQL Server databases and allow one to easily pinpoint areas of concern. By itself, the Profiler allows one to capture or trace problem SQL statements to a system file or a database table. The trace table or file will keep track of SQL statements falling into a limited range of predefined problem areas, or filters. These statements can be captured in a trace, but only one problem or event can be investigated per trace. Another concern is that the SQL Server Profiler can use significant system resources, causing a performance problem itself. Therefore, the more events traced, the more system overhead will be required. The NetIQ and LECCOTECH Solution NetIQ s DiagnosticManager for SQL Server overcomes the limitations of the Microsoft tools by enabling DBAs to rapidly diagnose and correct performance problems. Fast, accurate resolution is vital when the enterprise depends on database performance and availability to provide customer service. NetIQ and LECCOTECH: Integrated Solutions for Optimal SQL Server Performance 1

4 DiagnosticManager will automatically locate problem SQL statements tuning those SQL statements is the next logical step. To tune SQL, we first need to understand how the SQL Server database optimizer processes SQL statements. The SQL Server database optimizer determines the execution plan for all SQL statements. The optimizer uses a step by-step method to determine the execution plan: parsing the statement, checking indexes for search arguments, deriving the execution plan, and estimating the cost of execution. The optimizer has only milliseconds to choose an execution plan to avoid affecting user response time. Given the constraints that are placed on system resources, the optimizer may not have adequate resources allotted to it for performing this task in a thorough manner, as it would only contribute to an overall performance issue itself. Compounding this shortcoming of the optimizer, the complexity of a query, database structure, and volume changes can also limit its effectiveness. Given the nature of today s OLTP and DSS environments, one can imagine the challenges that are not met by relying solely on the database optimizer to dictate query performance. For a developer or DBA to work around any shortcomings of the database optimizer and approach SQL tuning manually requires not only a strong skill in SQL development, but also an abundance of time. The first step in this process is to identify SQL statements that are causing bottlenecks. Once these statements are identified, the next step requires one to be able to produce as many variations of the problematic SQL statement as possible, with execution plans that are unique to each newly produced alternative. After these alternatives are produced, they must be executed to determine their actual runtime. This process is very time consuming, even for the most capable developer or DBA. To address the challenge of providing an end-to-end database application tuning process, NetIQ and LECCOTECH have partnered to provide a complete performance tuning solution. Problematic SQL statements are automatically identified with DiagnosticManager and then sent to LECCO SQL Expert for optimization. This seamless integration of monitoring and tuning allows developers and DBAs alike to streamline performance initiatives and overcome any limitations of not only SQL Server, but also any limitations within the development, quality assurance, and production environments with regard to SQL performance. The LECCO SQL Expert product, integrated with DiagnosticManager, offers a total solution in SQL performance bottleneck detection and optimization. Once SQL performance bottlenecks are identified with DiagnosticManager, users can simply right click in DiagnosticManager s Worst-Performing SQL screen and send problematic SQL statements to LECCO SQL Expert for optimization. As stated previously, application performance issues are primarily the result of problematic SQL statements. The DiagnosticManager and LECCO SQL Expert solution focuses on this aspect of performance and provides a comprehensive solution that makes it possible to optimize application performance with a simple three-step process: 1. Locate problematic SQL statements with DiagnosticManager 2. Rewrite SQL statements with LECCO SQL Expert 3. Benchmark and test SQL alternatives within LECCO SQL Expert Step 1: DiagnosticManager: Locate Problem SQL Statements DiagnosticManager collects and sorts data, allowing administrators to view information for one server or for the entire environment (Figure 1), making it easy to identify the worst performing SQL statements. Data collection is entirely automated with a background service an administrator can review information across dozens of servers from one console or drill deep into any server to spot problem areas. 2 White Paper

5 When the DiagnosticManager console is open, it is easy to navigate to any area of performance that is of concern to a SQL Server administrator. An area of key importance is locating the SQL statements that may be under-performing in the database and therefore causing performance bottlenecks. To locate these statements, DiagnosticManager provides a simple point and click menu that will display any statements which exceed predefined performance thresholds. Figure 1: DiagnosticManager provides a graphical interface into the health of SQL Server databases. By choosing to view the performance diagnostics of a SQL Server database within DiagnosticManager, statements that are acting as bottlenecks are easily pinpointed. The flexibility of DiagnosticManager is evident immediately. Various performance items can be selected as criteria for capturing statements for further analysis. Selection options can be made to capture any poorly performing stored procedures, triggers, individual SQL statements, or even SQL statement batches. When making selections, exclusion of any statements can be specified as well. This way, any statements, procedures or triggers executed by an application such as the SQL Server profiler will not be included in those that are captured for analysis. This frees up any time that would be required for an administrator to build complex filters within multiple traces. By specifying the runtime and resource utilization of executed SQL statements, one can further filter those statements that are captured by DiagnosticManager for later optimization and tuning efforts (Figure 2). NetIQ and LECCOTECH: Integrated Solutions for Optimal SQL Server Performance 3

6 Figure 2: DiagnosticManager provides flexibility in filtering the SQL statements that will be selected for optimization and tuning. While accumulating the worst performing SQL statements, DiagnosticManager also allows administrators to regularly delete collected statements (Figure 3). Therefore, as the accumulated history of the offending statements gets larger, older statements can be deleted automatically, thereby reducing query result response times. Purging of any of the accumulated history can be done manually or set to occur automatically. Figure 3: Specify when dated information should be removed from the worst performing statements that are collected. Once DiagnosticManager collects SQL statements that violate the administrator-defined thresholds specified in the collection-filtering tab, these SQL statements are presented in the Worst Performing screen. Based on the collected SQL performance statistics, the decision can be made as to whether further optimization of the offending statement is required. If so, a SQL statement can be easily sent to LECCO SQL Expert for optimization through a click-through option (Figure 4). 4 White Paper

7 Figure 4: DiagnosticManager identifies performance bottlenecks so LECCO SQL Expert can tune queries to optimize performance. Step 2: LECCO SQL Expert: Rewriting SQL Statements Once a SQL statement has been submitted to LECCO SQL Expert from DiagnosticManager, LECCO SQL Expert begins the optimization process (Figure 5). Due to the complexity of the SQL language, there are many ways to write a SQL statement to return the same result set, but small SQL code variations can have a great impact on performance. LECCO SQL Expert s Syntactical SQL Optimizer uses a SQL transformation engine that completely transforms a SQL statement in every possible equivalent SQL variation, preserving the same logic in each alternative statement. The SQL rewrite process includes the use of syntactical SQL transformations and SQL Server hints which are optional to the user. Once the SQL statement has been transformed, LECCO SQL Expert obtains the execution plan [the step-by-step representation of the sequence of operations that SQL Server performs to execute a SQL statement] for each SQL statement. It then narrows the optimized statements to those with a unique execution plan, since an execution plan is what determines the performance of a SQL statement. This comprehensive SQL transformation process occurs in the DBA s desktop PC, thus not affecting database server resources (Figure 6). From this process, every semantic equivalent to the source SQL statement can be produced quickly and automatically. NetIQ and LECCOTECH: Integrated Solutions for Optimal SQL Server Performance 5

8 Figure 5: Once a statement is chosen to be a good candidate for tuning and optimization, the LECCO SQL Expert option within DiagnosticManager will automatically begin generating SQL alternatives. Figure 6: LECCO SQL Expert for SQL Server quickly generates a comprehensive selection of SQL alternatives. Step 3: LECCO SQL Expert: Benchmarking SQL Upon completion of the optimization process, a number of alternative SQL statements will be made available as potential replacements for the problematic source SQL statement. To better determine which statement to use in the SQL Server environment, these statements need to be tested for runtime and I/O statistical information so that the best alternative is identified. The Batch Run function in LECCO SQL Expert provides the capabilities to execute all or some of the SQL alternatives and collect this runtime information. 6 White Paper

9 When performing the Batch Run function, the developer or DBA will be able to quickly and accurately determine if there is a better performing statement for their application. All statements with unique execution plans are executed and their execution times are compared to that of the source statement. Since the optimization process is so comprehensive, there can be many alternatives to test, and LECCO SQL Expert allows the user to establish how the statements being tested should be terminated if they do not satisfy performance requirements (Figure 7). Figure 7: The ability to set termination times for SQL alternatives makes LECCO SQL Expert extremely flexible. LECCO SQL Expert gives DBAs and developers flexibility when performing the batch job function. This test can be set to retrieve the elapsed or response time for each statement being executed (Figure 8). The elapsed time is most relevant to batch jobs and reports when all the records need to be retrieved to display the result. The response time is important for web-based applications since once the first record has been retrieved it can be displayed on the end-user s browser. In addition, there is a setting for how many times each statement should be executed. Depending on the run time for the original SQL statement, the option is available to execute each alternative once or twice. Data caching is an important consideration when optimizing statements used in a web-based application. When the selection is made to execute each statement twice, the first run will be ignored and timing information will be taken from the second run (Figure 8). This gives a more realistic view of how they will perform on a daily basis. NetIQ and LECCOTECH: Integrated Solutions for Optimal SQL Server Performance 7

10 Figure 8: The Batch Run module offers a variety of run time selections. Once the Batch Run is started, progress can be monitored and the results viewed, as they are collected in real time. Once the Batch Run completes, the results of the test are presented (Figure 9). Detail is provided about each statement, along with which alternative performs best in the environment. Figure 9: LECCO SQL Expert allows DBAs and developers to review all run time information. Upon completion of the optimization process with LECCO SQL Expert, a DBA or developer can use the gathered information to learn and increase their knowledge in the area of SQL performance. The SQL Comparer module within the solution is an excellent training alternative (Figure 10). It allows all of the tested statements to be viewed side-by-side in order to display the syntax and execution plan differences. If a faster SQL statement was found, then viewing the changes in SQL syntax will allow one to learn how to write better SQL statements in the future. In addition, Plan Help explains what is being communicated in the execution plan. Understanding what the execution plan conveys regarding SQL execution is very beneficial for both DBAs and developers. 8 White Paper

11 Figure 10: LECCO SQL Expert is an excellent training tool for anyone dealing with application performance Conclusion In the SQL Server environment, performance tuning involves locating and solving bottlenecks at both the database and application levels. The tools available in SQL Server to alleviate these bottlenecks are really not enough when considering the complex nature of today s database environments and varying skills of DBAs. Also, the majority of performance issues on the application side are still going to be due to poorly performing SQL statements, and tuning is a time consuming process for even an expert DBA. NetIQ DiagnosticManager for SQL and LECCO SQL Expert for Microsoft SQL Server provides relief to these pressures by offering solutions that deliver a comprehensive tool to enable database administrators to not only rapidly diagnose performance problems, but also to automate the process of SQL optimization. Database administrators are able to work with DiagnosticManager to locate problematic SQL statements which are most often the root cause of performance problems, and then employ LECCO SQL Expert to provide SQL alternatives that will overcome the deficiencies of currently used statements. Time is saved in the process, and knowledge is gained in the area of database monitoring and SQL tuning. DiagnosticManager by NetIQ and LECCO SQL Expert for Microsoft SQL Server are a complete performance suite that offers a solution to DBAs and developers who understand the importance of database and application performance. Using DiagnosticManager as a monitoring solution and LECCO SQL Expert for SQL optimization will ensure that the highest performing database applications are available to the organization. Download a free trial of NetIQ DiagnosticManager and LECCO SQL Expert for Microsoft SQL Server. DiagnosticManager Product Trial: LECCO SQL Expert Product Trial: NetIQ and LECCOTECH: Integrated Solutions for Optimal SQL Server Performance 9