ABAP SQL Monitor Implementation Guide and Best Practices

Transcription

1 ABAP SQL Monitor Implementation Guide and Best Practices

2 TABLE OF CONTENTS ABAP SQL Monitor - What is it and why do I need it?... 3 When is it available and what are the technical requirements?... 5 In which system shall I use the SQL Monitor?... 5 SQL Monitor will run in production What is the performance overhead and average data volume?... 5 What are the steps to switch on SQL Monitor in my productive system?... 6 How do I use the data of SQL Monitor?... 8 Which data is displayed in SQL Monitor and how can I read it? What is the best strategy to analyze SQL Monitor data? What are the steps to switch off SQL Monitor and to get rid of all the data collected?

3 ABAP SQL Monitor - What is it and why do I need it? During the SAP HANA migration of a productive Business Suite system the performance of main business processes shall be optimized. Focus of this document is the optimization of custom ABAP code. A first step is of course to run and analyze the main processes (e.g. "Create sales order" using transaction VA01) by using standard performance analysis tools like SQL trace or ABAP runtime analysis. However, this will only cover one path through the transaction and the retrieved performance data may not even be realistic since the data volume, scenario and usage of this transaction might be different in the day-by-day business in production. Additionally you may miss many processes (e.g. RFCs, batch processes...) which represent main productive business scenarios containing a lot of SQL performance potential. In conclusion, we need a tool, which monitors the performance of ALL ABAP SQL executions in production, and links back to the driving business processes. This would allow answering questions like: Which of my business processes have the highest total SQL execution time - and are responsible for the main load on the database? What is the productive SQL profile of my main business process e.g. report ZXY and which SELECT is dominating this process? Which SQL statement has the highest execution time and which business processes are affected by this slow SQL statement? Which of my business processes run a huge number of SQL statements or work with a lot of data? Which are the most often executed SQL statements in my custom code? This SQL Monitoring task cannot be done by the standard analysis tools since: SQL trace, ABAP Runtime analysis, etc. contain all this information but they are designed to trace single processes and not a complete system. The trace files would get too big / overwritten and the performance overhead would be not acceptable when using these tools even for a short monitoring period. Statistics (STAD), Workload analysis (ST03) etc. monitor a complete system and deliver aggregated performance data for processes (transaction, report, RFC...) but they do not deliver detailed performance information about the different SQL statements. Database monitoring tools like SQL statement cache deliver very detailed information about the executed SQL statement, but the data is overwritten after some time, identical SQL statements coming from different ABAP programs are condensed to one SQL trace line and there is no link to the driving business process (e.g. transaction, ABAP report...) The new ABAP SQL Monitor fills this gap and provides total transparency of the running SQL statements in a productive environment. 3

4 The ABAP SQL Monitor: Traces each and every SQL statement coming from an ABAP program - this includes OPEN SQL, native SQL and SQL statements coming from the ABAP kernel Can be switched on for all or dedicated servers of an ABAP system Can run in a productive system in parallel to the productive usage since the performance overhead for the traced business processes is negligible. (The data collection is highly optimized by using kernel functionality which buffers the performance data in different memory layers before storing it asynchronously on database) Collects performance data for each traced SQL statement including: o number of executions o execution time (maximum, minimum, average, standard deviation) o fetched/changed rows (maximum, minimum, average, standard deviation) Derives the entry point of each process driving the traced SQL statement. The entry point can be a transaction, report, RFC module or URL Writes new records for an executed SQL statement if one of the following key fields is different (if not then the performance data of the executed SQL is added to an existing entry (aggregation)): o DB table name (s) - you may have several for e.g. joins or subqueries o SQL Statement location in ABAP (program, include, line) o Process type and process name ( e.g. transaction VA01) Allows displaying the performance data aggregated by business process (request) or ABAP code location Allows downloading SQL Monitor data in order to import it in a development system where the code corrections are done. The following screenshot shows an example for the SQL data display of SQL Monitor without aggregation and ranked by executions. 4

5 When is it available and what are the technical requirements? SQL Monitor comes in two (more or less identical) flavors. As a standard delivery of NW (SAP_BASIS software component) and via the ST-PI add on. The ST-PI add on variant has been added to serve older NW releases (NW releases >=7.00) where the standard SQL Monitor is not available. The standard NW SQL Monitor is recommended for the new NW releases like 702,731,740. SQL Monitor variant NW NW 702, NW 703/731, NW 740 ST-PI add on Supported NW Release NW 700, NW 701, NW 702, NW 703/731, NW 710, NW 711, NW 720, NW 730, NW 740 Kernel version For NW releases < 740: No support for 720 Kernel, use 721 kernel instead For NW releases < 740: No support for 720 Kernel, use 721 kernel instead Please refer to OSS note for all details about availability and necessary preparation steps. The following SQL Monitor description focuses on the feature set shipped with NW 740 SP3 or via the ST-PI add on (2008_1_700 SP8) In which system shall I use the SQL Monitor? The SQL Monitor can run in any ABAP system but since it's task is to collect realistic SQL performance data it is normally used in a productive system or in a test system where "realistic" tests are running. SQL Monitor will run in production What is the performance overhead and average data volume? The SQL Monitor data collection is highly optimized to ensure that it can run in parallel to production. From a technical perspective, the SQL Monitor relies on a memory based data collection process. If e.g. transaction VA01 executes a SELECT statement then SQL Monitor kernel functionality stores the measured performance data in the local memory of this internal session. If the same SELECT statement is executed in this session again then there is no new data record but the data of the second execution is aggregated into the same data record created before. If this session is rolled out or closed then the collected data is transferred and aggregated into shared memory of the server. Only if the assigned shared memory area is totally filled up a file is used to handle this memory overflow situation. Finally, the shared memory data is transferred to the database asynchronously by a batch job. Since the data collection is dominated by fast memory accesses the overhead for the running processes is very low (in average < 3 %.) The SQL Monitor is already active in several big productive systems without any noticeable performance impact for the running business processes. Separated from the running business processes, SQL Monitor batch jobs collect periodically the data from shared memory to the dedicated SQL Monitor database tables. The amount of SQL Monitor entries depends on the traffic on the system and the diversity of processes executing SQL statements. First experience shows that after one day (24h) you see between and SQL Monitor entries. The number SQL Monitor entries then quickly reach a kind of saturation so that after 2 weeks we reach between and some million entries. Important: The processes in each and every productive system are different and e.g. masses of requests with GUID like URLs would prevent aggregation/saturation and may create a lot of SQL Monitor entries. This may lead to a long runtime of the SQL Monitor data replication batch job. Therefore we recommend monitoring the number of created SQL Monitor entries and the runtime of SQL Monitor jobs (see page 7) during the first hours/first day. If more than 2 Million SQL Monitor entries (visible on the start screen of SQL Monitor) were created, we recommend switching off SQL Monitor. 5

6 In general we do not recommend measurement periods longer than 2 weeks since the quality of the SQL Monitor data decreases if the data aggregation covers many code changes, new business phases etc. Therefore, after one or two weeks, the SQL Monitor shall be stopped, reset and then restarted. What are the steps to switch on SQL Monitor in my productive system? In the following table you find a step by step description on how to implement and switch on the SQL Monitor in your productive system: Step Software Deployment Attach Authorizations to SQL Monitor users SQLM Variant / Release NW Description Implement the Kernel version, OSS notes, NW SP/ST-PI add on as described in OSS note Separate user who administrate the SQL Monitor and user who only display SQL Monitor data Standard authorization object for trace tools (ST05 etc...): authority object S_ADMI_FCD Dedicated new authorization values for SQL Monitor. SQMA (administrate SQL Monitor: switch on/off etc...) SQMD (display SQLM Data) Attach Authorizations to SQL Monitor users SQL Monitor: Main switch profile parameter = on ST-PI NW / ST-PI Authorization for transaction code SQLM (administrate SQL Monitor) Authorization for transaction code SQLMD (display SQL Monitor data) Standard transaction for trace tools (ST05 etc...): authority object S_ADMI_FCD with values: ST0M: Authorization to change trace switches ST0R: Authorization to analyze traces Authorization for transaction code /SDF/ZQLM (administrate SQL Monitor) Authorization for transaction code /SDF/ZQLMD (display SQL Monitor data) Display profile parameter abap/sqlm/main_switch in transaction code RZ11. The value of the profile parameter shall be on. (= default) You can use this profile parameter in an "emergency" situation to switch off the SQL Monitor. Run transaction SQLM (ST-PI: /SDF/ZQLM) Standard is to switch on SQL Monitor on all servers to collect all running SQL statements in the system: Choose: SQL Monitor->Activate->all servers Switch on SQL Monitor and define an expiry date NW /ST- PI If you want to switch on SQL Monitor only on dedicated servers then choose SQL Monitor->Activate->Selected servers and choose the servers in the provided server list. You can use the same functionality to change the server activation at any time. In both scenarios you are prompted for the SQL Monitor activation expiry date - meaning when the SQL Monitor shall be switched off automatically. The default setting is to switch off SQL Monitor after 7 days. 6

7 You may run SQL Monitor for one day, 1 week or 2 weeks. Longer periods are not recommended since the data is aggregated over the complete period and the value/significance of the data may decrease because of code changes, new processes etc. After activation you can always change the activation expiry date using SQL Monitor->Maintain Deactivation Schedule. Verify server activation status NW /ST- PI Run transaction SQLM (ST-PI: /SDF/ZQLM) and choose SQL Monitor -> Server State -> Check Server State. As a result you will see a server list with information about the actual SQL Monitor activation state and whether the actual state matches the expected state. Verify/Schedule SQL Monitor jobs Verify/Schedule SQL Monitor jobs Only for NW 740 SP2 NW 740 SP3, NW 702, NW 731 ST-PI If there is an inconsistency then you can try to correct this using SQL Monitor -> Server State -> Ensure Server State. Or you deactivate and activate SQL Monitor again. Verify in the Job overview (SM37) that the job RTM_PERIODIC_JOB is running periodically (once an hour). If this job is not scheduled, then run transaction SRTM and choose Runtime Monitor-> Administration -> Settings. Here you see the job administration for the job RTM_PERIODIC_JOB which should have been scheduled automatically when accessing Runtime monitor (SRTM) the first time. Verify that the SQL Monitor data replication is scheduled: Check in the job overview if the job RSQLM_UPDATE_DATA is scheduled periodically. If this job is not scheduled, then follow the instructions of OSS note No action necessary. The following jobs are scheduled/verified automatically when SQL Monitor is switched on: RTM_PERIODIC_JOB Runs periodically once an hour. Basic job which is in most cases running anyhow (independent of SQL Monitor). It collects data from shared memory and stores it in DB tables. SQLM_UPDATE_DATA/SCHEDULE (ST-PI: /SDF/ZQLM_UPDATE_DATA/SCHEDULE) Runs periodically once an hour. Replicates data to SQL Monitor tables. SQLM_DEACTIVATE/SCHEDULE (ST-PI: /SDF/ZQLM_DEACTIVATE/SCHEDULE) Runs at the specified date and time to deactivate the SQL Monitor. Important: If the runtime of the data replication job SQLM_UPDATE_DATA/SCHEDULE is getting too high (reaching 1h) then you may change the period to e.g. 4 hours. As a consequence it will take 4 hours until you see new data in SQL Monitor but the load of data replication goes down. If the job scheduling fails then this status information is visible in the SQL Monitor start screen. 7

8 How do I use the data of SQL Monitor? In order to solve this just deactivate and then activate SQL Monitor. If this does not help please verify that you are allowed to schedule batch jobs. After activating the SQL Monitor it takes up to one hour until the first data can be displayed. This delay comes up because the periodic SQL Monitor batch jobs must run to collect the data. The SQL Monitor data can be accessed via transaction SQLMD (/SDF/ZQLMD for ST-PI version) or by pressing the button "Display data" in SQL Monitor start screen. The following table contains usage scenarios and how these can be realized using SQL Monitor: Step Analyze SQL Monitor Log to know when the SQL Monitor was activated Description Run transaction SQLM or /SDF/ZQLM Choose SQL Monitor -> Display Log/History Specify a start date/time for the log display As a result you will get a chronological list of SQL Monitor events like activation/deactivation of SQL Monitor, deletion of SQL Monitor records, replication of data done by the periodic jobs etc In order to analyze the SQLM data you can use this information to find out when the measurement started or if someone deleted the data in between. Additionally, this information can be used to monitor the SQL Monitor activity and to check for errors or performance issues e.g. in the periodic SQL Monitor batch jobs. (E.g. you may change the frequency of the SQLM_UPDATE_DATA/SCHEDULE from one to e.g. four hours if the job needs too much time) 8

9 You normally download SQL Monitor data when you want to analyze it in another system or if you intend to switch off SQL Monitor and delete all the data to start from scratch (e.g. after you corrected already many SQL errors). To download SQL Monitor data press button "Download Data" in the SQL Monitor start screen. NW 740 SP2 only: Run report RSQLM_DOWNLOAD_DATA to download the current SQL Monitor data to a local file. Download SQL Monitor data Access SQL Monitor data Display the top n custom code SQL statement s of the system sorted by Total Time / Executions In the selection screen you can choose what you want to download - default is everything. But you may limit the download to e.g. only customer code packages (Z*, Y*). Additionally you specify the local file for the download. As a result you get a zip file containing the SQL Monitor data in XML format. In transaction SWLT (SQL Performance Tuning Work List) you can upload the file in a so called SQL Monitor snapshot: Create a SQL Monitor snapshot: Button Manage snapshot, on the next screen choose Create snapshot by file-import. Display SQL Monitor snapshot: Button Select snapshot, choose NONE in frame Static Check settings -> Run (F8) Run transaction SQLMD or /SDF/ZQLMD or press the button "Display data" in the SQLM or /SDF/ZQLM monitor start screen. In the SQL Monitor data selection screen limit the packages to your custom code (e.g. Z*, Y*) and aggregate by source position. Order by total time or executions. 9

10 In the SQL Monitor data selection screen choose aggregation by request and order by Total Time/ Executions: Display the top processes with the highest SQL load / highest SQL traffic 10

11 You can drill down from the top n process list (see scenario "Display the top processes with the highest SQL load") or directly limit the display to one process (e.g. transaction VA01). Display the SQL profile of a process Limit the display to one database table to evaluate which customer ABAP code is accessing the table: Display the usage of a DB table 11

12 Which data is displayed in SQL Monitor and how can I read it? Example screenshot (aggregation = none, order by executions, no filter). All not self explaining columns have a F1 help to provide the necessary information. Here is a short summary of the most interesting and more complex fields: Field Example value Meaning Mean Time[ms] 0,322 ms Average (aggregated) execution time of the SQL statement in milliseconds Mean Records 0,419 Average number of rows selected or changed by the SQL statement Table Names SQL Operation Int. Sessions Executions/Session 587,917 V_PTRV_APPR,PTRV_SHDR Update (Open SQL) DB tables used during execution of the SQL statement. For joins, sub queries or dynamic access you may see several DB tables SQL Operation kind - Here you can separate Read, Write access and Open, Native SQL Number of different internal sessions (roll areas) where this SQL statement was executed. For a simple report (request type Submit report ) the number of internal sessions equals the number of report executions. Average execution counter of the SQL statement per session. In this example the SQL was executed in average 588 times per internal session High execution/session values are often an indication for a nested select. Hint: All time values are displayed in milliseconds. All numbers are displayed with 3 decimal places. (Don't overlook the comma before the last 3 digits - e.g ,570 ms are ~ milliseconds) 12

13 What is the best strategy to analyze SQL Monitor data? In principal, we recommend to focus on total time of the SQL statements to optimize the performance of the business processes. Whereas you would concentrate additionally on the top SQL statements ranked by executions before the SAP HANA migration in order to find the often executed SQL statements. Steps Details In the selection screen of transaction SQLMD: Choose aggregation by request - no filter (and no max nr of records limitation!). Step 1: Process view Get an overview of your main SQL driven requests Step 2: System view Get the most often and most expensive SQL statements (limited to customer code) Step 3: System view Get the SQL statements reading/writing most of the data Step 4: Process view Analyze dedicated processes which you want to optimize Now you have the complete list of requests, which did run SQL statements in your system! Rank this list by Executions to get the requests which run most of the SQL statements. Rank this list by Total time to get the processes with the highest total SQL execution time. Rank this list by Total Records to get the processes with the highest total fetch/change row count. Now check the top 10 of these rankings (you may additionally filter by business criticality of the process) and memorize the aggregated value of the measurement value you are interested in (e.g. Total time). Get the SQL profile of the process by drilling down (link in "Records" column) and rank again by the column of interest (e.g. Total Time). Analyze the top entries which contribute most to the total value you found on process level. Experience shows that the first top 5 SQL statements in the SQL profile contribute up to 80% to the measurement value of interest. In the selection screen of transaction SQLMD: Choose aggregation by source position. No filter (and no max nr of records limitation!). Now you have the complete list of ABAP SQL statements which did run in your system! Sort by Total time or Executions. Check the min/max/standard deviation values of these top entries to figure out if e.g. only one SQL execution was extremely slow because it was hanging at a lock or if almost all executions are slow... You can drill down using the "Records" column to see which processes are driving these top SQL statements. For top execution you can use the drill down to focus on the top entries which have high numbers in column "Executions/session" since these are most often SQL statements in loops which have optimization potential. In the selection screen of transaction SQLMD: Choose aggregation by source position. No filter (and no max nr of records limitation!). Sort by Total Records. Check the min/max values for records. If the maximum and minimum is always the same (and equals the total number of lines stored in the DB table) then this might be a SQL statement without a where clause. If you see a big difference between max and min value then this might be a SQL statement which encounters an empty FOR ALL ENTRIES table or empty RANGE tables from time to time... In the selection screen of transaction SQLMD: Choose aggregation by request. Filter by Request Type and Request entry point (e.g. transaction VA01). Memorize the total values of the fields you are interested in (e.g. Total time) - Drill down and rank by the fields of interest -> see step 1. 13

14 This approach will lead to a work list of tuning proposals. After importing all the SQL/ABAP corrections we recommend to start SQL Monitoring from scratch to check if the corrections were beneficial (How did the SQL profile of your process change?). But remember: Create a SQL Monitor snapshot before deleting SQL Monitor data! What are the steps to switch off SQL Monitor and to get rid of all the data collected? The following table describes how to switch off the SQL Monitor on all servers and delete the SQL Monitor data. Step Switch off SQL Monitor SQLM Variant / Release NW/ST-PI Description In SQL Monitor (transaction SQLM or /SDF/ZQLM for ST-PI version) use menu SQL Monitor -> Deactivate Run transaction SQLM (ST-PI: /SDF/ZQLM) and choose SQL Monitor -> Server State -> Check Server State. As a result you will see a server list with information about the current SQL Monitor activation state and if the actual state matches the expected state. Verify server NW /ST-PI activation status Delete SQL Monitor data NW /ST-PI If there is an inconsistency then you can try to correct this using SQL Monitor -> Server State -> Ensure Server State. Or you directly logon to the inconsistent server and deactivate SQL Monitor for this server again. In SQL Monitor (transaction SQLM or /SDF/ZQLM for ST-PI version) use menu SQL Monitor -> Data-> Delete 14

15 SAP AG or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP AG and its affiliated companies ( SAP Group ) for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Please see for additional trademark information and notices.