Agentry and SMP Metadata Performance Testing Guidelines for executing performance testing with Agentry and SAP Mobile Platform Metadata based

Transcription

1 Agentry and SMP Metadata Performance Testing Guidelines for executing performance testing with Agentry and SAP Mobile Platform Metadata based applications

2 AGENTRY PERFORMANCE TESTING V 1.0 TABLE OF CONTENTS USING THE ATE BASED LOAD TEST TOOL... 3 Location... 3 Setup and Usage... 3 MAXIMUM LOAD TESTING... 5 Win32 Clients... 5 Other Device Clients... 5 SINGLE USER / SCREEN FLOW TEST... 6 Average Response Times... 6 RECOMMENDED READING... 6 OTHER USES... 6 ATTACHMENT

3 This document provides guidelines on how to execute performance testing on an Agentry application. It covers the load test tool and the different kinds of performance and load tests that may need to be performed and their uses. USING THE ATE BASED LOAD TEST TOOL The Agentry Test Environment can be used to automate testing using pre-written scripts (AGS files) to perform a specific set of actions for an instance. Many copies of the ATE can be run concurrently on the same machine allowing it to be used for performance testing as well as unit testing. Location The Agentry Test Environment Load Test Tool can be downloaded from the bottom of this document. Please note that this tool is to be used at your own risk and is provided without any warranties or support. If contains the following files: 1. loadtest.bat Executable that initiates a test run and contains configuration parameters 2. loadtest_clone.bat Internal file that is used to duplicate the client files and assign parameters 3. loadtest_run.ags Contains the login script 4. Strings.ini Works in conjunction with the ATE to enable multiple instances and contains the device id mask. 5. Script.ags The script that defines the actions to be simulated 6. Users.txt Contains usernames, passwords and other variables 7. Ssr.exe Internal File for text replace 8. Client subfolder contains the ATE files Setup and Usage The tool can be unzipped into any location and does not require installation. However it does require that an Agentry Test Environment (ATE) has already been installed once on the machine for correct operation. Edit users.txt file. This file should contain the usernames and passwords of the users being used for the test run. The format is comma separated values with one line per instance and semi-colon indicating a commented line. It also contains additional columns that can be used for parameters/values that could vary for each user or instance. Place a copy of the ATE in the client subfolder; you may need to overwrite the pre-existing files because your version may be different from the version in the package. The ATEs first connection to the server will cause it to load the application from scratch You may want to use a pre-built client if your test does not require an initial load, for example unit tests. Create an AGS script. The script file is an xml file that should contain the actions that you want to be performed after the user logs in. One method to create the script is to use the ATEs record function to record the actions to be performed. However, you will still to need to edit the script manually to ensure that the proper waits and delays are put in otherwise the script may fail. The Agentry Test Script language is explained in the Agentry Language Reference Guide It is best to run the ATE in the client folder once, to ensure that it will run correctly later. Changes may need to be made (using a text file editor) to the loadtest.bat file. Following are parameters that may need to be modified: a. LOAD_TEST_SCRIPT- This is a *.ags file that will be called to start the tests b. LOAD_TEST_APP_DESCRIPTION This is Agentry application name (also the program name is the bar e.g. Microsoft Word). c. LOAD_TEST_SERVER - The dns name of the server you are connecting d. LOAD_TEST_MODULE_NAME The name of the module being tested in the Agentry application (available from the editor) 3

4 e. LOAD_TEST_MODULE_DESCRIPTION - The description of the module being tested in the Agentry application (available from the editor) loattest_clone.bat only needs to be modified if there are changes to the parameters and variables being using the loadtest.bat and script.ags. Use a dos command prompt to run loadtest.bat. This will case a number to instances of the ATE to be run simultaneously. The number of instances is determined by the number of users in the users.txt file. A file called times.txt will be created that will contain the start time and end time for each instance. The individual ATE instances may also write their own logs in their respective folders. 4

5 MAXIMUM LOAD TESTING This is used to test that the system will have an acceptable, albeit above average, response time and not crash under peak loads. This test will assist in validating Agentry server sizing and ensure that the ERP backend is able to handle the load. This can be done my modeling the maximum number of concurrent users (as opposed to total number of users). However, in most cases, this modeling can only be an approximation of the real world scenario. Since one machine can simulate multiple clients this allows us to do a simulation without having to manually go through each one. It is recommended that performance monitors be run on both the Agentry server machine(s) and the ERP system to track CPU load percentage, memory usage percentage, errors, dumps and other performance indicators. The load test may be run multiple times and often may need to re-run to validate that backend system configurations are providing the desired response time improvements. Win32 Clients Since the ATE is a Win32 client as well, it is possible to more accurately mimic behavior. It is, however, important to mimic connectivity as closely as possible as well to get a measure of the connection latency. Thus in the case that the end-users are connected over a LAN it is recommended that the testing be done over LAN. and Wireless LAN for Wireless LAN. If the users are connected over Mobile Data Network, it would be best to set up devices that use the Mobile Data Network. If that is not feasible, it is recommended that the test machines be places in a network outside the internal LAN to provide latency and test the corporate internet capacity. If attached documents are going to be a regular part of the download, then the load test downloads should include those too. In these cases, have an better replica of the production network setup will assist in monitoring and detecting future bottlenecks. Other Device Clients The ATE can also be set up to simulate other devices. However, because the mobile device may have more limited processing capability, the ATE may run through the actions quicker than the device. This still allows a valid load test scenario to be run since the server load, latency and bandwidth are the primary indicators being measured. The previous sections recommendations about test machine network setup still apply in this case. 5

6 SINGLE USER / SCREEN FLOW TEST It is often useful to run through the whole application with stopwatches and logging turned on to find other whether particular actions are taking above average execution times. Following are some of the scenarios to watch out for:- Screens (list, transaction, detail) taking too long to show up after the button press List screens taking too long to scroll through Transaction screens being very slow to navigate Complex tables taking too long to load Object fetches taking too long Large attached document download and upload times Agentry Server running out of memory Agentry Server taking too long to process data ERP server responding too slowly to RFC calls BAPI execution taking too long BAPI running out of memory Average Response Times If the difference between single user and maximum user execution time is quite large, it may be useful to run an average load test run. This will assist in identifying the areas that would benefit the most from some extra scrutiny. RECOMMENDED READING There is more information and recommendations about performance testing available at SAP s performance and scalability page - OTHER USES Since this tool runs pre-recorded scripts it can also be used for running regression tests. One of the reason this tools uses batch files is so that they can be easily modified and extended to call other programs and outputs. The loadtest_launch.bat is already set up to output results of the test run into loadtest.log in text format. This output can be either pre- or post-processed to get the data into its desired viewing format. ATTACHMENT LoadTest.zip 6

7 SAP AG. All rights reserved. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, ianywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company. Crossgate, EDDY, B2B 360, and B2B 360 Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.