White Paper. Fabasoft app.test Load Testing. Fabasoft app.test 2015 Update Rollup 2. Fabasoft app.test Load Testing 1

Transcription

1 White Paper Fabasoft app.test Load Testing Fabasoft app.test 2015 Update Rollup 2 Fabasoft app.test Load Testing 1

2 Copyright Fabasoft R&D GmbH, Linz, Austria, All rights reserved. All hardware and software names used are registered trade names and/or registered trademarks of the respective manufacturers. No rights to our software or our professional services, or results of our professional services, or other protected rights can be based on the handing over and presentation of these documents. Fabasoft app.test Load Testing 2

3 Contents 1 Introduction 4 2 Software Requirements 4 3 Infrastructure Preconditions Management Server Backend Server Load Test Clients Telemetry/Database Server 7 4 Preparing Fabasoft app.test Tests Assign Unique ID for Each Statement One Big Test Versus Multiple Small Tests Use Unique Tokens Scoped Users Delay on Start Up Importing Documents Object Lists: Display All Quick Search / Combo Box Cache Limit Search Scope Clipboard Executions for External Programs Ensure Correct Language 9 5 Setting Up a Sample Load Test Environment Resource Manager Infrastructure Configuration Profile Job Related Documentation 15 6 Analyzing Reports 15 Fabasoft app.test Load Testing 3

4 1 Introduction This document describes how to set up a sample load testing environment using Fabasoft app.test. Load testing means running a huge amount of Fabasoft app.test tests simultaneously against the backend and measuring the performance of the server. 2 Software Requirements System environment: All information contained in this document implicitly assumes a Microsoft Windows or Linux environment. Supported platforms: For detailed information on supported operating systems and software see the software product information on the Fabasoft distribution media. Descriptions in this document are based on the following software: Apache Ant ( Jenkins ( Microsoft Windows Server 2008 R2 Enterprise SP1 Microsoft Windows 7 Enterprise SP1 or Microsoft Windows 8.1 Enterprise 3 Infrastructure Preconditions 3.1 Management Server A management server is required to execute and manage the load test jobs. Installation requirements: Fabasoft app.test Infrastructure Apache Ant Jenkins Fabasoft app.test Load Testing 4

5 For details how to use the Fabasoft app.test Installer: See white paper An Introduction to Fabasoft app.test. 3.2 Backend Server You need a backend server which is used as load test target. Installation requirements: Fabasoft Folio with Web Services For help how to set up a Folio backend server see white papers Fabasoft on Linux Installation of Fabasoft Folio Services and Fabasoft on Microsoft Windows Installation of Fabasoft Folio Services. 3.3 Load Test Clients For this example you need three load test clients on which the tests are executed and where the load to the backend server is generated and the performance is measured. You can use any operating system supported by Fabasoft app.test as client (see documentation of Fabasoft app.test installer mentioned above). Installation requirements: Fabasoft app.test Agent Do not forget to start the Fabasoft app.test Agent after installation and add it to the auto start on UNIX systems (the Fabasoft app.test Installer does this only on Windows systems). Running Fabasoft app.test Agent on Different Ports If you require running the Fabasoft app.test Agent multiple times (one per Windows session/user) on the same machine (e.g. your test uses the system clipboard) you can run the Agent in invisible Windows sessions on different ports. But note that this is officially not supported Steps to use parallel Agents: Create/Configure users: Using RDP connections on Windows Server OS: o Group Policy: "Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Connections Increase Limit number of connections (default: 2) o Create local user accounts for each connection or use domain users and do an initial login with each user to create the user environment. Client Windows operating system: o Create local user accounts for each connection or use domain users and do an initial login with each user to create the user environment. Adjust the port number for each agent: Ensure that no Fabasoft app.test Agent is running on the machine in another Windows session on the default port (5024) (else you cannot start the Agent in the current session as it detects that it is already running on that port). Log in with all users and do following steps which each: Start the app.test Agent Fabasoft app.test Load Testing 5

6 Change the listening port by opening the context menu on the app.test Agent system tray icon, selecting Set Port, changing the port number and clicking Change. Avoid using the default port as mentioned above. For information: The port number is stored per user at following locations: o Windows: HKEY_CURRENT_USER\Software\JavaSoft\Prefs\com\fabasoft\apptest\launcher\pref s_agent_port o UNIX: ~/.java/.userprefs/com/fabasoft/apptest/launcher/prefs.xml - prefs_agent_port The Agent will automatically start at the specified port number if the user is logged in again. Using RDP connections on Windows Server OS: o Close the connection after log in (do not log off) to keep the Windows session running disconnected. The Fabasoft app.test Agent will keep running and each has its own clipboard. But you have still to ensure that only one test runs simultaneously per Agent. Client Windows operating system: o Do not log off the users to keep the Windows session running disconnected. The Fabasoft app.test Agent will keep running and each has its own clipboard. But you have still to ensure that only one test runs simultaneously per Agent. Adjust the port number for each agent (alternative to steps described above): Delete the Fabasoft app.test Agent start up link from all users (e.g. C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Startup\Fabasoft app.test Agent.lnk) Copy the Fabasoft app.test Agent launcher command file from the Fabasoft app.test Agent s install directory (e.g. C:\Program Files (x86)\fabasoft\apptest\agent\apptestagent.cmd) to each user s start up folder e.g. (C:\Users\<user>\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\apptest-agent.cmd). Set the port number by editing the last line: start "" %APPTEST_JAVA% -jar "%APPTEST_HOME%\apptest-launcher.jar" -port:<port> Using RDP connections on Windows Server OS: o Start a RDP connection and log in with each user in order to start the Fabasoft app.test Agent at the dedicated ports. Close the connection after log in (do not log off) to keep the Windows session running disconnected. The Agent will keep running and each has its own clipboard. But you have still to ensure that only one test runs simultaneously per Agent. Client Windows operating system: o Log in with each user locally by switch user in order to start the Fabasoft app.test Agent at the dedicated ports. Do not log off the users to keep the Windows session running disconnected. The Agent will keep running and each has its own clipboard. But you have still to ensure that only one test runs simultaneously per Agent. Important: Note that Fabasoft app.test can t create screenshots (they remain black) when running in an invisible Windows session. Configure the app.test Agent hosts in your infrastructure (as described later): Note: if you are running the Agent on the default port (5024), the port number is not required to be configured. Fabasoft app.test Load Testing 6

7 infrastructure.xml <ClientPool... > <Host name="<hostname>:<port1>" /> <Host name="<hostname>:<port2>" /> <Host name="<hostname>:<port3>" /> <Host name="<hostname>:<port4>" />... </ClientPool> 3.4 Telemetry/Database Server To analyze the test results you need a Fabasoft app.telemetry Server and a database where to store the performance results. For help how to set up an app.telemetry Server (and connect it with the Folio backend) see white paper Installation Guide for Fabasoft app.telemetry. 4 Preparing Fabasoft app.test Tests A Fabasoft app.test Test needs to conform to several conditions to be suitable for load testing. 4.1 Assign Unique ID for Each Statement In order to analyze the results of the load test from a database and to make load test runs against different versions respectively different tests comparable you should assign a unique ID for each statement. Example <Test name="load Test"> <Sequence name="first Load Test Sequence" id="load_seq01"> <UseCase name="sample Use Case" id="load_seq01uc01"> <Execution action="..." id="load_seq01uc01exec01" /> <Execution action="..." id="load_seq01uc01exec02" /> <Execution action="..." id="load_seq01uc01exec03" /> <Execution action="..." id="load_seq01uc01exec_next" />... </UseCase>... </Sequence>... </Test> Tags like Set statements or ParameterTable statements are not required to have unique IDs as they aren t recorded in Fabasoft app.telemetry, but they can also be assigned with an ID for better unity. Do not forget to set the configuration value Telemetry.useexecutionid to true (e.g. see global parameters section in the infrastructure.xml file mentioned below). If this setting is false (default), or a statement does not define an ID, a generated GUID for each executed statement (different GUIDs for the same statement if a test is executed multiple times) will be passed to app.telemetry. The advantage if using IDs instead of GUIDs is to be able to assign app.telemetry records to test executions over multiple test runs and changed test files (e.g. running an adjusted test against a different backend version). Fabasoft app.test Load Testing 7

8 4.2 One Big Test Versus Multiple Small Tests It is recommended for load testing to use just one single big test (which contains all test sequences) instead of executing several small tests to ensure that the use cases are all executed in a predefined order (the execution order of tests within the test matrix is not defined) and to keep the analyzing more clearly. 4.3 Use Unique Tokens To ensure that tests can be executed parallel, even if a test crashes and leaves objects (like documents, folders on the desk, activities...) in the domain you should use unique tokens for all objects a test creates. At least a unique token for root objects (like a test folder created on the desktop in which the test creates objects) should be used. Example <Set parameter="token" value="{~timestamp~}-{~scope~}-{~run~}" options="autoreplace" /> <Execution id="load_seq01_set_testfoldername" action="set" location="pages.create Folder.Name" value="convtest - {~token~}" /> 4.4 Scoped Users Due to the parallelism of the test execution scoped users have to be used. They can be used as follows: Example <Sequence name="my Sequence" username="kinder{~scope~}"... In order to run 10 tests parallel there have to exist the users kinder0001 to kinder0010. If a test needs more than one user, for each scope there should be also exist other users like sacher{~scope~}, leitner{~scope~}, kainz{~scope~}, adler{~scope~} etc. 4.5 Delay on Start Up Running the same parallel against the same backend server can result in unrealistic performance behaviors when all tests do the same thing at the same time, e.g. a search query is executed simultaneously multiple times. To avoid this you should add a delay on top of a test, dependent on the user scope. Example <Test... <Sequence... <Set parameter="delay_time" eval="({~scope~} - 1) * 1000" /> <Execution action="delay" value="{~delay_time~}" if="{~run~}==1" /> 4.6 Importing Documents There are two possibilities to import documents into Fabasoft Folio. The first is to select the menu Import and set the path to the file into the displayed file input control. The better way is to import documents directly into the object list (corresponds to the drag & drop Fabasoft app.test Load Testing 8

9 import) as opening a file open dialog causes problems when tests are running simultaneously on the same machine. Example <Execution id="load_seq01_import01" action="import" location="pages.explore.object List" value="{~importdocs~}documents/fabasoft app.test.docx" /> 4.7 Object Lists: Display All When there is a need to find certain objects in an object list, like clicking on a certain activity in the work list, you have to consider that all objects should be displayed, e.g. when the number of entries in the list increases during test execution. To solve this problem you can set /a into the navigation field of the list. 4.8 Quick Search / Combo Box Cache A common mistake is that when recording a test combo boxes are just opened and a value is selected. But you have to consider that the displayed objects in a combo box are cache dependent. Therefore always a quick search should be performed before selecting values from a combo box. 4.9 Limit Search Scope Due to performance reasons you should never perform a search for all objects from an object class. Always limit the search as much as possible via setting the fields on the search form Clipboard Using the clipboard in a test running parallel multiple times on the same machine is currently not supported. If you are using the clipboard in the load tests you have to prepare your test clients/infrastructure as only one test runs simultaneously per test client (setting the maxparallel configuration of the client pool in the infrastructure configuration file to 1 (and adjusting the other test count values according to that)). Alternatively you can run the Fabasoft app.test Agent in different Windows sessions on different ports on the same machine. But note that running app.test in an invisible Windows session is officially not supported. Details see 3.3 Load Test Clients Executions for External Programs Try to avoid opening documents or starting other external programs during a test if there is no need to test that explicitly. Note that opening documents (and closing by app.test) simultaneously on the same machine (Windows session) is not recommended and can cause several unexpected problems Ensure Correct Language A common source of errors is also that a test is recorded in another environment than it is executed. Ensure that the test client uses the same locale settings as the test was created with. Fabasoft app.test Load Testing 9

10 5 Setting Up a Sample Load Test Environment Directory/Files Structure (located on the management server): <app.test Infrastructure installation directory> o resourcepool/resourcepool_config/infrastructure.xml o anttasks/anttasks_config/apptest-profilejob.xml <profile job directory> o run_load_tests.xml <load tests directory> o loadtests.ducx-testset o loadtests_reference.ducx-testset o *.ducx-test 5.1 Resource Manager Infrastructure Configuration Defines: One load test client pool (2 clients with 3 parallel tests each) and one load test reference client pool (1 client with 1 parallel test) and one load test backend server (1 host with (at least) 2 (clients) * 3 (slots) + 1 (client) * 1 (slot) = 7 scopes). Important: There exists a parameter Telemetry.active, which tells Fabasoft app.test to start a Fabasoft app.telemetry session for every single test run, if set to true (default setting: false). If you are planning to execute a huge amount of test runs, you should start a Fabasoft app.telemetry session before running all tests and stop it afterwards and leave this option on false. infrastructure.xml <Infrastructure version="0.1"> <!-- Property Definitions --> <PropertyDefinitions> <PropertyDefinition id="firefox_edge"> <Property key="clienttype"> <Entry value="html_mozilla" parameter="run.clienttype"> <Version name="18.0" /> </Entry> </Property> </PropertyDefinition> <PropertyDefinition id="ie9"> <Property key="clienttype"> <Entry value="html_ie" parameter="run.clienttype" version="10" /> </Property> </PropertyDefinition> <PropertyDefinition id="safari"> <Property key="clienttype"> <Entry value="html_safari" parameter="run.clienttype"/> </Property> </PropertyDefinition> </PropertyDefinitions> <!-- Clients Specification --> <ClientPools> <ClientPool id="loadtestclients" inherits="firefox_edge" maxparallel="3"> <Host name="[loadtestclient1]" /> <Host name="[loadtestclient2]" /> <Property key="loadtest" value="true" /> </ClientPool> </ClientPools> Fabasoft app.test Load Testing 10

11 <ClientPools> <ClientPool id="loadtestreferenceclients" inherits="firefox_edge" maxparallel="1"> <Host name="[loadtestreferenceclient1]" /> <Property key="loadtestreference" value="true" /> </ClientPool> </ClientPools> /> <!-- Applications Speficiation --> <Applications> <Application id="loadtestbackend" inherits="firefox_edge" scoperangefrom="1" scoperangeto="7"> <Parameter name="login.authentication" value="basic" /> <Parameter name="login.domain" value="" /> <Parameter name="login.username" value="kinder{~scope~}" /> <Parameter name="login.password" value="[password]" /> <Parameter name="run.address" value=" /> <Parameter name="run.webserver" value="[loadtestwebserver]" primary="true" <Property key="loadtest" value="true" /> </Application> </Applications> <!-- Global Parameters --> <Parameter name="telemetry.useexecutionid" value="true" /> <Parameter name="run.onerror" value="continue" /> <Parameter name="run.onerror.screenshot" value="true" /> <Parameter name="run.onerror.htmldump" value="true" /> <Parameter name="run.onerror.controldump" value="true" /> <Parameter name="run.maxerrors.sequence" value="15" /> <Parameter name="run.maxerrors.test" value="15" /> <Parameter name="run.timeout" value="120000" /> <Parameter name="run.foliopluginsenabled" value="true" /> <Parameter name="browsersettings.pluginsenabled" value="true" /> <Parameter name="agent.transfer" value="true" /> <Parameter name="adaptor.width" value="1400" /> <Parameter name="adaptor.height" value="750" /> </Infrastructure> 5.2 Profile Job Configure the test matrix execution to run the console player for each test in-process to avoid performance- and memory problems when executing a huge amount of tests simultaneously. apptest-profilejob.xml <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE properties SYSTEM " <properties> <!-- define if the console players for each test run should be run in profile job process or started as own process this preference is used for starting console players in the tasks apptestinfrastructuretestmatrix and runtest used if not defined in task default: false (start player as own process) --> Fabasoft app.test Load Testing 11

12 <entry key="com.fabasoft.apptest.infrastructure.playersinprocess">true</entry> </properties> Create a profile job ant build file for a Jenkins job which executes the tests, loads the results into a database and copies them into the Jenkins job workspace. run_load_tests.xml <?xml version="1.0" encoding="utf-8"?> <project name="run Tests" default="test"> <property name="apptest.dir" value="[app.test Infrastructure installation directory]/anttasks" /> <taskdef classpath="${apptest.dir}/apptest-ant.jar" resource="tasks.properties" /> <taskdef classpath="${apptest.dir}/apptest-antinfrastructure.jar" resource="tasks.properties" /> <property environment="env" /> <target name="test"> <apptest-profilejob verbose="true" jobname="simple Load Test"> <echo message="starting Load Test Matrix..." /> <parallel> <sequential> <!-- Execute tests on load test clients --> <apptest-infrastructuretestmatrix stage="test" verbose="false" allowstatic="true"> <fileset dir="[load tests directory]/loadtests"> <include name="**/loadtests.ducx-testset" /> </fileset> </apptest-infrastructuretestmatrix> </sequential> <sequential> <!-- Execute tests on load test performance reference clients --> <apptest-infrastructuretestmatrix stage="reference" verbose="false" allowstatic="true"> <fileset dir="[load tests directory]/loadtests"> <include name="**/loadtests_reference.ducx-testset" /> </fileset> </apptest-infrastructuretestmatrix> </sequential> </parallel> <!-- Load results into database --> <apptest-configuration id="database"> <config key="reportanalyzer.dbtype" value="postgresql" /> <config key="reportanalyzer.address" value="[databaseserver]" /> <config key="reportanalyzer.dbname" value="[databasename]" /> <config key="reportanalyzer.username" value="[databaseuser]" /> <config key="reportanalyzer.password" value="[databasepassword]" /> <config key="reportanalyzer.createtables" value="true" /> <config key="reportanalyzer.onerror" value="continue" /> </apptest-configuration> <apptest-reportanalyzersessionstart verbose="true"> <Configuration refid="database"> <config key="name" value="load-tests" /> </Configuration> </apptest-reportanalyzersessionstart> <apptest-reportanalyzer verbose="true"> <Configuration refid="database"> Fabasoft app.test Load Testing 12

13 <config key="session.id" value="${session.id}" /> <config key="product" value="folio" /> <config key="edition" value="editionloadtests" /> <config key="version" value=" " /> <config key="stage" value="test" /> </Configuration> <fileset dir="${apptest.workspace}/stages/reference"> <include name="**/*.ducx-rep" /> </fileset> </apptest-reportanalyzer> <apptest-reportanalyzersessionend verbose="true"> <Configuration refid="database"> <config key="session.id" value="${session.id}" /> </Configuration> </apptest-reportanalyzersessionend> <echo message="database Session ID: ${session.id}" /> <!-- Create report summary --> <apptest-reportsummary verbose="true"> <config key="name" value="load Test Report" /> <config key="result.path" value="${apptest.workspace}/stages" /> <config key="result.name" value="reportsummary.xml" /> <config key="result.transformation" value="reportsummary.html" /> <config key="result.stylesheet" value="${apptest.dir}/reportstyles/apptest-errorreport.xsl" /> <config key="fragments.stylesheet" value="${apptest.dir}/reportstyles/apptest-reportdetail.xsl" /> <config key="fragments.prefix" value="detail_" /> <config key="fragments.suffix" value=".html" /> <config key="level" value="overview"/> <config key="onerror.continue" value="true" /> <config key="image.compress" value="true" /> <config key="image.compress.ratio" value="0.7" /> <fileset dir="${apptest.workspace}/stages"> <include name="**/*.ducx-rep" /> </fileset> </apptest-reportsummary> <!-- Copy Fabasoft app.test reports into workspace --> <echo message="copying Results..." /> <delete dir="${env.workspace}" /> <extcopy src="${apptest.workspace}/stages" dest="${env.workspace}" /> <echo message="done." /> </apptest-profilejob> </target> </project> Execute load test 30 times (= 2 clients * 3 slots * 5 test runs) on load test client pool. loadtests.ducx-testset <TestSets> <TestSet id="load Tests Demo" name="load Tests Demo"> <Applications> <Application> <Require key="loadtest" value="true" /> </Application> </Applications> Fabasoft app.test Load Testing 13

14 <ClientPools> <ClientPool runonhosts="30"> <Require key="clienttype" value="html_mozilla" /> <Require key="loadtest" value="true" /> </ClientPool> </ClientPools> <Tests projectpath="."> <fileset dir="."> <include name="**/*.ducx-test" /> </fileset> </Tests> </TestSet> </TestSets> Execute load test for performance reference 5 times (= 1 client * 1 slot * 5 test runs) on load test performance reference client pool. loadtests_reference.ducx-testset <TestSets> <TestSet id="load Tests Reference Demo" name="load Tests Reference Demo"> <Applications> <Application> <Require key="loadtest" value="true" /> </Application> </Applications> <ClientPools> <ClientPool runonhosts="5"> <Require key="clienttype" value="html_mozilla" /> <Require key="loadtestreference" value="true" /> </ClientPool> </ClientPools> <Tests projectpath="."> <fileset dir="."> <include name="**/*.ducx-test" /> </fileset> </Tests> </TestSet> </TestSets> Jenkins Create a (free-style) Jenkins job which executes the run_load_tests.xml ant build file. Fabasoft app.test Load Testing 14

15 5.3 Related Documentation For more details on how to configure the Fabasoft app.test Resource Manager (e.g. auto start configuration) and how to use the app.test Ant Tasks see white paper Fabasoft app.test Continuous Integration. For more details on how to use app.test in general see white paper An Introduction to Fabasoft app.test. 6 Analyzing Reports Using Fabasoft app.test Loading the test results from the app.test report files into a database (as shown in the sample above) is not recommended as it is not optimized to load a huge amount of report data and the duration of executions is measured on the client but not on the backend server. Using Fabasoft app.telemetry You should use app.telemetry to analyze the performance of the load tests: Start a Fabasoft app.telemetry session Run all load tests Stop the Fabasoft app.telemetry session and analyze them. Fabasoft app.test Load Testing 15