Performance and Load Testing For ArcGIS based systems Ian Sims and John Meza

OVERVIEW

What is Performance and Load Testing What is the objective Acceptance Testing Ongoing Development Areyoutheclient Want to ensure you are getting what you pay for Are you the consultant/system integrator/developer Need to demonstrate the capabilities of the software Independent testing consultant Hired for testing only Additional information Microsoft patterns & practices Performance Testing Guidance for Web Applications Guide http://www.codeplex.com/perftestingguide/release/projectreleases.aspx?releaseid=6690

Quick Introduction to Load Testing What is load testing? Why do we load test? What tools do we use? Preparing for load testing? How do we load test? What to do with the results of a load test? 4

What is Load Testing? Load testing is a controlled method of exercising artificial workload against a running system. A system can be hardware or software oriented. A system can be both. Load testing can be executed in a manual or automated fashion. Automated Load Testing can mitigate inconsistencies and not compromise scientific reliability of data. 5

Why do we Load Test? Determine performance and scalabiltiy capabilities of ArcGIS-based system Measure specific areas of interest Understanding the resource demands of the deployment environment. Satisfy end-user, clients, and contractual agreements 6

Preparing for load testing? Define Performance Objectives Use Case Definition Performance Scenarios Data Modeling Scripting and Parameterization 7

Define Performance Objectives Every Load Test Should Have a Purpose of Measurement. Common Objectives Sessions Per Hour Transactions Per Hour Throughput Per Transaction and Per Hour Response Time Calibration Resource Saturation Calibration CPU Consumption Memory usage 8

Define Use Cases to test Workflows Use Cases should be prioritized based on the following: Perceived Risk Frequency of use Level of effort to perform 9

Define Performance Scenarios Collection of one or more use cases sequenced in a logical manner (compilation of a user session) Scenarios should be realistic in nature and based on recurring patterns identified d in session behavior models. Avoid simulating extraneous workload. Iterate when necessary. 10

Scripting and Parameterization Script Programmatically Focus on Reusability, Encapsulation and Testability Componentize the Action Step of the Use Case Use Explicit Naming Conventions Example: Measure the Performance of Opening a Word Document. Authenticate(), NavPortal(), NavCourse(), NavCourseMenu(Docs), ReadDoc() 11

Load Testing: How To Initially Configure Optimize the Environment from the Start Consider it your baseline configuration Knowledge of embedded sub-systems systems Previous Experience with Blackboard and/or current deployment Configuration Think twice about using the out of the box configuration. 12

USER INTERFACE

User Interface Tests functionality of entire software stack From browser or ArcMap down to DBMS And everything in between Software to perform this well is Not too expensive Can be tricky to use Fine for single unit performance test Difficult to use for scalability/load test

Software AutomatedQA TestComplete SilkPerformer LoadRunner

HTTP STREAM

HTTP Stream Record/Playback Microsoft Visual Studio 2008 Team Edition for Testers Create Test Project Right-click on Test Project (UC1) Add Web Test Browser will be shown, navigate to webpage Perform desired functionalty Stop recording Package in a Load Test

Demo: Web Test and Load Test with MS Visual Studio 2008 Team Edition for Testers

Microsoft Visual Studio 2008 Team Edition for Testers Ease of use: For HTTP stream record/playback Extremely easy to use Must know what to test, how, what to expect, etc.

CODED

Coded Load Tests Begin with developing coded test Test covers very specific and narrow band of functionality Labor intensive Code must be designed, developed, built for the test Possible to convert existing unit tests, if applicable Maintenance of test is ongoing MS VS 2008 Team Edition for Testers Once coded load test is in place, it s fairly reliable Rich API to configure Integrates well with

Coded Testing Code must provide all needed functionality

Coded Testing Configuration of load test similar to HTTP Assignment of counters to collect test metrics same as HTTP Able to mix multiple coded tests into Scenarios Use with Engine functionality, customization

Microsoft Visual Studio 2008 Team Edition for Testers Ease of use: For Code load test Not terribly bad Must know exact functionality to test Design, develop and test the test code Must know what to test, how, what to expect, etc. Integrates with MS Reporting Services SQLServer Team Foundation Server Customizable with

MS VS 2008 Team Edition for Testers Configuration Load Test Configuration Straight forward Easy to use Must know the details prior Scenarios Consists of Unit Tests % mix of Unit Tests How load is applied Counters Counters Specific Perfmon counter sets Which machines to monitor

MS VS 2008 Team Edition for Testers Unit Test Unit Tests Basic part of load Test Defines specific area of functionality to test Engine, ADF, customization Specific test defined in by TestMethod tag

MS VS 2008 Team Edition for Testers Unit Test Unit Test / TestMethod Executes or calls code to be tested

MS VS 2008 Team Edition for Testers Real-Time Test Results Defined in Configuration Perfmon Counters Perfmon Counters Stored in a repository

MS VS 2008 Team Edition for Testers

Demo: Coded Load Test with MS Visual Studio 2008 Team Edition for Testers