Performance Testing Mobile and Multi-Tier Applications

Transcription

1 mverify A Million Users in a Box Performance Testing Mobile and Multi-Tier Applications Chicago Quality Assurance Association June 26, 2007 Robert V. Binder mverify Corporation [email protected] x1001

2 Goals of Performance Testing Validate time requirements/expectations Validate utilization requirements/expectations Validate capacity requirements/expectations Reveal load-related bugs Prove compliance: SLAs, contracts, competitive rankings Fire-drill for recovery Assess robustness to shocks 2007 mverify Corporation 2

3 Business Impact/ROI In 2002, slow e-commerce downloads lead to an estimated $25 billion of abandoned transactions In 2005, Google s 15 minute outage estimated to have cost at least $150,000 in lost ad revenue Recent study: nearly two-thirds of mobile employees rank poor response time as a significant inhibitor to working remotely over a VPN 2007 mverify Corporation 3

4 Business Impact/ROI Avoidable Costs Lost revenue Lost user productivity Lost IT productivity Overtime payments Wasted goods Fines Solution Cost Software Tools Hardware Staffing Services Training Risk Mitigation Enterprise demise Law Suits Negative Publicity Personnel morale One 9 Two 9s Three 9s Four 9s Five 9s Six 9s IBM. "Maximizing Web site Availability," February mverify Corporation 4

5 Basic Objectives: RASP Reliability Availability Scalability Probability of a failure occurring within a certain period of time Percent achieved up-time, not including scheduled downtime The range load for which an incremental input consumes the same resources Performance The rate at which work is done 2007 mverify Corporation 5

6 Reliability Reliability: probability of non-failure Total operational hours or transactions Entire user population Can be estimated during test, if tests are sufficiently realistic 2007 mverify Corporation 6

7 Availability the nines Annual Unscheduled Downtime Six nines % 32 seconds five nines % 5 minutes four nines 99.99% 53 minutes Three nines 99.9% 8.8 hours two nines 99% 87 hours (3.6 days) One nine 90% 876 hours (36 days) Availability = percent up-time MTTR: mean time to recover, repair, restart Availability = 1 / 1 + (MTTR Reliability) 2007 mverify Corporation 7

8 Some Data Points Reliability (Failures/million hours) Availability, 6 min MTTR NT 4.0 Desktop 82, Windows 2K Server 36, Common Light Bulb 1, Stepstone OO Framework Telelabs Digital Cross Connect mverify Corporation 8

9 Performance Metrics Response Time Round-trip time Throughput Aggregate transaction processing rate Utilization Average % busy Failure Intensity Recovery Time Avg Resp, Sec Trans/Sec 0% 100% Utilization 2007 mverify Corporation 9

10 Strategies Performance Testing Assess compliance with performance goals Assess compliance with resource utilization goals Provides data to estimate reliability, availability Stress Testing, Load Testing Assess response to over-load scenarios Assess recovery from failure modes 2007 mverify Corporation 10

11 Strategies Benchmarks Assess throughput for open standard test suite Scalability Assess performance linearity Profiling Identify utilization bottlenecks by component 2007 mverify Corporation 11

12 Typical Server Side Setup Emulated Emulated Server(s) Under Test Internal LAN Emulated 2007 mverify Corporation 12

13 Issues Emulation Machines Synchronization, overall test execution Capacity Multi-homed, test control subnet Test vs. Production Systems Separate server farm? Network contention Isolation versus scale/scope Version/Configuration control System under test Test environment Database set/reset 2007 mverify Corporation 13

14 Issues Actual end-user/customer experience? Network latency, QoS Thin clients? Browser, client software versions OS? 2007 mverify Corporation 14

15 Edge Monitoring Emulated Emulated Server(s) Under Test Monitored Internal LAN Internet Emulated Monitored 2007 mverify Corporation 15

16 Issues Monitoring Machines Synchronization Achieving desired test input at desired time Capacity Data collection Availability Security (beta test agreement?) Network configuration DMZ Equipment, setup, security considerations 2007 mverify Corporation 16

17 Connectivity a Wild Card Emulated Emulated Emulated Internal LAN Server (s) Under Test Random latency, jitter, lost packets, re-ordered packets, re-routed packets, duplicate packets, bandwidth restrictions, bit-error, background load, QoS, operational events Internet Monitored Monitored 2007 mverify Corporation 17

18 Controlled Connectivity Emulated Emulated Emulated Internal LAN Server (s) Under Test Controlled latency, jitter, lost packets, re-ordered packets, re-routed packets, duplicate packets, bandwidth restrictions, biterror, background load, QoS, operational events Network Emulator Monitored Monitored 2007 mverify Corporation 18

19 Issues Complexity Impairment modeling Impairment emulator programming Coordination with emulated clients Coordination with monitored clients Specialized Skills Wire Shark (Ethereal) TCP log analysis 2007 mverify Corporation 19

20 How to Maximize Reliability Combine realistic functional and load testing Representative variation in load and usage Supports reliability/availability estimation Saves time: more test goals supported with fewer tests Typically effective in finding weird bugs Security? Add abuse cases to the usage profile Interleave with normal traffic You play like you practice 2007 mverify Corporation 20

21 Events Per Second Use Dynamic Loading The real world isn t flat Vary behavior rate for actor/actor group Arc Flat Internet fractal Negative ramp Positive ramp Random Spikes Square wave Waves Time (seconds) Actual Waves Loading 2007 mverify Corporation 21

22 Case Study Event Simulator Script Writer SilkTest Java GUI DB Test Object Serializer Java Driver Java API Java Servers TX Formatter 3270 Driver MainFrame Test Oracle Custom Test Component 3 rd Party Product System Under Test Test Run Reports Comparator 2007 mverify Corporation 22

23 Case Study Every test run unique and realistic Simulated user behavior to generate transactions Automatically submit in real time ~100,000 test cases per hour ~200 complete daily cycles Evaluated functionality and performance Controlled distributed heterogeneous test agents (Java, 4Test, Perl, SQL, Prolog) driving Java/CORBA GUI/API Five person team, huge productivity increase Achieved proven high reliability Last pre-release test run: ~500,000 events in two hours, no failures detected No production failures 2007 mverify Corporation 23

24 Notes Capture/replay scripts Static think-time Can distort load and response time Performance Analysis Neil Gunther books and web site 2007 mverify Corporation 24

25 Tools Open Source opensta PushToTest Grinder Scripting systems: Tcl, Perl, Ruby, Python Built-in Windows Perfmon *nix SNMP, others 2007 mverify Corporation 25

26 mverify Testing System End-to-End Edge to Core Integrated functional and performance testing Test objects XML performance measurements Adapters for Windows Mobile, Web Services, ODBC, *nix command line Forthcoming Profile-based test generation Adapters and plug-ins for many other platforms 2007 mverify Corporation 26

27 MTS/RPM Console Host Agent Host MTS Console TEST RUN REPORTS MTS Test Agent Agent Host MTS Test Agent MTS Remote Agent RPM Plug In Under Test MTS Remote Agent RPM Plug In Host Under Test Under Test MTS Remote Agent RPM Plug In Server Host Under Test Server Under Test Host Under Test MTS Remote Agent Host Under Test may be RPM Server ü Cell Phone Plug In Under Test ü PDA ü Desktop Server Host Under Test ü Server ü Embedded Processor ü Network Equipment ü Access Point ü Base Station 2007 mverify Corporation 27

28 MTS/RPM Integrates Functional and Performance Test MTS 1.5: RPM Plug-in for Windows Mobile Plug-ins for Win32, *nix coming soon 2007 mverify Corporation 28

29 Q & A 2007 mverify Corporation 29