mverify A Million Users in a Box Performance Testing Mobile and Multi-Tier Applications Chicago Quality Assurance Association June 26, 2007 Robert V. Binder mverify Corporation Bob_Binder@mverify.com 312 881-7337 x1001 www.mverify.com
Goals of Performance Testing Validate time requirements/expectations Validate utilization requirements/expectations Validate capacity requirements/expectations Reveal load-related bugs Prove compliance: SLAs, contracts, competitive rankings Fire-drill for recovery Assess robustness to shocks 2007 mverify Corporation www.mverify.com 2
Business Impact/ROI In 2002, slow e-commerce downloads lead to an estimated $25 billion of abandoned transactions In 2005, Google s 15 minute outage estimated to have cost at least $150,000 in lost ad revenue Recent study: nearly two-thirds of mobile employees rank poor response time as a significant inhibitor to working remotely over a VPN 2007 mverify Corporation www.mverify.com 3
Business Impact/ROI Avoidable Costs Lost revenue Lost user productivity Lost IT productivity Overtime payments Wasted goods Fines Solution Cost Software Tools Hardware Staffing Services Training Risk Mitigation Enterprise demise Law Suits Negative Publicity Personnel morale One 9 Two 9s Three 9s Four 9s Five 9s Six 9s IBM. "Maximizing Web site Availability," February 2002 2007 mverify Corporation www.mverify.com 4
Basic Objectives: RASP Reliability Availability Scalability Probability of a failure occurring within a certain period of time Percent achieved up-time, not including scheduled downtime The range load for which an incremental input consumes the same resources Performance The rate at which work is done 2007 mverify Corporation www.mverify.com 5
Reliability Reliability: probability of non-failure Total operational hours or transactions Entire user population Can be estimated during test, if tests are sufficiently realistic 2007 mverify Corporation www.mverify.com 6
Availability the nines Annual Unscheduled Downtime Six nines 99.9999% 32 seconds five nines 99.999% 5 minutes four nines 99.99% 53 minutes Three nines 99.9% 8.8 hours two nines 99% 87 hours (3.6 days) One nine 90% 876 hours (36 days) Availability = percent up-time MTTR: mean time to recover, repair, restart Availability = 1 / 1 + (MTTR Reliability) 2007 mverify Corporation www.mverify.com 7
Some Data Points Reliability (Failures/million hours) Availability, 6 min MTTR NT 4.0 Desktop 82,000 0.999000000 Windows 2K Server 36,013 0.999640000 Common Light Bulb 1,000 0.999990000 Stepstone OO Framework 5 0.999999500 Telelabs Digital Cross Connect 3 0.999999842 2007 mverify Corporation www.mverify.com 8
Performance Metrics Response Time Round-trip time Throughput Aggregate transaction processing rate Utilization Average % busy Failure Intensity Recovery Time Avg Resp, Sec Trans/Sec 0% 100% Utilization 2007 mverify Corporation www.mverify.com 9
Strategies Performance Testing Assess compliance with performance goals Assess compliance with resource utilization goals Provides data to estimate reliability, availability Stress Testing, Load Testing Assess response to over-load scenarios Assess recovery from failure modes 2007 mverify Corporation www.mverify.com 10
Strategies Benchmarks Assess throughput for open standard test suite Scalability Assess performance linearity Profiling Identify utilization bottlenecks by component 2007 mverify Corporation www.mverify.com 11
Typical Server Side Setup Emulated Emulated Server(s) Under Test Internal LAN Emulated 2007 mverify Corporation www.mverify.com 12
Issues Emulation Machines Synchronization, overall test execution Capacity Multi-homed, test control subnet Test vs. Production Systems Separate server farm? Network contention Isolation versus scale/scope Version/Configuration control System under test Test environment Database set/reset 2007 mverify Corporation www.mverify.com 13
Issues Actual end-user/customer experience? Network latency, QoS Thin clients? Browser, client software versions OS? 2007 mverify Corporation www.mverify.com 14
Edge Monitoring Emulated Emulated Server(s) Under Test Monitored Internal LAN Internet Emulated Monitored 2007 mverify Corporation www.mverify.com 15
Issues Monitoring Machines Synchronization Achieving desired test input at desired time Capacity Data collection Availability Security (beta test agreement?) Network configuration DMZ Equipment, setup, security considerations 2007 mverify Corporation www.mverify.com 16
Connectivity a Wild Card Emulated Emulated Emulated Internal LAN Server (s) Under Test Random latency, jitter, lost packets, re-ordered packets, re-routed packets, duplicate packets, bandwidth restrictions, bit-error, background load, QoS, operational events Internet Monitored Monitored 2007 mverify Corporation www.mverify.com 17
Controlled Connectivity Emulated Emulated Emulated Internal LAN Server (s) Under Test Controlled latency, jitter, lost packets, re-ordered packets, re-routed packets, duplicate packets, bandwidth restrictions, biterror, background load, QoS, operational events Network Emulator Monitored Monitored 2007 mverify Corporation www.mverify.com 18
Issues Complexity Impairment modeling Impairment emulator programming Coordination with emulated clients Coordination with monitored clients Specialized Skills Wire Shark (Ethereal) TCP log analysis 2007 mverify Corporation www.mverify.com 19
How to Maximize Reliability Combine realistic functional and load testing Representative variation in load and usage Supports reliability/availability estimation Saves time: more test goals supported with fewer tests Typically effective in finding weird bugs Security? Add abuse cases to the usage profile Interleave with normal traffic You play like you practice 2007 mverify Corporation www.mverify.com 20
Events Per Second Use Dynamic Loading The real world isn t flat Vary behavior rate for actor/actor group Arc Flat Internet fractal Negative ramp Positive ramp Random Spikes Square wave Waves 3000.000 2500.000 2000.000 1500.000 1000.000 500.000 0.000-5000 0 5000 10000 15000 20000 25000-500.000 Time (seconds) Actual Waves Loading 2007 mverify Corporation www.mverify.com 21
Case Study Event Simulator Script Writer SilkTest Java GUI DB Test Object Serializer Java Driver Java API Java Servers TX Formatter 3270 Driver MainFrame Test Oracle Custom Test Component 3 rd Party Product System Under Test Test Run Reports Comparator 2007 mverify Corporation www.mverify.com 22
Case Study Every test run unique and realistic Simulated user behavior to generate transactions Automatically submit in real time ~100,000 test cases per hour ~200 complete daily cycles Evaluated functionality and performance Controlled distributed heterogeneous test agents (Java, 4Test, Perl, SQL, Prolog) driving Java/CORBA GUI/API Five person team, huge productivity increase Achieved proven high reliability Last pre-release test run: ~500,000 events in two hours, no failures detected No production failures 2007 mverify Corporation www.mverify.com 23
Notes Capture/replay scripts Static think-time Can distort load and response time Performance Analysis Neil Gunther books and web site 2007 mverify Corporation www.mverify.com 24
Tools Open Source opensta PushToTest Grinder http://opensourcetesting.org/performance.php Scripting systems: Tcl, Perl, Ruby, Python Built-in Windows Perfmon *nix SNMP, others 2007 mverify Corporation www.mverify.com 25
mverify Testing System End-to-End Edge to Core Integrated functional and performance testing Test objects XML performance measurements Adapters for Windows Mobile, Web Services, ODBC, *nix command line Forthcoming Profile-based test generation Adapters and plug-ins for many other platforms 2007 mverify Corporation www.mverify.com 26
MTS/RPM Console Host Agent Host MTS Console TEST RUN REPORTS MTS Test Agent Agent Host MTS Test Agent MTS Remote Agent RPM Plug In Under Test MTS Remote Agent RPM Plug In Host Under Test Under Test MTS Remote Agent RPM Plug In Server Host Under Test Server Under Test Host Under Test MTS Remote Agent Host Under Test may be RPM Server ü Cell Phone Plug In Under Test ü PDA ü Desktop Server Host Under Test ü Server ü Embedded Processor ü Network Equipment ü Access Point ü Base Station 2007 mverify Corporation www.mverify.com 27
MTS/RPM Integrates Functional and Performance Test MTS 1.5: RPM Plug-in for Windows Mobile Plug-ins for Win32, *nix coming soon 2007 mverify Corporation www.mverify.com 28
Q & A 2007 mverify Corporation www.mverify.com 29