Performance Testing Crash Course Dustin Whittle @dustinwhittle
The performance of your application affects your business more than you might think. Top engineering organizations think of performance not as a nice-to-have, but as a crucial feature of their product. Those organizations understand that performance has a direct impact on user experience and, ultimately, their bottom line. Unfortunately, most engineering teams do not regularly test the performance and scalability of their infrastructure. Join this session to find out about the latest performance testing tools and why your team should add performance testing to your agile development process. 2
Dustin Whittle dustinwhittle.com @dustinwhittle San Francisco, California, USA Technologist, Traveler, Pilot, Skier, Diver, Sailor, Golfer 3
What I have worked on Developer Evangelist @ Consultant & Trainer @ Developer Evangelist @ 4
Why does performance matter? 5
Microsoft found that Bing searches that were 2 seconds slower resulted in a 4.3% drop in revenue per user 6
When Mozilla shaved 2.2 seconds off their landing page, Firefox downloads increased 15.4% 7
Making Barack Obama s website 60% faster increased donation conversions by 14% 8
Amazon and Walmart increase revenue 1% for every 100ms of improvement 9
10
Performance directly impacts the bottom line 11
12
HealthCare.gov HealthCare.gov 13
Treat performance as a feature! 14
AN EXAMPLE SPRING APP TO TEST PERFORMANCE 15
16
17
GIT CLONE HTTPS://GITHUB.COM/ CLOUDFOUNDRY- SAMPLES/HELLO- SPRING- CLOUD 18
MVN PACKAGE 19
20
21
CF PUSH 22
HTTP://SPRINGONE- DEMO.CFAPPS.IO/ 23
Tools of the trade for performance testing 24
Understand your baseline performance 25
Static vs Hello World vs Application 26
Apache Bench 27
ab -c 1 -t 10 -k http://acmedemoapp.com/ 28
Benchmarking acmedemoapp.com (be patient) Finished 187 requests Server Software: Nginx/1.6 Server Hostname: acmedemoapp.com Server Port: 80 Document Path: / Document Length: 5217 bytes Concurrency Level: 1 Time taken for tests: 10.039 seconds Complete requests: 187 Failed requests: 0 Keep-Alive requests: 187 Total transferred: 1021768 bytes HTML transferred: 975579 bytes Requests per second: 18.63 [#/sec] (mean) Time per request: 53.687 [ms] (mean) 29
ab -c 10 -t 10 -k http://acmedemoapp.com/ 30
Benchmarking acmedemoapp.com (be patient) Finished 659 requests Server Software: Nginx/1.6 Server Hostname: acmedemoapp.com Server Port: 80 Document Path: / Document Length: 5217 bytes Concurrency Level: 10 Time taken for tests: 10.015 seconds Complete requests: 659 Failed requests: 0 Keep-Alive requests: 659 Total transferred: 3600776 bytes HTML transferred: 3438003 bytes Requests per second: 65.80 [#/sec] (mean) Time per request: 151.970 [ms] (mean) 31
Siege 32
siege -c 10 -b -t 10S http://acmedemoapp.com/ 33
** SIEGE 3.0.6 ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 623 hits Availability: 100.00 % Elapsed time: 9.57 secs Data transferred: 3.10 MB Response time: 0.15 secs Transaction rate: 65.10 trans/sec Throughput: 0.32 MB/sec Concurrency: 9.91 Successful transactions: 623 Failed transactions: 0 Longest transaction: 0.30 Shortest transaction: 0.10 34
Crawl the entire app to discover all urls 35
sproxy -o./urls.txt 36
SPROXY v1.02 listening on port 9001...appending HTTP requests to:./urls.txt...default connection timeout: 120 seconds 37
wget -r -l 0 -t 1 --spider -w 1 -e "http_proxy = http://127.0.0.1:9001" "http://acmedemoapp.com/" sort -u -o urls.txt urls.txt 38
http://acmedemoapp.com/ http://acmedemoapp.com/about http://acmedemoapp.com/cart http://acmedemoapp.com/currency/change/eur http://acmedemoapp.com/currency/change/gbp http://acmedemoapp.com/currency/change/usd http://acmedemoapp.com/login http://acmedemoapp.com/register http://acmedemoapp.com/t/brand/bookmania http://acmedemoapp.com/t/category/books http://acmedemoapp.com/t/category/mugs http://acmedemoapp.com/t/category/stickers http://acmedemoapp.com/terms-of-service 39
Benchmark traffic across all unique urls with siege 40
siege -v -c 50 -i -t 3M -f urls.txt -d 10
** SIEGE 3.0.6 ** Preparing 10 concurrent users for battle. The server is now under siege... Lifting the server siege... done. Transactions: 623 hits Availability: 100.00 % Elapsed time: 9.57 secs Data transferred: 3.10 MB Response time: 0.15 secs Transaction rate: 65.10 trans/sec Throughput: 0.32 MB/sec Concurrency: 9.91 Successful transactions: 623 Failed transactions: 0 Longest transaction: 0.30 Shortest transaction: 0.10 42
Apache JMeter 43
44
45
Multi-Mechanize is an open source framework for performance and load testing 46
pip install multi-mechanize 47
multimech-newproject demo 48
import requests class Transaction(object): def run(self): r = requests.get( http://acmedemoapp.com/) r.raw.read() 49
import mechanize import time class Transaction(object): def run(self): br = mechanize.browser() br.set_handle_robots(false) start_timer = time.time() resp = br.open( http://acmedemoapp.com/) resp.read() latency = time.time() - start_timer self.custom_timers['homepage'] = latency start_timer = time.time() resp = br.open( http://acmedemoapp.com/cart') resp.read() latency = time.time() - start_timer self.custom_timers['cart'] = latency assert (resp.code == 200) 50
[global] run_time = 10 rampup = 5 51
multimech-run demo 52
53
What about when you need more than one machine? 54
Who lives in the cloud? 55
Bees with Machine Guns 56
A utility for arming (creating) many bees (micro EC2 instances) to attack (load test) targets (web applications) 57
pip install beeswithmachineguns 58
# ~/.boto [Credentials] aws_access_key_id=xxx aws_secret_access_key=xxx [Boto] ec2_region_name = us-west-2 ec2_region_endpoint = ec2.us-west-2.amazonaws.com 59
bees up -s 2 -g default -z us-west-2b -i ami-bc05898c -k appdynamics-dustinwhittle-aws-uswest-2 -l ec2-user 60
Connecting to the hive. Attempting to call up 2 bees. Waiting for bees to load their machine guns....... Bee i-3828400c is ready for the attack. Bee i-3928400d is ready for the attack. The swarm has assembled 2 bees. 61
bees report 62
Read 2 bees from the roster. Bee i-3828400c: running @ 54.212.22.176 Bee i-3928400d: running @ 50.112.6.191 63
bees attack -n 1000 -c 50 -u http://acmedemoapp.com/ 64
Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will fire 500 rounds, 25 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. Offensive complete. Complete requests: 1000 Requests per second: 306.540000 [#/sec] (mean) Time per request: 163.112000 [ms] (mean) 50% response time: 151.000000 [ms] (mean) 90% response time: 192.000000 [ms] (mean) Mission Assessment: Target crushed bee offensive. The swarm is awaiting new orders. 65
bees attack -n 100000 -c 1000 -u http://acmedemoapp.com/ 66
Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will fire 50000 rounds, 500 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. Offensive complete. Complete requests: 100000 Requests per second: 502.420000 [#/sec] (mean) Time per request: 360.114000 [ms] (mean) 50% response time: 451.000000 [ms] (mean) 90% response time: 402.000000 [ms] (mean) Mission Assessment: Target crushed bee offensive. The swarm is awaiting new orders. 67
Read 2 bees from the roster. Connecting to the hive. Assembling bees. Each of 2 bees will fire 50000 rounds, 500 at a time. Stinging URL so it will be cached for the attack. Organizing the swarm. Bee 0 is joining the swarm. Bee 1 is joining the swarm. Bee 0 is firing his machine gun. Bang bang! Bee 0 lost sight of the target (connection timed out). Bee 1 lost sight of the target (connection timed out). Offensive complete. Target timed out without fully responding to 2 bees. No bees completed the mission. Apparently your bees are peace-loving hippies. The swarm is awaiting new orders. 68
bees down 69
Read 2 bees from the roster. Connecting to the hive. Calling off the swarm. Stood down 2 bees. 70
locust.io https://github.com/locustio/locust 71
PIP INSTALL LOCUSTIO 72
73
74
LOCUST HOST= HTTP://ACMEDEMOAPP.COM/ 75
76
77
There are many load testing tools.. Gatling.io Wrk Vegeta Tsung 78
What about the client side? 79
In modern web applications more latency comes from the clientside than the server-side. 80
Google PageSpeed 81
Google PageSpeed Insights 82
83
84
WBench
gem install wbench 86
wbench http://dustinwhittle.com/ 87
Automate clientside performance testing with Grunt
Use Bower (for dependencies), Grunt/Gulp (for automation), and Yeoman (for bootstrapping) (checkout jhipster too) 90
How many people understand exactly how fast their site runs in production? 91
Track performance in development and production 92
Instrument everything = code, databases, caches, queues, third party services, and infrastructure. 93
Statsd + Graphite + Grafana 94
95
Track performance of end users 96
97
webpagetest.org 98
99
100
101
SiteSpeed.io 102
103
104
105
106
107
108
Load testing services from the cloud 109
110
111
112
Test for failures NetFlix Simian Army + Chaos Monkey What happens if you lose a caching layer? What happens if dependencies slow down? 113
Best Practices Treat performance as a feature Capacity plan and load test the server- side Optimize and performance test the client- side Understand your starting point Instrument everything Monitor performance in development and production Measure the difference of every change Automate performance testing in your build and deployment process Understand how failures impact performance 114
Integrate automated performance testing into continuous integration for server-side and client-side 115
Understand the performance implications of every deployment and package upgrade 116
Monitor end user experience from end to end in production 117
Questions? 118
Find these slides on SpeakerDeck https://speakerdeck.com/dustinwhittle 119