Continuous Performance Testing Confoo.ca Kore Nordmann (@koredn) 27. Feb 2013 Continuous Performance Testing 1 / 30
About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30
About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30
About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30
About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30
Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. http://qafoo.com Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30
Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. http://qafoo.com Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30
Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. http://qafoo.com Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30
Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. http://qafoo.com Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30
Outline Motivation Conclusion Continuous Performance Testing 4 / 30
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30
Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30
Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30
Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30
Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30
Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30
Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30
The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Testing 7 / 30
The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Testing 7 / 30
The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Testing 7 / 30
A webshop Awesome Shop 0 articles 0.00 Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Testing 8 / 30
A webshop 2 articles 42.32 Continuous Performance Testing 8 / 30
A webshop Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- 5 items in stock Continuous Performance Testing 8 / 30
A webshop Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Testing 8 / 30
A webshop 1337,-- 5 items in stock Continuous Performance Testing 8 / 30
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30
A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30
JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30
JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30
JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30
JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30
JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30
Getting started Create a test plan Example: What do users actually do on your site? Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Testing 11 / 30
Getting started Create a test plan Example: What do users actually do on your site? Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Testing 11 / 30
Using JMeter Demo Continuous Performance Testing 12 / 30
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30
Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30
Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> 29 30 < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > 216 217 < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30
Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> 29 30 < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > 216 217 < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30
Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> 29 30 < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > 216 217 < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30
Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> 29 30 < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > 216 217 < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30
Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> 29 30 < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > 216 217 < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30
Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > 458 459 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > 465 466 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30
Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > 458 459 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > 465 466 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30
Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > 458 459 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > 465 466 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30
Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > 458 459 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > 465 466 <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30
Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> 422 423 < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> 425 426 <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > 429 430 <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > 434 435 < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Testing 16 / 30
Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> 422 423 < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> 425 426 <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > 429 430 <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > 434 435 < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Testing 16 / 30
Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> 422 423 < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> 425 426 <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > 429 430 <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > 434 435 < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Testing 16 / 30
Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> 355 356 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> 359 360 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> 363 364 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> 367 368 </ target > Continuous Performance Testing 17 / 30
Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> 355 356 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> 359 360 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> 363 364 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> 367 368 </ target > Continuous Performance Testing 17 / 30
Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> 355 356 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> 359 360 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> 363 364 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> 367 368 </ target > Continuous Performance Testing 17 / 30
Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> 355 356 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> 359 360 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> 363 364 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> 367 368 </ target > Continuous Performance Testing 17 / 30
Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> 355 356 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> 359 360 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> 363 364 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> 367 368 </ target > Continuous Performance Testing 17 / 30
Test Setup What do we actually test? Continuous Performance Testing 18 / 30
Test Setup jmeter Continuous Performance Testing 19 / 30
Test Setup iftstat, iftop jmeter Continuous Performance Testing 19 / 30
Test Setup iftstat, iftop jmeter Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd Continuous Performance Testing 19 / 30
Test Setup iftstat, iftop jmeter Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd vmstat, top Slave Master Continuous Performance Testing 19 / 30
Test Setup iftstat, iftop jmeter Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd vmstat, top Slave Master Services Continuous Performance Testing 19 / 30
Test Setup iftstat, iftop jmeter Response Times Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd Profiling, xhprof, Response Times vmstat, top Slave Master Slow Query Log, Services Continuous Performance Testing 19 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30
Extended Statistics Demo Continuous Performance Testing 21 / 30
Continuous Performance Plugins available for: Jenkins Sonar Maintaining all those servers can be expensive Continuous Performance Testing 22 / 30
Continuous Performance Plugins available for: Jenkins Sonar Maintaining all those servers can be expensive Continuous Performance Testing 22 / 30
Continuous testing with Jenkins Continuous Performance Testing 23 / 30
Questions Common questions: Can we survive christmas? What is the maximum we can accomplish with the current setup? Does (APC XCache MemCache) really help us? Continuous Performance Testing 24 / 30
Questions Common questions: Can we survive christmas? What is the maximum we can accomplish with the current setup? Does (APC XCache MemCache) really help us? Continuous Performance Testing 24 / 30
Questions Common questions: Can we survive christmas? What is the maximum we can accomplish with the current setup? Does (APC XCache MemCache) really help us? Continuous Performance Testing 24 / 30
Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30
Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30
Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30
Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30
Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30
Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Example calculation Customer provided values, for a classic webshop: 1.000.000 PIs per month 30.000 sold articles per month 45.000 registrations per month Per day: 1.000.000/26 = 38.500 (non-business) Per hour: 38.500/12 = 3.200 (national shop) Peak hour: 3.200 8 = 25.500 (18:00 to 19:00) Per second: 25.500/3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So... 50 PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30
Verification Do your requests actually model customer requirements? Compare generated accesss logs with real access logs Compare user registrations / checkouts per hour with requested values Continuous Performance Testing 27 / 30
Outline Motivation Conclusion Continuous Performance Testing 28 / 30
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30
Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30
Thanks for Listening Rate this talk: https://joind.in/7842 Stay in touch Kore Nordmann kore@qafoo.com @koredn / @qafoo Rent a web quality expert: http://qafoo.com Continuous Performance Testing 30 / 30
Thanks for Listening Rate this talk: https://joind.in/7842 Stay in touch Kore Nordmann kore@qafoo.com @koredn / @qafoo Rent a web quality expert: http://qafoo.com Continuous Performance Testing 30 / 30