Continuous Performance Testing

Transcription

1 Continuous Performance Testing Confoo.ca Kore Nordmann 27. Feb 2013 Continuous Performance Testing 1 / 30

2 About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30

3 About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30

4 About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30

5 About me Degree in computer sience Professional PHP since 2000 Open source enthusiast Passion for Software Design Automated Testing Continuous Performance Testing 2 / 30

6 Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30

7 Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30

8 Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30

9 Co-founder of Qafoo passion for software quality Helping people to create high quality web applications. Expert consulting Individual training Get a training on object oriented design for your team! Continuous Performance Testing 3 / 30

10 Outline Motivation Conclusion Continuous Performance Testing 4 / 30

11 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30

12 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30

13 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30

14 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Testing 5 / 30

15 Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30

16 Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30

17 Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30

18 Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30

19 Often used tools Often misused tools siege ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Useful tools, I won t talk about xdebug Profiling xhprof Database profiling tools System metrics (Graphite,... ) Continuous Performance Testing 6 / 30

20 The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Testing 7 / 30

21 The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Testing 7 / 30

22 The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Testing 7 / 30

23 A webshop Awesome Shop 0 articles 0.00 Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Testing 8 / 30

24 A webshop 2 articles Continuous Performance Testing 8 / 30

25 A webshop Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- 5 items in stock Continuous Performance Testing 8 / 30

26 A webshop Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Testing 8 / 30

27 A webshop 1337,-- 5 items in stock Continuous Performance Testing 8 / 30

28 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30

29 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30

30 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30

31 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30

32 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Testing 9 / 30

33 JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30

34 JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30

35 JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30

36 JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30

37 JMeter 101 Thread Group An execution plan (like a user registration) Controller Controls how samplers are executed (loop, random,... ) Config Element Timer Sampler Configration, optionally from external sources Defining timing constraints for sampler execution Perform the actual work (like HTTP requests) Continuous Performance Testing 10 / 30

38 Getting started Create a test plan Example: What do users actually do on your site? Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Testing 11 / 30

39 Getting started Create a test plan Example: What do users actually do on your site? Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Testing 11 / 30

40 Using JMeter Demo Continuous Performance Testing 12 / 30

41 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30

42 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30

43 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30

44 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30

45 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30

46 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30

47 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Testing 13 / 30

48 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30

49 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30

50 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30

51 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30

52 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Testing 14 / 30

53 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30

54 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30

55 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30

56 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Testing 15 / 30

57 Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Testing 16 / 30

58 Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Testing 16 / 30

59 Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Testing 16 / 30

60 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Testing 17 / 30

61 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Testing 17 / 30

62 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Testing 17 / 30

63 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Testing 17 / 30

64 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Testing 17 / 30

65 Test Setup What do we actually test? Continuous Performance Testing 18 / 30

66 Test Setup jmeter Continuous Performance Testing 19 / 30

67 Test Setup iftstat, iftop jmeter Continuous Performance Testing 19 / 30

68 Test Setup iftstat, iftop jmeter Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd Continuous Performance Testing 19 / 30

69 Test Setup iftstat, iftop jmeter Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd vmstat, top Slave Master Continuous Performance Testing 19 / 30

70 Test Setup iftstat, iftop jmeter Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd vmstat, top Slave Master Services Continuous Performance Testing 19 / 30

71 Test Setup iftstat, iftop jmeter Response Times Load Balancer Verify Sessions vmstat, top FrontEnd FrontEnd FrontEnd Profiling, xhprof, Response Times vmstat, top Slave Master Slow Query Log, Services Continuous Performance Testing 19 / 30

72 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

73 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

74 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

75 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

76 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

77 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

78 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

79 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Testing 20 / 30

80 Extended Statistics Demo Continuous Performance Testing 21 / 30

81 Continuous Performance Plugins available for: Jenkins Sonar Maintaining all those servers can be expensive Continuous Performance Testing 22 / 30

82 Continuous Performance Plugins available for: Jenkins Sonar Maintaining all those servers can be expensive Continuous Performance Testing 22 / 30

83 Continuous testing with Jenkins Continuous Performance Testing 23 / 30

84 Questions Common questions: Can we survive christmas? What is the maximum we can accomplish with the current setup? Does (APC XCache MemCache) really help us? Continuous Performance Testing 24 / 30

85 Questions Common questions: Can we survive christmas? What is the maximum we can accomplish with the current setup? Does (APC XCache MemCache) really help us? Continuous Performance Testing 24 / 30

86 Questions Common questions: Can we survive christmas? What is the maximum we can accomplish with the current setup? Does (APC XCache MemCache) really help us? Continuous Performance Testing 24 / 30

87 Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: PIs per month sold articles per month registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30

88 Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: PIs per month sold articles per month registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30

89 Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: PIs per month sold articles per month registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30

90 Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: PIs per month sold articles per month registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30

91 Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: PIs per month sold articles per month registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30

92 Getting realistic settings Ask for real access logs: Extract exact request model from those Your customer usually only knows very broad values, like: PIs per month sold articles per month registrations per month Very seldom, that you get more: Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Testing 25 / 30

93 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

94 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

95 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

96 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

97 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

98 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

99 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

100 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Testing 26 / 30

101 Verification Do your requests actually model customer requirements? Compare generated accesss logs with real access logs Compare user registrations / checkouts per hour with requested values Continuous Performance Testing 27 / 30

102 Outline Motivation Conclusion Continuous Performance Testing 28 / 30

103 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30

104 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30

105 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30

106 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Testing 29 / 30

107 Thanks for Listening Rate this talk: Stay in touch Kore Rent a web quality expert: Continuous Performance Testing 30 / 30

108 Thanks for Listening Rate this talk: Stay in touch Kore Rent a web quality expert: Continuous Performance Testing 30 / 30