Continuous Performance Tests

Transcription

1 Continuous Performance Tests IPC Spring 2012 Kore Nordmann Manuel Pichler June 6, 2012 Continuous Performance Tests 1 / 25

2 About us Degree in computer sience More than 10 years of professional PHP Open source enthusiasts Contributing to various FLOSS projects Qafoo passion for software quality Continuous Performance Tests 2 / 25

3 About us Degree in computer sience More than 10 years of professional PHP Open source enthusiasts Contributing to various FLOSS projects Qafoo passion for software quality Continuous Performance Tests 2 / 25

4 About us Degree in computer sience More than 10 years of professional PHP Open source enthusiasts Contributing to various FLOSS projects Qafoo passion for software quality Continuous Performance Tests 2 / 25

5 About us Degree in computer sience More than 10 years of professional PHP Open source enthusiasts Contributing to various FLOSS projects Co-founders of Qafoo passion for software quality Continuous Performance Tests 2 / 25

6 About us Degree in computer sience More than 10 years of professional PHP Open source enthusiasts Contributing to various FLOSS projects Co-founders of Qafoo passion for software quality We help people to create high quality web applications. Continuous Performance Tests 2 / 25

7 About us Degree in computer sience More than 10 years of professional PHP Open source enthusiasts Contributing to various FLOSS projects Co-founders of Qafoo passion for software quality We help people to create high quality web applications. Continuous Performance Tests 2 / 25

8 Outline Motivation Conclusion Continuous Performance Tests 3 / 25

9 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25

10 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25

11 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25

12 Motivation Why should we do performance tests? Locate unknown bottlenecks Mesaure behaviour of the full stack Why should we do that continuously? Find performance regressions Ensure optimizations are persistent Continuous Performance Tests 4 / 25

13 Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25

14 Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25

15 Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25

16 Often used tools Often misused tools sieve ApacheBench (ab) Testing for micro-optimizations Evaluating Hello-World-examples of Frameworks Continuous Performance Tests 5 / 25

17 The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25

18 The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25

19 The problem is more complex Your task: Create a new webshop Assume it s march The deadline is October this year, right before Christmas Continuous Performance Tests 6 / 25

20 A webshop Awesome Shop 0 articles 0.00 Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25

21 A webshop Awesome Shop Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 2 articles ,-- 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25

22 A webshop Awesome Shop 2 articles Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Comments 1337,-- 5 items in stock Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25

23 A webshop Awesome Shop 2 articles Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 1337,-- 5 items in stock Comments Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25

24 A webshop Awesome Shop 2 articles Smartphone Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Comments 1337,-- 5 items in stock Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor. Continuous Performance Tests 7 / 25

25 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25

26 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25

27 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25

28 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25

29 A real tool for performance tests JMeter Complex user paths Concurrend requests Record on proxy Clustering Continuous Performance Tests 8 / 25

30 JMeter 101 Thread Group Controller Config Element Timer Sampler Continuous Performance Tests 9 / 25

31 Getting started Create a test plan Example: What do users actually do on your site? Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Tests 10 / 25

32 Getting started Create a test plan Example: What do users actually do on your site? Random browser User registration Sign on Shopping with checkout Commenting products Continuous Performance Tests 10 / 25

33 Using JMeter Demo Continuous Performance Tests 11 / 25

34 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25

35 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25

36 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25

37 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25

38 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25

39 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25

40 Automation Ant JMeter integration Automation of your environment Setting up different software versions Testing with different extensions Running different database setups Handling multi-node environments Plain Ant or Puppet, Chef, Vagrant Continuous Performance Tests 12 / 25

41 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Tests 13 / 25

42 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Tests 13 / 25

43 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Tests 13 / 25

44 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Tests 13 / 25

45 Apache Ant example 26 < target name= build apc bytecode cache user cache f i l e 27 depends= setup apc bytecode f i l e cache, build 28 description= > Run with APC opcode cache and f i l e based cache. /> < target name= build apc bytecode cache user cache apc 31 depends= setup apc bytecode apc cache, b u i l d 32 description= > Run with APC opcode cache and APC based cache. /> 211 < target name= remote exec p a r a l l e l > 212 <subant t a r g e t = $ { t a r g e t } i n h e r i t a l l = t r u e > 213 < f i l e s e t d i r = $ { p r o j e c t. d i r } i n c ludes= server. xml /> 214 </ subant> 215 </ target > < target name= remote exec > 218 <sshexec command= $ {command} 219 username= $ { ssh. username } 220 password= $ { ssh. password } 221 host= $ { hostname } 222 trust = true /> 223 </ target > Continuous Performance Tests 13 / 25

46 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Tests 14 / 25

47 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Tests 14 / 25

48 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Tests 14 / 25

49 Apache Ant example 454 < target name= restart host > 455 < antcall target= remote exec > 456 <param name= command value= shutdown r now /> 457 </ a n t c a l l > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } to stop... /> 460 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 461 <not> 462 <http url = http : / / $ { hostname } /> 463 </ not> 464 </ w a i t f o r > <echo taskname= w a i t f o r message= Wait f o r $ { hostname } i s up again... /> 467 < waitfor maxwait= 5 maxwaitunit= minute checkevery= 100 > 468 <http url = http : / / $ { hostname } /> 469 </ w a i t f o r > 470 </ target > Continuous Performance Tests 14 / 25

50 Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Tests 15 / 25

51 Apache Ant example 421 < target name= jmeter depends= settings i n i t, start jmeter /> < t a r g e t name= s t a r t jmeter > 424 < antcall target= start jmeter before hook /> <jmeter jmeterhome= $ { l o c a l. jmeter.home. d i r } 427 r e s u l t l o g = $ { l o c a l. jmeter. log. f i l e } 428 t e s t p l a n = $ { l o c a l. jmeter. t e s t. d i r } / $ { jmeter. f i l e } > <p r op e r ty name= jmeter. data. d i r value= $ { l o c a l. p r o j e c t. data. d i r } /> 431 <property name= jmeter. rampup. time value= $ { jmeter. rampup. time } /> 432 <property name= jmeter. execution. time value= $ { jmeter. execution. time } /> 433 </ jmeter > < antcall target= start jmeter after hook /> 436 </ target > Continuous Performance Tests 15 / 25

52 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Tests 16 / 25

53 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Tests 16 / 25

54 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Tests 16 / 25

55 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Tests 16 / 25

56 Apache Ant example 351 < target name= backup data from host > 352 <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. e r r o r. log } 353 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} e r r o r. log 354 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. webserver. access. log } 357 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} access. log 358 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. php. e r r o r. log } 361 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} php errors. log 362 trust = true /> <scp r e m o t e f i l e = $ { ssh. username } : $ { ssh. password }@${ hostname } : $ { remote. basedir } / s c r i p t s / load. log 365 l o c a l t o f i l e = $ { l o c a l. b u i l d d i r } / $ { hostname} load. log 366 trust = true /> </ target > Continuous Performance Tests 16 / 25

57 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

58 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

59 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

60 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

61 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

62 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

63 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

64 Hardware Test in a realistic environment If your software runs in the cloud test against virtual environment If you use real hardware, also test against real hardware JMeter might have serious hardware requirements Use real hardware Use the biggest VM available Ensure that not the JMeter hardware is the bottleneck Be sure that the network is not the bottleneck See ifstat, iftop Measure several system metrics See vmstat, top Continuous Performance Tests 17 / 25

65 Extended system metrics Response time Requests per second 75 Seconds 300 R/s :00 05:00 09:00 13:00 17:00 21:00 01:00 00:00 05:00 09:00 13:00 17:00 21:00 01:00 Minimum Average Maximum Average CPU usage Memory usage 100 Percent 100 Percent System 05:00 09:00 13:00 17:00 21:00 01:00 05:00 User 05:00 09:00 13:00 17:00 21:00 01:00 05:00 Used Memory Shared Cached Continuous Performance Tests 18 / 25

66 Continuous Performance Jenkins Sonar Plugin available JMeter Plugin available Continuous Performance Tests 19 / 25

67 Continuous Performance Jenkins Sonar Plugin available JMeter Plugin available Continuous Performance Tests 19 / 25

68 Continuous Performance Jenkins Sonar Plugin available JMeter Plugin available Continuous Performance Tests 19 / 25

69 Continuous Performance Jenkins Sonar Plugin available JMeter Plugin available Continuous Performance Tests 19 / 25

70 Continuous testing with Jenkins Demo Continuous Performance Tests 20 / 25

71 Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: PIs per month sold articles per month registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25

72 Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: PIs per month sold articles per month registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25

73 Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: PIs per month sold articles per month registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25

74 Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: PIs per month sold articles per month registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25

75 Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: PIs per month sold articles per month registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25

76 Getting realistic settings Your customer usually only knows his AGOV/IVW values, like: PIs per month sold articles per month registrations per month Believe us, you will get nothing more :-) Ask for access statistics before Christmas and the ratio compared with regular months Ask for hours with the highest conversion rates Maybe get the aggregated access logs from existing similar shops Continuous Performance Tests 21 / 25

77 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

78 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

79 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

80 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

81 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

82 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

83 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

84 Example calculation Customer provided values, for a classic webshop: PIs per month sold articles per month registrations per month Per day: /26 = (non-business) Per hour: /12 = (national shop) Peak hour: = (18:00 to 19:00) Per second: /3600 = 7PI/s Add Christmas / Easter bonus Add launch bonus So PI/s should be safe? Spare resources for scaling are always a business decision Provide with trade-off: Costs vs. downtime / slowness Fail gracefully Continuous Performance Tests 22 / 25

85 Outline Motivation Conclusion Continuous Performance Tests 23 / 25

86 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25

87 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25

88 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25

89 Conclusion Plan your test scenario Use realisitic thresholds Choose the right tool with care Continuous Performance Tests 24 / 25

90 Thanks for listening Rate this talk More about us: Continuous Performance Tests 25 / 25