IS YOUR WEBSITE STRESSED? LOAD TESTING FOR PEAK PERFORMANCE. Noelle A. Stimely, University of California - San Francisco

Transcription

1 IS YOUR WEBSITE STRESSED? LOAD TESTING FOR PEAK PERFORMANCE Nelle A. Stimely, University f Califrnia - San Francisc SPEAKER BACKGROUND Nelle Stimely is presently a Senir Perfrmance Test Engineer at the University f Califrnia San Francisc. Nelle has been an Oracle Database Administratr, hlds certificatins in Oracle 8/8i/9i/10g and HP LadRunner. She has held varius psitins including Lead Database Administratr, Senir Database Administratr and Oracle Instructr during her sixteen year career. INTRODUCTION Have yu ever experienced an unexpected rush f users n a prductin system which nearly grund it t a halt? Have yu ever had t apprach management with a request fr mre resurces due t pr perfrmance during a perid f budgetary cnstraints within the rganizatin? This presentatin will discuss hw t ptentially avid these situatins in advance thugh lad testing. A lad test can be run t gauge end user experience r stress the database t the pint f breakage t determine where the limit lies. This presentatin will discuss the purpse f lad testing; hw, when and why t test using bth freeware and cmmercially available tls. It will als describe hw a prperly designed lad test stresses the system and hw t utilize the results fr tuning purpses. I will als cmpare actual lad testing versus Oracle s Real Applicatin Testing tl. Finally, I examine the reprts prduced and hw t best use thse results t determine current r future resurce needs. WHAT IS LOAD TESTING? Lad testing, stress testing, perfrmance testing can all mean different things t different peple. It is extremely imprtant when embarking n a lad test scenari t prperly define the scpe f the test yu are planning t perfrm. It is als equally imprtant t have management/stakehlder buy-in prir t mving frward t ensure the test has credibility. Lad testing can be defined as placing demands n a system t gauge its respnse under lad. The questin becmes which type f lad t use. Is the purpse f the test t push the system t the pint f breakage? Is the test being run with real wrld lads? Is it being run with anticipated future lads? Is there a certain aspect f the applicatin which needs t be tested? There are many questins which are raised when determining which type f lad test t perfrm. Determining the type is ne f the mst critical pints f lad testing. It will drive the design and makeup f the test. TYPES OF LOAD TESTS Transactin Respnse Testing A lad test may be useful when there is an issue where certain transactins perfrm prly n a cnsistent basis. D yu get calls frm end-users at certain pints in the mnth when specific transactins are run? Was the applicatin recently upgraded and transactins which perfrmed in minimal time nw perfrm unacceptability? Wuldn t yu want t knw and fix prly perfrming transactins prir t ging live n the new release/system? Define acceptable respnse time requirements. A gd methd t use is t state an average and a 90th percentile respnse time fr critical transactins. Fr example, if the maximum respnse time is 15 secnds and the 90 th percentile is 9 secnds it means there are 10% f respnse times between 9 t 15 secnds meaning there are sme transactins in this area which shuld be further investigated. If the test was run and fund t have a 90 th percentile f 14 secnds with the maximum being 15 it wuld shw the largest respnse times t be cnsistent. Examples f Transactin Respnse Testing: Order entry transactin must cmplete within 8 secnds. AP query must return results within 5 secnds. Airline flight reservatin must cmplete within 7 secnds. PDF attachment must uplad within 5 secnds. 1

2 End User Experience Testing Testing end user experience under lad is als an excellent methdlgy t use as actual human user satisfactin will be measured. During end user experience testing, the lad test is executed while a subset f users wrks n the system. By cnducting end user experience testing, sme applicatin issues which may nt appear in the final numbers can be identified. Examples f End User Experience Testing: Run full-scale web lad test while a subset f users lgs nt system t cnduct nrmal wrk activity. Have a sub-set f end users lg nt the system in the middle f the lad test t gauge perfrmance. Stress Test Under a stress test, a system will be pushed beynd the limits f nrmal peratin t find the breaking pint where the system will fail. This type f test wuld be used fr an envirnment designated as missin critical that cannt tlerate dwntime. The test wuld be fcused n discvering an applicatin s perfrmance under extreme cnditins t determine its availability and reliability. Stress testing wuld include cnditins f heavy lads, high cncurrency and limited resurces t find the system breaking pint. Operating system bugs, resurce, lcking and cde bugs can be lcated and crrected. The results f this test wuld prvide the necessary infrmatin as t where additinal resurces may be needed t avid dwntime. Examples f Stress Testing: Multiple users lgging int the system at ne time (100 users lg in at ne time) Heavy transactin lads. Users lgging int the system very rapidly (e.g. 1 user every secnd) Extended cncurrency times (25 users remain lgged int the system running heavy transactins fr an extended perid) Befre a lad test can be cnducted a baseline must first be established. The current system perfrmance must be dcumented and knwn prir t a lad test being perfrmed. Once a baseline has been gathered, system perfrmance imprvement r decline can be effectively identified. After running the lad test results can be cmpared against the baseline and perfrmance prblems can be easily identified and tuned. WHY LOAD TEST? There are many questins which may be answered thrugh lad testing. These questins shuld be asked whenever deplying a new applicatin, upgrading hardware/sftware and increasing user/data lads. Des the applicatin perfrm acceptably with a minimum amunt f users n the system? Des the applicatin respnd quickly with additinal cncurrent users? Are there hardware bttlenecks in the system? Can the applicatin handle a grwing number f users/data? What is the maximum amunt f lad the system can handle? These are nly a few f the questins which need t be asked when designing a lad test. The purpse f the lad test shuld be t answer these questins s the system will nt crash when the unexpected hits. Hw many f us have gtten the dreaded phne calls right after an upgrade where the end users are cmplaining abut bad perfrmance? Everyne wuld like t avid this situatin and if a prperly defined lad test was cnducted prir t g-live there is a very high likelihd these calls may be avided. PURPOSE OF LOAD TESTING Befre embarking n any lad test the purpse and gal(s) must be clearly defined and dcumented. All lad tests are run t eliminate risk and t mitigate ptential perfrmance prblems. Sme f questins asked might be: Will the system perfrm better r the same after the upgrade? 2

3 Can the system handle anther 300 users? Open enrllment is 6 weeks away can we handle the additinal traffic? The system is nt currently meeting perfrmance standards. Can we reprduce the issue and crrect it n an identical test system? The bttm line is perfrmance: specifically, end-user satisfactin. We all knw end users are never satisfied but if we can reprduce their experience then bttlenecks can be identified and crrected. The ultimate gal is t best predict the unexpected. In wrking t define the gal(s) f the lad test it is very imprtant t wrk with the applicatin wner(s)/management t gain buy-in. Mst f the time these are the peple n the frnt lines wh best knw the applicatin and knw where the prblems lie. These are the peple wh can tell yu where t fcus s the prblem gets slved and the test has validity. If the wrng area f the applicatin is tested, then it is wasted time and the prper test will nt be run. When designing the test it is imperative that a prper test plan be develped which requires the input f the applicatin wner. The applicatin wner will assist in designing the test cases necessary fr a successful test. They will be pint f cntact in creating a realistic test which adequately stresses the web applicatin. WEB APPLICATION COMPONENTS A web applicatin is made up f three distinct, but interrelated cmpnents. These cmpnents are: Web server presentatin tier Applicatin server business lgic tier Database server infrmatin tier. Any ne f these cmpnents culd act as a bttleneck in the applicatin. The key is t discver which cmpnent is the culprit. Anther way f saying things is that displaying the pretty parts f web pages is ne cmpnent. The thinking and business decisin making anther with cre infrmatin the last. All f these cmpnents wrk tgether in a jint unit that makes up the web applicatin. DEVELOPING THE LOAD TEST LOAD TEST TEAM After the lad test has been defined, then it is time t develp the test itself. First is t identify the lad test team. These will be the peple wh will assist in develping the test, utlining the gals, defining the hardware/sftware invlved and mnitring the system during the test fr perfrmance issues. Critical team members are as fllws (every envirnment is different, but these are sme f the primary participants): Applicatin wner(s) Database Administratr(s) Web Server Administratr(s) System Administratr(s) Netwrk Administratr(s) Develper(s) Prject Manager(s) Applicatin wners(s) will assist with creating test cases, identifying applicatin perfrmance prblems, defining/creating test data, creating a lad test prfile, setup ids and granting access. Database Administratr(s) knw hw the database functins n a day t day basis and will need t mnitr the database during the lad test. They will als be invlved with identifying perfrmance issues and tuning the system. Web Server Administratr(s) understand the web server envirnment and can assist yu with determining the current and prjected website hits. They will bth mnitr and tune the system. 3

4 System Administratr(s) understand the server envirnment be it Windws, UNIX r Linux. They are mstly invlved with mnitring and tuning the system during and after the lad test. They can als prvide yu with the current baseline system perfrmance. Netwrk Administratr(s) knw the netwrk and understand hw traffic perates in yur envirnment. They can als prvide baseline perfrmance and will mnitr and tune the netwrk after the test. Develper(s) knw the applicatin cde and hw it functins. If yur applicatin was designed in-huse r yu have access t the surce cde f the applicatin develpers will be invlved with tuning the cde after the test has been run. They will als be invlved with successive cde tuning effrts when/if the test is repeated. Prject Manager(s) may r may nt be invlved with the lad test depending n the scale. If a Prject Manager is invlved with the lad test make sure they understand the definitin f the test and the cmplexities invlved. They can either be yur best friend (eliminating scpe creep/n schedule) r wrst nightmare (d nt understand the prject/technlgy). Develping the prper team and team members is critical t the success f the prject. They will guide yu in develping, executing and final tuning f the lad test. Cmmunicatin is critical. Listening t the viewpints f the team members will help yu t identify perfrmance prblems s yu can create a rbust test which will accmplish all gals set. IDENTIFY THE TEST ENVIRONMENT The test envirnment used fr the lad/stress test shuld be as clse t an exact replica as the prductin envirnment as pssible. Exact replicas f prductin envirnments are extremely rare, but fr a reliable test a clse ne is preferred. When cnsidering the system t use keep in mind t cmpare the lgical and physical prductin and test architectures and the available tls n each. Under ideal cnditins, the test envirnment is an exact cpy f the prductin system nly with additinal lad testing tls and mnitring sftware installed. A significant cnsideratin f the cmplexity, lad and size f test t run is a measure f the similarity f the hardware, sftware and netwrk cnfiguratin f the test system. It is imprtant t keep in mind the results f the lad test are nly as gd as the test envirnment in which it is run. IDENTIFY THE PERFORMANCE ACCEPTANCE CRITERIA Befre embarking n the lad test design and creatin it is imprtant t have the desired perfrmance characteristics f the applicatin/database established. These perfrmance metrics can be defined by a variety f surces such as applicatin wners, end-users, database administratrs r a cmbinatin f stakehlders. Typical measures f perfrmance satisfactin may include respnse time, thrughput and resurce utilizatin. An example f respnse time wuld be a custmer s rder details must display in less than 3 secnds. Thrughput may be defined as being able t supprt 30 bk rders per secnd. Resurce utilizatin is defining CPU usage t be under 80% during the lad test. Other key pints which shuld als be cnsidered in defining perfrmance criteria are as fllws: Service Level Agreements (SLAs) End user expectatins Business requirements Wrklad key scenaris representing reality Perfrmance ptimizatin bjectives Scalability Range f anticipated wrklad cnditins Resurce utilizatin bjectives These are nly a few f the elements t keep in mind when develping perfrmance acceptance criteria. It is imprtant t have these bjectives nailed dwn s t design a rbust test which will meet everyne s gals. Having the prper criteria defined and agreed upn in advance will make identifying the perfrmance bttlenecks and tuning much easier and make fr a successful lad test. IDENTIFY APPLICATION SCENARIOS 4

5 The key t a lad test are the applicatin scenaris chsen t test. It is imperative t chse the prper scenaris and mix f scenaris t prperly stress the system t identify the perfrmance issues and bttlenecks. Many times the lad tester des nt have intimate knwledge f the applicatin and needs t heavily rely n end user and applicatin wner feedback. There are many questins t be asked in determining the scenari(s) t use. The first questin shuld be is when and where d the perfrmance prblems ccur? The peple mst likely t answer this questin wuld be the end users. They are n the frntlines and knw when and where the perfrmance declines. Als, the applicatin wners may have knwledge as well since in many cases, cmmunicate with the end users when the system des nt perfrm up t expectatin. It is als imprtant the applicatin wner/end user understand the gal(s) f the perfrmance test. If the peple assisting/designing the scenaris have a basic understanding f hw the lad test is ging t wrk, they can better assist with the prject. Cnsider what will be tested: Which aspect f applicatin ging t test? Break dwn actins in applicatin task int separate tests What actins are perfrmed n the same system cncurrently when main task is taking place (i.e. peple brwsing pages, running reprts) The primary applicatin scenaris are anticipated user actins that generally incrprate multiple applicatin activities. Primary scenaris are thse where specific perfrmance gals have been set r have a significant perfrmance impact. These scenaris are chsen because they are cmmnly executed r because they are extremely resurce intensive. An example f an applicatin testing scenari is as fllws: Lg int the applicatin. Search airline flights. Select specific flight. Bk flight. Pay fr flight with credit card. Lgut f applicatin. Wuld the scenari mix include nrmal day t day actins r specific perfrmance issues? The cmbinatin f scenaris t be used is critical t prperly lad testing the system and identifying bttlenecks. I typically wrk with the applicatin wner t prvide me with screenshts f each step fr the scenari. Each screensht ties t a page in the applicatin and the steps taken. Belw is an example: 5

6 Typically when I run a lad test, I am unfamiliar with the intracies f the applicatin. I need a radmap t fllw s I knw hw it wrks. After receiving all screenshts fr the scenaris yu can nw recrd the test in the tl f yur chice. Recrding the scenari is the easy part. Setting up the lad test prfile, UIDs and data feeds is much mre difficult and is the mst imprtant part f the test. LOAD TEST PROFILE After the scenaris fr the lad test have been defined it is nw necessary t create a lad prfile identify the perfrmance characteristics r wrklad assciated with each f the defined scenaris. Each f the scenari(s) must cnsider the fllwing as part f the lad prfile: Hw many users have been cncurrently lgged nt the system at the busiest time? Hw many cncurrent users will be used fr the test? Hw will users be divided amng the script tasks t be tested, i.e. test 4 actins during test and divide users amng scripts 100 fr script 1, 200 fr script 2, 50 fr script 3 and 150 fr script 4. Number f users: the ttal number f cncurrent and simultaneus users wh access the applicatin in a given time frame Rate f requests: the requests received frm the cncurrent lad f users per unit time. Patterns f requests: A given lad f cncurrent users may be perfrming different tasks using the applicatin. Patterns f requests identify the average lad f users, and the rate f requests fr a given functinality f an applicatin. Will user pause befre re-lgging in again t perfrm actin, i.e. wait 30 secnds befre lgging in again? Hw lng will users spend n each web page f applicatin during test (think time) Hw lng will the test run? 6

7 APPLICATION USAGE PATTERNS Nw yu need t find ut the usage patterns f the scenaris being run fr the lad test. Fr example, ur airline reservatin lad test will be cmprised f 4 scenaris. Search and bk flight. Search fr flights withut bking. Change an existing reservatin. Ticket agent running a reprt. Amng these scenaris, yu wuld need t divide yur ttal number f users. The best way t find the divisin f users is t talk t the applicatin wner. Applicatin wners typically knw the applicatin the best and can give yu the best guidance n usage numbers. After knwing the numbers then yu can tailr the test t meet the perfrmance testing gals. Belw is a distributin f scenaris fr this lad test: Script % f users Ttal Search and bk flight Search fr flights withut bking Change an existing reservatin Ticket agent running a reprt Ttal After creating a lad prfile, begin lad testing with a ttal number f users distributed against yur user prfile, then start t incrementally increase the lad fr each test cycle. Cntinue t increase the lad, and recrd the behavir until yu reach the threshld fr the resurces identified in yur perfrmance bjectives. Yu can als cntinue t increase the number f users until yu hit yur service level limits, beynd which yu wuld be vilating yur service level agreements fr thrughput, respnse time, and resurce utilizatin. When designing the lad test prfile the fllwing needs t be taken int cnsideratin: User activity Sme transactins ccur mre ften than thers and therefre shuld be a larger part f the test data and scenaris. Mst sites ften experience large swings in usage depending n the type f users wh access it. Fr example, U.S. retail custmers, fr example, typically use a web site in the evenings (7:00 p.m. EST t 11:00 p.m. PST). Business custmers typically use a web site during regular wrking hurs (9:00 a.m. EST t 4:00 p.m. PST). The functinality f a web site can als have a significant impact n usage patterns. U.S. stck qutes, fr example, are typically requested during market trading hurs (9:30a.m. EST t 4:00p.m. EST). When attempting t mdel the real wrld, yu shuld cnduct research t determine peak usage ramp-up and ramp-dwn, peak usage duratin, and whether any ther lad prfile parameters vary by time f day, the day f the week, r anther time increment. After all the research has been cmpleted yu can create a lad prfile which mst resembles real life transactins. Ramp-upTime A majr cmpnent f stress n a site/applicatin is hw many and hw quickly users lg int the system. If a site/applicatin/database is nt prperly tuned r sized perfrmance can severely 7

8 degrade r crash if an unexpected rush f users lg int the system at nce. An example wuld be a 24 hur airfare sale. If the system was nt tested t handle a mass rush f users it culd crash. A rigrus lad test wuld have the users lgging in at a rate f 1 user per secnd. Depending n the number f users fr the test this rate culd easily bring a system dwn. A mre typical ramp-up rate wuld be 1 user lgging in every secnds. Users wuld lg in accrding t the ramp-up rate until all users are lgged int the system. Ramp-dwn Time Lad test ramp dwn implies gradually stpping the threads during a lad run in rder t detect memry leaks and check system recvery after the applicatin has reached a threshld. Ramp-dwn time is the amunt f time it take users t lg ff the system after it has ran with cncurrent users fr the defined lad test time. Think Times The time it takes fr a user t respnd t an applicatin page has a significant impact n the number f users the applicatin can supprt withut perfrmance issues. Peple have a wide range f different skill levels and abilities. Individual users require varying amunts f time t think abut the infrmatin that they are viewing. One user may mve frm ne applicatin page t anther, barely pausing lng enugh t cmprehend what they ve seen; while thers need time t cntemplate what they ve read befre mving n t the next page. The length f this pause is called think time. Think time is generally cnsidered t be the length f time frm when a user receives the last data packet fr the web page currently being viewed t the mment that the user requests a new page. Typically a applicatin/database which can supprt a maximum f secnd aggressive users shuld be able t supprt secnd casual users because bth types f users result in rughly the same (600) transactins per minute. Only the mst basic f applicatins/databases wuld hld true t this thinking as mre interactive applicatins require resurces fr bth active and passive users, meaning that the 300 casual users are mre likely t require extra resurces than the 100 aggressive users. When recrding r writing test scripts fr perfrmance testing, yu shuld cnsider hw much time each f the varius types f users might spend waiting n each page. Having an idea f the mix f applicatin users (aggressive vs. passive), yu shuld be able t create a realistic distributin f timings and, pssibly, a randmizing algrithm. A gd surce f infrmatin fr estimating think time are web lgs. If yu find in the web lgs users spend abut 50 secnds n the site pages I wuld recmmend setting think time fr a randm time f 20 and 60 secnds. Run Length The length f the actual lad test can vary based n many factrs. Yu shuld aim t run the test lng enugh t get a representative example f true system perfrmance. Typically we run mst lad tests fr a perid f 30 minutes and extraplate perfrmance t the prductin system. There are sme instances where yu may want t run the test fr a lnger duratin until the applicatin hits its breaking pint. It depends n what yu are testing fr and what yu hpe t achieve. Wrklad/users Calculating the number f users fr the lad test is critical in rder t have a valid test which accurately frecasts perfrmance. Fr example, let s say a travel reservatin site reprts that the site either has r expects t have an average f 100,000 users per mnth. The first calculatin needed is t find the number f visitrs per day. A cnsideratin is site usage. Is the site is used equally seven days a week? Is it used mstly during the week? If it is used n a seven day basis divide the number f users by 30. If the site is used mstly during the week divide the number f users by 21.7 which is the average number f days per mnth minus weekends. Assuming the site is used equally thrughut the week fr a per-day user number f 3,333 frm the example abve. Nw it is necessary t figure ut hw many hurs f the day the site is busiest which will differ depending n the site/applicatin. In rder t determine peak hur activity the applicatin 8

9 wner r webserver administratr can be very helpful in finding peak usage. If yu are unable t identify peak usage hurs then use a 12 hurs as a basis. Average visits per hur = 3,333 average visitrs per day / 12 hurs = 278 visits per hur In rder t find ut the average cncurrent users fr the lad test it is necessary t deturmine the average length a user stays n the site. User site visits can be fund in the web lgs r by yur visiting the site yurself, running thrugh the scenaris and timing it. If yu knw the average visit length, then it is pssible t calculate the average number f cncurrent users. An example f cncurrency is if ne run thrugh a scenari takes ten minutes then ver a sixty minute lad test the level f cncurrency wuld be six visits per hur per user. Cncurrent Users = Visits per hur/60 min/average visit If yu use the ten minute figure frm the scenari abve an hurly visit rate f 278 returns an average cncurrent user cunt f 46 which is quite lwer than the 100,000 per mnth figure f user visits. After yu knw hw many cncurrent users the applicatin shuld handle the next step is t select the lad test amunt. Using the example abve the applicatin experiences a maximum f 46 cncurrent users. If fr the purpse f the lad test yu want t test that the applicatin can handle ccasinal peaks f 50% mre traffic, then yu wuld want t test using 92 cncurrent users. An example f a lad cnfiguratin might be 4 separate lad tests which test different lad levels. The first test culd start at 25 cncurrent users the next ramping up t 50, then 75 and lastly 100 cncurrent users. Each test wuld shw yu the respnse times users wuld experience in lw (25 users) and high (50 users) traffic cnditins. The lad tests f 75 and 100 users wuld demnstrate what wuld happen when usage grws r yu see an unexpected burst f traffic due t sme external event. Scenari 1 25 Users Lad Prfile Script Actin # Iteratins Think Time sec Users Pacing # Users Ramp Up # Users Ramp Dwn Duratin (mins) Script01 Script02 Search & Bk Flight 1 Search fr flights withut bking 1 Use randm percentage f recrded think time: Min. 50% - Max 150%. Limit think time t 10 secnds. Use randm percentage f recrded think time: Min. 50% - Max 150% Secnds 8 0 Secnds 1 User every 10 secnds 1 User every 10 secnds 1 User every 10 secnds 30 1 User every 10 secnds 30 9

10 Script03 Script04 Change existing reservatin 1 Ticket agent running reprt 1 Limit think time t 10 secnds. Use randm percentage f recrded think time: Min. 50% - Max 150%. Limit think time t 10 secnds. Use randm percentage f recrded think time: Min. 50% - Max 150%. Limit think time t 10 secnds. 5 0 Secnds 2 0 Secnds 1 User every 10 secnds 1 User every 10 secnds 1 User every 10 secnds 30 1 User every 10 secnds 30 LOAD TEST DESIGN After the perfrmance gals f the test have been defined, scenaris established and dcumented, and a lad test tl has been chsen it is nw time t start develping the test. The recrding f the scenaris is different with each tl chsen, but mst likely will invlve clicking a buttn t recrd and pening a brwser t a lgin page. The fllwing are imprtant items t take int cnsideratin when implementing the test design: Applicatin accunts are created fr the test with apprpriate privileges. Test data feeds are prperly implemented fr testing. Test data feeds are data which is inputed int the system during the lad test t create variances in activity thus mimicking real life. Validatin f scenari transactin n the webserver, database, applicatin server and hsting server. Special data used in applicatin which needs t be incremented fr prper executin such as prduct ID. TEST IDS When beginning t recrd the test it is imperative t have at least ne lgin id which has the apprpriate privileges t exectute the transactins in the scenari. Withut the prper applicatin permissins yu may nt get far beynd the lgin page. Use this id t recrd and execute the initial script. It smetimes takes a while t receive the full list f test ids s this gives yu the ability t get a running start n scripting the test. Always after receiving the initial test applicatin id lg int the applicatin and g thrugh each scenari step manually t verify the scenari funtins as predicted and the id has the apprpriate level f permissins. There is nthing wrse than kicking ff a lad test nly t have it fail due t permissin issues! The initial scripting f the test is usually the mst difficult part f a lad test due t things such as permissin issues, data issues, crrelatins and a variety f ther prblems which can arise. 10

11 TEST DATA FEEDS Anther cnsideratin in designing the test are the test data feeds. In rder t have a lad test which is dynamic and mst like a true prductin enivirnment it is imperative t have test data t use. An example f such wuld be an airline reservatin system. When running the lad test yu wuld nt want t cntinue rebking the same flight ver and ver again since it wuld nt realistic f the applicatin and cached data. Fr the test yu wuld wuld want t have parameter files with names f different cities/passengers, etc. t select during the curse f the test. An example wuld be intitially recrding the scenari bking a flight frm Bangkk t New Yrk. If yu did nt parameterize the custmer and cities when testing the scenari wuld run ver and ver again bking the same custmer and flight every time. Using the same data fr every transactin is nt realistic and the data wuld be cached therefre nt expsing ptential perfrmance prblems. Parameter files need t be created with dummy data fr testing purpses. In this scenari a list f custmer names, departure and arrival cities wuld need t be generated. When creating the data it is necessary t keep in mind applicatin cnstraints and wrk with yur applicatin team in generating the files. It is als imprtant t run trial tests using the newly created parameter files t make sure the data wrks with the applicatin. Anther cnsideratin is whether the data is resuable. If any f the data entry fields are unique the amunt f dummy data can be quite extensive. In the airline reservatin example abve unique fields shuld nt be an issue, but if yu were testing an emplyee new hire system using scial security numbers it wuld be an issue. If the new hire applicatin used a cmbinatin f first name, last name and scial security number as the unique identifier it wuld be necessary t have a list f unique dummy scial security numbers s the test wuld nt fail. Each time the scenari script is run it updates the applicatin with ne f the pieces f data. Think f the parameter file as a spreadsheet with a rw fr each piece f data. The first time the script is run it will select Jhn, Smith, The next time the script is ran it will select the next line Jane, Jnes, and s frth. If the script is then ran again it will start at line ne selecting Jhn Smith and the test will fail with a unique cnstraint vilatin applicatin errr. If yu are testing against an Oracle database ne ptin yu have is t set up Oracle Flashback. ORACLE FLASHBACK I typically setup database flashback when I am testing against an Oracle system. Flashback technlgy was first intrduced in Oracle 10g and with this feature in place yu can restre t a particular pint in time within minutes. I use flashback database because f its ease f use by relying n flashback lgs and infrmatin mined frm the archive lgs t access past versin f the changed blcks instead f und data. Using flashback lgs and archive data is a much quicker alternative and less disruptive due nt having t restre frm backup. Flashback is easily setup by enabling archiving and setting up the flash recvery area size. One thing t keep in mind is the amunt f und which may be generated frm the lad test. The amunt f und generated needs t be kept in mind when cnfiguring the flash recvery area size. The tw initializatin parameters used t setup the flash recvery area are: DB_RECOVERY_FILE_DEST_SIZE DB_RECOVERY_FILE_DEST The first parameter defines the maximum space available fr the flash recvery area files and the secnd parameter pints t the destinatin n disk. The size parameter needs t be set befre the destinatin can be cnfigured. The size f the flashback area can be estimated by taking int cnsideratin the size f the datafiles, red lgs, cntrl files and the average frequency and number f blck changes during a typical lad test run. Setting up flashback prir t running a lad test script which uses unique data can save a gd deal f time and wrk bth fr the database and applicatin administratrs. Fr example, say yu are lad testing a new hire applicatin where scial security number is the primary key. If yu d nt have a list f multiple unique numbers fr this field it will fail when run fr multiple iteratins. Yu need t make sure there is enugh data t supprt multiple runs f the script. Many lad tests run fr at least 30 minutes r mre where the test users lgin execute the script and lgut. Depending n the amunt f think-time, pacing and length the scenari can repeat 50 times r mre within this perid. It is therefre necessary t knw the data needed and 11

12 have enugh unqiue test values t cver the testing perid. If yu use Oracle database flashback a restre pint can be set in the database prir t the test executin and data updates. After running the test the database can then be flashed back t its riginal state prir t any updates saving much time and hassle. If testing against a nn-oracle system yu need t keep in mind the rle f unique data in the test and hw t run trials withut errr. It may be necessary t have hundreds r thusands pieces f test data t enter int the system r a way t restre the system back t its riginial state. When designing and building a lad test it is necessary t run multiple trials t validate the scripts, test data, timing and flw f the test t mentin a few items. It is extremely imprtant all scenari scripts run smthly withut errr prir t the executin f the actual test. It is very embassasing and a waste f many peple s time if n the day f the lad test a data errr r applicatin errr is encuntered mid-test which causes it t fail due t nnvalidatin. LOAD TESTING TOOLS FREEWARE VS. COMMERCIAL There are many different tls which can be chsen t cnduct a lad test. The tls range frm freeware tls such as JMeter, HammerOra r OpenSTA t cmmercial tls such as LadRunner r Silk Perfrmer t name a few. FREEWARE TOOLS Freeware tls such as JMeter, HammerOra, Swingbench and OpenSTA ffer many features fr n cst, but may nt ffer the cmplexity needed fr a truly rbust lad test. Many f these tls require additinal prgramming skills which may r may nt be available. If the time and skills are available then these tls can ffer a truly cst-effective, flexible manner by which t lad test a system. The first freeware tl I will discuss is Apache JMeter. Apache JMeter is a 100% java, pen surce desktp applicatin designed t test applicatin functinal behavir and perfrmance. Apache JMeter may be used t test functinal and perfrmance n many different types f resurces (files, Servlets, Perl scripts, Java Objects, databases and queries, FTP servers and mre). The tl can be used t simulate a substantial lad n a server, netwrk r bject t test its availability r t analyze system perfrmance under varying lads. JMeter can be used t perfrm a functinal test n websites, databases, LDAPs, webservices r many ther areas. JMeter des nt act as a brwser. When using web services and testing remte services fr all intents and purpses, JMeter des lk like a brwser, but JMeter des nt perfrm all actins supprted by mst brwsers. Specifically, JMeter des nt execute the Javascript fund in HTML pages. It des nt render the HTML pages as a brwser des. HammerOra is anther freeware tl which can be used t lad test a system. HammerOra is an pen surce lad test tl which can be used against Oracle, Micrsft SQL Server, MySQL Databases and Web applicatins. HammerOra is built with the Tcl scripting language and can be used t simulate a real wrklad using as many client user sessins as yur system can handle. HammerOra includes in its install a pre-installed and cnfigured schema with lad tests based n the industry standard TPC-C and TPC-H benchmarks. These benchmark tests can be deplyed against the Oracle database t test with multiple users. The tl can als cnverts and replay Oracle trace files and enable Web-tier testing t build rbust lad tests fr an entire Oracle applicatin envirnment. Swingbench is a Java based lad test tl specifically designed t be used fr lad testing Oracle databases. The Swingbench install includes fur benchmark tests, OrderEntry, SalesHistry, CallingCircle and StressTest. The first tw tests are based n the default test schemas included with Oracle s e 11g database. The secnd tw tests are custm written fr the applicatin and are designed t simulate a call center and randm inserts, updates and deletes t stress the system. The final pen surce tl I will discuss is OpenSTA. OpenSTA is written in COBRA and designed t run n Windws. Lad tests are architected using the tester's wn brwser which creates simple scripts that can be edited and cntrlled with a OpenSTA s scripting language. After the scripts have been finished it can then be played back t simulate many users using a lad generatin engine. OpenSTA lad tests can be designed t scale t hundreds f thusands f virtual users. 12

13 COMMERCIAL TOOLS Cmmercial tls ffer bth pluses and minuses. One f the big pluses f using a cmmercial tl t lad test is prduct supprt. If yu run int an issue scripting r running the lad test yu can simply pick up the phne and call the sftware vendr whereas when using freeware ne is stuck with either figuring it ut themseleves r with Ggle supprt. A majr minus is cst. Many cmpanies d nt value the imprtance f lad testing and d nt want t spend precius cmpany resurces n pricy sftware. Time is mney in mst cases and having a website r database crash can be a very cstly event which is why it is imperative t lad test databases/applicatins. Using cmmercial tls t lad test can significantly limit the time invlved in creating and running the lad test depending n tester skill level. If the lad tester has a strng prgramming backgrund using freeware tls might be a gd way t g. On the ther hand if the tester has weak skills r limited time, cmmercial might be the best ptin. HP LOADRUNNER The first cmmercial tl I will discuss is HP Ladrunner which is seen by many as being the industry standard fr lad testing. Ladrunner was acquired by HP in 2006 as a part f its purchase f Mercury Interactive. It can be used t test a brad range f applictins and up t 51 prtcls. Prtcls enable cmmunicatin between a client and server. Examples f this are an airline ticketing system which can use HTTP/HTTPS with Java and Web Services r an applicatin using Citrix as its basis. These scripts can be recrded bth in single and multi-prtcl mdes. Ladrunner is a very versitile tl written primarily in c with prtcl extensins enabling it t be used against a variety f applicatins and databases (at extra cst, f curse). The prduct cnsists f 4 primary pieces: Virtual User Generatr (VuGen) Cntrller Lad Generatr Analysis These parts are cmbined t enable end t end lad testing and analysis f an appliatin r database. ORACLE APPLICATION TESTING SUITE Oracle Applicatin Testing Suite (ATS) is anther cmmercial lad test sftware package. ATS is Oracle s lad testing fulllifecycle prduct built t validate applicatin functinality and test availability under lad. Applicatin Testing Suite cnsists f the fllwing pieces: Oracle Lad Testing fr scalbility, perfrmance and lad testing. Oracle Functinal Testing fr autmated functinal and regressin testing. Oracle Test Manager fr test prcess management, including test requirements management, test management, test executin and defect tracking. Oracle Applicatin Testing Suite als cntains integrated testing acceleratrs which are used fr testing Oracle packaged and SOA applicatins. The purpse f the acceleratrs is t enhanced scripting t create mre rbust tests. SILKPERFORMER Silk Perfrmer by Brland Sftware is anther cmmnly used lad test tl t diagnse applicatin perfrmance. Silk Perfrmer is mstly used t lad test Web 2.0 applicatins which rely n AJAX lgic. The latest release, nw allws fr the use f Web brwers (Internet Explrer) t generate lad n the system. Using AJAX technlgy built int Web brwsers enables a test which wuld mst likely mimic end-user brwsing behavir. 13

14 Silk Perfrmer prides itself n its ease f use and a small testing ftprint. Ease f use is maintained thugh the use f simple wizards which allw the tester t input a variety f fields. The lad test can als be made mre rbust during recrding by using variying virtual user activities t simulate real wrld activities t test web cnnectin speeds and netwrk equipment perfrmance. The installatin f Silk Perfrmer n the test server nly requires a minimal cnfiguratin t supprt the running f thusands f virtual users against an applicatin. The agent sftware and r cntrller nly require a minium f 512 MB RAM, Pentium 4 prcessr and 580 MB disk space. After the sftware has been installed it is nw pssible t create realistic applicatin scenaris with thusands f users using the latest Web 2.0 technlgy. Silk Perfrmer als has the fllwing additinal cmpnents available t use with its lad testing stack: SilkPerfrmer Diagnstics perfrmance analysis package fr evaluating Java and.net applicatins. SilkPerfrmer Cludburst a clud-based extensin which allws a tester t scale a lad test t thusands f users withut having t setup and manage cmplex infrastructures. SilkCentral Test Manager integrates multiple aspects f testing activities under a unified umbrella allwing central f the lad test management envirnment. ORACLE REAL APPLICATION TESTING VS. LOAD TESTING When lad testing tl cnsideratin is discussed amng Oracle prfessinals the tl which mst likey might be mentined is Oracle Real Applicatin Testing (RAT). RAT was first intrduced with Oracle 11g. It is a separately licensed ptin and is nly available with Oracle Enterprise Editin. RAT cnsists f Database Replay and SQL Perfrmance Analyzer. RAT is NOT fr lad testing, it is mre a regressin testing tl due t its inability t scale t multiple users. Oracle Real Applicatin Testing is a very pwerful tl t use in testing infrastructure changes. It des nt have the ability t ramp-up users as needed when cnducting a regular lad test. It is pssible t use Oracle Real Applicatin Testing alng with Oracle s Applicatin Testing Suite t create a rbust lad test which uses captured data frm yur prductin envirnment. The database test scripts wuld be created manually r autmatically in Applicatin Testing Suite by imprting a wrklad capture frm Oracle Real Applicatin Testing replay cmpnent. Bttm line is Oracle Real Applicatin Testing cannt be used by itself t create a lad test. Bth RAT and Oracle Applicatin Testing Suite must be used tgether fr lad testing purpses. MONITORING THE LOAD TEST Once the lad prfile has been created it is nw necessary t identify the metrics which are derived frm the defined perfrmance bjectives. The metrics identified will be used t measure the applicatins perfrmance t cmpare with yur perfrmance bjectives. Clearly defined metrics will make identifying any perfrmance bttlenecks easier and better t reslve. Specific areas t be mnitred are as fllws: OS metrics which culd include CPU, memry, disk I/O and netwrk I/O statistics. Database metrics t watch fr are waits, tp SQL, Buffer gets, etc. Netwrk metrics wuld include the verall health f yur ruters, switches and gateways. Webserver metrics wuld include hits per secnd, number f requests, etc. After identifying the areas t mnitr fr the lad test it is nw necessary t identify the individual team members wh will take part in mnitring the test. These peple will mnitr their respective systems fr perfrmance issues while the test is run. These peple will mnitr during the test and pssibly assist in tuning the system after any bttlenecks have been discvered. Typically when we run lad tests, there is a bridge call pen which includes all team members s they can speak up when issues are encuntered as the test is run. Having real-time mnitring ccuring while yu are running a lad test is invaluable as yu can see hw the additin f mre users r faster ramp-up time is directly affecting the system. ANALYZE AND REPORT LOAD TEST RESULTS 14

15 After the lad test has been cnducted it is time t analyze the results f the test. This presentatin is nt n tuning and will nly shw hw t use the lad test results t pinpint system bttlenecks. The perfrmance bjectives initially set at the beginning f the prject shuld be used t judge the results. Fr example, checkut shuld nt take lnger than 4 secnds under 15 hits per secnd wrklad. Other measurements culd be CPU utilizatin fr the test shuld nt exceed 80% during any transactin. All f these measurements shuld have been defined and agreed upn at the beginning f the prject in the perfrmance acceptance criteria. Sme things t cnsider when analyzing the results are: Transactins in average respnse time graph breach the perfrmance acceptance criteria. Respnse time that is wrse than what the SLA fr the custmer allws. Crrelate transactins with ther graphs and measurements t find bttlenecks. Invlve develpers, system and netwrk administratrs in investigatin. Overlay respnse time graph with mnitring graphs and lk fr perfrmance trends. Check standard deviatin f transactins if transactins behave incnsistently identify the cause. Transactins availability: (cunt f pass transactins/cunt f ttal transactins) x 100. Usually applicatins have arund 99.9% availability. Lwer values dn t necessarily indicate an issue, but shuld be verified why lwer. The graph abve shws average respnse time cmbined with running virtual users. The graph was taken frm a lad test perfrmed using LadRunner. Analysis f this graph shws a perfrmance irrgularity ccurred between 14 and 18 minutes int the test. The perfrmance hit lks t have taken place when the cncurrent level f 50 virtual users was hit. Yu wuld then want t take a lk at the transactins which were mst affected during this time. In this case it lks like the transactin 12_Nn_Payrll_Update tk the lngest. Yu wuld als want t investigate the web hits t see if there was a large hit n the webserver during this time. Belw is a graph f web hits frm the lad test: 15

16 As yu can see 15 minutes int the test there was a huge drp ff in web server hits. At this pint yu wuld want t get with yur webserver administratr t review the webserver lgs fr this pint f the test. On the database side f things yu wuld want t mnitr and pull an ASH reprt fr the lad test perid. Belw is a graph shwing database perfrmance fr the lad test: 16

17 As yu can see during the perid the lad test was ran database perfrmance was within an acceptible range. If perfrmance is negative yu can review the tp SQL and Tp User Events t see where the waits were fund. REPORTING AUDIENCE When cmpiling the analysis reprts fr the lad test it is necessary t keep in mind yur audience. Many times management requests the lad test reprt and depending n hw technical the audience will dictate what infrmatin is included. If yur audience is nn-technical in nature (lves t see graphs and nt numbers), gd graphs t include are as fllws: Running Users Shws the impact f ramp-up and ramp-dwn as well as when errrs begin t ccur during the lad test. Hits per Secnd Shws the amunt f lad generated by vitual users in terms f hits t web servers. Thrughput Shws the amunt f lad virtual users create n netwrk resurces raw amunt f bytes client receives each secnd as a result f hits n the web server. Average Transactin Respnse Time Shws the actual lad generated by each transactin thrughut the entire lad test. The cmbinatin f the graphs abve will give bth a technical and nn-technical audience a gd verview f the utcme f the lad test. The graphs will shw where the bttlenecks lie and where t best fcus the tuning effrts. These charts can be generated using Excel with data btained frm yur system administratrs, netwrk administratrs and web server administratrs. If yu are using a cmmercial tl such as LadRunner these graphs are included with the sftware and are easily generated. Fr a mre technical audience yu may want t include the utput f an Oracle Active Sessin Histry (ASH), Web server, OS and Netwrk reprts. Be sure t wrk with the respective system wners t bil the reprts dwn t a pint where the perfrmance bttleneck can be easily identified fr tuning purpses. LOAD TEST SUMMARY At the end f the lad test summary reprt shuld als be included which explains the key results in an easy t read layut. It shuld list the slwest transactins and explain any crrelatins with ther graphs. A gd example f items t list are as fllws: Tp 5 Slwest Pages Tp 5 Slwest Transactins Tp 5 Slwest SQL Statements The tp 5 slwest pages wuld include the slwest pages encuntered in the lad test. These pages are nes which d nt meet page duratin time during the lad test. The tp 5 slwest transactins are the transactins which d nt meet the perfrmance criteria defined at test inceptin. Identifying these transactins nt nly shw management the areas which need wrk, but shw the technical team where t cncentrate their effrts. The tp 5 slwest SQL statements will give the DBAs the infrmatin needed t tune the applicatin SQL which is typically 80% f perfrmance prblems. FIND BOTTLENECKS, TUNE SYSTEM AND POSSIBLY RETEST An nline applicatin can cnsist f multiple cmpnents which need t be cnsidered in a lad test, such as lad balancer, web server, applicatin server, database, etc. Perfrmance f the entire applicatin is usually affected by ne f these cmpnents. Imprving r fixing a single cmpnent can result in perfrmance imprvement, while imprving ther items 17

18 may nt change anything r will reduce perfrmance. Many times perfrmance imprvement can be achieved by changing settings, tuning applicatin cde r the purchase f additinal hardware. After any tuning adjustments are cmpleted the decisin can be made whether t re-run the test t see if the bttleneck has been reslved. The iterative tests shuld be run with the same ptins as the initial test s there is an apples t apples cmparisn f the results. Lad testing is an iterative prcess and tests may be executed several times befre the perfrmance bttleneck is crrected. CONCLUSIONS Lad testing, like mst f IT is bth an art and a science. It takes a team effrt t design and execute a lad test. Lad testing enables yu t identify applicatin/database issues which wuld nly be visible under extreme cnditins: bttlenecks such as SQL executin plans, memry, CPU-bund systems, netwrk bandwidth and disk capacity. And, these are nly a few f the ptential bttlenecks r issues which culd be expsed. Lad testing is fcused n the identified perfrmance bjectives and key applicatin scenaris with a fcus n reliability and stability f the applicatin. It is imprtant that crrect methdlgy is used and all data extracted are prperly analyzed. The methdlgy emplyed fr the lad test needs t take int cnsideratin the mst realistic reprductin f prductin wrklad, vlume f data, identificatin f key scenaris and interpretatin f perfrmance metrics. 18