Perfrmance Test Mdeling with ANALYTICS Jeevakarthik Kandhasamy Perfrmance test Lead Cnsultant Capgemini Financial Services USA jeevakarthik@gmail.cm Abstract Websites and web/mbile applicatins have becme the imperative sales enablers fr mst rganizatins. The ptimized user experience f a website becmes critical fr rganizatins t utperfrm their cmpetitrs. This demands perfrmance testing t be practive and shift dynamics frm measuring respnsiveness t imprving user experience. User Experience (UX) index f a sftware prduct can be measured by the ease f use, prviding navigatinal cmfrt and quick respnsiveness t end user In a nutshell better perfrmance f an applicatin results in better user experience but unfrtunately even after a huge spend critical perfrmance issues are detected / bserved in prductin. Sme f the critical factrs fr these include Perfrmance testing bjective remains t be meeting respnse time SLA s Ambiguity in lad test mdeling Nt emulating actual user behavir during lad tests Lack f simulatin f devices used by end users Web Analytics tls like Ggle Analytics, Yah analytics etc have cme a lng way and simplified the prcess f analyzing, cllecting and reprting user behavir which serves as a input in creating a cmprehensive perfrmance test strategy and test bed which included lads f cmpulsive features. Bigraphy Jeevakarthik Kandhasamy certified, test engineering Leader & Architect with experience in Functinal & Nn-Functinal Testing. With ver 12 years f experience in Perfrmance testing, cnsulting, strategic partnering & prgram management, executing transfrmatin, change management prgrams in testing space Jeeva has served financial services majrs in India, China, Eurpe and Nrth America. His interests include champining rganizatinal wide activities like IP lead test assets creatins, Framewrks, assessments, Innvatin drives & testing rad shws. He currently based in Chicag and leads a testing engagement fr a leading glbal insurance custmer Cpyright <Jeevakarthik Kandhasamy> <06-14-2015> Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 1
1 Intrductin A Recent Cisc survey reprts that 78% f the Nrth American shppers use internet, scial media r mbile devices fr research and purchase f prducts and services Alng with the brand value, quality and ther marketing campaigns mst firms rely n their web applicatins t yield revenue DISGRUNTLED user experience may turn ut t be a perfect recipe fr IT rganizatins failure. Sme f the risks assciated include Increased spend n IT Failure t meet sales gals and business bjective Decreasing revenue Denting rganizatins brand value The think tanks in mst nline shps have realized the imprtance f imprved UX Index and have stressed the need and imprtance f nn-functinal testing and specifically perfrmance testing. This invlves a lt f mney fr Setting up infrastructure fr perfrmance testing Tls and licenses Resurce cst In spite f the rganizatins spending tremendus mney fr perfrmance testing mst f the Perfrmance issues are detected in prductin The key reasns f failure are Perfrmance testing bjective remains t be meeting respnse time SLA s Ambiguity in lad test mdeling Nt emulating actual user behavir during lad tests Lack f simulatin f devices used by end users All these clearly indicate that lack f analytical data n the actual user behavir in prductin. The apprach f determining user and system behavir with server lgs becmes tedius and uncertain. Perfrmance testing strategized withut understanding the actual user behavir results in finding ut issues in prductin. 2 Slutining with Analytics Web Analytics tls like Ggle Analytics and Yah! Web Analytics have cme a lng way and simplified the prcess f analyzing, cllecting and reprting user behavir which serves as an input in creating a cmprehensive perfrmance test strategy and test bed (lad test mdel). Sme f the cmpulsive features include: Prviding a hlistic view f business Decisin making Custm and integrated reprting User cnversin rate Bunce rate and reasns fr users exiting the website Fr applicatins yet t be develped experimental sites are available with analytical data based n trends frm cmpetitrs and industry standards. Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 2
Understand visitr behavirs like Hw visitrs use yur website Hw they arrived n yur website Gegraphical lcatins f users Devices used by users accessing the website Web Analytics simplifies the prcess by prviding the nn functinal tester inputs n: - Wh the end users are, - What they d - Entry and exit pints f the applicatin. - What they like and dn t like in the applicatin. It s fair t say that web analytics tls can help determine the perfrmance testing bjective, help creating the perfrmance testing strategy, lad test mdeling and executing lad tests simulating actual user behavir. 3 Mdeling a Perfrmance Test using Ggle Analytics The sectin belw explains hw a perfrmance test culd be designed and executed using Ggle Analytics. I have prvided snippets and screen shts frm Ggle Analytics. The flw diagram belw indicates the end t end perfrmance test prcess. Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 3
Perfrmance test planning will include designing, executing and reprting a perfrmance test. Sme f the key factrs fr designing f a perfrmance test will depend n: Type f the perfrmance test t determine the virtual user lad and the duratin f the test e.g. lad, stress, endurance, spike etc. User flw r traversal flw Platfrm Client and server systems Gegraphical distributin f the users Determining the type f test Fr a prtal that is already existing in prductin user, sessin, cntent metrics are available n hurly, daily, weekly, mnthly and yearly basis. This behavir analysis shuld help a perfrmance engineer t crrelate and identify the test lad, duratin and bjective f the test. In the sectin belw we will walk thrugh a few scenaris fr mdeling a perfrmance test, and the advantages f using Analytics tl. User prfile One f the main flaws in perfrmance test design is nt mimicking the user prfile but with analytics varius factrs is factred in type f user, demgraphic, gegraphical lcatin and the mst imprtant factr being - % f new users vs % f returning users which will depict the actual user behavir. Traversal flw While identifying the perfrmance test flws it s usually based n the Key Perfrmance Indicatrs and ensuring they are cvered as part f the flws. User flw analytics prvides users with the fllwing infrmatin: User landing page that includes ttal number f sessins active and inactive alng with the drp-ffs First Interactins, secnd and subsequent interactins This will help us t identify the user flw, interacting internal and external systems and services. Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 4
Technlgy, platfrm and devices Applicatin perfrmance in mst situatins are based n server perfrmance, but with mdern technlgies perfrmance f a particular applicatin will depend n the system that is used by the end user. Sme f the factrs influencing this wuld include: Brwser, Operating Systems, Screen reslutin, Flash versin and Script cntrl f the user: With the increase in use f mbile devices mst applicatins tday are mbile cmpatible. Analytics tl prvides infrmatin n the devices (Brwser, OS), netwrk speed, carriers etc. Gegraphical distributin With the business mdel tday being glbal, perfrmance testing shuld als fllw and be distributed. This culd be achieved by understanding the user gegraphies and simulating it using latency and netwrk simulatins during a perfrmance test. Alng with these benefits, analytics dashbards can be used fr mnitring prductin envirnment real time. Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 5
4 Case Study This sectin cntains a case study describing the need and benefits f using web analytics tls fr perfrmance testing fr leading aut insurance prvider in Nrth America insuring 18 millin vehicles acrss USA. 4.1 Situatin Clients Aut insurance applicatin was the applicatin under test. This was Java / J2EE based system. Based n the Nn-functinal requirements the bjective f perfrmance testing was t ensure the respnse times f critical Key Perfrmance Indicatrs (KPI) were less than 7 secnds. The dwnsized test bed was t ramp up 700 cncurrent users and create 2500 aut plicies with different driver / vehicle cmbinatin in the perfrmance test envirnment. 4.2 Issues Observed The client was spending clse t 1.5 millin dllars annually fr perfrmance testing that includes the cst f resurces, infrastructure, tls etc. The fllwing issues were reprted by business and prductin team Respnse times fr page navigatin tk mre than 11 secnds Multiple search failures reprted between nn 3 PM EST 2 prductin utages Declining nline sales cnversin 4.3 Assessment bservatins A detailed assessment was executed t understand the reasns fr failure by using Ggle Analytics in bth prductin and Perfrmance test envirnment. Sme f the key bservatins related t lad test mdeling failures are listed belw The test simulated 700 users creating 2500 plicy in test envirnment but the actual situatin was 2000 + users accessing the system and nly 700 users navigate and prceed till purchase USER AND VOLUME SIMULATION issues. Clse t 65% f the agents were using brwsers IE 7 and belw which did nt supprt the caching and ther UI based perfrmance acceleratrs CLIENT & BROWSER related issues. The client data centre was lcated in Gergia while mre than 40% f the users were frm nrth east and west cast lcatins the lad test did nt reciprcate the NETWORK LATENCY similar t prductin. Search initiated frm hme page tk mre time while search frm ther pages were fine SITUATIONAL & USER BEHAVIOUR issues. The peak vlume reached increased by mre than 30% during prmtinal ffers failure t execute SPIKE test due t prmtinal / un-expected vlume. 4.4 Assessment bservatins Nn-functinal requirements and lad test mdel, test scenari design was updated based n the actual user behavir frm Analytics reprt the user pattern was nt available frm server lgs. Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 6
Additinal plug-in Recmmendatins prvided t agents fr brwsers User pattern was mdified t include 120 MS and 245 MS netwrk latency respectively fr east and west cast users. Multiple perfrmance issues related t cde and cnfiguratin were bserved during perfrmance tests which resulted in the fllwing perfrmance tuning recmmendatins DB indexing Client side caching Search ptimizatin Lad balancing Netwrk perfrmance 4.5 Results Fllwing are the psitive results by using web analytics fr perfrmance testing: Realistic simulatin f user behavir frm prductin Detected 82% f perfrmance issues during perfrmance testing phase reduced risk f identifying perfrmance issued in perfrmance Imprved user experience btained based n custmers and agent survey. Observed trend f imprved sales by 6% 8% n a half early basis. 5 Cnclusin Using web analytics tls adds value t the perfrmance testing prcess. Sme f key benefits include: Efficient lad mdeling Imprving user experience Imprve sales and user cnversin Decisin making Supprt and adapt based n evlving changes in prductin Cst effective slutins availability f pen surced and licensed tls & reprts In tugh ecnmic situatins, rganizatins tday are frced t deliver high perfrming applicatins t retain existing custmer base and attract new custmers. The gal f Perfrmance testing needs t shift frm measuring respnsiveness t imprving the custmer experience and web analytics tls can help achieve these bjectives and help rganizatins succeed. Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 7
References http://en.wikipedia.rg/wiki/web_analytics_tls www.ggle.cm/analytics http://www.adbe.cm/slutins/digital-analytics.html Excerpt frm PNSQC 2015 Prceedings Cpies may nt be made r distributed fr cmmercial use Page 8