July/August 2009 Volume 3 SOFTWARE This issue of Softwre Testing delves into mobile ppliction nd regression testing. HOTSPOTS
1 Mobile pplictions nd regression testing p WANT JOB SECURITY? This issue of Softwre Testing explores the rewrds of being thoroughly-modern, thoroughly-trined nd lwys-diligent mobile ppliction nd regression testers. In the mobile ppliction development mrket, the role of regression testing is proving to be mission criticl in the Web 2.0, Rich Internet Appliction ge. Both mobile pps nd regression go better with gile nd Softwre Development Lifecycle (SDLC) pproches s well s some cretive thinking so just being code wizrd or pretty good tester won t get you the gold. Those who get nd sty in the mobile ppliction trining circuit could hve job security for mny yers, ccording to Enterprise Mngement Assocites (EMA) Julie Crig, who wrote this issue s led rticle. Just don t expect to quit school t ny time, becuse the mobile technologies will continue to be complex nd vried, with new devices ppering fster thn bubbles in seltzer wter. Beyond discussing the gret creer potentil in mobile ppliction testing, Crig lys out the mobile pp peculirities tht require gret development cretivity nd more svvy testing. In this ezine s rticle on testing nd retesting defects, Dvid W. Johnson connects the dots between mobile ppliction nd regression testing by citing the Blckberry blckouts of 2007 nd 2008, which were n unfortunte exmple of production issues tht should hve been detected by regression test suite in combintion with other forml testing efforts. Helping testers void common mistkes is Johnson s key gol in this rticle. Too often, user reports bug; regression testing is done; fixed report goes to the user; nd the user experiences the sme problem. Johnson suggests prctices tht cn help end this embrrssing scenrio. Regression testers, like mobile ppliction testers, must invest in continuing eduction due to the constnt emergence of new breeds of softwre, such s mobile pplictions. This issue is good first step on tht trining tril. JAN STAFFORD, Executive Editor jstfford@techtrget.com 2 SOFTWARE JULY/AUGUST 2009
1 The chllenges of testing wireless mobile pplictions Mobile ppliction testing requires not just skill, but cretivity nd resourcefulness. It lso requires products nd services specificlly designed for the chllenges of mobile technology. BY JULIE CRAIG p DEVICES ARE hot, hot, hot, nd pplictions re the reson why. The lure of lwys-vilble Web browsing, emil, document viewing nd instnt messging hs proven to be irresistible to business nd personl users like. Not to be outdone, entertinment-relted pplictions re flourishing s well, with mobile gmes, Twitter nd Fcebook, nd photo uplods becoming fundmentl elements of the new mobile lifestyle. This is reflected in the economy. Annul globl sles of mobile phones in units re now roughly equl to sles of lptops, with smrtphones ccounting for more thn 10% of the totl. Vendors re cpitlizing on this growth with host of mobile-enbled pplictions. Numr Softwre, Inc. recently introduced Numr Foot- Prints Mobile, which llows support personnel to ccess Numr s Service Desk softwre using the Web browser on their Microsoft Windows Mobile, RIM Blckberry, or Apple iphone devices. TD Ameritrde offers mobile stock trding softwre which enbles customer to execute stock trdes from his or her phone. Workdy.com, Softwre s Service (SS)-bsed Enterprise Resource Plnning (ERP) provider, recently nnounced indevelopment softwre tht will provide ccess to Workdy s Humn Resources (HR) nd finncil pplictions from n iphone. Clerly there is strong demnd for growing host of mobile pplictions, nd s result, mobile development nd testing re flourishing. This explosion cn be bonnz for mobile literte development nd testing professionls. The outlook for growth in this spce is so fvorble tht job security is virtully gurnteed for the foreseeble future. The 3 SOFTWARE JULY/AUGUST 2009
bd news, however, is tht these jobs re hrder thn they used to be, thnks to wide rry of hndheld hrdwre, mobile-specific softwre, nd the sheer force of constnt chnge. Mobile pplictions isn t your grndmother s COBOL, nd testing them requires not just skill, but cretivity nd resourcefulness. It lso requires products nd services specificlly designed for the chllenges of mobile technology. : SIMILAR TO CONNECTED Even without mobile technology, softwre development is notoriously filure-prone ctivity. I don t need to go into the detils since everybody reding this is likely wre of the high percentge of softwre development projects tht fil to deliver in terms of functionlity, budget or schedule. Mobile pplictions dd to ppliction complexity, nd in doing so, increse development nd Qulity Assurnce (QA) risks. Mobile development requires the sme Softwre Development Life Cycle (SDLC) pproch tht dds discipline nd governnce to trditionl softwre development. Agile methodologies re rpidly replcing older Wterfll processes to become the de fcto stndrd governing the SDLC. Agile principles include incrementl nd itertive cycles, testing erly nd often, nd incorporting end-user input throughout the lifecycle, nd they pper to be pying off. BMC Softwre dopted Agile strting in 2004, nd were ble to increse product time to mrket by two to three times over the industry norm (go to www.enterprisemngement. com 1 for more informtion on this story). From the perspective of mobile pplictions, Agile methodologies re often lso the best choice, primrily becuse of the ongoing focus on customer involvement nd itertive testing. The steps required requirements gthering, design, development, testing, nd deployment re similr for both connected nd wireless pplictions. However, the lifecycle must be tweked for mobile pplictions, nd it is importnt to nswer some key questions from the outset. They include: Which mobile devices nd operting system versions will this ppliction support? How do we test pplictions to mke sure they run on those pltforms? Wht modifictions must be mde to ccommodte the differences mong pltforms? How will industry innovtions be supported going forwrd, since new mobile devices, technologies, nd pplictions re constntly being introduced? How will development nd testing processes ccommodte the inherent differences mong wireless network protocols nd mobile service providers? How do we know how much testing is enough? 4 SOFTWARE JULY/AUGUST 2009 1 Agile Mngement Wht Mngers cn Lern from Agile Methodologies
Ech orgniztion will hve to nswer these questions internlly, bsed on their overll gols nd objectives. Mobile development requires the sme Softwre Development Lifecycle (SDLC) pproch tht dds discipline nd governnce to trditionl softwre development. KEY DIFFERENCES Chllenges with mobile testing tend to center round severl key res. One is the wide vriety of mobile pltforms nd the complexity round testing n ppliction s comptibility with ll supported pltforms. Another is the chllenge of developing pplictions tht compenste for the fcts tht the mobile device is not lwys on nd tht the wireless connection my not be s relible s wired one. Another key considertion is the uniqueness of ech user. Customers subscribe to different crrier networks nd hve different use cses. The lst mile to their device is likely to run t different speeds nd over slightly different protocols depending on user loction, roming, vendor, network, nd device. The bottom line is tht, if you nd I re running the sme mobile ppliction, our experiences might be very different. Fctoring such vritions into expected testing outcomes cn become quite complex. The primry problem is one of testing every possible scenrio, given tht few development orgniztions own one of every device, version, nd crrier connection. And even the most tlented QA tem will probbly not be ble to simulte every combintion of these vribles, especilly when the idiosyncrsies of users re fctored in. Fortuntely, lively mrketplce hs risen round mitigting these chllenges, nd some of the products nd services detiled below my be helpful to orgniztions struggling with mobile testing. HETEROGENEITY AND VENDORS RESPONSES Sy wht you will bout Microsoft, for those of us who remember the old dys when operting systems (OSes) were virtully vendor-specific, the ubiquity of Windows hs certinly mde it esier to develop nd test. Of course, Apple is, nd lwys hs been, n exception to the stndrdiztion rule, nd mny Independent Softwre Vendors (ISVs) develop pplictions for both Windows nd Mc OS. This stndrdiztion hs been boon in terms of limiting pltform prolifertion. Assuming support for both vendors nd given number of operting system levels for ech, the worst cse testing scenrio is still firly limited. In contrst, mobile computing is just bout where personl computing 5 SOFTWARE JULY/AUGUST 2009
ws 20 yers go. Let s see, you hve Symbin, Blckberry OS, Windows Mobile, Plm OS, Apple OS X, Linux, etc., etc. Add to this the fct tht there re multiple browsers nd little stndrdiztion cross devices, throw Mobile pplictions isn t your grndmother s COBOL, nd testing them requires not just skill, but cretivity nd resourcefulness. in 2 or 3 relese levels ech nd the result is firly complex testing scenrio. This cn be problem for testing tems since, unless they hve one of ech combintion of device, OS, version, etc., it is virtully impossible to ensure tht given ppliction works everywhere. Multiple vendors re ddressing this chllenge with wide rry of solutions. Some solutions re imed t inhouse testing: Hewlett Pckrd (HP): HP is integrting mobile ppliction testing into its existing Quick Test Professionl (QTP) solution by prtnering with vendors tht specilize in mobile testing. The first prtnership is with Jmo Solutions, which offers n end-user experience mngement solution for Windows Mobile devices. HP is working on other prtnerships s well, including one with DeviceAnywhere. DeviceAnywhere mrkets both cloudbsed nd on-site versions of its mobile Qulity of Service (QoS) solution with support for more thn 2000 hndsets nd 30 mobile opertors. Microsoft: Microsoft is rich source of testing resources for Windows mobile pplictions, with brod rnge of tools supporting development nd testing of wireless pplictions built using Microsoft technology. Microsoft s mobile development site is good strting point. Resources include set of mobile emultor imges tht cn be used with or without Microsoft Visul Studio nd provide testing pltform for the Windows Mobile operting system. Dexterr : Dexterr is different product entirely, nd its flgship product is the Dexterr Concert mobile ppliction development pltform. Dexterr is vilble in two versions. Enterprise Edition is n open, stndrds-bsed development pltform designed to be deployed within the enterprise. Crrier Edition is trgeted t service providers who pln to host the solution s service. Both pltforms support wide rnge of mobile devices s well s softwre requirements mngement. Dexterr promises to stremline nd simplify development of mobile pplictions, nd my be well worth evluting for compnies which re hevily invested in mobile development. 6 SOFTWARE JULY/AUGUST 2009
There re lso vriety of options for Qulity Assurnce (QA) mngers looking for hnds-on testers for specific devices or to supplement inhouse tems. This cn be lifesver when testing schedules bck up, or to certify solutions ginst ctul devices insted of emultions. Two vendors offering such services include: utest : utest is positioned squrely in the softwre testing relm, nd recently nnounced support for mobile pplictions. The compny s business model is distinctive one. It consists of host of softwre testers (more thn 14,000 worldwide) who test mobile pplictions cross vriety of operting systems nd pltforms. Testers get pid by the bug. They execute testing scenrios nd report unexpected issues to the customer, who then decides whether the issue is, in fct, defect. If so, the customer pys the tester who found the bug. If not, the customer does not py. This model cn be vluble becuse humn testers ctully simulte the kinds of issues tht re likely to rise once the softwre is relesed to the public. It is lso cost effective, in tht utest s customers don t py for testing fetures the tester interprets s bugs. Keynote Systems, Inc.: Keynote is one of the best known Web testing compnies in the world, due in prt to its frequent press releses detiling performnce of well-known online vendors. Click here for n exmple of recent Keynote relese. Keynote hs now expnded into mobile testing s well, leverging testers from loctions worldwide to deliver n impressive bredth of solutions. They include: Mobile Device Perspective (MDP): Provides feedbck on how effectively mobile devices interct with their service content Mobile Appliction Perspective (MAP): Provides informtion bout how well mobile sites function during trnsmission nd delivery SITE Test System: Provides mobile opertors with metrics tht quntify the helth of the mobile network Globl Romer: Provides n indepth understnding of how pplictions function from vrious points cross the world, bsed on testing from multiple locles Web Effective for iphone: Pltform to run lrge-scle, tsk-bsed iphone Web usbility studies PROBLEMS AND PAY-OFFS There re obviously mny more vendors nd solutions in the mobile mrket thn there is spce to discuss them. The point to think bout is the ide of using utomtion wherever possible to dd efficiency to mobile development nd testing. In recent yers, the cost of IT hs risen to the point where, for some compnies, it is limiting the bility to grow nd compete. Softwre development, for mny compnies, is cost 7 SOFTWARE JULY/AUGUST 2009
center within IT, nd efficiencies in the development nd testing spces re s importnt to overll cost reduction s efficiencies in the dt center. Tht being sid, rising IT costs ren t due to IT specilists, developers, or testers who wste time. They re due to technology complexity. The pst 10 yers hve seen n explosion of heterogeneous technology, with ech new technology requiring specilized personnel, tools, nd infrstructure. Mobile is no exception. Mobile pplictions represent new relity in terms of pplictions, in tht mny of the pplictions currently under development cnnot be 100% tested by in-house personnel nd resources. They will be used by literlly millions of users, nd it is impossible for testing tem with finite resources to duplicte every use cse, pltform, network, nd device. Becuse of this, investments in utomted testing products nd ugmenttion of in-house test tems with third-prty vendors will likely py off in terms of both testing efficiency nd qulity. Reserch hs repetedly shown tht softwre bugs detected Mobile pplictions represent new relity in terms of pplictions, in tht mny of the pplictions currently under development cnnot be 100% tested by in-house personnel nd resources. erly in the lifecycle re much cheper to remedy thn those found in production. Delivering qulity softwre is one wy tht QA tems cn directly impct costs, nd this hs become even more true with the prolifertion of mobile pplictions. ABOUT AUTHOR: JULIE CRAIG, senior nlyst t EMA, hs over 20 yers of experience in softwre engineering, IT infrstructure engineering nd enterprise mngement. Julie's res of focus currently include best prctices, configurtion mngement, ppliction mngement, service-oriented rchitecture (SOA) nd Softwre s Service (SS). Her gol is to interpret industry trends with common sense pproch nd simplified outlook. 8 SOFTWARE JULY/AUGUST 2009
1 Regression testing hot spots: Coverge, common mistkes This rticle discusses regression test opportunities, coverge policies nd development, s well s filling you in on five common regression testing mistkes. BY DAVID W. JOHNSON p TEST is one of the most misunderstood softwre testing concepts. Ineffective regression test policies led to incresed postdeployment risk nd resulting business losses. Severl recent softwre filures should hve been detected nd mitigted by effective regression test suites. For exmple, the Blckberry blckouts of 2007 nd 2008 re n unfortunte exmple of production issues tht should hve been detected by regression test suite in combintion with other forml testing efforts. The convergence of dptive development prctices (gile, extreme, etc.) nd complex integrted ppliction spces (Internet, cloud computing, etc.) crete growing risk profile tht needs to be mitigted by disciplined implementtion of effective regression testing policies. Testing erly nd often significntly Severl recent softwre filures should hve been detected nd mitigted by effective regression test suites. reduces remedition costs while mitigting potentil production risks. In this rticle, I will wlk you through regression test opportunities, regression test coverge policies nd regression test development, s well s filling you in on five common regression testing mistkes. Regression testing is often referred to s Test Phse or Test Type, but it is relly the execution of one or more tests within given Test Phse. For the purposes of this rticle, regres- 9 SOFTWARE JULY/AUGUST 2009
sion test is the execution of tests to verify tht chnges in the ppliction lndscpe or rchitecturl frmework hve not negtively impcted existing, unchnged functionlity. Generlly, ccording to WhtIs, regression test involves the execution of tests in current Test Phse tht vlidtes the chnges in the current ppliction lndscpe such s softwre, hrdwre, dt, nd/or metdt hve not negtively impcted existing, unchnged functionlity. TEST OPPORTUNITIES In the pst, regression test hs been closely ssocited with the Function Test Phse nd System Test Phse. For the most prt, the IT community hs come to relize tht testing erly nd often significntly reduces remedition costs while mitigting potentil production issues. The following tble illustrtes regression test opportunities s the solution moves from being conceptul to production For the most prt, the IT community hs come to relize tht testing erly nd often significntly reduces remedition costs while mitigting potentil production issues. relese/pckge. The solution set cn include softwre, hrdwre, dt, metdt, or mixture of these nd ny other spects of the ppliction lndscpe. In the chrt below, Solution Set SOLUTION SET IS EXISTS AS OPPORTUNITY Conceptul (no code) A Model Design Review Phse Under Construction Discrete Units Unit Test Phse Instrumenttion Constructed Functionl Function Test Phse Components Functionl Test Sets Functionl Integrted System Test Phse System System Test Sets Confirmed Pckge User Acceptnce or Relese Test Phse UA Test Sets Relesed Production Production Monitoring Solution 10 SOFTWARE JULY/AUGUST 2009
refers to the current mturity of the solution set. Exists s is the current deliverbles supporting the solution set. Regression Opportunity notes the vilble regression test opportunities. The primry opportunities re in itlics. Unit Test Phse: Instrumenttion The first opportunity to perform substntive regression testing occurs in the unit test phse. The primry ctor responsible for regression testing within the Unit Test Phse is the developer. The developer ccomplishes this by implementing the Adptive development principles nmely gile, extreme nd rpid ppliction development (RAD) of continuous integrtion nd code instrumenttion. Remedition of defects t this level of solution mturity will yield significnt downstrem benefits nd significntly reduce remedition cost. I hve recently hd the plesure of working with severl disciplined gile development tems tht hve fully implemented code instrumenttion nd continuous integrtion. The results of their efforts were significnt decrese in defects lter in the development lifecycle. Function Test Phse: Functionl Test Sets The second opportunity to perform substntive Regression Testing occurs in the Function Test Phse this is the opportunity tht is most often leverged by orgniztions. The primry ctor responsible for Regression Testing within the Function Test Phse is the Tester. The Tester should ccomplish this by implementing suite of utomted scripts tht test the functionl res of the ppliction deemed to be in-scope for the current test There is often not enough time in the test schedule for significnt mnul regression testing. enggement these utomted scripts should be constructed to be reusble test rtifcts. Remedition of defects t this level of solution mturity will reduce remedition costs. Mnul regression testing cn lso be performed, but the overll return on investment is gretly reduced. Mnul tests tke longer to execute nd re prone to filure over multiple executions. There is often not enough time in the test schedule to llow for significnt mnul regression testing. System Test Phse: System Test Sets The third opportunity to perform substntive regression testing occurs in the System Test Phse. The System Test Phse should include severl types of testing; from Regression testing perspective, the two most importnt would be Business functionlity nd performnce. The primry ctor responsible for regression testing business functionlity within the System Test Phse is 11 SOFTWARE JULY/AUGUST 2009
the tester. The tester should once gin ccomplish this by implementing suite of utomted scripts tht test the business functionlity deemed to be in scope for the current test enggement; these utomted 1 Avoiding Common Regression Test Mistkes MOST regression test mistkes come from misplced ssumption tht regression tests: ARE STATIC» Regression test mintennce must be ongoing. Regression tests need to be mintined s IP (intellectul property) tht meet the current orgniztionl test coverge needs.» I hve worked severl enggements in the lst few yers tht hve treted existing regression test suites s sttic rtifcts. In one cse, the client ended up with over five-thousnd test cses tht could never be executed within the time-frme provided for testing. Once we did mintennce sweep of these mnul regression test scripts we hd less thn one-thousnd utomted test cses. ARE A NICE-TO-HAVE» Regression testing is only nice-to-hve if the current orgniztionl coverge needs such s regultory, IT governnce nd risk does not require ny supportive regression test metric. This would be rther unusul circumstnce, but it does occur; i.e., n in-house reporting tool.» Anyone who works in the testing spce hs seen the consequences of this mindset. In severl cses I hve seen clients do little or no regression testing nd py the cost in post-production support; cost tht fr exceeds the cost of ctully doing the testing. NEED TO RETEST EVERYTHING» This is usully not prcticl gol unless the nture of the ppliction spce (i.e. medicl devices) nd the level of risk, justify complete end-to-end regression test the testing tem simply runs out of time. Unfortuntely this usully mens tht the esier tests re executed first while the hrder regression tests re deferred. Ensure you crete test schedule tht fits into the vilble Regression test time-box nd ddresses the higher priority coverge items first. 12 SOFTWARE JULY/AUGUST 2009
scripts should be constructed to be reusble test rtifcts. Ensure these tests do not replicte other function testing efforts. The primry ctor responsible for regression testing performnce within the System Test Phse is the performnce ssurnce test engineer. The performnce ssurnce test engineer should ccomplish this tsk by implementing regression suite of performnce tests nd ssocited performnce monitors these suites should be constructed to be reusble performnce test rtifcts. Mnul regression testing of business functionlity cn lso be performed but the overll return on investment is gretly reduced. Mnul tests tke longer to execute nd re prone to filure over multiple executions. There is often not enough time in the test schedule to llow for significnt mnul regression testing. User Acceptnce Test Phse The finl opportunity to perform regression testing occurs during the User Acceptnce Test Phse. These mnul tests re performed by the end-user. The primry purpose of regression testing during this test phse should be ssuring the enduser tht no unexpected chnges hve occurred in previously existing functionlity. TEST COVERAGE Regression test coverge policies guide the orgniztion in determining wht regression test rtifcts re required to be built nd, more importntly mintined. Regression test policies should be bsed on three key fctors: 1. Regultory Spce Rules, Regultions, Lws, nd Stndrds tht pply to the ppliction spce.» i.e. FDA, FAA, SOX, etc. 2. IT Governnce Corporte governnce s it pplies to the regultory spce. Corporte governnce s it pplies to corporte rules, regultions, nd stndrds. 3. Risk Profile The cceptble level or risk for ny given spect of the ppliction spce. Regultory Spce The regultory spce cn hve significnt impct on the scope of Regression testing within ny given test phse. Mngers nd test leds responsible for testing need to ensure they understnd their legl nd fiduciry responsibilities within the regultory spce nd the impcts of this responsibility on the overll scope of the testing effort. IT Governnce IT governnce implies system in which ll stkeholders hve the necessry input into the IT decision mking process preventing IT from independently mking nd lter being held solely responsible for poor decisions. 13 SOFTWARE JULY/AUGUST 2009
Therefore bord needs to: understnd the overll rchitecture of its compny s IT pplictions portfolio; ensure tht mngement knows wht informtion resources re out there; know wht condition informtion resources re in; understnd wht role informtion resources ply in generting revenue. There re severl definitions of IT governnce: Weill nd Ross focus on Specifying the decision rights nd ccountbility frmework to encourge desirble behvior in the use of IT. IT Governnce Institute expnds the definition to include foundtionl mechnisms: the ledership nd orgniztionl structures nd processes tht ensure tht the orgniztion s IT sustins nd extends the orgniztion s strtegies nd objectives. AS8015, the Austrlin Stndrd for Corporte Governnce of Informtion nd Communiction Technology (ICT) defines Corporte Governnce of ICT s The system by which the current nd future use of ICT is directed nd controlled. It involves evluting nd directing the plns for the use of ICT to support the orgniztion nd monitoring this use to chieve plns. It includes the strtegy nd policies for using ICT within n orgniztion. From regression testing perspective ppliction of IT Governnce to the ppliction lndscpe mens the pproprite level of informtion must be consistently supplied to support decision mking t the bord level. Regression testing provides severl opportunities to collect pproprite informtion s IT Governnce prctices mture the mount of informtion collected needs to be ssessed for completeness. Risk Profile Risk is the net negtive impct of vulnerbility tking into considertion both the probbility nd impct/cost if the risk becomes n issue/event. Risk mngement is the process of identifying risk, ssessing risk, nd Regression tests should be consistently reviewed nd dpted to meet the current coverge needs. tking steps to reduce risk to n cceptble level. Risk mngement often encompsses three min processes: risk ssessment, risk mitigtion, nd risk evlution Regression testing dels with supplying dt 14 SOFTWARE JULY/AUGUST 2009
to support risk evlution. The orgniztion should hve lredy considered wht level of risk is cceptble for prticulr ppliction. It is the responsibility of the orgniztion to understnd the bsis for the current risk ssessment nd pply the pproprite level of Regression testing rigor, to meet the specified level of cceptble risk. Acceptble levels of risk cn be ctegorized s: Tier 1: Criticl Appliction Spce Miniml or no interruptions in service re cceptble Tier 2: Importnt Appliction Spce Short uncommon interruptions in service re cceptble Tier 3: Norml Appliction Spce Short common interruptions in service re cceptble Tier 4: Non-Criticl Appliction Spce Interruptions in service re cceptble Appliction cn be pulled Regression Test Policies: Coverge 1. Regression test coverge in combintion with other forml testing efforts shll provide sufficient test coverge to stisfy the Regultory Spce. 2. Regression test coverge in combintion with other forml testing efforts shll provide sufficient test coverge to stisfy the current IT Governnce requirements. 3. Regression test coverge in combintion with other forml testing efforts shll provide sufficient test coverge to stisfy the current cceptble Risk Profile. TEST DEVELOPMENT Regression tests should ddress ll meningful: Code Brnches: Regression testing t the level of Unit Test Functionl Business events: Regression testing t the level of Function Test End-to-End Business events: Regression testing t the level of System Test User Experience: Regression testing t the level of User Acceptnce Test In ech cse pproprite tests need to be designed, built, nd constructed. These tests need to be esy to mintin, with definitive pss/fil conditions, nd when possible utomted. Regrding regression test utomtion, the User Acceptnce test should be defined nd executed by representtives of the user community - these types re rrely utomted. All other levels of regression testing should be either utomted using test utomtion tools or instrumented using Adptive development techniques. On the mintennce front, regression tests re often treted s sttic 15 SOFTWARE JULY/AUGUST 2009
set of rtifcts to be pplied during ech relese cycle. Regression tests should be consistently reviewed nd dpted to meet the current coverge needs (regultion, governnce, nd risk). This review should include dding to, removing, nd merging regression tests in the regression test suite. If test cse mintennce is not incorported into Regression testing, the tendency is to end up with n unmngeble number of tests tht my not ddress the coverge need. In softwre development, there re some key regression test policies tht you cn t overlook without bd consequences. From my experience, these re the most importnt: Regression tests shll be utomted using test utomtion tools or instrumented into the solution using Adptive development techniques. Regression tests shll undergo scheduled mintennce designed to right size the Regression test suite to meet the current coverge needs, such s regultion, governnce nd risk. Regression test dependencies plns, schedules, test dt, test environments, tools, etc. must be considered prt of the regression test suite. Regression testing needs to be incorported into the overll testing strtegy. The gol should be to mitigte risks to production cused by unexpected impcts ginst unchnged functionlity; creting regression test sves erly in the system development life cycle. A sve is the detection of defect before it reches production thus providing the opportunity to remedite the defect before it is relesed. The regression testing effort lso needs to be rightsized to meet the corporte coverge requirements nd fit into the vilble regression testing time-box. ABOUT AUTHOR: DAVID (DJ) W. JOHNSON is senior test rchitect with over 22 yers of experience in Informtion Technology cross severl industries hving plyed key roles in business needs nlysis, softwre design, softwre development, testing, trining, implementtion, orgniztionl ssessments, nd support of business solutions. Developed specific expertise over the pst 12 yers on implementing Test wre including - test strtegies, test plnning, test utomtion (functionl nd performnce), nd test mngement solutions. 16 SOFTWARE JULY/AUGUST 2009