Transcription

1 TechnicalReportRSTR GaryMcGraw,ChristophMichaelandMichaelSchatz GeneratingSoftwareTestDataby Evolution RSTCorporation Suite#250,21515RidgetopCircle Sterling,VA20166 February9,1998 datageneration.thisresearchextendspreviousworkondynamictestdatageneration wheretheproblemoftestdatagenerationisreducedtooneofminimizingafunction[miller andspooner,1976,korel,1990].inourwork,thefunctionisminimizedbyusingoneof Thispaperdiscussestheuseofgeneticalgorithms(GAs)forautomaticsoftwaretest Abstract twogeneticalgorithmsinplaceofthelocalminimizationtechniquesusedinearlierresearch. thisapproachonanumberofprograms,oneofwhichissignicantlylargerthanthosefor ofprogramcomplexityonthetestdatagenerationproblembyexecutingoursystemona WedescribetheimplementationofourGA-basedsystem,andexaminetheeectivenessof whichresultshavepreviouslybeenreportedintheliterature.wealsoexaminetheeect numberofsyntheticprogramsthathavevaryingcomplexities.

2 1 Introduction Animportantaspectofsoftwaretestinginvolvesjudginghowwellaseriesoftestinputstestsapieceof code.usuallythegoalistouncoverasmanyfaultsaspossiblewithapotentsetoftests,sinceatestseries thathasthepotentialtouncovermanyfaultsisobviouslybetterthanonethatcanonlyuncoverafew. Unfortunately,itisalmostimpossibletopredicthowmanyfaultswillbeuncoveredbyagiventestset.This isnotonlybecauseofthediversityofthefaultsthemselves,butbecausetheveryconceptofafaultisonly vaguelydened.testadequacycriteria,meanttodistinguishgoodtestsetsfrombadones,weredeveloped toaddressthisproblem. Onceatestadequacycriterionhasbeenselected,thequestionthatarisesnextishowtogoaboutcreating atestsetthatisgoodwithrespecttothatcriterion.sincethiscanbediculttodobyhand,thereisaneed forautomatictestdatageneration. Inthispaper,weintroduceGADGET(theGeneticAlgorithmDataGEnerationTool),whichusesatest datagenerationparadigmcommonlyknownasdynamictestdatageneration.dynamictestdatageneration wasoriginallyproposedby[millerandspooner,1976]andtheninvestigatedfurtherby[korel,1990],[fergusonandkorel,1996]and[gallagherandnarasimhan,1997].thisparadigmtreatspartsoftheprogramas functionsthatcanbeevaluatedbyexecutingtheprogram,andwhosevalueisminimalforthoseinputsthat satisfytheadequacycriterion.inthisway,theproblemofgeneratingtestdatareducestothewell-understood problemoffunctionminimization. Theapproachusuallyproposedforperformingthisminimizationisgradientdescent,butgradient-descent suersfromsomewell-understoodweaknesses.thusitisappealingtousemoresophisticatedtechniquesfor functionminimization,suchasgeneticsearch[holland,1975],simulatedannealing[kirkpatricketal.,1983], ortabusearch[glover,1989].inthispaper,weinvestigatetheuseofgeneticsearchtogeneratetestcases byfunctionminimization. Inthepast,automatictestdatagenerationschemeshaveusuallybeenappliedtosimpleprograms(e.g., mathematicalfunctions)usingsimpletestadequacycriteria(e.g.,branchcoverage).randomtestgeneration performsadequatelyontheseproblems.nevertheless,itseemsunlikelythatarandomapproachcouldalso performwellonrealistictest-generationproblems,whichoftenrequireanintensivemanualeort.indeed, ourresultssuggestthatrandomtestgenerationperformspoorlyonrealisticprograms.thebroaderimplicationisthatduetotheirsimplicity,toyprogramsfailtoexposethelimitationsofsometest-datageneration techniques.therefore,suchprogramsprovidelimitedutilitywhencomparingdierenttestgenerationmethods.becausegadgetwasdesignedtoworkonlargeprogramswrittenincandc++,itispossiblefor ustoexaminetheaectsofprogramcomplexityonthedicultyoftestdatageneration. Byusinglargerprograms,wealsouncoverapotentiallyimportantdierencebetweendynamictest datagenerationandotherfunctionminimizationproblems.onthelargerprogramswetested,coincidental discoveryoftestinputssatisfyingnewcriteriawasmuchmorecommonthantheirdeliberatedetection.in fact,theabilityoftestdatageneratorstosatisfycoveragecriteriacoincidentallyseemstoplayanimportant roleindeterminingtheireectiveness. Randomtestgenerationdidnotperformwellinourexperiments,althoughonemightexpectittobe goodatdiscoveringthingsbycoincidence.theguidedsearchperformedbygeneticalgorithmsappearsto begoodatsettingupthecoincidentaldiscoveryoftestssatisfyingnewcriteria.thisislikelyduetothefact that,dependingonitstnessfunction,geneticsearchcanbemadetoconcentrateitseortonfruitfulareas oftheinputspace. 2 Testadequacycriteriaandtestdatageneration Empiricalresultsindicatethattestsselectedonthebasisoftestadequacycriteriaaregoodatuncovering faults[horganetal.,1994,chilenskiandmiller,1994,demilloandmathur,1992].furthermore,testadequacy criteriaareobjectivemeasuresbywhichthequalityofsoftwaretestscanbejudged.neitherofthesebenets canberealizedunlessadequatetestdata(i.e.,testdatathatsatisfytheadequacycriteria)canbefound. Manualgenerationofsuchtestscanbequitetime-consuming,soitwouldbeappealingtohavealgorithms thatcanexamineaprogram'sstructureandgenerateadequatetestsautomatically. Unfortunately,testdatagenerationleadstoanundecidableproblem:itisnotpossibletotakeatest adequacycriterionanddeterminewhetheraninputexiststhatsatisesit.tocircumventthisdilemma, {1{

3 amountofcomputationtimethealgorithmmayexpend).ifnosolutionisfoundbeforetheallotedresources alwayssucceedinndinganadequatetestinput.comparisonsofdierenttestdatagenerationschemesare practicaltestdatagenerationalgorithmsareusuallygivenresourcelimitations(e.g.,limitationsonthe usuallyaimedatdeterminingwhichmethodcanprovidethemostbenetgivenxedresources. beingmorecapableofndingadequatetests.ourresearchspecicallyaddressesthisneed. areexhausted,thenthealgorithmisconsideredtohavefailed.inshort,testgenerationalgorithmsdonot 2.1Codecoverageandtestadequacycriteria Clearly,itisdesirabletohavetestdatagenerationalgorithmsthataremorepowerfulinthesenseof Mosttestadequacycriteriarequirecertainfeaturesofaprogram'ssourcecodetobeexercised.Asimple exampleisacriterionthatsays,\eachstatementintheprogramshouldbeexecutedatleastoncewhenthe programistested."testmethodologiesthatusesuchcriteriaareusuallycalledcoverageanalyses,because certainfeaturesofthesourcecodearetobecoveredbythetests. example,toobtainbranchcoverageofthecodefragment: coverage.thiscriterionrequireseveryconditionalbranchintheprogramtobetakenatleastonce.for Theexamplegivenabovedescribesstatementcoverage.Aslightlymorerenedapproachisbranch if(a>=b){doonething} ofb,andanotherthatcausesthevalueofatobelessthanthatofb.oneeectofthisrequirementisto requiresoneprograminputthatcausesthevalueofthevariableatobegreaterthanorequaltothevalue else {dosomethingelse} ensurethatboththe\doonething"and\dosomethingelse"sectionsoftheprogramareexecuted. inaprogram.weshallrefertothishierarchyasdeninglevelsofcoverage.atthetopofthehierarchy ismultipleconditioncoverage,whichrequiresthetestertoensurethateverypermutationofvaluesforthe Booleanvariablesineveryconditionoccursatleastonce.Atthebottomofthehierarchyisfunctioncoverage whichrequiresonlythateveryfunctionbecalledonceduringtesting(sayingnothingaboutthecodeinside Thereisahierarchyofincreasinglycomplexcoveragecriteriahavingtodowiththeconditionalstatements eachfunction).somewherebetweentheseextremesiscondition-decisioncoverage,whichisthecriterionwe useinourtest-datagenerationexperiments. valuedexpressions,whileadecisionisanexpressionthatinuencestheprogram'sowofcontrol.toobtain condition-decisioncoverage,atestsetmustmakeeachconditionevaluatetotrueforatleastoneofthe tests,andeachconditionevaluatetofalseforatleastoneofthetests.furthermore,thetrueandfalse branchesofeachdecisionmustbeexercised.putanotherway,condition-decisioncoveragerequiresthateach Aconditionisanexpressionthatevaluatestotrueorfalse,butdoesnotcontainanyothertrue/false- onceẇithanyofthesecoveragecriteria,wemustaskwhattodowhenatestsetfailstomeetthechosen branchinthecodebetakenandthateveryconditioninthecodebetrueatleastonce,andfalseatleast criterion.inmostcases,thenextstepistotrytondatestsetthatdoessatisfythecriterion.sinceitcan areusedtoautomatethisprocess. 2.2Previousworkintestdatageneration bequitediculttomanuallysearchfortestinputssatisfyingcertaincriteria,testdatagenerationalgorithms testdatageneration.inthenextthreesubsections,wewilldescribeeachofthesetechniquesinturn.the datagenerationasabaselineforcomparison. counteredarerandomtestdatageneration,symbolic(orpath-oriented)testdatageneration,anddynamic GADGETsystemwedescribeinthispaperisadynamictestgenerator.Inourexperiments,weuserandom Therearemanyexistingparadigmsforautomatictestdatageneration.Perhapsthemostcommonlyen- {2{

4 Randomtestdatagenerationsimplyconsistsofgeneratinginputsatrandomuntilausefulinputisfound Randomtestdatageneration testinputmayhavetosatisfyveryspecicrequirements.insuchacasethenumberofadequateinputsmay Theproblemwiththisapproachisclear:withcomplexprogramsorcomplexadequacycriteria,anadequate beverysmallcomparedtothetotalnumberofinputs,sotheprobabilityofselectinganadequateinputby chancecanbelow. [Korel,1996]foundthatrandomtestgenerationwasoutperformedbyothermethodsevenonsmallprograms wherethegoalwastoobtainstatementcoverage.morecomplexprogramsormorecomplexcoveragesare likelytopresentevengreaterproblemsforrandomtestdatagenerators.nonetheless,randomtestdata generationmakesagoodbaselineforcomparison,becauseitiseasytoimplementandcommonlyreported Thisintuitionisconrmedbyempiricalresults(includingthosereportedinSection5).Forexample Manytestdatagenerationmethodsusesymbolicexecutiontondinputsthatsatisfyanadequacycriterion intheliterature. (e.g.,[clarke,1976,ramamoortyetal.,1976,outt,1991]).symbolicexecutionofaprogramconsistsof 2.2.2Symbolictestdatageneration assigningsymbolicvaluestovariablesinordertocomeupwithanabstract,mathematicalcharacterization ofwhattheprogramdoes.thus,inanidealcase,testcasegenerationcanbereducedtoaproblemofsolving analgebraicexpression. arisesinindeniteloops,wherethenumberofiterationsdependsonanon-constantexpression.toobtain isneverentered,ifititeratesonce,ifititeratestwice,andsoonadinnitum.inotherwords,thesymbolic executionoftheprogrammayrequireaninniteamountoftime. acompletepictureofwhattheprogramdoes,itmaybenecessarytocharacterizewhathappensiftheloop Anumberofproblemsareencounteredinpracticewhensymbolicexecutionisused.Onesuchproblem symbolicallyforonecontrolpathatatime.pathsmaybeselectedbytheuser,byanalgorithm,orthey maybegeneratedbyasearchprocedure.ifonepathfailstoresultananexpressionthatyieldsanadequate testinput,anotherpathistried. Testdatagenerationalgorithmssolvethisprobleminastraightforwardway:theprogramisonlyexecuted otherobstaclestoapracticaltestdatagenerationalgorithmbasedonsymbolicexecution.problemscan arisewhendataisreferencedindirectly,asinthestatement: Loopsarenottheonlyprogrammingconstructsthatcannoteasilybeevaluatedsymbolically;thereare a=b[c+d]/10 fragment: Here,itisunknownwhichelementofthearrayBisbeingreferredtobyB[c+d],becausethevariablescand darenotboundtospecicvalues. Pointerreferencesalsopresentaproblembecauseofthepotentialforaliasing.ConsiderthattheCcode *a=12; resultsinctakingthevalue12unlessthepointersaandbrefertothesamelocation,inwhichcasecis *b=13; assignedthevalue13.sinceaandbarenotboundtonumericvaluesduringsymbolicexecution,thenal c=*a; valueinccannotbedetermined. ofsymbolicallyexecutingrealprograms. referencesarenotatheoreticalimpedimenttotheuseofsymbolicexecution,theycomplicatetheproblem notnormalpracticetoavoidtheseconstructswhenwritingaprogram.thus,althougharrayandpointer Technically,anycomputablefunctioncanbecomputedwithouttheuseofpointersorarrays,butitis {3{

5 2.2.3Dynamictestdatageneration Athirdclassoftestdatagenerationparadigmsisdynamictestdatageneration,introducedin[Millerand Spooner,1976]andexempliedbytheTESTGENsystemof[Korel,1990,Korel,1996]aswellastheADTEST systemof[gallagherandnarasimhan,1997].thisparadigmisbasedontheideathatpartsofaprogram canbetreatedasfunctions.onecanexecutetheprogramuntilacertainlocationinthecodeisreached, recordthevaluesofoneormorevariablesatthatlocation,andtreatthosevaluesasthoughtheywerethe valueofafunction. Forexample,supposethatahypotheticalprogramcontainsthecondition if(pos>=21)... online324,andthatthegoalistoensurethatthetruebranchofthisconditionistaken.wemustndan inputthatwillcausethevariablepostohaveavaluegreaterthanorequalto21whenline324isreached. Asimplewaytodeterminethevalueofposonline324istoexecutetheprogramuptoline324andthen recordthevalueofpos. Letpos324denotethevalueofpos,recordedonline324whentheprogramisexecutedintheinputx. Thenthefunction F(x)=21?pos324;ifpos324<21; 0; otherwise isminimalwhenthetruebranchistakenonline324.thus,theproblemoftestdatagenerationisreduced tooneoffunctionminimization:tondthedesiredinput,wemustndavalueofxthatminimizesf(x). Unfortunately,thisisanoversimplication,becauseline324maynotbereachedforsomeinputs.There are,however,twocommonsolutionstothisproblem.first,onecanamendthedenitionoff(x)sothatit willhaveaverylargevaluewheneverthedesiredconditionisnotreached.second,onecantreattheproblem ofreachingthedesiredlocationasasubproblemthatmustbesolvedbeforetheminimizationoff(x)can commence. IntheTESTGENsystemof[Korel,1990],theminimizationofF(x)beginsbyestablishinganoverall goal,whichissimplythesatisfactionofthetestadequacycriterion.theprogramisexecutedonaseed input,anditsbehavioronthisinputisusedasthebasisofasearchforasatisfactoryinput(thatis,ifthe seedinputisnotsatisfactoryitself). Thesubsequentactiondependsonwhetherthetheexecutionreachesthesection(s)ofcodewherethe adequacycriterionissupposedtohold(forexample,whetheritreachesline324intheexampleabove).ifit does,thenfunctionminimizationmethodscanbeusedtondanadequateinputvalue. Ifthecodeisnotreached,asubgoaliscreatedtobringabouttheconditionsnecessaryforfunction minimizationtowork.thesubgoalconsistsofredirectingtheowofcontrolsothatthedesiredsectionof codewillbereached.thealgorithmndsabranchthatisresponsible(whollyorinpart)fordirectingthe owofcontrolawayfromthedesiredlocation,andattemptstomodifytheseedinputinawaythatwill forcethecontrolofexecutioninthedesireddirection. Thenewsubgoalcanbetreatedinthesamewayasothertestadequacycriteria.Thusthesearchforan inputsatisfyingasubgoalproceedsinthesamewayasthesearchforaninputsatisfyingtheoverallgoal. Likewise,moresubgoalsmaybecreatedtosatisfytherstsubgoal.Thisrecursivecreationofsubgoalsis calledchainingin[korel,1990]and[fergusonandkorel,1996]. Korel'sapproachisadvantageouswhenthereismorethanonepaththatreachesthedesiredlocationin thecode.thetest-data-generationalgorithmisfreetochoosewhicheverpathitwants(aslongasitcan forcethatpathtobeexecuted),andsomepathsmaybebetterthanothers.forexample,supposewewant totakethetruebranchofthecondition if(b>10)... butsupposethatbhasadefaultvalueof3.itmaybethatoneexecutionpathgivesbanewvalue,while adierentpathsimplyleavesthevalueofbalone.aslongasweonlytakethepaththatleavesbwithits defaultvalue,wewillneverbeabletomaketheconditiontrue;nochoiceofinputscanmakethedefault valueofbbeanythingotherthan3.therefore,thetestgenerationalgorithmmustknowhowtoselectthe executionpaththatassignsanewvaluetob.inthetestgensystem,heuristicsareusedtoselectthe paththatseemsmostlikelytohaveanimpactonthetargetcondition. {4{

6 andthegoaloftestdatagenerationistondaninputthatexecutesthedesiredpath.sinceitisknown whichbranchmustbetakenforeachconditiononthepath,alloftheseconditionscanbecombinedina singlefunctionwhoseminimizationleadstoanadequatetestinput.forexample,ifthedesiredpathrequires takingthetruebranchofthecondition IntheADTESTsystemof[GallagherandNarasimhan,1997],anentirepathisspeciedinadvance, if(b>=10)... online11,andtakingthefalsebranchofthecondition online13,thenonecanndanadequatetestinputbyminimizingthefunctionf1(x)+f2(x),where if(c>=8) F1(x)=10?b11;ifb11<10online11; 0; otherwise F2(x)=c13?8;ifc13>8online13; theadtestsystembeginsbytryingtosatisfytherstconditiononthepath,addingthesecondcondition Unfortunately,thisfunctioncannotbeevaluateduntilline10andline13arebothreached.Therefore 0; otherwise onlyaftertherstconditionhasbeensatised.asmoreconditionsarereached,theyareincorporatedin thefunctionthatthealgorithmseekstominimize. ThissystemrequiresinstrumentationoftheAdaprogrambyhand.Oncethecodeisinstrumentedandranges Changetal.,1996].ThisisahybridsystemcombiningrandomtestinganddynamictestingforAdacode. andtypesofinputvariableshavebeenprovided,thesystemcreatestestdatausingrule-basedheuristics. Forexample,valuesofparametersareadjustedaccordingtoonesuchruletoincreaseordecreasebyaxed AnothertestgenerationsystemrelevanttoourworkistheQUEST/Adasystemof[Deasonetal.,1991, constantpercentage.thetestadequacycriterionchosenbychangetal.isbranchcoverage.thesystem createsacoveragetableforeachbranchandmarksthosethathavebeensuccessfullycovered.table1 branchestotargetfortesting.partially-coveredbranchesarealwayschosenovercompletely-non-covered providesanexampleofsuchabranchtable.thetableisconsultedduringanalysistodeterminewhich branches.(thisstrategywillbeexplainedindetailinsection4.1.) Decisiontruefalse 12 Branch 34 X Table1:AsamplecoveragetableafterChang.Thistableisgeneratedfromaprogramowchart.Thetableprovides 56 - X informationregardingthecoveredbranchesanddirectsfuturetestcasegeneration.weadaptthisapproachforour generatoraswell. - coverwhateverconditionsitcanreach.althoughthisisinecientwhenoneonlywantstoexerciseacertain featureofthecodeundertest,itcansavequiteabitofunnecessaryworkifonewantstoobtaincomplete dealingwiththesituationwhereadesiredconditionisnotreached.insteadofpickingaparticularcondition astestgendoes,orpickingaparticularpathlikeadtest,thisstrategyisopportunisticandseeksto Thecoveragetableof[Changetal.,1996]isrelevanttoourresearchbecauseitprovidesastrategyfor coverageaccordingtosomecriterion. Theresearchdescribedinthispaperaddressestwolimitationscommonlyfoundindynamictest-datagenerationsystems.First,manysystemsmakeitdiculttogeneratetestsforlargeprograms,eitherbecausethe 2.3Contributionsofthispaper Weindependentlydevelopedthecoverage-tablestrategy,anduseitinourtest-generationsystem. {5{

7 programmustbeinstrumentedbyhandorbecausetheyonlyworkonsimpliedprogramminglanguages. Second,manysystemsusegradientdescenttechniquestoperformfunctionminimization,andthereforethey canstallwhentheyencounterlocalminima(thisproblemisdescribedbelowingreaterdetail). LimitedprogramcomplexityisadrawbackoftheTESTGENandQUEST/Adasystems.QUEST/Ada requirestheprogramundertesttobeinstrumentedbyhand.testgenonlyallowsprogramswrittenina subsetofthepascallanguage.theproblemwithsuchlimitationsisthattheypreventonefromstudying howthecomplexityofaprogramaectsthedicultyofgeneratingtestdata.theunchallengingdemands ofsimpleprogramscanmakenaveschemeslikerandomtestgenerationappeartoworkbetterthanthey actuallydo. Inthispaper,wereportonGADGET,atestgenerationsystemdesignedforprogramswritteninCor C++.GADGETautomaticallygeneratestestdataforarbitraryC/C++programs,withnolimitationson thepermissiblelanguageconstructsandnorequirementforhand-instrumentation.wereporttestresultsa programcontainingover2000linesofsourcecodeexcludingcomments.toourknowledge,thisisthelargest programforwhichresultshavebeenreported.(although[gallagherandnarasimhan,1997]reportedthat theirsystemhadbeenrunonprogramsaslargeas60,000linesofsourcecode,noresultswerepresented.) TheabilitytogeneratetestsforprogramsusingallC/C++constructshastheaddedbenetofallowingus tostudytheeectsofprogramcomplexityonthedicultyoftestdatageneration. TheGADGETsystemusesgeneticalgorithmstoperformthefunctionminimizationneededduring dynamictestdatageneration.inthisrespectitdiersfromthetestgenandadtestsystems,which usegradientdescent.theadvantageofusinggeneticalgorithmsisthattheyarelesssusceptibletolocal minima,whichcancauseatest-generationalgorithmtohaltwithoutndinganadequateinput. Geneticalgorithmswereusedintestgenerationpreviouslyby[Schultzetal.,1993],butthegoalthere wasnottoachieveacertainlevelofcoverage.instead,thatsystemsoughttoinducebehaviorconsidered likelytomakethesoftwarefail. Themostfrequentlycitedadvantageofgeneticalgorithms,whentheyarecomparedtogradientdescent methods,isthatgeneticalgorithmsarelesslikelytostallinalocalminimum aportionoftheinput spacewheref(x)appearstobeminimalbutisnot.thereisalsoasecondadvantagewhenseveralpathsto thedesirednodeareavailable.unlikegradientdescentmethods,whichmustconcentrateonasinglepath, theimplicitparallelismofgeneticalgorithmsallowsthemtoexaminemanypathsatonce.thispresentsa simplesolutiontothepath-selectionproblemdescribedinsection Certainlimitationsarecommontoalldynamictestgenerationsystems,includingourown.Existing systemsarelimitedtoprogramswhoseinputsarescalartypes.furthermore,wearenotawareofanydynamic test-datagenerationsystemthatcanintelligentlyhandletrue/false-valuedvariablesorenumeratedtypes. Programsusingsuchvariableswithinconditionalstatementsdonotseemtohavebeenusedinpastresearch. (Theproblempresentedbysuchvariableswillbeexaminedinsection5.4.) Inspiteoftheselimitations,oursistheonlydynamictestdatagenerationsystemweknowofthatcanbe runonlargeprograms,andthatatthesametimeusesanoptimizationmethodmorepowerfulthangradient descent.inthispaper,wetakeadvantageofthiscapabilitybyexaminingtheimpactofprogramcomplexity ontheproblemofdynamictestdatageneration. Weexamineafeatureofthedynamictestgenerationproblemthatdoesnothaveananaloginmostother functionminimizationproblems.ifwearetryingtosatisfymanyadequacycriteriaforthesamesoftware,we havetoperformmanyfunctionminimizations,butthefunctionsbeingminimized(sincetheycorrelatewith programsemantics)aresometimesquitesimilar.thatmakesitpossibletosolveoneproblembycoincidence whiletryingtosolveanother.inotherwords,thetestgeneratorcanndinputsthatsatisfyonecriterion eventhoughitissearchingforinputstosatisfyadierentone. Ourresultssuggestthattheserendipitousdiscoveryofinputssatisfyingnewcriteriamaygreatlyinuence thecomplexionofdynamictestdatageneration.itseemsthattheabilitytondgoodtestsbycoincidence playsacrucialroleinthesuccessofadynamictest-datagenerationtechnique.butwhenweexaminethis behaviorinmoredetail(section5.4.3),wewillseethatserendipityofthesortweobserveisfarfromarbitrary. Thisexplainsthefactthatrandomtestgeneration,whichoughttobegoodatndinginputscoincidentally, performspoorlyinourexperiments.theabilityofgeneticalgorithmstoconcentrateonsemantically-relevant partsoftheinputspacethroughevolutionarypressuresseemstogivethemanadvantagewhenitcomesto discoveringnewinputsaccidentally.similaremergentbehaviorofgashasalsobeendocumentedforother kindsofproblems[?]. {6{

8 GADGETdrawsonpastworkindynamictestdatageneration.Itcombinesthefunctionminimization 3andtheuseofacoveragetableafter[Changetal.,1996].Themarriageofthecoverage-tablestrategywith approachof[millerandspooner,1976,korel,1990](thoughusinggastoavoidtheproblemoflocalminima) Geneticalgorithmsfortestdatageneration geneticalgorithmsisparticularlyadvantageousbecausethecoveragetableisacheapalternativetokorel's 3.1Functionminimization maythereforeletusretainsomeoftheadvantagesofexplicitpathselectionwithlesscomputationalcost. path-selectionstrategyandthegacanexaminemanypathsatonce.thecombinedparallel-searchstrategy Theabilitytonumericallyminimizeafunctionvalueisthekeytodynamictestdatageneration.Various numericcomputationalgorithmsareintendedforthispurpose(newton'smethod,etc.),buttheserequire thefunctiontobeberelativelysmooth.thismaynotbethecasewhenthefunctionisonecomputedbyan arbitrarysectionofsomeprogram. ofthesemethodsisgradientdescent,whichmodiestheinputvaluesinsuchawaythatthefunction'svalue alwaysdecreases.theprocessstopswhennofurtherdecreaseinthefunctionvaluecanbeobtainedbyany ofthemodicationsthataretried. Therefore,moregenericmethods,makinglessstringentassumptions,aredesirable.Oneofthesimplest inputvaluearemadeinitiallytodetermineagooddirectionformakinglargermoves.whenanappropriate directionisfound,increasinglylargestepsaretakeninthatdirectionuntilnofurtherimprovementisobtained (inwhichcasethesearchbeginsanewwithsmallinputmodications),oruntiltheinputnolongerreaches thedesiredlocation(inwhichcasethemoveistriedagainwithasmallerstepsize).whennofurtherprogress Theminimizationmethodsuggestedin[Korel,1996]isaformofgradientdescent.Smallchangesinone madeforanyinputvalue. canbemade,adierentinputvalueismodied,andtheprocessterminateswhennomoreprogresscanbe region in which inputs are sought that decrease f(x). f(x) Figure1:Illustrationofalocalminimum.Analgorithmtriestondavalueofxthatwilldecreasef(x),butthere foundaglobalminimumoff. arenosuchvaluesintheimmediateneighborhoodofthecurrentx.thealgorithmmayfalselyconcludethatithas Gradientdescentcanfailifalocalminimumisencountered.Alocalminimumoccurswhennoneofthe global current input x {7{

9 changesofinputvaluesthatarebeingconsideredleadtoadecreaseinthefunctionvalue,andyetthevalue inputvalues(i.e.,asmallsectionofthesearchspace)duetoresourcelimitations.theinputvaluesthatare isnotgloballyminimized.theproblemarisesbecauseitisonlypossibletoconsideralimitednumberof consideredmaysuggestthatanychangeofvalueswillcausethefunction'svaluetoincrease,evenwhenthe currentvalueisnottrulyminimal.thissituationisillustratedinfigure1. notblindlypursuethesteepestgradient.notableamongthesearesimulatedannealing[kirkpatricketal., 1989,Mitchell,1996]).Inthispaper,weapplyageneticalgorithmtotheproblemoftestdatageneration. 1983],tabusearch[Glover,1989,Skorin-Kapov,1990],andgeneticalgorithms[Holland,1975,Goldberg, Theproblemoflocalminimahasledtothedevelopmentoffunctionminimizationmethodsthatdo 3.2Geneticalgorithms Ageneticalgorithm(GA)isarandomizedparallelsearchmethodbasedonevolution.GAshavebeenapplied toavarietyofproblemsandareanimportanttoolinmachinelearningandfunctionoptimization.[goldberg, 1989]and[Mitchell,1996]givethoroughintroductionstoGAsandprovidelistsofpossibleapplicationareas. Themotivationbehindgeneticalgorithmsistomodeltherobustnessandexibilityofnaturalselection. alleles.thestringproducedbytheconcatenationofalltheencodedparametersformsagenotype.each biology,anencodedparametercanbethoughtofasagene,wheretheparameter'svaluesarethegene's aninitialpopulationofindividuals,eachrepresentedbyarandomlygeneratedgenotype.thetnessof genotypespeciesanindividualwhichisinturnamemberofapopulation.thegastartsbycreating InaclassicalGA,eachofaproblem'sparametersisrepresentedasabinarystring.Borrowingfrom individualsisevaluatedinsomeproblem-dependentway(inourcaseindividualsaremoretiftheyseem closertosatisfyinganewcriterion),andthegatriesevolvehighlytindividualsfromtheinitialpopulation. Thethreebasicgeneticoperators:selection,crossover,andmutationcarryoutthesearch. duringeachiteration(generation)untilreachingsometerminationcondition.thebasicalgorithm,where P(t)isthepopulationofstringsatgenerationt,is: Thegeneticsearchprocessisiterative:evaluating,selecting,andrecombiningstringsinthepopulation while(terminationconditionnotsatised)do evaluatep(t) initializep(t) t=t+1 evaluatep(t+1) recombinep(t+1) selectp(t+1)fromp(t) vidual)isbasedonatnessfunctionthatisproblemdependent.determiningtnesscorrespondstothe environmentaldeterminationofsurvivabilityinnaturalselection,andinourcase,itisdeterminedbythe tnessfunctiondescribedinsection Inrststep,evaluation,thetnessofeachindividualisdetermined.Evaluationofeachstring(indi- generation.selectionofastringdependsonitstnessrelativetothatofotherstringsinthepopulation. Mostoften,thetwoindividualsareselectedatrandom,buteachindividual'sprobabilityofbeingchosenis relativetness.itprobabilisticallycullsfromthepopulationindividualshavingrelativelylowtness. proportionaltoitstness.thisisknownasroulette-wheelselection.thus,selectionisdoneonthebasisof Thenextstep,selection,isusedtondtwoindividualsthatwillbematedtocontributetothenext nature.onetypeofsimplecrossoverisimplementedbychoosingarandompointinaselectedpairofstrings (encodingapairofsolutions)andexchangingthesubstringsdenedbythatpointasshowningure3.2. thepermanentlossofalleles.mutationsimplyresultsintheippingofabitswithagenome,butthis Thethirdstepiscrossover(orrecombination),whichllstheroleplayedbysexualreproductionin ippingofbitsonlyoccursinfrequently. Inadditiontoevaluation,selection,andrecombination,geneticalgorithmsusemutationtoguardagainst {8{

10 Crossover Point Parents A B ofpartsfrombothparentsleadingtoinformationexchange. Figure2:Single-pointcrossoverofthetwoparentsAandBproducesthetwochildrenCandD.Eachchildconsists Children population,usuallyleadingthegaawayfromunpromisingareasofthesearchspaceandtowardspromising TheindividualsinthepopulationactasaprimitivememoryfortheGA.Geneticoperatorsmanipulatethe C D ones,withoutthegahavingtoexplicitlyrememberitstrailthroughthesearchspace[rawlins,1991]. genotype(string)tophenotype(pointinsearchspace)isusuallytrivial.forexample,inordertooptimize thefunctionf(x)=x,individualscanberepresentedasbinarynumbersencodedinnormalfashion.in thiscase,tnessvalueswouldbeassignedbydecodingthebinarynumbers.ascrossoverandmutation manipulatethestringsinthepopulationtherebyexploringthespace,selectionprobabilisticallyltersout ItiseasiesttounderstandGAsintermsoffunctionoptimization.Insuchcases,themappingfrom stringswithlowtness,exploitingtheareasdenedbystringswithhightness.notethatthissearch techniqueisnotsubjecttotheproblemsassociatedwithlocalminima,whichweredescribedabove. 4thatourapproachisbasedontheconversionofthetest-datagenerationproblemtoafunctionminimization problem,whichallowsageneticalgorithmtobeapplied.thetwodetailsthatmustbeaddressedarethe Inthissection,wedescribeourapproachtotestdatagenerationbygeneticsearch.Recallfromsection2.2.3 GADGET:GeneticAlgorithmDataGEnerationTool ourtestadequacycriteriontobesatised,andwemustconvertthatcriterionintoafunctionthatcanbe sameasthosedescribedinsection2.2.3:wemustndawaytoreachthecodelocationwherewewant minimized. 4.1Reachingthetargetcondition. Recallthatindynamictestgeneration,functionminimizationcannotbeperformedunlesstheowofcontrol reachesacertainpointinthecode.forexample,ifweareseekinganinputthatexercisesthetruebranch ofaconditioninline954ofaprogram,weneedinputsthatreachline954beforewecanbegintodofunction minimization. concentrateonndingaspecicpathtothedesiredlocation.ourgoalis(amongotherthings)tocoverall branchesinaprogram.thismeanswecansimplydelayourattemptstosatisfyacertainconditionuntilwe havefoundteststhatreachthatcondition. Ourapproachisslightlydierentfromthatof[Korel,1990]and[GallagherandNarasimhan,1997],which 1:if(state_error>state_bndry[0]) 2: Thiscanbeillustratedusingthefollowingcodefragment: 3:elseif((state_error>state_bndry[2])) 4: 5:elseif((state_error<state_bndry[1])) 6: rule_area=area(1.0,(state_bndry[0]-state_bndry[1])); state_weight=rule(state_error,state_bndry[2],state_bndry[0]); state_weight=rule(state_error,state_bndry[3],state_bndry[1]); {9{

11 Theconditiononline5canonlybereachedwhentheconditiononline1andtheconditiononline3are thatcausetherstconditiontobetrueaswellassomethatcauseittobefalse.therefore,whenwehave bothfalse. Inordertoreachtheifconditiononline3,atestcasemustcausetheconditiononline1tobefalse. line3.wearethusreadytobeginlookingfortestcasesthatcoverbothbranchesofthedecisiononline3. achievedbranchcoverageofline1,wewillalreadyhaveatleastonetestcasethatreachestheconditionon Likewise,whenwehavecoveredtheconditiononline3,wewillhavetestcasesthatreachline5,and Ifthegoaloftestgenerationistoexerciseallbranchesinthecode,thentheremustbothbetestcases thereforewewillbereadytobeginlookingfortestcasesthatexercisethebranchesoftheconditiononline5. functionminimizationtothatcondition.ifbothbrancheshavebeentaken,thencoverageissatisedforthat isgeneratedtokeeptrackoftheconditionalbranchesalreadycoveredbyexistingtestcases.ifneither branchofaconditionhasbeentaken,thenthatdecisionhasnotbeenreached,sowearenotreadytoapply conditionandweneednotexamineitfurther.however,ifonlyonebranchofaconditionhasbeenexercised, Thisleadstoatestgenerationapproachsimilartotheoneemployedby[Changetal.,1996].Atable online1hasbeencoveredbecauseboththetrueandfalsebrancheshavebeenexercised.sincethefalse thentheconditionhasbeenreached,anditisappropriatetoapplyfunctionminimizationinsearchofan branchesofthatdecisionhasnecessarilybeenexercised.table3showsasituationwheretheconditionon branchofthatconditionwasexercised,theconditiononline3hasbeenreachedaswell,andoneofthetwo inputthatwillexercisetheotherbranch. line3wasalsofalse,meaningthattheconditiononline5wasreachedandoneofthebranchesofthat Forthecodefragmentshownabove,thesituationisillustratedbyTable2.Inthetable,thecondition conditionhasbeenexercisedȧ Decisiontruefalse 13 X Thistablecanbeautomaticallygeneratedfrom Table2:AsamplecoveragetableafterChang. 5 - thecodefragmentshownabove.suchtablesare X Decisiontruefalse 13 B 5 X- X - tionduringautomatictestdatageneration. usedtodecidewheretoapplyfunctionminimiza- Table3:Asimilartableshowingasituation wheretestshavetakenadierentcombinationof branches. 3,oronethatexercisesthetruebranchonline5.Bothconditionsarereachedandsobotharecandidates forfunctionminimization.table3showsadierentsituationwhereline5wasnotreachedbecauseonlythe truebranchwastakenonline3.inthatcase,wecannotapplyfunctionminimizationonline5because notestcasesreachthere.however,wecanapplyfunctionminimizationatline3.ifwedososuccessfully, InTable2,wemaybeginlookingforatestcasethatexercisesthetruebranchoftheconditiononline wewillndatestthatexercisesthefalsebranchofthatcondition,andthattestwillreachline5.after wehaveobtainedoneormoresuchtests,theconditiononline5willbereachableandwewillbereadyto beginfunctionminimizationatthatpoint. usethegatoperformanewfunctionminimizationforeachconditionwereach.forexample,inthecase illustratedbytable3wewouldusethegatocoverthefalsebranchonline3.withluck,wemaynd online5.ifthishappens,wewillbenished.ontheotherhand,ifourtestcasesonlycoveroneofthe severalteststocoverthisbranch,anditmayevenbethatthesetestscoverbothbranchesofthecondition Aswehavealreadystated,GADGETperformsfunctionminimizationusingageneticalgorithm.We branchesonline5,wewillhavetorunthegaagaintocovertheremainingbranch. example,aninputtotheb737autopilotcomponentweusedtotestthegadgetsystem(seesection5.4) contains186inputvalues,soanindividualisaspeciclistof186variableswhenthegaisgeneratingtests forthatproblem.theseindividualscompetewitheachotheraccordingtonaturalselectionrules.thebest individualsareretainedandbredtogether,whileuselessindividualsareculledfromthepopulation.the TheGAworksbycreatingapopulationofindividualsthatcorrespondtoindividualtestcases.For {10{

12 goodnessortnessofanindividualiscomputedasafunctionofthesourcecode(aswithalldynamictest datagenerationmethods;seesection2.2.3).thegastopsintwosituations:oneinwhichitissuccessful, andanotherinwhichnoforwardprogresshasbeenmadeforaspeciedperiodoftime.inthemostuseful situation,thegastopswhenavalidtestcasemeetingtheminimizationcriteriahasbeendiscovered(e.g., thegastopswhenaconditionitistryingtohitfalseactuallyis).itmayalsostopiftherehasbeenno improvementinoveralltnessforaspeciednumberofgenerations.inthiscase,thegahasfailedand eitheranewruncanbetriedwithanewrandomnumberseed,orelsethegoalcanbechanged. thetestsreachingthatcondition.theinitialpopulationmayalsobecontainsomerandomtestcasesifthe conditionwasnotreachedsucientlyoften. Whenwerunthegeneticalgorithmonagivencondition,ourinitialpopulationoftestcasescontains 4.2Calculationoftnessfunctions. Recallfromsection2.2.3thatdynamictestgenerationinvolvesreducingthetestgenerationproblemtoone ofminimizingatnessfunctionf.therststepistodenef.likemostotherdynamictestgeneration techniques,oursbeginsbyinstrumentingthecodeundertest.thepurposeofthisinstrumentationisto allowustocalculatethetnessfunctionbyexecutingtheinstrumentedcode. tablebelowshowshowf(x)iscalculatedforsometypicalrelationaloperatorswhenweareseekingtotake thetruebranchofacondition(thefunctionsforthefalsebranchareanalogous). Ateachcondition,weaddinstrumentationtoreportF(x)whenexecutionreachesthatcondition.The decisiontype inequality example if(c>=d)... F(x)=d-c;ifdc; tnessfunction true/falsevalueif(c)... if(c==d)...f(x)=jd?cj F(x)=1000;ifc=false; 0; 0; otherwise executedthestatement)thenthetnessfunctiontakesitsworstpossiblevalue. Iftheprogram'sexecutionfailstoreachthedesiredlocation(itterminatesortimesoutwithouthaving otherwise if((c>d)&&(c<f))... canbehandledbyexploitingc/c++shortcircuitevaluation.forexample,thesecondclauseofthecondition Sinceoursystemseekstogeneratecondition/decisionadequatetestsets,conjunctionsanddisjunctions clausemustbothbetrueisreplacedbyacriterionstatingthatthesecondclausemustbereached,and isnotreachedunlesstherstclauseevaluatestotrue,sothecriterionthatstatestherstandsecond mustevaluatetotrue.ifbothclausesarereached,andbothclausestakeonboththevaluetrueandthe valuefalse(asrequiredbycondition/decisioncoverage),thenbothbranchesoftheconditionalbranchwill necessarilyhavebeentaken. 4.3Executioncontrol IntheGADGETsystem,anexecutioncontrollerisinchargeofrunningtheinstrumentedcode,coordinating GAsearches,andcollectingcoverageresultsandnewtestcases.Itbeginsbyexecutingallpreliminary testcases.(thesepreliminarycasescanbesuppliedbytheuserorgeneratedrandomlybytheexecution controller.)afterrunningallinitialtestcases,theexecutioncontrollerusesthecoveragetabletonda conditionthatcanbereached,buthasnotbeencoveredyet(thatis,noinputhasmadethecondition true,orelsenoinputhasmadeitfalse).thegeneticalgorithmisinvokedtomakethisconditiontake havebeencovered). thecondition(thoughtheydidnotgivetheconditionthedesiredvalue,orelsetheconditionwouldalready onthevaluethatwasnotalreadyobserved.thegaisseededwithtestcasesthatcansuccessfullyreach {11{

13 untilallconditionsthathavehadonlyonevalue(eithertrueorfalse)havebeensubjectedtogasearch. TheexecutioncontrollerkeepstrackofallGArunsthatcovernewprogramcode,regardlessofwhether TheGAiscalledagainwiththetaskofndinganinputthatcoversthiscondition.Thisprocesscontinues theexecutioncontrollerusesthecoveragetabletondanewconditionthathasnotbeencoveredcompletely. WhentheGAterminates,eitherbyndingasuccessfultestcaseorbyreachingaterminationcondition, ornottheysatisfythecriterionthatthegaiscurrentlyworkingon.(inotherwords,gadgettakes advantageofallserendipitouscoverages.)thesetestcasesarestoredforlateruse. TheGADGETsystemcurrentlycontainstwoGAimplementations.TherstisastandardGAusingonepointcrossover,alowmutationvalue,andaroulette-wheelselectionoperator.ThiskindofGAisdescribed 4.4Functionminimizationmodules numericalminimizationproblemsthanthestandardga. insection3.2.thesecondgaisadierentialga[storn,1996],whichismorespecicallytailoredfor representedas432=128bits.reproductionproceedsaccordingtothefollowingalgorithm: 1.Everyindividualxreceivesatnessvalue,obtainedasinsection4.2(sincewewantgoodinputsto InthestandardGA,eachinputisrepresentedasabitstring.Forexample,four32-bitintegersare 2.Twoindividualsarechosenatrandomaccordingtotness.Eachindividual'sprobabilityofbeing haveahightness,weuse1=f(x)asthetnessvalue,truncatingitifitbecomestoolarge). 3.Acrossoverpointinthebitstringischosenuniformlyatrandom. chosenisitsthetnessdividedbythesumofalltheindividuals'tnesses.thesameindividualisnot 4.Twonewindividualsarecreatedusingone-pointcrossover,startingwiththetwoparentsselectedin chosentwotimes. 5.Mutationisapplied:eachbitinthenewindividualshasasmallchanceofbeingipped. 6.Thetwomosthighlytofthefourindividuals(twoparentsandtwochildren)areselected. step2. 7.Thetwonewindividualsareaddedtothenextgenerationunlesstheyareidentical.Iftheyareidentical, 8.Steps2{7arerepeateduntilthenewgenerationhassamenumberofindividualsastheprevious arandomly-generatedindividualisaddedaswell. 9.Steps1{9arerepeateduntilaterminationconditionissatised(thetargetlocationishittrue/false, generation.(thisnumberisthepopulationsize,providedasaparametertothegeneticalgorithm.) Reproductionfollowsthealgorithm: ThesecondtypeofGAincorporatedinGADGETisthedierentialGAasreportedin[Storn,1996]. orngenerationshavebeenproducedwithnoimprovementintness.) 2.CreateanewindividualI0accordingtothefollowingplan:ForeachinputvariablevalueinI,copy 1.Foreachindividualinthepopulation,selectaparentI,andthreedierentdistinctindividualsforuse aspotentialmates.callthema,b,andc. 3.Ifallvariablesarecopiedverbatiminstep2,thenforceatleastonevariablevalue(chosenatrandom) thevariablevaluefromitoi060%ofthetime.theremaining40%ofthetime,createanewvariable valueusingtheformulaa+0:5(c?b)andcopyittoi0. 4.IfI0isbetterthanI,thenaddI0tothenewgeneration,otherwiseaddItothenewgeneration. 5.Repeatuntilthenewgenerationhassamenumberofindividualsasthepreviousgeneration.(This tobecreatedbythea+0:5(c?b)rule. numberisthepopulationsize.) {12{

14 generatorplaysfundamentallythesameroleasthega,butitcreatesnewinputvaluesentirelyatrandom. 6.Repeatuntilaterminationconditionissatised(eitherthetargetlocationishittrue/false,orN Thesevaluesaretestedtoseeiftheysatisfyanyoftheadequacycriteriabysimplyusingthemasinputsto InadditiontotheGAs,wealsoimplementedarandomtestcasegenerationmodule.Therandomtestcase generationshaveexecutedwithnotnessimprovement.) resourcesareexhausted. arestorediftheyincreasecoverage.thisisrepeateduntilallcoveragecriteriaaremetortheallocated theprogramandrecordingthecoverageobtained(withnotnesscalculation,ofcourse).successfulguesses 5randomtestgenerationperformonincreasinglycomplexsyntheticprograms.Thethirdexperimentshows Inthissection,wereportonfoursetsoftestdatagenerationexperiments.Therstsetofexperimentsinvolves programsthatcalculatesimplenumericfunctions.thesecondexperimentinvestigateshowthegasand Experimentalresults theimpactthatcoveragelevelhasonempiricalresults.finally,wepresentresultsobtainedbyanalyzinga real-worldautopilotcontrolprogramcalledb Simpleprograms Webeganourexperimentationonasetofsimplefunctionsmuchlikethosereportedintheliterature[DeMillo andoutt,1993,changetal.,1996,fergusonandkorel,1996].theprogramsanalyzedwere: Binarysearch Numberofdaysbetweentwodates Bubblesort Insertionsort Computingthemedian Euclideangreatestcommondenominator Warshall'salgorithm Triangleclassication Quadraticformula Theseprogramsareroughlyofthesamecomplexity,averaging30linesofcodeandallhavingrelatively and[fergusonandkorel,1996];inthosepapers,randomtestgenerationalsoperformednearlyaswellas bothapproacheshavethesameeectiveness.theseresultsresemblethosereportedin[changetal.,1996] moresophisticatedtechniquesonsimpleprograms. simpledecisions. Randomtestcasegenerationhastheupperhandintheseexperimentsbecauseitinvolvessignicantly Fortheseprograms,randomtestdatagenerationneveroutperformsgeneticsearch,thoughsometimes lesscomputation.however,ineverycaseoneofthegasperformsthebestoverall.table4showsthe results.thenumbersreportedinthetablerepresentthehighestcoverageobtainedduringaseriesofve runs. classicationisshownbelow.(thecommentsontherightmarginwillbeusedlatertoshowwhichinputs satisedwhichconditions.)notethatmanydecisionsonlycontainasinglecondition. ItisusefultoanalyzeasampleprogramtounderstandwhattheGAsaredoing.ThecodeforTriangle {13{

15 Program Binarysearch Bubblesort1 Bubblesort2 random 80 GA 70 dierential-ga Numberofdaysbetweentwodates Euclideangreatestcommondenominator Insertionsort 87.5 Computingthemedian 92.3 Quadraticformula Table4:Acomparisonofthreetestdatagenerationapproachesonsimpleprograms.Thedierential-GAexhibits Warshall'salgorithm Triangleclassication thebestperformanceoverall,butnotbyasignicantamount.thesedatarepresentthehighestcoverageobtained byeachmethodduringaseriesofveruns. #include<stdio.h> inttriang(inti,intj,intk){ //4ifnotatriangle //3iftriangleisequilateral //2iftriangleisisosceles //1iftriangleisscalene //returnsoneofthefollowing: if(i==j)tri+=1; inttri=0; if(i==k)tri+=2; if((i<=0) (j<=0) (k<=0))return4; //g //h //acd if(j==k)tri+=3; if(tri==0){ //i returntri; } elsetri=1; if((i+j<=k) (j+k<=i) (i+k<=j))tri=4;//be (tri>3)tri=3; //f//bef returntri; } elsetri=4; elseif((tri==3)&&(j+k>i))tri=2; elseif((tri==2)&&(i+k>j))tri=2; elseif((tri==1)&&(i+j>k))tri=2; //h voidmain(){ if(t==1) intt=triang(a,b,c); inta,b,c; scanf("%d%d%d",&a,&b,&c); printf("enter3integersforsidesoftriangles\n"); }Figure3showshowthethreedierentsystems standardga,dierentialga,andrandomtestgeneration performonthetriangleprogramshownabove. resultslargelyfromthisphenomenon;lessexertionwasrequiredofthegabecausesomanycriteriawere whentheyweretryingtosatisfyadierentcriterion.infact,theshorterexecutiontimeofthestandardga coveredserendipitously.wendthistobearecurringphenomenoninourexperiments,andhavemoreto AninterestingresultoftheseexperimentswasthattheGAsoftensatisedonetestcriterionbycoincidence elseif(t==4)printf("thisisnotatriangle\n"); elseif(t==3)printf("triangleisequilateral\n"); elseif(t==2)printf("triangleisisosceles\n"); printf("triangleisscalene\n"); //abcdegi //h //f {14{

16 % covered triangle GA differential random triangleprogram.thegraphshowshowperformance(intermsofthepercentageofcriteriasatised)improves asthenumberofrunsincreases.randomtestgenerationhitsitspeakearly,butfailstoimproveafterthat.the Figure3:AconvergencegraphcomparingtheperformanceofthetwoGAsandrandomtestgenerationonthe pointwisemeanperformanceoververunsforeachsystem. dierentialgahasabetterperformanceandexecutesforalongeramountoftime,butthestandardgahasthe bestperformanceoverall,coveringabout93%ofthecodeonaverageinabout8,000runs.thecurvesrepresentthe runs x sayaboutitinsection5.4. RunnumberKeyInteger1 234 bac Integer2 Integer dgef Table5:AtableofsampleinputcasesgeneratedbythestandardGAfortriangle.Thesedatacanbemappedto hi conditionalexpressionsinthecodeshownaboveusingthekeyeld beinganalyzedwasrelativelysimple.in[changetal.,1996],randomtestdatagenerationwasreported canbemappedtothesourcecodeshownabovebyusingthelettersshowninthecomments. tocover93:4%oftheconditionsonaverage.althoughitsworstperformance onaprogramcontaining11 AsampleofresultsobtainedbytheGAtestdatagenerationalgorithmisshowninTable5.Thesedata decisionpoints was45:5%,itoutperformedmostoftheothertestgenerationschemesthatweretried.only symbolicexecutionhadbetterperformancefortheseprograms.[fergusonandkorel,1996]reportsoneleven Ourresultsforrandomtestcasegenerationresemblethosereportedelsewhereforcaseswherethecode achieving100%statementcoverageforveprograms,andaveraging76%coverageontheothersix. programsaveragingjustover100linesofcode.overall,randomtestdatagenerationwasfairlysuccessful, {15{

17 Itisalsointerestingtocompareourresultswiththoseobtainedby[Korel,1996]forthreeslightlylarger programs.again,simplebranchcoveragewasthegoal.randomtestgenerationachieved67%,68%,and 79%coverage,respectively,onthethreeprogramsanalyzed.Symbolictestgenerationachieved63%,60%, and90%coverage,whiledynamictestgenerationachieved100%,99%,and100%coverage. Theseresultsshowacommontrend:randomtestgenerationhasatleastanadequateperformanceon suchprograms,butforlargerprogramsormoredemandingcoveragecriteria,itsperformancedeteriorates. Theprogramsusedin[Korel,1996]werelargerthanthoseusedin[Changetal.,1996],andrandomtest generationhadpoorerperformanceonthelargerprograms. Theresultsreportedinthissectionusesmallprogramsthoughourtest-adequacycriterionismore stringent.insomecases(likethetriangleprogram)randomtestgenerationisfarlessadequateeventhough resourcelimitationswerenotatellingfactor:thenal90%oftherandomly-generatedtestsfailedtosatisfy anynewcoveragecriteria.inoursubsequentexperimentsonlargerprograms,wendthistobeacontinuing trend:programsize,programcomplexity,andcoveragelevelalldecreasethepercentageofadequacycriteria thatcanbesatisedusingrandomtestgeneration. Sincethepercentageofadequacycriteriasatisedbyrandomtestgenerationactuallygoesdownwhen theprogramsbecomemorecomplex,thesuggestionisthatsomethingaboutlargeprograms,otherthanthe sheernumberofconditionsthatmustbecovered,makesitdiculttogeneratetestdataforthem.thisis conrmedinourlaterexperiments;afterexaminingtheeectsofprogramcomplexityinthenextsection, weinvestigatetherolethatcoveragelevelplaysinthisresultinsection5.3.insection5.4weexaminethe performanceoftest-datagenerationtechniquesonamedium-sizedreal-worldprogram. 5.2Theroleofcomplexity Inoursecondsetofexperiments,wecreatedsyntheticprogramswithconditionsanddecisionswhosecharacteristicswecontrolled.Thetwocharacteristicswewereinterestedincontrollingwere:1)howdeeply conditionalclauseswerenested(wecallthisthenestingfactor),and2)thenumberofbooleanconditionsin eachdecision(whichwecallconditionfactor).forexample,aprogramwithnonestedconditionalclauses wouldlooklikethebeginningpartofthetriangleprogram,inthatifstatementsarenotnestedinsideother ifstatements.thenatureoftheconditionalexpressionsineachconditionalclauseiscontrolledbythesecondparameter.thedecision((i<=0) (j<=0) (k<=0))fromthetriangleprogramranksas a3onthisscalebecauseitcontainsthreeconditions.wecanclassifyprogramsaccordingtotheircomplexity withafunctioncompl(nest,cond)wherenestisthenestingfactorandcondistheconditionfactor.inourexperiments,programsweregeneratedwithallcomplexitiescompl(nest,cond),nest2f0;3;5g;cond2f1;2;3g. ThisexperimentreportsresultsofthedierentialGA.OurresultsshowthatneitherofthetwoconditionsaloneaccountfordierencedbetweenrandomandGA-basedtestgeneration.Infact,alltechniques performedsimilarlyforthesetscompl(;1)andcompl(0;).themostinterestingbehaviorisobservedby increasingthenestingcomplexityatthesametimeastheconditioncomplexity(thecomplexitiescompl(0;1), compl(3;2),andcompl(5;3)). Figures4{6showthreegraphsrepresentingconvergencebehaviorofthetwotypesofsystems.TheGA showsremarkablybetterperformancethanrandomtestdatagenerationonthecomplexcases.themore complextheprograms,thebetterthegaperformswithrespecttorandomtestdatageneration. 5.3Theimpactofcoveragelevel Theaectofcoveragelevel thatis,thedicultyofsatisfyinganadequacycriterion onthediculty ofautomaticallygeneratingtestdataisdemonstratedwiththefollowingexperiment.figure7showshow coveragelevelsaectthesuccessofdierenttestdatagenerationmethods.thegureshowsresultsfor theb737program(seesection5.4),butthisbehaviorisobservedinallexperiments.ingeneral,alltest datagenerationtechniquesperformedbestonfunctioncoverage,whichwasourleastdemandingcriterion. Performancedeterioratedforallthreetechniqueswhenmoredemandingcoveragecriteriawereused.Additionally,experimentswithmoredicultcoveragecriteriawerebetterabletodiscriminatebetweenourthree testgenerationtechniques. Notethattheleastamountofdiscriminationisshowninthefunctioncoveragecondition,andthemostin theconditiondecisioncoveragecondition.clearly,deeperlevelsofcoverageanalysisarehardertoachieve. {16{

18 % covered Nesting Complexity (1,1) random GA Figure4:RandomandGAperformrelativelyequallyonthenon-complexprogram.compl(0,1) runs % covered Nesting Complexity (2,2) Figure5:Ascomplexityincreases,theGAbeginstodobetter.compl(3,2). GA random runs x % covered Nesting Complexity (3,3) GA Figure6:Highcomplexityprogramsarehardforbothgenerators,buttheGAoutperformsrandombyevenmore. random runs x compl(5,3). {17{

19 % covered b737 coverage level differential function coverage random function coverage differential branch coverage differential C/D coverage testadequacycriteria.highercoveragelevelssuchascdcincreasediscriminabilitybetweenapproachesandprovide Figure7:ConvergencegraphcomparingperformanceoftheGAandrandomtestdatagenerationusingdierent random branch coverage betteranalysisdata. random C/D coverage b737:real-worldcontrolsoftware runs x Inthisstudy,weusedtheGADGETsystemonb737,aCprogramwhichispartofanautopilotsystem. Thiscodehas69decisionpointsand2046sourcelinesofcode(excludingcomments).Itwasgeneratedby acasetool. weremadeusingeachmethod.forthetwogeneticalgorithms,wemadesomeattempttotuneperformance byadjustingthenumberofindividualsinthepopulationandthenumberofgenerationsthathadtoelapse withoutanyimprovementbeforethegaswouldgiveup.thegoalofthisne-tuningwastomaximizethe percentageofconditionscovered,whilekeepingtheexecutiontimelow. WegeneratedtestsusingthestandardGA,thedierentialGA,andrandomtestdatageneration.Tenruns ationstoelapsewhennoimprovementintnesswasseen.forthedierentialgaweusedpopulationsof 25individualseach,andallowed20generationstoelapsewhentherewasnoimprovementintness(the smallernumberofindividualsallowsmoregenerationstobeevaluatedwithagivennumberofprogram executions,andwefoundthatthedierentialgatypicallyneededmoregenerationsbecauseitconverges Forthestandardgeneticalgorithm,weusedpopulationsof100individualseach,andallowed15gener- moreslowlythanthestandardga).asbefore,weattemptedtogeneratetestcasesthatsatisfyconditiondecisioncoverage.notethataconditionalstatementmaycontainseveralconditions,andcondition-decision coveragerequireseachoftheseconditionstotakeonbothpossiblevalues.thisisastrongercriterionthan branch-adequacy,whichonlyrequiresthatthetrueandfalsebranchesoftheconditionalstatementbe executed. wepermittedthesamenumberofprogramexecutionsaswasusedbythegeneticsearch.thisamountsto thousandsofrandomtests,oneforeachtimethetnessfunctionwasevaluatedduringgeneticsearch.note, onceeachtimethetnessfunctionisevaluatedforsometestcase.inthecaseofrandomtestgeneration, datagenerationtothesameprogram.theprogramforwhichtestdataarebeinggeneratedmustbeexecuted First,wetriedtoachievecondition-decisioncoveragewiththeGADGET.Next,weappliedrandomtest however,thatrandomtestgenerationstopsmakingprogressveryquickly. showthebest,worst,andpointwisemeanperformanceovertenseparaterunsofeachsystem. ThebestperformanceisseenononeofthetenstandardGAruns.Inthisrun,geneticsearchwasableto Figure8showstheconvergencegraphcomparinggeneticandrandomtestdatageneration.Thegraphs {18{

20 % covered b737 GA best, worst, mean diff. GA best GA differential random diff. GA mean diff. GA worst Figure8:Convergencegraphcomparingperformanceofthreesystemsontheb737code.Threecurvesareshown foreachsystem.theyrepresentthebestperformance,thepointwisemeanovertenruns,andtheworseperformance TheGAsbothshowmuchbetterperformancethanrandomtestdatageneration,withthestandardGAgenerally random best, worst, & mean outperformingthedierentialga runs x 10 3 achievemorethan93%cdccodecoverage signicantlybetterthanrandomtestgeneration,whichonly achieved55%coverage.overall,thestandardgaperformsbest,withameanperformancenoticeablyhigher thanthatofthedierentialga,andfarabovethatofrandomtestgeneration.therewaslittlevariabilityin theperformanceofthestandardgaandalmostnoneintheperformanceoftherandomtestgenerator,but populationweusedforthedierentialga.(withfewerindividualsinthepopulation,itislesslikelythat thedierentialgaexhibitedsurprisingvariabilitybetweenruns.thismayhavebeencausedbythesmaller Here,asinourotherexperiments,mostinputswerediscoveredserendipitously.(Notethatwhenatest 5.4.1Detailedanalysisofmodelruns statisticaluctuationsintnesswillcanceleachotherout.) criterionissatisedserendipitously,itoftenhappensbeforethegeneticalgorithmmakesaconcertedattempt tosatisfythatcriterion.therefore,thefactthatmanycriteriaweresatisedbychancedoesnotimplythat dierentgaimplementations.aquicklookatpartsofthesourcecodeelucidatesthisbehavior.wewill notsatisedatall.inthisrespect,theb737experimentsshedsomelightonthetruebehaviorofthetwo thegawouldhavefailedtosatisfythemotherwise.) rstexplainatypicalindividualrunofthestandardgaandthendiscussthedierentialga. Thefactthatmostinputswerediscoveredbyluckmeansthatmostcriterianotsatisedbychancewere executionsofb737,thisrunsoughttosatisfyadequacycriteriaontwelvedierentconditionsinthecode. Ofthesetwelveattempts,onlyonewassuccessful.Theremainingelevenattemptsshowedlittleforward areclosetothemeanvalueofallcoverageresultsproducedbythestandardgaonb737.inits11,409 progressduringtengenerationsofevolution. TheexecutionofthestandardGAweexamineasamodelwasselectedbecauseitscoverageresults criteriaotherthantheonesitwasworkingonatthetime.thehighcoveragelevelthatwasnallyattained wasmostlyduetothosefourteeninputs.indeed,themostsuccessfulexecutionshavetheshortestrun-times preciselybecausesomanyinputswerefoundserendipitously manyconditionshadalreadybeencovered bychancebeforethegawasreadytobeginworkingonthem.(aswehavestatedalready,thisdoesnot Whilemakingtheseattempts,however,theGAcoincidentallydiscoveredfourteenteststhatsatised {19{