ARGONNENATIONALLABORATORY 9700SouthCassAvenue Argonne,Illinois BenchmarkingOptimizationSoftwarewithPerformanceProles

Transcription

1 ARGONNENATIONALLABORATORY 9700SouthCassAvenue Argonne,Illinois60439 BenchmarkingOptimizationSoftwarewithPerformanceProles MathematicsandComputerScienceDivision ElizabethD.DolanandJorgeJ.More PreprintANL/MCS-P January2001 Energy,underContractW Eng-38,andbytheNationalScienceFoundation(ChallengesinComputationalScience)grantCDA and(InformationTechnologyResearch)grantCCR ThisworkwassupportedbytheMathematical,Information,andComputationalSciences DivisionsubprogramoftheOceofAdvancedScienticComputing,U.S.Departmentof

2 Contents 4CaseStudy:OptimalControlandParameterEstimationProblems 2PerformanceEvaluation 3BenchmarkingData 1Introduction CaseStudy:TheFullCOPS 6CaseStudy:LinearProgramming 7Conclusions Acknowledgments References 12

3 BenchmarkingOptimizationSoftwarewithPerformanceProles ElizabethD.DolanyandJorgeJ.Morez 1Introduction Weproposeperformanceproles distributionfunctionsforaperformancemetric asatoolforbenchmarkingandcomparingoptimizationsoftware.weshowthatperformanceprolescombinethebestfeaturesofothertoolsforperformanceevaluation. Abstract withtheevaluationandperformanceofoptimizationcodes.asrecentexamples,wecite [1,2,3,4,6,12,17]. Mittelmann'seortshavegainedthemostnotice,otherresearchershavebeenconcerned Thebenchmarkingofoptimizationsoftwarehasrecentlygainedconsiderablevisibility.Hans themaintechnicalissuesaddressedinthispaper.mostbenchmarkingeortsinvolvetables Mittlemann's[13]workonavarietyofoptimizationsoftwarehasfrequentlyuncovered displayingtheperformanceofeachsolveroneachproblemforasetofmetricssuchascpu decienciesinthesoftwareandhasgenerallyledtosoftwareimprovements.although time,numberoffunctionevaluations,oriterationcountsforalgorithmswhereaniteration impliesacomparableamountofwork.failuretodisplaysuchtablesforasmalltestset wouldbeagrossomission,buttheytendtobeoverwhelmingforlargetestsets.inallcases, theinterpretationoftheresultsfromthesetablesisoftenasourceofdisagreement. Theinterpretationandanalysisofthedatageneratedbythebenchmarkingprocessare researcherstotryvarioustoolsforanalyzingthedata.thesolver'saverageorcumulative mance[1,4,6].asaresult,asmallnumberofthemostdicultproblemscantendto dominatetheseresults,andresearchersmusttakepainstogiveadditionalinformation.anotherdrawbackisthatcomputingaveragesortotalsforaperformancemetricnecessitates discardingproblemsforwhichanysolverfailed,eectivelybiasingtheresultsagainstthe totalforeachperformancemetricoverallproblemsissometimesusedtoevaluateperfor- Thequantitiesofdatathatresultfrombenchmarkingwithlargetestsetshavespurred subprogramoftheoceofadvancedscienticcomputing,u.s.departmentofenergy,undercontract W Eng-38,andbytheNationalScienceFoundation(ChallengesinComputationalScience)grant valuecanbeassignedforfailedsolverattempts,butthisrequiresasubjectivechoicefor mostrobustsolvers.asanalternativetodisregardingsomeoftheproblems,apenalty separatetable. thepenalty.mostresearcherschoosetoreportthenumberoffailuresonlyinafootnoteor CDA and(InformationTechnologyResearch)grantCCR ThisworkwassupportedbytheMathematical,Information,andComputationalSciencesDivision zmathematicsandcomputersciencedivision,argonnenationallaboratory,argonne,illinois60439 ydepartmentofelectricalandcomputerengineering,northwesternuniversity,andmathematicsand 1

4 dierencebetweensolvertimes[4])appearstobeaviablewayofensuringthataminority place,usuallyfork=1;2;3.rankingthesolvers'performanceforeachproblemhelps [4,6,15,17].Inotherwords,theycountthenumberoftimesthatasolvercomesinkth preventaminorityoftheproblemsfromundulyinuencingtheresults.informationonthe sizeoftheimprovement,however,islost. Toaddresstheshortcomingsofthepreviousapproach,someresearchersrankthesolvers oftheproblemsdonotdominatetheresults,butinourtestingwehavewitnessedlargeleaps inquartilevaluesofaperformancemetric,ratherthangradualtrends.ifonlyquartiledata isused,theninformationontrendsoccurringbetweenonequartileandthenextislost;and wemustassumethatthejourneyfromonepointtoanotherproceedsatamoderatepace. Also,inthespeciccaseofcontrastingthedierencesbetweensolvertimes,thecomparison Comparingthemediansandquartilesofsomeperformancemetric(forexample,the isthatifresultsaremixed,interpretingquartiledatamaybenoeasierthanusingtheraw data;anddealingwithcomparisonsofmorethantwosolversmightbecomeunwieldy. failstoprovideanyinformationontherelativesizeoftheimprovement.analdrawback denethebordersofverycompetitiveandcompetitive. appearsin[2],withsolversratedbythepercentageofproblemsforwhichasolver'stimeis termedverycompetitiveorcompetitive.theratioapproachavoidsmostofthediculties thatwehavediscussed,providinginformationonthepercentimprovementandeliminating thenegativeeectsofallowingasmallportionoftheproblemstodominatetheconclusions. Themaindisadvantageofthisapproachliesintheauthor'sarbitrarychoiceoflimitsto Theideaofcomparingsolversbytheratioofonesolver'sruntimetothebestruntime mulative)distributionfunctionforaperformancemetric.inthispaperweusetheratioof thecomputingtimeofthesolverversusthebesttimeofallofthesolversastheperformance metric.section3providesananalysisofthetestsetandsolversusedinthebenchmark theperformanceofoptimizationsoftware.theperformanceproleforasolveristhe(cu- resultsofsections4and5.thisanalysisisnecessarytounderstandthelimitationsofthe benchmarkingprocess. InSection2,weintroduceperformanceprolesasatoolforevaluatingandcomparing parameterchoicesandtheneedtodiscardsolverfailuresfromtheperformancedata. inuenceofasmallnumberofproblemsonthebenchmarkingprocessandthesensitivityof resultsassociatedwiththerankingofsolvers.performanceprolesprovideameansofvisualizingtheexpectedperformancedierenceamongmanysolvers,whileavoidingarbitrary withversion2.0ofthecops[7]testset.weshowthatperformanceproleseliminatethe WeconcludeinSection6byshowinghowperformanceprolesapplytothedata[13] Sections4and5demonstratetheuseofperformanceproleswithresults[8]obtained theuseofperformanceprolesandalsoshowsthatperformanceprolescanbeappliedto ofmittelmannforlinearprogrammingsolvers.thissectionprovidesanothercasestudyof awiderangeofperformancedata. 2

5 informationofinterestsuchasthenumberoffunctionevaluationsandthecomputingtime. Inthissectionweintroducethenotionofaperformanceproleasameanstoevaluateand comparetheperformanceofthesolversonatestsetp. BenchmarkresultsaregeneratedbyrunningasolveronasetPofproblemsandrecording 2PerformanceEvaluation measures.foreachproblempandsolvers,wedene putingtimeasaperformancemeasure;although,theideasbelowcanbeusedwithother Weassumethatwehavenssolversandnpproblems.Weareinterestedinusingcom- performanceratio bysolverswiththebestperformancebyanysolveronthisproblem;thatis,weusethe If,forexample,thenumberoffunctionevaluationsistheperformancemeasureofinterest, settp;saccordingly. Werequireabaselineforcomparisons.Wecomparetheperformanceonproblemp tp;s=computingtimerequiredtosolveproblempbysolvers. solversdoesnotsolveproblemp.theperformanceofsolversonanygivenproblemmay solver.ifwedene beofinterest,butwewouldliketoobtainanoverallassessmentoftheperformanceofthe WeassumethataparameterMp;sforallp;sischosen,andp;s=Mifandonlyif ps()=1npsizenp2p:p;so; p;s= minftp;s:1snsg: possibleratio.thefunctionpsisthe(cumulative)distributionfunctionfortheperformance problemsthatarelikelytooccurinapplications,thensolverswithlargeprobabilityps() ratioẇeusethetermperformanceproleforthedistributionfunctionofaperformancemetric.ourclaimisthataplotoftheperformanceprolerevealsallofthemajorperformance characteristics.inparticular,ifthesetofproblemspissuitablylargeandrepresentativeof thenps()istheprobabilitythataperformanceratiop;siswithinafactorofthebest aretobepreferred. versusaproblemparameter.forexample,higham[11,pages296{297]plotstheratio =kak1,whereistheestimateforthel1normofamatrixaproducedbythelapack conditionnumberestimator.notethatinhigham'suseofthetermperformanceprole thereisnoattemptatdeterminingadistributionfunction. function,continuousfromtherightateachbreakpoint.thevalueofps(1)istheprobability Thetermperformanceprolehasalsobeenusedforaplotofsomeperformancemetric numberofwins,weneedonlytocomparethevaluesofps(1)forallofthesolvers. thatthesolverwillwinovertherestofthesolvers.thus,ifweareinterestedonlyinthe Theperformanceproleps:R7![0;1]forasolverisanondecreasing,piecewiseconstant 3

6 istheprobabilitythatthesolversolvesaproblem.thus,ifweareinterestedonlyinsolvers resultofthisconvention,ps(m)=1,and thatp;s2[1;m]andthatp;s=monlywhenproblempisnotsolvedbysolvers.asa Thedenitionoftheperformanceproleforlargevaluesrequiressomecare.Weassume withahighprobabilityofsuccess,thenweneedtocomparethevaluesofps( M)forall solversandchoosethesolverswiththelargestvalue.thevalueofps( M)canbereadilyseen for2[s;m)forsomes<m. inaperformanceprolebecausepsatlinesforlargevaluesof;thatis,ps()=ps( M) ps( M)lim! Mps() asourceofdisagreementbecausethereisnoconsensusonhowtochooseproblems.thecops problemsareselectedtobeinterestinganddicult,butthiscriteriaissubjective.because 3BenchmarkingData themainpurposeofourtestingistouncoverthebestgeneralsolveronawiderangeof problems,weincludeproblemswithmultipleminimaandproblemswithalargenumberof ThetimingdatausedtocomputetheperformanceprolesinSections4and5isgenerated degreesoffreedom.foreachoftheapplicationsinthecopssetweusefourinstancesofthe withthecopstestset,whichcurrentlyconsistsofseventeendierentapplications,all applicationobtainedbyvaryingaparameterintheapplication,forexample,thenumber modelsintheampl[9]modelinglanguage.thechoiceofthetestproblemsetpisalways mationapplicationsinthecopsset,whilethediscussioninsection5coversthecomplete ofgridpointsinadiscretization. performanceresults.accordingly,weprovidehereinformationspecictothissubsetofthe COPSproblemsaswellasananalysisofthetestsetasawhole.Table3.1givesthequartiles forthreeproblemparameters:thenumberofvariablesn,thenumberofconstraints,and theratio(n ne)=n,whereneisthenumberofequalityconstraints.intheoptimization literature,n neisoftencalledthedegreesoffreedomoftheproblem,sinceitisanupper Section4focusesononlythesubsetoftheelevenoptimalcontrolandparameteresti- throughoutthetestsetandshowsthatatleastthree-fourthsoftheproblemshavethe boundonthenumberofvariablesthatarefreeatthesolution. AnotherfeatureoftheCOPSsubsetisthattheequalityconstraintsaretheresultofeither intherange[1;50]becauseotherbenchmarkingproblemsetstendtohaveapreponderance numberofvariablesnintheinterval[400;5000].ouraimwastoavoidproblemswherenwas ofproblemswithninthisrange.themaindierencebetweenthefullcopssetandthe COPSsubsetisthattheCOPSsubsetismoreconstrainedwithnen=2foralltheproblems. ThedatainTable3.1isfairlyrepresentativeofthedistributionoftheseparameters settingtheoutputlevelsothatwecangatherthedataweneed,increasingtheiteration dierenceorcollocationapproximationstodierentialequations. Weranournalcompleterunswiththesameoptionsforallmodels.Theoptionsinvolve 4

7 to5000.noneofthefailureswerecordinthenaltrialsincludeanysolvererrormessages abouthavingviolatedtheselimits. limitsasmuchasallowed,andincreasingthesuper-basicslimitsforminosandsnopt Numberofconstraints Degreesoffreedom(%) Numberofvariables minq1q2q3maxminq1q2q3max Table3.1:ProblemdataforCOPStestset FullCOPS COPSsubset soastominimizetheeectofuctuationinthemachineload.thescripttracksthewallclocktimefromthestartofthesolve,killinganyprocessthatruns3,600seconds,which recordingthewall-clocktimeaswellasthecombinationofamplsystemtime(tointerpret themodelandcomputevaryingamountsofderivativeinformationrequiredbyeachsolver) andsolvertimeforeachmodelvariation.werepeatthecycleforanymodelforwhichone wedeclareunsuccessful,andbeginningthenextsolve.wecyclethroughalltheproblems, Thescriptforgeneratingthetimingdatasendsaproblemtoeachsolversuccessively, ofthesolver'sampltimesandthewall-clocktimesdierbymorethan10percent.to furtherensureconsistency,wehaveveriedthattheampltimeresultswepresentcouldbe requirementscanalsoplayanimportantrole.inparticular,solversthatusedirectlinear criteriaarenotlikelytochangecomputingtimesbymorethanafactoroftwo.memory intheamplenvironment.inanycase,dierencesincomputingtimeduetothestopping thesolvers.ideallywewouldhaveusedthesamestoppingcriteria,butthisisnotpossible runningsolaris7. reproducedtowithin10percentaccuracy.allcomputationsweredoneonasparcultra2 equationsolversareoftenmoreecientintermsofcomputingtimeprovidedthereisenough memory. Wehaveignoredtheeectsofthestoppingcriteriaandthememoryrequirementsof mation.theuseofsecond-orderinformationcanreducethenumberofiterations,but useonlyrst-orderinformation,whilelancelotandloqoneedsecond-orderinfor- thecostperiterationusuallyincreases.inaddition,obtainingsecond-orderinformationis morecostlyandmaynotevenbepossible.minosandsnoptarespecicallydesigned LANCELOTandLOQO.Asanexampleofcomparingsolverswithsimilarrequirements, forproblemswithamodestnumberofdegreesoffreedom,whilethisisnotthecasefor Thesolversthatwebenchmarkherehavedierentrequirements.MINOSandSNOPT Section6showstheperformanceoflinearprogrammingsolvers. 5

8 4CaseStudy:OptimalControlandParameterEstimationProblems WenowexaminetheperformanceprolesofLANCELOT[5],MINOS[14],SNOPT[10], andloqo[16]onthesubsetoftheoptimalcontrolandparameterestimationproblemsin thecops[7]testset.figures4.1and4.2showtheperformanceprolesindierentranges providethenecessaryobjectiveinformationforreasonablesubjectiveanalysisofalargetest set.figure4.1showstheperformanceprolesofthefoursolversforsmallvaluesof. Byshowingtheratiosofsolvertimes,weeliminateanyweightofimportancethattaking toreectvariousareasofinterest.ourpurposeistoshowhowtheperformanceproles ofbeingtheoptimalsolver)andthattheprobabilitythatloqoisthewinneronagiven straighttimedierencesmightgivetotheproblemsthatrequirealongruntimeofevery solversreceivetheirduecreditforcompletingproblemsforwhichoneormoreoftheother solversfails.inparticular,1 ps()isthefractionofproblemsthatthesolvercannotsolve withinafactorofthebestsolver,includingproblemsforwhichthesolverinquestionfails. solver.wendnoneedtoeliminateanytestproblemsfromdiscussion.forthisreason, showsthattheprobabilitythatthesetwosolverscansolveajobwithinafactor4ofthe MINOS,butitsperformancebecomesmuchmorecompetitiveifweextendourofinterest bestsolverisonlyabout70%.snopthasalowernumberofwinsthaneitherloqoor problemisabout:61.ifwechoosebeingwithinafactorof4ofthebestsolverasthescope ofourinterest,theneitherloqoorminoswouldsuce;buttheperformanceprole FromthisgureitisclearthatLOQOhasthemostwins(hasthehighestprobability successfully,thensnoptcapturesourattentionwithitsabilitytosolveover90%ofthis areinterestedinthesolverthatcansolve75%oftheproblemswiththegreatesteciency, thenminosstandsout.ifweholdtomorestringentprobabilitiesofcompletingasolve COPSsubset,asdisplayedbytheheightofitsperformanceprolefor>40.Thisgraph to7ḟigure4.2showstheperformanceprolesforallthesolversintheinterval[1;100].ifwe interval[15;40]. percentageoftheproblems.anotherpointofinterestisthatloqo,minos,andsnopt eachhavethebestprobabilityps()forinsomeinterval,withsimilarperformanceinthe displaysthepotentialforlargediscrepanciesintheperformanceratiosonasubstantial betweenquartilesdoesnotnecessarilyproceedlinearly;hence,wereallyloseinformation forsnoptand13:9forloqo.bylookingatfigures4.1and4.2,weseethatprogress MINOSalsosharerstquartilevaluesof1.Inotherwords,thesetwosolversarethe bestsolversonatleast25%oftheproblems.loqobestsminos'smedianvaluewith1 comparedwith2:4,butminoscomesbackwithathirdquartileratioof4:3versus12:6 valuesoftimeratios.thetopthreesolversshareaminimumratioof1,andloqoand Anobservationthatemergesfromtheseguresisthelackofconsistencyinquartile andnoobviousalternativevalueexists.asanalternativetoprovidingonlyquartilevalues, ifwedonotprovidethefulldata.also,themaximumratiowouldbemforourtesting, 6

9 1 Performance Profile on Subset of COPS P( ( t p,s / min {t p,s : 1 s n s } ) τ) 0.7 Figure4.1:Performanceproleon[0;10] LANCELOT MINOS 0.3 SNOPT LOQO τ 1 Performance Profile on Subset of COPS 0.9 P( ( t p,s / min {t p,s : 1 s n s } ) τ) LANCELOT MINOS SNOPT τ Figure4.2:Performanceproleon[0;100]LOQO

10 however,theperformanceproleyieldsmuchmoreinformationaboutasolver'sstrengths andweaknesses. of2forthescale.inotherwords,weplot fullimplicationsofourtestdataregardingthesolvers'probabilityofsuccessfullyhandling aproblem.sincewearealsointerestedinthebehaviorforclosetounity,weuseabase thesolvers.evenextendingto100,wefailtocapturethecompleteperformancedata proles.inthisway,wecanshowallactivitythattakesplacewith<mandgraspthe forlancelotandloqo.asanaloption,wedisplayalogscaleoftheperformance Wehaveseenthatatleasttwographsmaybeneededtoexaminetheperformanceof thegraphisnotasintuitive,sinceweareusingalogscale. infigure4.3.thisgraphrevealsallthefeaturesoftheprevioustwographsandthus capturestheperformanceofallthesolvers.thedisadvantageisthattheinterpretationof 7!1npsizefp2P:log2(p;s)g 1 Log 2 Scaled Performance Profile on Subset of COPS 0.9 P( log 2 ( t p,s / min {t p,s : 1 s n s } ) τ) interval[0;log2(1043)]infigure4.3toincludethelargestp;s<m.weextendtherange theothertwoguresand,inaddition,showsthateachofthesolversfailsonatleast8%of slightlytoshowtheatliningofallsolvers.thenewgurecontainsalltheinformationof Figures4.1and4.2aremappedintoanewscaletoreectalldata,requiringatleastthe Figure4.3:Performanceproleinalog2scale LANCELOT 0.2 MINOS 8 SNOPT LOQO ρ M τ

11 problemsweregenerallychosentobedicult. theproblems.thisisnotanunreasonableperformanceforthecopstestsetbecausethese 5CaseStudy:TheFullCOPS 1 Log 2 Scaled Performance Profile on Full COPS Set 0.9 P( log 2 ( t i,j / min {t i,j : 1 j n s } ) τ) theperformanceprolesinsection4,thisgureshowsthatperformanceproleseliminate Figure5.1:PerformanceproleforfullCOPSset 0.2 theundueinuenceofasmallnumberofproblemsonthebenchmarkingprocessandthe LANCELOT sensitivityoftheresultsassociatedwiththerankingofsolvers.inaddition,performance Wenowexpandouranalysisofthedatatoincludealltheproblemsinversion2.0ofthe MINOS 0.1 SNOPT prolesprovideanestimateoftheexpectedperformancedierencebetweensolvers. COPS[7]testset.WepresentinFigure5.1alog2scaledviewoftheperformanceproles forthesolversonthattestset. Figure5.1givesaclearindicationoftherelativeperformanceofeachsolver.Asin LOQO ρ testsetloqodominatesallothersolvers:theperformanceproleforloqoliesabove ThemostsignicantaspectofFigure5.1,ascomparedwithFigure4.3,isthatonthis M τ allothersforallperformanceratios.theinterpretationoftheresultsinfigure5.1is important.inparticular,theseresultsdonotimplythatloqoisfasteroneveryproblem. Theyindicateonlythat,forany1,LOQOsolvesmoreproblemswithinafactorof ofanyothersolvertime.moreover,byexaminingps(1)andps(m),wecanalsosaythat 9

12 LOQOisthefastestsolveronapproximately58%oftheproblems,andthatLOQOsolves themostproblems(about87%)tooptimality. Figure5.1withFigure4.3andnotingthattheperformanceprolesofMINOSandSNOPT SNOPTtodeteriorateonthefullCOPSset.Thisdeteriorationcanbeseenbycomparing setismuchlargerthanfortherestrictedsubsetofoptimalcontrolandparameterestimation problems.since,asnotedinsection3,minosandsnoptaredesignedforproblemswith amodestnumberofdegreesoffreedom,weshouldexpecttheperformanceofminosand factors.firstofall,ascanbeseenintable3.1,thedegreesoffreedomforthefullcopstest ThedierencebetweentheresultsinSection4andtheseresultsisduetoanumberof thatminosandsnoptuseonlyrst-orderinformation,whileloqousessecond-order information.thebenetofusingsecond-orderinformationusuallyincreasesasthenumber ofvariablesincreases,sothisisanotherfactorthatbenetsloqo. aconvenienttoolforcomparingandevaluatingtheperformanceofoptimizationsolvers,but, aresimilarbutgenerallylowerinfigure5.1. likealltools,performanceprolesmustbeusedwithcare.aperformanceprolereects Theresultsinthissectionunderscoreourobservationthatperformanceprolesprovide AnotherreasonforthedierencebetweentheresultsinSection4andtheseresultsis theperformanceonlyonthedatabeingused,andthusitisimportanttounderstandthe testsetandthesolversusedinthebenchmark. casestudy,weusedataobtainedbymittelmann[13].figure6.1showsaplotoftheperformanceproleforthetimeratiosinthedatabenchmarkoflpsolversonalinux-pmanceprolesaremostusefulincomparingseveralsolvers.becauselargeamountsofdata aregeneratedinthesesituations,trendsinperformanceareoftendiculttosee.asa 6CaseStudy:LinearProgramming ( ),whichincludesresultsforCOPLLP-1.0,PCx-1.1,SOPLEX-1.1,LPABO-5.6, Performanceprolescanbeusedtocomparetheperformanceoftwosolvers,butperfor- withoutconvergenceunderthesolver'sstoppingcriteria.onefeatureweseeinthegraphof Mittelmann'sresultsthatdoesnotappearintheCOPSgraphsisthevisualdisplayofsolvers thatneveratline.inotherwords,thesolversthatclimbothegrapharethosethatsolve allofthetestproblemssuccessfully.aswithfigure4.3,alloftheeventsinthedatatinto thosesolvesthataremarkedintheoriginaltableasstoppingclosetothenalsolution MOSEK-1.0b,BPMPD-2.11,andBPMPD thislog-scaledrepresentation.whilethisdatasetcannotbeuniversallyrepresentativeof InkeepingwithourgraphingpracticeswiththeCOPSset,wedesignateasfailures particular,thetestsetusedtogeneratefigure6.1includesonlyproblemsselectedby benchmarkingresultsbyanymeans,itdoesshowthatourreportingtechniqueisapplicable beyondourownresults. oftheperformanceoflpsolversonlyonthedatasetusedtogeneratetheseresults.in AsinthecasestudiesinSections4and5,theresultsinFigure6.1giveanindication 10

13 1 Log 2 Scaled Performance Profile for LP Solvers 0.9 BPMPD 2.14 P( log 2 ( t p,s / min {t p,s : 1 s n s } ) τ) 0.8 MOSEK 0.7 PCx 0.6 SOPLEXC LPABO 0.5 BPMPD 2.11 Mittelmannforhisbenchmark.Theadvantageoftheseresultsisthat,unlikethesolversin COPL_LP 0.4 Sections4and5,allsolversinFigure6.1havethesamerequirements Conclusions Figure6.1:Performanceproleforlinearprogrammingsolvers 0.2 Wehaveshownthatperformanceprolescombinethebestfeaturesofothertoolsforbenchmarkingandcomparingoptimizationsolvers.Clearly,theuseofperformanceprolesisnot 0.1 restrictedtooptimizationsolversandcanbeusedtocomparesolversinotherareas. 0 ThePerlscriptperf.plontheCOPSsite[7]generatesperformanceprolesformatted ρ M τ adjustedwithinmatlabtoreectparticularbenchmarkinginterests. toshowthefullareaofactivity.theareadisplayedandscaleofthegraphcanthenbe asmatlabcommandstoproduceacompositegraphasinfigures4.1and4.2.thescript acceptsdataforseveralsolversandplotstheperformanceproleonanintervalcalculated 11

14 Acknowledgments Version1.0oftheCOPSproblemswasdevelopedbyAlexanderBondarenko,DavidBortz, LizDolan,andMichaelMerritt.Theircontributionswereessentialbecause,inmanycases, References thewayforcopswithhisbenchmarkingwork. ouslysharedtheiramplexpertisewithus.finally,wethankhansmittelmannforpaving version2.0oftheproblemsarecloselyrelatedtotheoriginalversion. andspiriteddiscussionsonproblemformulation,whilebobfoureranddavidgaygener- AlexBondarenko,NickGould,SvenLeyerandBobVanderbeicontributedinteresting [1]H.Y.Benson,D.F.Shanno,andR.J.Vanderbei,Interior-pointmethodsfor [3]A.S.Bondarenko,D.M.Bortz,andJ.J.More,COPS:Large-scalenonlinearlyconstrainedoptimizationproblems,TechnicalMemorandumANL/MCS-TM-237, ArgonneNationalLaboratory,Argonne,Illinois,1998(RevisedOctober1999). large-scalemixedcomplementarityproblems,comp.optim.appl.,7(1997),pp.3{25. nicalreportorfe-00-02,princetonuniversity,princeton,newjersey,2000. [2]S.C.Billups,S.P.Dirkse,andM.C.Ferris,Acomparisonofalgorithmsfor nonconvexnonlinearprogramming:jammingandcomparativenumericaltesting,tech- [5]A.R.Conn,N.I.M.Gould,andP.L.Toint,LANCELOT,no.17inSpringer [6],NumericalexperimentswiththeLANCELOTpackage(ReleaseA)forlarge-scale [4]I.Bongartz,A.R.Conn,N.I.M.Gould,M.A.Saunders,andP.L.Toint, AnumericalcomparisonbetweentheLANCELOTandMINOSpackagesforlarge-scale numericaloptimization,report97/13,namuruniversity,1997. [8]E.D.DolanandJ.J.More,BenchmarkingoptimizationsoftwarewithCOPS, [7]COPS.Seehttp:// SeriesinComputationalMathematics,Springer-Verlag,1992. TechnicalMemorandumANL/MCS-TM-246,ArgonneNationalLaboratory,Argonne, Illinois,2000. nonlinearoptimization,math.programming,73(1996),pp.73{110. [10]P.E.Gill,W.Murray,andM.A.Saunders,SNOPT:Analgorithmforlargescaleconstrainedoptimization,ReportNA97-2,UniversityofCalifornia,SanDiego, MathematicalProgramming,TheScienticPress,1993. [9]R.Fourer,D.M.Gay,andB.W.Kernighan,AMPL:AModelingLanguagefor

15 [11]N.J.Higham,AccuracyandStabilityofNumericalAlgorithms,SIAM,Philadelphia, [12]H.Mittelmann,BenchmarkinginteriorpointLP/QPsolvers,Opt.Meth.Software, Pennsylvania,1996. [14]B.A.MurtaghandM.A.Saunders,MINOS5.4user'sguide,ReportSOL83-20R, [13],Benchmarksforoptimizationsoftware,2000.Seehttp://plato.la.asu.edu/ bench.html. StanfordUniversity, (1999),pp.655{670. [17]R.J.VanderbeiandD.F.Shanno,Aninterior-pointalgorithmfornonconvex [16]R.J.Vanderbei,LOQOuser'smanual{Version3.10,Opt.Meth.Software,12 [15]S.G.NashandJ.Nocedal,AnumericalstudyofthelimitedmemoryBFGSmethod (1999),pp.485{514. andthetruncatednewtonmethodforlargescaleoptimization,siamj.optim.,1 (1991),pp.358{372. nonlinearprogramming,comp.optim.appl.,13(1999),pp.231{