Transcription

1 DynamicLoadBalancing ExploitingProcessLifetimeDistributionsfor MorHarchol-Balter and AllenB.Downey tionwhetherpreemptivemigration(migratingactiveprocesses)isnecessary,orwhetherremote WeconsiderpoliciesforCPUloadbalancinginnetworksofworkstations.Weaddresstheques- UniversityofCalifornia,Berkeley execution(migratingprocessesonlyatthetimeofbirth)issucientforloadbalancing.weshow thatresolvingthisisssueisstronglytiedtounderstandingtheprocesslifetimedistribution.our measurementsindicatethatthedistributionoflifetimesforunixprocessispareto(heavy-tailed), withaconsistentfunctionalformoveravarietyofworkloads.weshowhowtoapplythisdistributiontoderiveapreemptivemigrationpolicythatrequiresnohand-tunedparameters.weusea trace-drivensimulationtoshowthatourpreemptivemigrationstrategyisfarmoreeectivethan remoteexecution,evenwhenthememorytransfercostishigh. CategoriesandSubjectDescriptors:unknown[unknown]:unknown unknown bution workloadmodeling,trace-drivensimulation,networkofworkstations,heavy-tailed,paretodistri- GeneralTerms:unknown AdditionalKeyWordsandPhrases:Loadbalancing,loadsharing,migration,remoteexecution, 1.INTRODUCTION Mostsystemsthatperformloadbalancinguseremoteexecution(i.e.non-preemptive migration)basedonaprioriknowledgeofprocessbehavior,oftenintheformofa listofprocessnameseligibleformigration.althoughsomesystemsarecapableof NSFgrantnumberCCR AllenDowneypartiallysupportedbyNSF(DARA)grant MorHarchol-BaltersupportedbyNationalPhysicalScienceConsortium(NPSC)Fellowshipand suchaspreservingautonomy.apreviousanalyticstudybyeageretal.discourages migratingactiveprocesses,mostdosoonlyforreasonsotherthanloadbalancing, DMW AnearlierversionofthispaperappearedintheProceedingsoftheACMSigmetricsConference Permissiontomakedigitalorhardcopiesofpartorallofthisworkforpersonalorclassroomuseis onmeasurementandmodelingofcomputersystems(may23-26,1996)pp grantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotordirectcommercial advantageandthatcopiesshowthisnoticeontherstpageorinitialscreenofadisplayalong withthefullcitation.copyrightsforcomponentsofthisworkownedbyothersthanacmmust servers,toredistributetolists,ortouseanycomponentofthisworkinotherworks,requiresprior behonored.abstractingwithcreditispermitted.tocopyotherwise,torepublish,toposton specicpermissionand/orafee.permissionsmayberequestedfrompublicationsdept,acm

2 2implementingpreemptivemigrationforloadbalancing,showingthattheadditional performancebenetofpreemptivemigrationissmallcomparedwiththebenetof M.Harchol-BalterandA.B.Downey simplenon-preemptivemigrationschemes[eageretal.1988].butsimulationstudies,whichcanusemorerealisticworkloaddescriptions,andimplementedsystems haveshowngreaterbenetsforpreemptivemigration[kruegerandlivny1988] 1.1Loadbalancingtaxonomy andatrace-drivensimulationtoinvestigatetheseconictingresults. Onanetworkofsharedprocessors,CPUloadbalancingistheideaofmigratingprocessesacrossthenetworkfromhostswithhighloadstohostswithlowerloads.The andimprovetheutilizationoftheprocessors.analyticmodelsandsimulationstud- motivationforloadbalancingistoreducetheaveragecompletiontimeofprocesses ieshavedemonstratedtheperformancebenetsofloadbalancing,andtheseresults havebeenconrmedinexistingdistributedsystems(seesection1.4). determineswhenmigrationsoccurandwhichprocessesaremigrated.thisisthe questionweaddressinthispaper.theotherhalfofaloadbalancingstrategyis thelocationpolicy theselectionanewhostforthemigratedprocess.previous workhassuggestedthatsimplychoosingthetargethostwiththeshortestcpurun Animportantpartoftheloadbalancingstrategyisthemigrationpolicy,which [Baraketal.1993].Thispaperusesameasureddistributionofprocesslifetimes execution,alsocallednon-preemptivemigration,inwhichsomenewprocessesare relativeunimportanceoflocationpolicy. queueisbothsimpleandeective[zhou1987][kunz1991].ourworkconrmsthe (possiblyautomatically)executedonremotehosts,andpreemptivemigration,in whichrunningprocessesmaybesuspended,movedtoaremotehost,andrestarted. Innon-preemptivemigrationonlynewbornprocessesaremigrated. Processmigrationforpurposesofloadbalancingcomesintwoforms:remote Implicitmigrationpoliciesmayormaynotuseaprioriinformationaboutthe functionofprocesses,howlongtheywillrun,etc. Loadbalancingmaybedoneexplicitly(bytheuser)orimplicitly(bythesystem). mationaboutjoblifetimes.thisinformationisoftenimplementedasaneligibility lifetimeofprocesses,implicitnon-preemptivepoliciesrequiresomeaprioriinfor- listthatspeciesbyprocessnamewhichprocessesareworthmigrating[svensson 1990][Zhouetal.1993]. Sincethecostofremoteexecutionisusuallysignicantrelativetotheaverage sincethisisoftendiculttomaintainandpreemptivestrategiescanperformwell withoutit.thesesystemsuseonlysystem-visibledatalikethecurrentageofeach processoritsmemorysize. Incontrast,mostpreemptivemigrationpoliciesdonotuseaprioriinformation, balancingstrategiesthatassumenoaprioriinformationaboutprocesses. (1)Ispreemptivemigrationworthwhile,giventheadditionalcost(CPUandlatency)associatedwithmigratinganactiveprocess? Thispaperexaminestheperformancebenetsofpreemptive,implicitload Weanswerthefollowingtwoquestions: (2)Whichactiveprocesses,ifany,areworthmigrating?

3 1.2Processmodel Inourmodel,processesusetworesources:CPUandmemory(wedonotconsider ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing 3 scheduling;insection7wediscusstheeectofotherlocalschedulingpolicies.since completion).weassumethatprocessorsimplementtime-sharingwithround-robin I/O).Thus,weuse\age"tomeanCPUage(theCPUtimeaprocesshasused imposedonaprocessis processesmaybedelayedwhileontherunqueueorwhilemigrating,theslowdown thusfar)and\lifetime"tomeancpulifetime(thetotalcputimefromstartto wherewalltimeisthetotaltimeaprocessspendsrunning,waitinginqueue,or migrating. Slowdownofprocessp=wall-time(p) 1.3Outline CPU-time(p) Theeectivenessofloadbalancing eitherbyremoteexecutionorpreemptive balancingstrategies. trace-drivensimulationtoevaluatetheimpactofthisworkloadonproposedload observationsabouttheworkloadonanetworkofunixworkstations,andusesa migration dependsstronglyonthenatureoftheworkload,includingthedistributionofprocesslifetimesandthearrivalprocess.thispaperpresentsempiricabutionispredictablewithgoodnessoftgreaterthan99%andconsistentacross avarietyofmachinesandworkloads.asaruleofthumb,theprobabilitythata workloadsinanacademicenvironment,includinginstructionalmachines,research machines,andmachinesusedforsystemadministration.wendthatthedistri- Section2presentsastudyofthedistributionofprocesslifetimesforavarietyof processwithcpuageofonesecondusesmorethantsecondsoftotalcputime is1=t(seefigure1). 1986],butthispriorworkhasbeenincorporatedinfewsubsequentanalyticand simulatorstudiesofloadbalancing.thisomissionisunfortunate,sincetheresults ofthesearesensitivetothelifetimemodel(seesection2.2). OurmeasurementsareconsistentwiththoseofLelandandOtt[LelandandOtt loadbalancing: Theysuggestthatitispreferabletomigrateolderprocessesbecausetheseprocesseshaveahigherprobabilityoflivinglongenough(eventuallyusingenough CPU)toamortizetheirmigrationcost. eligibilityofaprocessformigrationasafunctionofitscurrentage,migration Ourobservationsoflifetimedistributionshavethefollowingconsequencesfor Afunctionalmodelofthedistributionprovidesananalytictoolforderivingthe withoutmigration.accordingtothiscriterion,aprocessiseligibleformigration slowdownimposedonamigrantprocessislowerinexpectationthanitwouldbe cost,andtheloadsatitssourceandtargethost. onlyifits InSection3wederiveamigrationeligibilitycriterionthatguaranteesthatthe CPUage>1 n?mmigrationcost

4 4wheren(respectivelym)isthenumberofprocessesatthesource(target)host. InSection5weuseatrace-drivensimulationtocompareourpreemptivemigrationpolicywithanon-preemptivepolicybasedonname-lists.Thesimulatoruses starttimesanddurationsfromtracesofarealsystem,andmigrationcostschosen fromameasureddistribution. M.Harchol-BalterandA.B.Downey evenwithsurprisinglylargemigrationcosts,despiteseveralconservativeassumptionsthatgivenon-preemptivemigrationanunfairadvantage. migrationstrategiesinmoredetail.wendthatpreemptivemigrationreduces migration.wealsoproposeseveralalternativemetricsintendedtomeasureusers' themeandelay(queueingandmigration)by35{50%,comparedtonon-preemptive Nextwechooseaspecicmodelofpreemptiveandnon-preemptivemigration migrationcostontherelativeperformanceofthetwostrategies.notsurprisingly, Nevertheless,preemptivemigrationperformsbetterthannon-preemptivemigration wendthatasthecostofpreemptivemigrationincreases,itbecomeslesseective. Weusethesimulatortorunthreeexperiments:rstweevaluatetheeectof costsbasedonrealsystems(seesection4),andusethismodeltocomparethetwo perceptionofsystemperformance.bythesemetrics,theadditionalbenetsof preemptivemigrationcomparedtonon-preemptivemigrationappearevenmore moreeectivethanevenawell-tunednon-preemptivemigrationpolicy.insection5.5weusethesimulatortocompareourpreemptivemigrationstrategywith previouslyproposedpreemptivestrategies. InSection5.4wediscussindetailwhyasimplepreemptivemigrationpolicyis signicant. 1.4Relatedwork insection7andconclusionsinsection8. WenishwithacriticismofourmodelinSection6,adiscussionoffuturework jobs,fewhaveimplementedimplicitloadbalancingpolicies.mostsystemsonly allowforexplicitloadbalancing.thatis,thereisnoloadbalancingpolicy;theuser decideswhichprocessestomigrate,andwhen.examplesincludeaccent[zayas 1987],Locus[Thiel1991],Utopia[Zhouetal.1993],DEMOS/MP[Powelland 1.4.1Systems.Althoughseveralsystemshavethemechanismtomigrateactive preemptivepolicies(activeprocessesareonlymigratedforpurposesotherthanload [DePaoliandGoscinski1995],andMIST[Casasetal.1995]. Miller1983],V[Theimeretal.1985],NEST[AgrawalandEzzet1987],RHODOS [Tanenbaumetal.1990],Charlotte[ArtsyandFinkel1989],Sprite[Douglisand balancing,suchaspreservingworkstationautonomy).examplesincludeamoeba Ousterhout1991],Condor[Litzkowetal.1988],andMach[Milojicic1993].In Afewsystemshaveimplicitloadbalancingpolicies,howevertheyarestrictlynon- aboutprocesses;e.g.,explicitknowledgeabouttheruntimesofprocessesoruserprovidedlistsofmigratableprocesses[agrawalandezzet1987][litzkowandlivny 1990][DouglisandOusterhout1991][Zhouetal.1993]. general,non-preemptiveloadbalancingpoliciesdependonaprioriinformation thattheirschemeiseectiveandrobust. loadbalancingismosix[baraketal.1993].ourresultssupportthemosixclaim Oneexistingsystemthathasimplementedimplementedautomatedpreemptive

5 ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing Studies.Althoughfewsystemsincorporatemigrationpolicies,therehave beenmanysimulationandanalyticalstudiesofvariousmigrationpolicies.most ofthesestudieshavefocusedonloadbalancingbyremoteexecution[livnyand Melman1982][WangandMorris1985][CasavantandKuhl1987][Zhou1987][Pulidasetal.1988][Kunz1991][BonomiandKumar1990][EvansandButt1993][Lin andraghavendra1993][mirchandaneyetal.1990][zhangetal.1995][zhouand Ferrari1987][HacandJin1990][Eageretal.1986]. Onlyafewstudiesaddresspreemptivemigrationpolicies[LelandandOtt1986] [KruegerandLivny1988].TheLelandandOttmigrationpolicyisalsoagebased, butdoesn'ttakemigrationcostintoaccount. Eageret.al.,[Eageretal.1988],concludethattheadditionalperformancebenet ofpreemptivemigrationistoosmallcomparedwiththebenetofnon-preemptive migrationtomakepreemptivemigrationworthwhile.thisresulthasbeenwidely cited,andinseveralcasesusedtojustifythedecisionnottoimplementpreemptive migration,asintheutopiasystem,[zhouetal.1993].ourworkdiersfrom [Eageretal.1988]inbothsystemmodelandworkloaddescription.[Eageretal. 1988]modelaserverfarminwhichincomingjobshavenoanityforaparticular processor,andthusthecostofinitialplacement(remoteexecution)isfree.thisis dierentfromourmodel,anetworkofworkstations,inwhichincomingjobsarrive ataparticularhostandthecostofmovingthemaway,evenbyremoteexecution, issignicantcomparedtomostprocesslifetimes.also,[eageretal.1988]usea degeneratehyperexponentialdistributionoflifetimesthatincludesfewjobswith non-zerolifetimes.whenthecoecientofvariationofthisdistributionmatches thedistributionsweobserved,fewerthan4%ofthesimulatedprocesseshavenonzerolifetimes.withsofewjobs(andbalancedinitialplacement)thereisseldom anyloadimbalanceinthesystem,andthuslittlebenetforpreemptivemigration. Furthermore,the[Eageretal.1988]processlifetimedistributionisexponentialfor jobswithnon-zerolifetimes,theconsequencesofwhichwediscussinsection2.2. Foramoredetailedexplanationofthisdistributionanditseectonthestudy,see [DowneyandHarchol-Balter1995]. KruegerandLivnyinvestigatethebenetsofsupplementingnon-preemptivemigrationwithpreemptivemigrationandndthatpreemptivemigrationisworthwhile.Theyuseahyperexponentiallifetimedistributionthatapproximatesclosely thedistributionweobserved;asaresult,theirndingsarelargelyinaccordwith ours.onedierencebetweentheirworkandoursisthattheyusedasynthetic workloadwithpoissonarrivals.theworkloadweobserved,andusedinourtracedrivensimulations,exhibitsserialcorrelation;i.e.itismoreburstythanapoisson process.anotherdierenceisthattheirmigrationpolicyrequiresseveralhandtunedparameters.insection3.1weshowhowtousethedistributionoflifetimes toeliminatetheseparameters. Likeus,BryantandFinkeldiscussthedistributionofprocesslifetimesandits eectonpreemptivemigrationpolicy,buttheirhypotheticaldistributionsarenot basedonsystemmeasurements[bryantandfinkel1981].alsolikeus,theychoose migrantprocessesonthebasisofexpectedslowdownonthesourceandtargethosts, buttheirestimationofthoseslowdownsisverydierentfromours.inparticular, theyusethedistributionofprocesslifetimestopredictahost'sfutureloadasa functionofitscurrentloadandtheagesoftheprocessesrunningthere.wehave

6 6examinedthisissueandfound(1)thatthismodelfailstopredictfutureloads becauseitignoresfuturearrivals,and(2)thatcurrentloadisthebestpredictorof M.Harchol-BalterandA.B.Downey futureload(seesection3.1).thus,inourestimatesofslowdown,weassumethat thefutureloadonahostisequaltothecurrentload. 2.DISTRIBUTIONOFLIFETIMES Thegeneralshapeofthedistributionofprocesslifetimesinanacademicenvironmenthasbeenknownforalongtime[Rosin1965]:therearemanyshortjobsand afewlongjobs,andthevarianceofthedistributionisgreaterthanthatofan theircurrentage[cabrera1986].thatsameyear,lelandandottproposeda functionalformfortheprocesslifetimedistribution,basedonmeasurementsofthe lifetimesof9.5millionunixprocessesbetween1984and1985[lelandandott 1986].TheyconcludethatprocesslifetimeshaveaUBNE(used-better-than-new- In1986,CabrerameasuredUNIXprocessesandfoundthatover40%doubled exponentialdistribution. aprocess,thegreateritsexpectedremainingcpulifetime.specically,theynd in-expectation)typeofdistribution.thatis,thegreaterthecurrentcpuageof \longprocesseshaveexponentialservicetimes."manysubsequentstudiesassume anexponentiallifetimedistribution. thatfort>3seconds,theprobabilityofaprocess'slifetimeexceedingtseconds isrtk,where?1:25<k<?1:05andrnormalizesthedistribution. Incontrast,Rommel[Rommel1991]claimsthathismeasurementsshowthat functionalformproposedbylelandandotttstheobserveddistributionswell,for processeswithlifetimesgreaterthan1second.thisfunctionalformisconsistent policies,weperformedanindependentstudyofthisdistribution,andfoundthatthe acrossavarietyofmachinesandworkloads,andalthoughtheparameter,k,varies Becauseoftheimportanceoftheprocesslifetimedistributionforloadbalancing from-1.3to-.8,itisgenerallynear-1.thus,asaruleofthumb, Theprobabilitythataprocesswithage1secondusesatleastTsecondsoftotal TheprobabilitythataprocesswithageTsecondsusesatleastanadditional TsecondsofCPUtimeisabout1=2.Thus,themedianremaininglifetimeofa processisequaltoitscurrentage. CPUtimeisabout1=T. tionpolicies. served.section2.2discussesothermodelsforthedistributionoflifetimes,and arguesthattheparticularshapeofthisdistributioniscriticalforevaluatingmigra- Section2.1describesourmeasurementsandthedistributionoflifetimesweob- 2.1Lifetimedistributionwhenlifetime>1s TodeterminethedistributionoflifetimesforUNIXprocesses,wemeasuredthe lifetimesofoveronemillionprocesses,generatedfromavarietyofacademicworkloads,includinginstructionalmachines,researchmachines,andmachinesusedfor systemadministration.weobtainedourdatausingtheunixcommandlastcomm, showsonlyprocesseswhoselifetimesexceedonesecond.thedotted(heavy)line whichoutputsthecputimeusedbyeachcompletedprocess. Figure1showsthedistributionoflifetimesfromoneofthemachines.Theplot

7 ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing Distribution of process lifetimes (fraction of processes with duration > T) Duration (T secs.) 1 Distribution of process lifetimes (log plot) (fraction of processes with duration > T) 1/2 1/4 1/8 1/16 onmachinepomid-semester.thedotted(thicker)lineshowsthemeasureddistribution;thesolid Fig.1.(a)Distributionoflifetimesforprocesseswithlifetimesgreaterthan1second,observed 1/32 (thinner)lineshowstheleastsquarescurvet.(b)thesamedistributiononalog-logscale.the straightlineinlog-logspaceindicatesthatthedistributioncanbemodeledbytk,wherekisthe 1/64 slopeoftheline Duration (T secs.)

8 8showsthemeasureddistribution;thesolid(thinner)lineshowstheleast-squarest tothedatausingtheproposedfunctionalformprflifetime>tg=tk. M.Harchol-BalterandA.B.Downey well.incontrast,figure2showsthatitisimpossibletondanexponentialcurve thattsthedistributionoflifetimesweobserved. Byvisualinspection,itisclearthattheproposedmodeltstheobserveddata oftheformtk,withkvaryingfrom?1:3to?0:8fordierentmachines.table1 showsthevalueoftheparameterforeachmachinewestudied,estimatedbyan iterativelyweightedleast-squarest(withnointercept,inaccordancewiththe functionalmodel).wecalculatedtheseestimateswiththeblsscommandrobust Forallthemachineswestudied,thedistributionofprocesslifetimestsacurve shownhereindicatethatthettedcurveaccountsforgreaterthan99%ofthe ofcertainty).ther2valueindicatesthegoodnessoftofthemodel thevalues thatparameter(alloftheseparametersarestatisticallysignicantatahighdegree [AbrahamsandRizzardi1988]. variationoftheobservedvalues.thus,thegoodnessoftofthesemodelsishigh Thestandarderrorassociatedwitheachestimategivesacondenceintervalfor andconditionaldistributionfunctionforprocesslifetimes.thesecondcolumn (foranexplanationofr2values,see[larsenandmarx1986]). showsthesefunctionswhenk=?1,whichwewillassumeforouranalysisin Section3. Table2showsthecumulativedistributionfunction,probabilitydensityfunction, thatitsmoments(mean,variance,etc.)areinnite.ofcourse,sincetheobserved distributions,though,becausecalculatedmomentstendtobedominatedbyafew distributionshavenitesamplesize,theyhavenitemean(0.4seconds)andcoecientofvariation(5{7).onemustbecautiouswhensummarizinglong-tailed Thefunctionalformweareproposing(thetteddistribution)hastheproperty likethemedian,ortheestimatedparameterk)tosummarizedistributions,rather outliers.inouranalysesweusemorerobustsummarystatistics(orderstatistics thanmoments. second,wedidnotndaconsistentfunctionalform;however,forthemachines westudiedtheseprocesseshadanevenlowerhazardratethanthoseofage>1 second.thatis,theprobabilitythataprocessofaget<1secondslivesanothert secondsisalwaysgreaterthan1=2.thusforjobswithlifetimeslessthan1second, 2.1.1Processwithlifetime<1second.Forprocesseswithlifetimeslessthan1 Manypriorstudiesofprocessmigrationassumeanexponentialdistributionofprocesslifetimes,bothinanalyticalpapers[LinandRaghavendra1993][Mirchandaney andmelman1982][zhangetal.1995][chowdhury1990].thereasonsforthisas- 1991][Pulidasetal.1988][WangandMorris1985][EvansandButt1993][Livny willnotaecttheresultsofloadbalancingstudies. lifetimedistributionisinfactnotexponential,assuminganexponentialdistribution sumptioninclude:(1)analytictractability,and(2)thebeliefthateveniftheactual 2.2Whythedistributioniscritical themedianremaininglifetimeisgreaterthanthecurrentage. etal.1990][eageretal.1986][ahmadetal.1991]andinsimulationstudies[kunz Regardingtherstpoint,althoughthefunctionalformthatweandLelandand

9 ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing 9 NameNumberNumberEstimated of Host po Procs>1sec of 4107 Distrib. T?0:97 Error Std R2 po2 po3 cory pors T?1:22 T?1:27 T?0: bugs T?0:94 T?0: Table1.Theestimatedlifetimedistributionforeachmachinemeasured,andtheassociated faith T?0: goodnessoftstatistics.descriptionofmachines:poisaheavily-useddecserver5000/240, usedprimarilyforundergraduatecoursework.po1,po2,andpo3refertomeasurementsmade onpomid-semester,late-semester,andend-semester.coryisaheavily-usedmachine,usedfor courseworkandresearch.porscheisalessfrequently-usedmachine,usedprimarilyforresearch onscienticcomputing.bugsisaheavily-usedmachine,usedprimarilyformultimediaresearch. Faithisaninfrequently-usedmachine,usedbothforvideoapplicationsandsystemadministration. 1 Observed distribution and two curve fits (fraction of processes with duration > T) 1/2 1/4 k T fit (one parameter) 1/8 exponential fit (two parameters) 1/16 Fig.2.Inlog-logspace,thisplotshowsthedistributionoflifetimesforthe13000processesfrom ourtraceswithlifetimesgreaterthansecond,andtwoattemptstotacurvetothisdata.one 1/32 ofthetsisbasedonthemodelproposedinthispaper,tk.theothertisanexponential curve,ce?t.althoughtheexponentialcurveisgiventhebenetofanextrafreeparameter, 1/64 iteratively-weightedleastsquares. itfailstomodeltheobserveddata.theproposedmodeltswell.bothtswereperformedby Duration (T secs.)

10 10 M.Harchol-BalterandA.B.Downey Distributionoflifetimesforprocesses>1secWhenk=?1 PrfT<L<T+dTsecg=?kTk?1dT PrfL>bsecjage=ag=?bak PrfL>Tsecg=Tk 1=T2dT distributionfunctionfortheprocesslifetimel.thesecondcolumnshowsthefunctionalformof Table2.Thecumulativedistributionfunction,probabilitydensityfunction,andconditional eachforthetypicalvaluek=1. a=b tion,itneverthelesslendsitselftosomeformsofanalysis,asweshowinsection3.1. Ottproposecannotbeusedinqueueingmodelsaseasilyasanexponentialdistribu- migrationpolicydependsonhowtheexpectedremaininglifetimeofajobvaries isimportanttomodelthisdistributionaccurately.specically,thechoiceofa distributionaectstheperformanceofmigrationpolicies,andthereforethatit withage.inourobservationswefoundadistributionwiththeubneproperty Regardingthesecondpoint,wearguethattheparticularshapeofthelifetime theexpectedremaininglifetimeofajobincreaseslinearlywithage.asaresult,we choseamigrationpolicythatmigratesonlyoldjobs. processanditsremaininglifetime.forexample,auniformdistributionhasthe NBUEproperty theexpectedremaininglifetimedecreaseslinearlywithage. choosetomigrateonlyyoungprocesses.inthiscase,weexpectnon-preemptive Thusifthedistributionoflifetimeswereuniform,themigrationpolicyshould Butdierentdistributionsyieldindierentrelationshipsbetweentheageofa migrationtoperformbetterthanpreemptivemigration. withthelowestmigrationcost,regardlessofage. lifetimeofajobisindependentofitsage.inthiscase,sinceallprocesseshavethe sameexpectedlifetimes,themigrationpolicymightchoosetomigratetheprocess Asanotherexample,theexponentialdistributionismemoryless theremaining distribution(auniformdistributioninlog-space)havearemaininglifetimethat increasesuptoapointandthenbeginstodecrease.inthiscase,thebestmigration policymightbetomigratejobsthatareoldenough,butnottooold. Asanalexample,processeswhoselifetimesarechosenfromauniformlog dierentmigrationpolicies.inordertoevaluateaproposedpolicy,itiscritical tochooseadistributionmodelwiththeappropriaterelationshipbetweenexpected remaininglifetimeandage. Thusdierentdistributions,evenwiththesamemeanandvariance,canleadto oflifetimes.thesedistributionsmayormaynothavetherightbehavior,depending onhowaccuratelytheytobserveddistributions.[kruegerandlivny1988]use athree-stagehyperexponentialwithparametersestimatedtotobservedvalues. ThisdistributionhastheappropriateUBNEproperty.Butthetwo-stagehyperexponentialdistribution[Eageretal.1988]useismemoryless;theremaininglifetime ofajobisindependentofitsage(forjobswithnonzerolifetimes).accordingtothis distribution,migrationpolicyisirrelevant;allprocessesareequallygoodcandidates Somestudieshaveusedhyperexponentialdistributionstomodelthedistribution formigration.thisresultisclearlyinconictwithourobservations.

11 preemptivemigration.theheavytailofourmeasuredlifetimedistributionimplies Assumingthewronglifetimedistributionmayalsounderestimatethebenetsof ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing 11 abilitytoidentifythosefewhogs.inalifetimedistributionwithoutsuchaheavy tail,preemptivemigrationmightnotbeaseective. Aswe'lldiscussinSection5.4,partofthepowerofpreemptivemigrationisits thatatinyfractionofthejobsrequiremorecputhanalltheotherjobscombined. 3.MIGRATIONPOLICY Amigrationpolicyisbasedontwodecisions:whentomigrateprocessesandwhich processestomigrate.therstquestionconcernshowoftenoratwhattimesthe Thefocusofthispaperisthesecondquestion,alsoknownastheselectionpolicy: systemchecksforeligiblemigrants.weaddressthisissuebrieyinsection Giventhattheloadatahostistoohigh,howdowechoosewhichprocess tive,migrationtimehasalargeimpactonresponsetime.aprocesswouldchoose lifetimes.themotivationforthisheuristicistwofold.fromtheprocess'sperspec- Ourheuristicistomigrateprocessesthatareexpectedtohavelongremaining tomigrate? tomigrateonlyifthemigrationoverheadcouldbeamortizedoveralongerlifetime. tion),becausetheseprocesseshavenoallocatedmemoryandthustheirmigration Fromtheperspectiveofthesourcehost,ittakesasignicantamountofworkto costislow.theideaofmigratingnewbornprocessesmightalsostemfromthe thatarelikelytobemoreexpensivetorunthantomigrate. packageaprocessformigration.thehostwouldonlychoosetomigrateprocesses processeshaveequalexpectedremaininglifetimesregardlessoftheirage,soone fallacythatprocesslifetimeshaveanexponentialdistribution,implyingthatall Manyexistingmigrationpoliciesonlymigratenewbornprocesses(nopreemp- shouldmigratethecheapestprocesses.theproblemwithonlymigratingnewborn processesisthat,accordingtotheprocesslifetimedistribution,newbornprocesses ourmeasurementsshowthatover70%ofprocesseshavelifetimessmallerthanthe areunlikelytolivelongenoughtojustifythecostofremoteexecution.infact, smallestnon-preemptivemigrationcost(seetable3). Wehavefound,though,thattheabilityofthesystemtopredictprocesslifetimes edgeaboutprocessesandcanselectivelymigrateprocesseslikelytobecpuhogs. bynameislimited(section5.4). Thusanewbornmigrationpolicyisonlyjustiedifthesystemhaspriorknowl- processes. processtorunlongerthanayoungprocess;thus,itispreferabletomigrateold jorityofprocessesareshort,theremightnotbeenougholdprocessestohavea Therearetwopotentialproblemswiththisapproach.First,sincethevastma- Canwedobetter?Thedistributionoflifetimesimpliesthatweexpectandold signicantloadbalancingeect.infact,althoughtherearefewlong-livedprocesses,theyaccountforalargepartofthetotalcpuload.accordingtoour measurements,typicallyfewerthan4%ofprocesseslivelongerthan2seconds,yet thelongtailoftheprocesslifetimedistribution.furthermore,wewillseethatthe abilitytomigrateevenafewlargejobscanhavealargeeectonsystemperfor- theseprocessesmakeupmorethan60%ofthetotalcpuload.thisisdueto

12 12 mance,sinceasinglelongprocessonabusyhostimposesslowdownsonmanyshort processes. M.Harchol-BalterandA.B.Downey migratingprocesseswithlongerexpectedlives. activeprocessismuchgreaterthanthecostofremoteexecution.ifpreemptive migrationisdonecarelessly,thisadditionalcostmightoverwhelmthebenetof Asecondproblemwithmigratingoldprocessesisthatthemigrationcostforan 3.1OurMigrationPolicy Forthisreason,weproposeastrategythatguaranteesthateverymigrationimprovestheexpectedperformanceofthemigrantprocessandtheotherprocesses atthesourcehost.thisstrategymigratesaprocessonlyifitimprovestheexpectedslowdownoftheprocess,whereslowdownisdenedasinsection1.2.of course,processesonthetargethostareslowedbyanarrivingmigrant,butona moderately-loadedsystemtherearealmostalwaysidlehosts;thusthenumberof processesatthetargethostisusuallyzero.inanycase,thenumberofprocesses atthetargetisalwayslessthanthenumberatthesource. migrationisdone.ifmigrationcostsarehigh,fewprocesseswillbeeligiblefor migration;intheextremetherewillbenomigrationatall.butinnocaseisthe performanceofthesystemworse(inexpectation)thantheperformancewithout migration. Ifthereisnoprocessonthehostthatsatisestheabovemigrationcriterion,no imposedonamigrantprocess,andusethisresulttoderiveaminimumagefor migrationbasedonthecostofmigration.denotingtheageofthemigrantprocess thenumberofprocessesatthesourcehostbyn;andthenumberofprocessesat bya;thecostofmigrationbyc;the(eventualtotal)lifetimeofthemigrantbyl, Usingthedistributionofprocesslifetimes,wecalculatetheexpectedslowdown thetargethost(includingthemigrant)bym,wehave: Efslowdownofmigrantg t=aprlifetimeof migrantistslowdowngiven =Z1 t=aprftl<t+dtjlagna+c+m(t?a) lifetimeistdt =12ca+m+n t=aat2na+c+m(t?a) t t formigrationonlyifitsexpectedslowdownaftermigrationislessthann(whichis theslowdownitexpectsintheabsenceofmigration). Iftherearenprocessesataheavilyloadedhost,thenaprocessshouldbeeligible Thus,werequire12(ca+m+n)<n,whichimplies Wecanextendthisanalysistothecaseofheterogeneousprocessorspeedsby Minimummigrationage=Migrationcost n?m

13 applyingascalefactortonorm. Thisanalysisassumesthatcurrentloadpredictsfutureload;thatis,thattheload ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing 13 atthesourceandtargethostswillbeconstantduringthemigration.inanattempt calculatingapredictionofsurvivorsandfuturearrivalsbasedonthedistribution toevaluatethisassumption,andpossiblyimproveit,weconsideredanumberof oftime),(2)summingtheagesoftheprocessesrunningonthehost,anda(3) predictor,andthatusingseveralpredictivevariablesincombinationdidnotgreatly alternativeloadpredictors,including(1)takingaloadaverage(overaninterval modelproposedhere.wefoundthatcurrent(instantaneous)loadisthebestsingle improvetheaccuracyofprediction.theseresultsareinaccordwithzhou[zhou 3.2PriorPreemptivePolicies Onlyafewpreemptivestrategieshavebeenimplementedinrealsystemsorproposed 1987]andKunz[Kunz1991]. inpriorstudies.thethreethatwehavefoundare,likeours,basedontheprinciple thataprocessshouldbemigratedifitisoldenough. freeparameterwhosevalueischosenwithoutexplanation,andthatwouldneedto bere-tunedforadierentsystemoranotherworkload. UnderLelandandOtt'spolicy,aprocesspiseligibleformigrationif Inmanycases,thedenitionofoldenoughdependsona\voodoo"constant1:a andlivny'spolicy,likeours,takesthejob'smigrationcostintoaccount.aprocess piseligibleformigrationif wherekisafreeparametercalledmincrit[lelandandott1986].krueger age(p)>agesofkyoungerjobsathost buttheydonotexplainhowtheychosethevalue0:1[kruegerandlivny1988]. TheMOSIXpolicyissimilar[Baraketal.1993];aprocessiseligibleformigration if age(p)>0:1migrationcost(p) ofamigrantprocessisnevermorethan2,sinceintheworstcasethemigrant completesimmediatelyuponarrivalatthetarget. Thechoiceoftheconstant(1:0)intheMOSIXpolicyensuresthattheslowdown age(p)>1:0migrationcost(p): theslowdownthatwouldbeimposedatthesourcehostintheabsenceofmigration(presumablythereismorethanoneprocessthere,orthesystemwouldnot beattemptingtomigrateprocessesaway).second,itisbasedontheworst-case Despitethisjustication,thechoiceofthemaximumslowdown(2)isarbitrary. WeexpecttheMOSIXpolicytobetoorestrictive,fortworeasons.First,itignores 1ThistermwascoinedbyProfessorJohnOusterhoutatU.C.Berkeley. onload. bestchoiceforthisparameter,forourworkload,isusuallynear0.4,butitdepends slowdownratherthantheexpectedslowdown.insection5.5,weshowthatthe

14 14 Migrationcosthassuchalargeeectontheperformanceofpreemptiveloadbalancing;thissectionpresentsthemodelofmigrationcostsweuseinoursimulation Wemodelthecostofmigratinganactiveprocessasthesumofaxedmigration 4.MODELOFMIGRATIONCOSTS M.Harchol-BalterandA.B.Downey proportionaltotheamountoftheprocess'smemorythatmustbetransferred. studies. costformigratingtheprocess'ssystemstateandamemorytransfercostthatis remotehost,logginginorotherwiseauthenticatingtheprocess,andcreatinganew Thecostofremoteexecutionincludessendingthecommandandargumentstothe shellandenvironmentontheremotehost. Wemodelremoteexecutioncostasaxedcost;itisthesameforallprocesses. r:thecostofremoteexecution,inseconds f:thexedcostofpreemptivemigration,inseconds b:thememorytransferbandwidth,inmbpersecond Throughoutthispaper,weusethefollowingnotation: andthus: m:thememorysizeofmigrantprocesses,inmb wherethequotientm=bisthememorytransfercost. costofpreemptivemigration=f+m=b costofremoteexecution=r [DouglisandOusterhout1991]haveanexcellentdiscussionofthisissue,andwe migrationdependsonpropertiesofthedistributedsystem.douglisandousterhout 4.1Memorytransfercosts borrowfromthemhere. Theamountofaprocess'smemorythatmustbetransferredduringpreemptive systemlikesprite,whichintegratesvirtualmemorywithadistributedlesystem, itisonlynecessarytowritedirtypagestothelesystembeforemigration.when thecostofmigrationisproportionaltothesizeoftheresidentsetratherthanthe theprocessisrestartedatthetargethost,itwillretrievethesepages.inthiscase Atthemost,itmightbenecessarytotransferaprocess'sentirememory.Ona sizeofmemory. ferredwhiletheprogramcontinuestorunatthesourcehost.whenthejobstops creased,thedelayimposedonthemigrantprocessisgreatlydecreased.additional dirtyduringtheprecopy.althoughthenumberofpagestransferredmightbein- executionatthesource,itwillhavetotransferagainanypagesthathavebecome Insystemsthatuseprecopying,suchasV[Theimeretal.1985],pagesaretrans- techniquescanreducethecostoftransferringmemoryevenmore[zayas1987]. 4.2Migrationcostsinrealsystems Thespecicparametersofmigrationcostdependnotonlyonthenatureofthe system(asdiscussedabove)butalsoonthespeedofthenetwork.tables3and4 showreportedcostsfromavarietyofrealsystems.laterwewilluseatrace-driven simulatortoevaluatetheeectoftheseparametersonsystemperformance.we

15 System Sprite ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing Hardware SPARCstation1 0.33sec Costofrexec,r 15 [DouglisandOusterhout1991] GLUNIX [Vahdatetal.1994][Vahdat1995]ATMnetwork MIST [Prouty1996] 10Mb/secEthernet Utopia HPworkstations HP9000/720 10Mb/secEthernet 0.25to0.5sec [Zhouetal.1993] DEC3100andSPARCIPC0.1sec 0.33sec Table3.Costofnon-preemptivemigrationinvarioussystems.Someofthesenumberswere obtainedfrompersonalcommunicationwiththeauthors. System Sprite [DouglisandOusterhout1991] MOSIX 10Mb/secEthernet Hardware SPARCstation1 Fixed 0.33sec Cost,f 2.00sec/MB Inverse [Baraketal.1993][Braverman1995]and48666MHz IntelPentium90MHz0.006sec0.44sec/MB Bandwidth,1=b Table4.Valuesforpreemptivemigrationcostsfromvarioussystems.Manyofthesenumbers wereobtainedfrompersonalcommunicationwiththeauthors.thememorytransfercostisthe MIST [Prouty1996] HP9000/720s 10Mb/secEthernet 0.24sec 0.99sec/MB willmakethepessimisticsimplicationthatamigrant'sentirememorymustbe transferred,although,aspointedoutabove,thisisnotnecessarilythecase. productoftheinversebandwidth,1=b,andtheamountofmemorythatmustbetransferred,m. gration.wecomparetwomigrationstrategies:ourproposedage-basedpreemptive Inthissectionwepresenttheresultsofatrace-drivensimulationofprocessmi- 5.TRACE-DRIVENSIMULATION migrationstrategy(section3.1)andanon-preemptivestrategythatmigratesnewbornprocessesaccordingtotheprocessname(similartostrategiesproposedby [Wangetal.1993]and[Svensson1990]).Withtheintentionofndingaconservativeestimateofthebenetofpreemptivemigration,wegivethename-based usethesimulatortorunthreeexperiments.first,insection5.2,weevaluate thesensitivityofeachstrategytothemigrationcostsr,f,b,andmdiscussed strategythebenetofseveralunrealisticadvantages;forexample,thename-lists arederivedfromthesametracedatausedbythesimulator. insection4.next,insection5.3,wechoosevaluesfortheseparametersthatare representativeofcurrentsystemsandcomparetheperformanceofthetwostrategies Section5.1describesthesimulatorandthetwostrategiesinmoredetail.We indetail.insection5.4wediscusswhythepreemptivepolicyoutperformsthe non-preemptivepolicy.lastly,insection5.5,weevaluatetheanalyticcriterion formigrationageproposedinsection3.1,comparedtocriteriausedinprevious studies.

16 16 5.1Thesimulator Wehaveimplementedatrace-drivensimulationofanetworkofsixidenticalworkstations.2 M.Harchol-BalterandA.B.Downey eachfrom9:00a.m.to5:00p.m.fromthesixtracesweextractedthestarttimes Weselectedsixdaytimeintervalsfromthetracesonmachinepo(seeSection2.1), andcpudurationsoftheprocesses.wethensimulatedanetworkwhereeachof thedaytimetraces. sixhostsexecutes(concurrentlywiththeothers)theprocessarrivalsfromoneof fractionofprocesses(0:1%)arrivetondallhostsbusy.inordertoevaluatethe activityduringtheeight-hourtrace.formostofthetraces,everyarrivingprocess mixanddistributionoflifetimes,thereisconsiderablevariationinthelevelof ndsatleastoneidlehostinthesystem,butinthetwobusiesttraces,asmall Althoughtheworkloadsonthesixhostsarehomogeneousintermsofthejob lowesttohighestload.run0hasatotalof15000processessubmittedtothesix intervals.werefertotheseasruns0through7,wheretherunsaresortedfrom simulatedhosts;run7has30000processes.theaveragedurationofprocesses(for eectofchangesinsystemload,wedividedtheeight-hourtraceintoeightone-hour allruns)is0.4seconds.thusthetotalutilizationofthesystem,,isbetween0.27 and0.54. givenrunandagivenhost,theserialcorrelationininterarrivaltimesistypically between.08and.24,whichissignicantlyhigherthanonewouldexpectfroma Poissonprocess(uncorrelatedinterarrivaltimesyieldaserialcorrelationof0.0; perfectcorrelationis1.0). ThebirthprocessofjobsatourhostsisburstierthanaPoissonprocess.Fora randomlyfromameasureddistribution(seesection5.2).thissimplicationobliteratesanycorrelationsbetweenmemorysizeandotherprocesscharacteristics,but thememorysizeofeachprocess,whichdeterminesitsmigrationcost,ischosen itallowsustocontrolthemeanmemorysizeasaparameterandexamineitseect Althoughthestarttimesanddurationsoftheprocessescomefromtracedata, onsystemperformance. areneverblockedoni/o.duringagiventimeinterval,wedividecputimeequally amongtheprocessesonthehost(processorsharing). Inoursystemmodel,weassumethatprocessesarealwaysreadytorun;i.e.they migrationtothesourcehost.thissimplicationisconservativeinthesensethat thetransferredpages,partintransitinthenetwork,andpartonthetargethost unpackingthedata.thesizeofthesepartsandwhethertheycanbeoverlapped dependondetailsofthesystem.inoursimulationwechargetheentirecostof Inrealsystems,partofthemigrationtimeisspentonthesourcehostpackaging itmakespreemptivemigrationlesseective. thepoliciesassimpleandassimilaraspossible.forbothtypesofmigration,we considerperformingamigrationonlywhenanewprocessisborn,eventhougha tion3.1withanon-preemptivemigrationstrategy,wherethenon-preemptivestrat- egyisgivenunfairadvantages.forpurposesofcomparison,wehavetriedtomake 5.1.1Strategies.WecomparethepreemptivemigrationstrategyproposedinSec- 2The trace-driven simulator and the trace data are available at

17 preemptivestrategymightbenetbyinitiatingmigrationsatothertimes.also, forbothstrategies,ahostisconsideredheavily-loadedanytimeitcontainsmore ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing 17 gration.finally,weusethesamelocationpolicyinbothcases:thehostwiththe lowestinstantaneousloadischosenasthetargethost(tiesarebrokenbyrandom thanoneprocess;inotherwords,anytimeitwouldbesensibletoconsidermi- selection). areconsideredeligibleformigration: Thustheonlydierencebetweenthetwomigrationpoliciesiswhichprocesses isbornataheavily-loadedhost,theprocessisexecutedremotelyontheselected itsnameisonalistofprocessesthattendtobelong-lived.ifaneligibleprocess host.processescannotbemigratedoncetheyhavebegunexecution. Name-basednon-preemptivemigration.Aprocessiseligibleformigrationonlyif choseathresholdonmeandurationthatisempiricallyoptimal(forthissetofruns). Addingmorenamestothelistdetractsfromtheperformanceofthesystem,asit durationandselectingthe15commonnameswiththelongestmeandurations.we Wederivedthislistbysortingtheprocessesfromthetracesaccordingtonameand Theperformanceofthisstrategydependsonthelistofeligibleprocessnames. allowsmoreshort-livedprocessestobemigrated.removingnamesfromthelist detractsfromperformanceasitbecomesimpossibletomigrateenoughprocesses agedforsomefractionofitsmigrationcost.basedonthederivationinsection3.1, list,ourresultsmayoverestimatetheperformancebenetsofthisstrategy. thisfractionis1 tobalancetheloadeectively.sinceweusedthetracedataitselftoconstructthe Age-basedpreemptivemigration.Aprocessiseligibleformigrationonlyifithas causeitdoesnotallowthesystemtoinitiatemigrationsexceptwhenanewprocess processesthatsatisfythemigrationcriterionaremigratedaway. source(target)host.whenanewprocessisbornataheavily-loadedhost,all Thisstrategyunderstatestheperformancebenetsofpreemptivemigration,be- n?m,wheren(respectivelym)isthenumberofprocessesatthe arrives. morecomplicatedpredictorsoffutureloads,butnoneofthesepredictorsyielded signicantlybetterperformancethantheinstantaneousloadweusehere. AsdescribedinSection3.1,wealsomodeledotherlocationpoliciesbasedon theperformanceofthemigrantwillimproveinexpectation).oneofthestrategies wheneveraprocessbecomeseligible(sincetheeligibilitycriterionguaranteesthat thanwhenanewprocessarrives.ideally,onewouldliketoinitiateamigration weconsideredperformsperiodicchecksofeachprocessonaheavily-loadedhost Wealsoconsideredtheeectofallowingpreemptivemigrationattimesother followingperformancemetrics: toseeifanysatisfythecriterion.theperformanceofthisstrategyissignicantly betterthanthatofthesimplerpolicy(migratingonlyatprocessarrivaltimes) Metrics.Weevaluatetheeectivenessofeachstrategyaccordingtothe metricofsystemperformance.whenwecomputetheratioofmeanslowdowns(as fromdierentstrategies)wewillusenormalizedslowdown,whichistheratioof (thus,itisalwaysgreaterthanone).theaverageslowdownofalljobsisacommon Meanslowdown.Slowdownistheratioofwall-clockexecutiontimetoCPUtime

18 18 M.Harchol-BalterandA.B.Downey Fraction of procs 0.5 Distribution of slowdowns smallslowdowns,buttheprocessesinthetailofthedistributionaremorenoticeableandannoying Fig.3.Distributionofprocessslowdownsforrun0(withnomigration).Mostprocessessuer CPUtime.Forexample,ifthe(unnormalized)meanslowdowndropsfrom2:0to tousers. inactivetime(theexcessslowdowncausedbyqueueingandmigrationdelays)to :5,theratioofnormalizedmeanslowdownsis0:5=1:0=0:5:a50%reductionin Slowdown ofthetwostrategies;itunderstatestheadvantagesofthepreemptivestrategyfor thesetworeasons: delay. Skeweddistributionofslowdowns:Evenintheabsenceofmigration,themajority Meanslowdownaloneisnotasucientmeasureofthedierenceinperformance Userperception:Fromtheuser'spointofview,theimportantprocessesarethe Thevalueofthemeanslowdownisdominatedbythismajority. onesinthetailofthedistribution,becausealthoughtheyaretheminority,they ofprocessessuersmallslowdowns(typically80%arelessthan3.0.seefigure3). haveasmalleectonthemeanslowdown,butalargeeectonauser'sperception causethemostnoticeableandannoyingdelays.eliminatingthesedelaysmight ofperformance. Therefore,wewillalsoconsiderthefollowingthreemetrics: tailofthedistribution;i.e.thenumberofjobsthatexperiencelongdelays.(see maybemoremeaningfultointerpretthismetricasameasureofthelengthofthe tryingtoscheduletasks.inlightofthedistributionofslowdowns,however,it dictabilityofresponsetime[silberschatzetal.1994],whichisanuisanceforusers Varianceofslowdown.Thismetricisoftencitedasameasureoftheunpre- severelyimpactedbyqueueingandmigrationpenalties.(seefigures5cand5d). abledelaysexplicitly,weconsiderthenumber(orpercentage)ofprocessesthatare Figure5b). Numberofseverelyslowedprocesses.Inordertoquantifythenumberofnotice-

19 than:5seconds)aremoreperceivabletousersthandelaysinshortjobs.(see Meanslowdownoflongjobs.Delaysinlongerjobs(thosewithlifetimesgreater ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing 19 Figure6). 5.2Sensitivitytomigrationcosts Inthissectionwecomparetheperformanceofthenon-preemptiveandpreemptive strategiesoverarangeofvaluesofr,f,bandm(themigrationcostparameters process)andb(thebandwidthofthenetwork).wechosethememorytransfercost Weconsideredarangeforthexedmigrationcostof:1<f<10seconds. denedinsection4). fromadistributionwiththesameshapeasthedistributionofprocesslifetimes, Thememorytransfercostisthequotientofm(thememorysizeofthemigrant Forthefollowingexperiments,wechosetheremoteexecutioncostr=:3seconds. featureofthisdistributionisthattherearemanyjobswithsmallmemorydemands seconds.theshapeofthisdistributionisbasedonaninformalstudyofmemory-use patternsonthesamemachinesfromwhichwecollectedtracedata.theimportant settingthemeanmemorytransfercost(mmtc)toarangeofvaluesfrom1to64 distributiondoesnotaecttheperformanceofeithermigrationstrategystrongly, butofcoursethemean(mmtc)doeshaveastrongeect. andafewjobswithverylargememorydemands.empirically,theexactformofthis migrationstrategiesusingnormalizedslowdown.specically,foreachoftheeight one-hourrunswecalculatethemean(respectivelystandarddeviation)oftheslowdownimposedonallprocessesthatcompleteduringthehour.foreachrun,we Figures4aand4barecontourplotsoftheratiooftheperformanceofthetwo thentaketheratioofthemeans(standarddeviations)ofthetwostrategies.lastly wetakethegeometricmeanoftheeightratios(fordiscussionofthegeometric mean,see[hennessyandpatterson1990]). tivemigration,namelythexedcost,f,andthemmtc,m=b.thecostofnon- preemptivemigration,r,isxedat0:3seconds.asexpected,increasingeitherthe xedcostofmigrationorthemmtchurtstheperformanceofpreemptivemigration.thecontourlinemarked1:0indicatesthecrossoverwheretheperformance ThetwoaxesinFigure4representthetwocomponentsofthecostofpreemp- non-preemptivemigration.whenthexedcostofmigrationorthemmtcare ofpreemptiveandnon-preemptivemigrationisequal(theratiois1:0).forsmaller valuesofthecostparameters,preemptivemigrationperformsbetter;forexample, ifthexedmigrationcostis0.3secondsandthemmtcis2seconds,thenormalizedmeanslowdownwithpreemptivemigrationisalmost40%lowerthanwith unaectedbythesecostssothenon-preemptivestrategycanbemoreeective. veryhigh,almostallprocessesareineligibleforpreemptivemigration;thus,the preemptivestrategydoesalmostnomigrations.thenon-preemptivestrategyis aregionwherepreemptivemigrationyieldsahighermeanslowdownthannonpreemptivemigration,butalowerstandarddeviation.thereasonforthisisthadowns.thecrossoverpoint wherenon-preemptivemigrationsurpassespreemp- Figure4bshowstheeectofmigrationcostsonthestandarddeviationofslowtivemigration isconsiderablyhigherherethaninfigure4a.thusthereis turnsouttobeshort-lived.theseprocessessuerlargedelays(relativetotheir non-preemptivemigrationoccasionallychoosesaprocessforremoteexecutionthat

20 20 M.Harchol-BalterandA.B.Downey (a) 10 Ratio of mean slowdowns Fixed migration cost (sec.) X NON PREEMPTIVE BETTER HERE 1.0 PREEMPTIVE BETTER HERE (b) Mean memory transfer cost (sec.) 10 Ratio of std of slowdowns 1.0 Fixed migration cost (sec.) Fig.4.(a)Theperformanceofpreemptivemigrationrelativetonon-preemptivemigrationdeterioratesasthecostofpreemptivemigrationincreases.Thetwoaxesarethetwocomponentsof PREEMPTIVE BETTER EXCEPT UPPER RIGHT X ofslowdownmaygiveabetterindicationofauser'sperceptionofsystemperformancethanmean theparticularsetofparameterswewillconsiderinthenextsection.(b)thestandarddeviation thepreemptivemigrationcost.thecostofnon-preemptivemigrationisheldxed.thexmarks slowdown.bythismetric,thebenetofpreemptivemigrationisevenmoresignicant Mean memory transfer (sec.)

21 runtimes)andaddtothetailofthedistributionofslowdowns.inthenextsection, weshowcasesinwhichthestandarddeviationofslowdownsisactuallyworsewith ExploitingProcessLifetimeDistributionsforDynamicLoadBalancing 21 non-preemptivemigrationthanwithnomigrationatall(threeoftheeightruns). tems(seesection4.2)andusethemtoexaminemorecloselytheperformanceof thetwomigrationstrategies.thevalueswechoseare: Inthissectionwechoosemigrationcostparametersrepresentativeofcurrentsys- 5.3Comparisonofpreemptiveandnon-preemptivestrategies r:thecostofremoteexecution,0.3seconds f:thexedcostofpreemptivemigration,0.3seconds m:themeanmemorysizeofmigrantprocesses,1mb b:thememorytransferbandwidth,0.5mbpersecond point(comparedtothecaseofnomigration). withanx.figure5showstheperformanceofthetwomigrationstrategiesatthis InFigures4aand4b,thepointcorrespondingtotheseparametervaluesismarked bylessthan20%formostruns(and40%forthetworunswiththehighestloads). Preemptivemigrationreducesthenormalizedmeanslowdownby50%formost runs(andmorethan60%fortwooftheruns).theperformanceimprovementof preemptivemigrationovernon-preemptivemigrationistypicallybetween35%and Non-preemptivemigrationreducesthenormalizedmeanslowdown(Figure5a) 50%. metricstotrytoquantifythesebenets.figure5bshowsthestandarddeviation and5dexplicitlymeasurethenumberofseverelyimpactedprocesses,accordingto statestheperformancebenetsofpreemptivemigration.wehaveproposedother ofslowdowns,whichreectsthenumberofseverelyimpactedprocesses.figures5c Asdiscussedabove,wefeelthatthemeanslowdown(normalizedornot)under- twodierentthresholdsofacceptableslowdown.bythesemetrics,thebenetsof non-preemptivemigrationappearsmuchgreater.forexampleinfigure5d,inthe migrationingeneralappeargreater,andthediscrepancybetweenpreemptiveand absenceofmigration,7{18%ofprocessesareslowedbyafactorof5ormore. Non-preemptivemigrationisabletoeliminate42{62%ofthese,whichisasignicantbenet,butpreemptivemigrationconsistentlyeliminatesnearlyall(86{97%) severedelays. nomigrationatall.forthepreemptivemigrationstrategy,thisoutcomeisnearly allprocessesinvolved(inexpectation).intheworstcase,then,thepreemptive migrationactuallymakestheperformanceofthesystemworsethaniftherewere impossible,sincemigrationsareonlyperformediftheyimprovetheslowdownsof AnimportantobservationfromFigure5bisthatforseveralruns,non-preemptive formanceasloadincreases(asshowninfigure5).inthepresenceofpreemptive strategywilldonoworsethanthecaseofnomigration(inexpectation). regardlessoftheoverallloadonthesystem. migration,boththemeanandstandarddeviationofslowdownarenearlyconstant, Anotherbenetofpreemptivemigrationisgracefuldegradationofsystemper-

22 22 M.Harchol-BalterandA.B.Downey 4.0 Mean slowdown no migration non-preemptive, name-based migration preemptive, age-based migration run number Standard deviation of slowdown no migration non-preemptive, name-based migration preemptive, age-based migration % run number Processes slowed by a factor of 3 or more no migration non-preemptive, name-based migration preemptive, age-based migration 20% 10% 0% 20% run number Processes slowed by a factor of 5 or more no migration non-preemptive, name-based migration preemptive, age-based migration 15% Fig.5.(a)Meanslowdown.(b)Standarddeviationofslowdown.(c)Percentageofprocesses slowedbyafactorof3ormore.(d)percentageofprocessesslowedbyafactorof5ormore. 10% 5% 0% run number