4MRJ,Inc.,NASAContractNAS MoettField,CA ,USA

Transcription

1 DrorG.Feitelson1,LarryRudolph1,UweSchwiegelshohn2,KennethC. inparalleljobscheduling TheoryandPractice TheHebrewUniversity,91904Jerusalem,Israel 1InstituteofComputerScience Sevcik3,andParksonWong4 UniversityofToronto,Toronto,Ontario,CanadaM5S3G4 UniversityDortmund,44221Dortmund,Germany 3DepartmentofComputerScience 2ComputerEngineeringInstitute divergenceoftheoryandpractice.wereviewtheoreticalresearchin Abstract.Theschedulingofjobsonparallelsupercomputerisbecomingthesubjectofmuchresearch.However,thereisconcernaboutthe 4MRJ,Inc.,NASAContractNAS MoettField,CA ,USA thisarea,andrecommendationsbasedonrecentresults.thisiscontrastedwithaproposalforstandardinterfacesamongthecomponents andasurveyofalargenumberofproposedandimplementedapproaches[19]. muchresearchactivity.see,forexampletheproceedingsofthreeworkshops[40], 1Introduction Theschedulingofjobsonparallelsupercomputersisbecomingthesubjectof Ithasbecomeadistinctresearchtopicfromthelargelyunrelated,butbetter ofaschedulingsystem,thathasgrownfromrequirementsintheeld. andindeedmanyrecentsystemsprovidehardwaresupportfordierentlevelsof However,thisdoesnotprecludeasharedaddressspacemodelofcomputation, junctionwiththeprocessors,ratherthanbeingtreatedasadistinctresource. allelprocessors(mpps),whichcurrentlydominatethesupercomputingarena. Intermsofscheduling,onsuchmachinesmemoryistypicallyallocatedincon- jobsaremostlyindependent. havexedandspecieddependencies,whereasjobschedulingassumesthatthe knownproblemofdagscheduling[1].dagschedulingassumesthatalltasks memorysharing. supportlargediverseworkloadsofparalleljobsonmulticomputersthathave Thereareagrowingnumberofhighperformancecomputingfacilitiesthat Thispaperisaboutschedulingjobsondistributedmemorymassivelypar-

2 tenstothousandsofprocessors.thetypicalwaythattheyarecurrentlyusedis that: 1.Thesystemisdividedinto\partitions"consistingofdierentnumbersof 2.A(large)numberofqueuesareestablished,eachonecorrespondingtoa partitionsforparalleljobsatnightoroverweekends,attheexpenseofthe workthroughtime-slicingofitsprocessors.anothermaybedevotedtoser- processors.mostprocessorsareallocatedtopartitionsdevotedtoserving paralleljobs.onepartitionistypicallysetasideforsupportofinteractive vicetasks,suchasrunningaparallellesystem.thecongurationofpar- titionsmaybechangedonaregularbasis(forexample,byprovidinglarger 3.Eachpartitionisassociatedwithoneormorequeues,anditsprocessors speciccombinationofjobcharacteristics.(forexample,onequeuemight interactivepartition). correspondtojobsthatrequireasmanyas32processors,andareexpected torunnolongerthan15minutes.)somequeuesareservedathigherpriority associatedqueuesaresearchedinorderofpriorityforonethatisnon-empty. thanothers,sotheusertendstosubmitajobtothehighestpriorityqueue forwhichthejobqualiesbasedonitsexpectedresourcerequirements. serveasapoolforthosequeues.wheneversomeprocessorsarefree,the Therstjobinthatnon-emptyqueueisthenactivatedinthepartition,and itrunsuntilitcompletes,providedthenumberoffreeprocessorsissucient. studies,however,showthatmoreexibilityinboththescheduleractionsandthe Thus: systems,suchasnon-fcfsserviceandsupportforswapping,thegeneraltrend istoretainthesameframework,andmoreover,tocastitintoastandard.many {thenumberofprocessorsassignedtoajobisxedbytheuser; {onceinitiatedthejobrunstocompletion. Whilethereexistsomeinnovationsthathavebeenintroducedintoproduction Withineachqueuejobsareprocessedstrictlyinrst-come-rst-servedorder. wayprogramsmakeuseofparallelismresultinbetterperformance.butthere practicalworkatlargeinstallationsisreviewedinsection4.finally,wediscuss mendationsthataremadeinsection3.thestandardizationeortbasedon thepartitionsizesinaccordancewiththecurrentsystemload.notably,much ofthisworkisbasedonworkloadmodelsthatarederivedfrommeasurements atsupercomputerinstallations. eectivenessofpreemptioninachievinglowaverageresponsetimes,andalso ishopeforconvergence[25].forexample,theoreticalanalysisunderscoresthe showsthatconsiderablebenetsarepossibleiftheschedulerisallowedtotailor 2SurveyofTheoreticalResults thisstateofaairsandpresentourconclusionsinsection5. Variouskindsofschedulingorsequencingproblemshavebeenaddressedsincethe ftiesbytheoreticalresearchersfromtheareasofcomputerscience,operations WesurveythetheoreticalbackgroundinSection2,andthespecicrecom-

3 witheachother. useinpractice,butalsothediversityofdivergencemakesthemhardtocompare ofthesetheoreticalresutlsrelyonacreativesetofassumptions,inordertomake theirproofstractable.thisdivergencefromrealitynotonlymakethemhardto ofproblems.thisisespeciallytrueforjobschedulingonparallelsystemswith alargenumberofprocessorsornodes.henceadirectuseofmanyofthese theoreticalresultsinrealapplicationswouldseemtobenatural.however,many research,anddiscretemathematics.thechallengeofecientjobmanagement oncomputershasfrequentlybeennamedasakeyreasontoaddressthiskind 2.1TheDiversityofDivergence Thissectioncoversmanyoftheassumptionsoftheoreticalwork,bypresenting aroughclassicationofdierenttheoreticalmodels.thisincludesdierentcost metrics,dierentfeaturesandoperationsavailableonthemodeledsystem,and dierentalgorithmicapproaches. CostmetricsForthediscussionofthevariouscostmetricsweusethefollowing notations: pletedworkonthisjob.notethatnoinformationisprovidedonwhetherthejob Thecompletiontimetiisthetimewhenthecomputersystemhasnallycom- wi=weightofjobi di=deadlineofjobi si=releasetimeofjobi ti=completiontimeofjobi hasbeensuccessfullycompletedorwhetherithasbeenremovedfromthesystem ofthenewjob.however,sometimesitisassumedthattheschedulingsystemis canstartworkingonjobi.usually,thereleasetimeofajobisidenticalwith forotherreasons.thereleasetimesiistheearliesttimethecomputersystem itssubmitalorarrivaltime,i.e.thersttimewhenthesystembecomesaware Theweightwiofajobisawaytoprioritizeonejoboveranother.Thedeadline awareofalljobsattime0,butjobicannotbestartedbeforesometimesi0. diisthetimebywhichajobmustcompleteitsexecution.thereisnogenerally sourcestocompetingjobs.ajobshouldsomehowbechargedforitsresource comsumption.oftenthecostofascheduleissimplythesumoftheindividual schedules.assumingajobsystemthefollowingmetricsarecommonlyused: accepteddenitionastowhathappensifthedeadlineisnotmetforaspecic job,i.e.ti>di. jobcosts.thiscostfunctionservesasbasistocompareandevaluatedierent Obviously,theroleofthescheduleristheallocationoflimittedsystemre-

4 Xi2wi(ti si) Xi2wimaxf0;ti dig=weightedtardiness Xi2witi jfi2jti>digj=deadlinemisses max i2ti =Weightedow(response)time =Weightedcompletiontime =Makespan(throughput) onasingleprocessorcanbeminimizedifthetasksarescheduledbyincreasing weightthisalgorithmbecomesthewellknownshortest-jobrstmethod. Notethatresponsetimeandowtimeusuallyhavethesamemeaning.Theorigin in1956thatthesumoftheweightedcompletiontimesforasystemofjobs executiontimetoweightratio,thesocalledsmithratio.ifalljobshaveunit ofthesecriteriaoftengoesbacktothefties.forinstancesmith[82]showed LeonardiandRaz[47]. owtime.thisequalitydoesnothold,however,whentheydeviatebyeven aconstantfactorfromtheoptimumasshownbykellereretal.[42]andby schedulewithoptimalweightedcompletiontimealsohastheoptimalweighted onereasonfortheirpopularity,buttherearesomesubtledierenceinthem.a 1.Satisfytheusers. Inreality,themetricsattempttoformalizetherealgoalsofascheduler: Thesemetricsallowarelativelysimpleevaluationofalgorithmswhichmaybe Forinstance,areductionofthejobresponsetimewillmostlikelyimproveuser time.iftheusersubmitsthejobinthemorning(9am)hemayexpecttoreceive satisfaction. Example1.Assumethatajobineedsapproximately3hoursofcomputation theresultsafterlunch.itprobablydoesnotmattertohimwhetherthejobis startedimmediatelyordelayedforanhouraslongasitisdoneby1pm.any 2.Maximizetheprot. reduced. increasecosts.thiscorrespondstotardinessscheduling.however,ifthejobis bewillingtopostponeexecutionofhisjobuntilnighttimewhenthechargeis notcompletedbefore5pmitmaybesucientiftheusergetshisresultsearly delaybeyond1pmmaycauseannoyanceandthusreduceusersatisfaction,i.e. expected.also,iftheuserischargedfortheuseofsystemresources,hemay isinformedatthetimeofsubmitalthatexecutionofthejobby5pmcannotbe nextmorning.moreover,hemaybeabletodealwiththesituationeasilyifhe policyforacommercialsystemmayrequirethatdierentmetricsareusedduring installationsmaybeduetothesimplicityoftheevaluation,oritmaybeasignof somenon-obviousinuencefromtheory.ontheotherhand,agoodmanagement Theuseofmetricssuchasthroughputandresponsetimeinmanycommercial

5 completionoftheirsubmittedjobs.thusaresponsetimemetricisappropriate. However,duringthenightitisbesttomaximizethethroughputofjobs. dierenttimesoftheday:duringdaytimemanyuserswillactuallywaitforthe Time 8am 6am 4am 2am 12am User activity moderate low very low Fig.1.Workloadofaparallelcomputeroverthecourseofaday. 10pm low 8pm moderate 6pm 4pm high 2pm moderate high activitysmalljobsaregivenpreferenceevenifsomeprocessorsremainidledue 12pm high 10am theshortestresponsetimeisachieved.ontheotherhandduringperiodsoflow useractivitylargebatchjobsarestarted.alsomoldablejobsareruninaway tofragmentationoftheprocessorspace.jobsareallocatedresourcessuchthat iseasiertovisualizejobs.asshowninthegure,duringperiodsofhighuser necessarilyrequirealinearone-dimensionalprocessorspace.butthiswayit ofsimplicityeachjobisdescribedasarectangle.blackrectanglesdenoteidle processorsduetofragmentation.however,notethatmultiprocessorsdonot Fig.1showstheloadofamultiprocessoroverthecourseofaday.Forreasons 8am moderate toincreaseeciency,i.e.usingfewerprocessorswhilerequiringlongerexecution time. Processor Space bicriteriaschedulingwhereschedulingmethodsareintroducedwhichgenerate Recentstudies,e.g.Charkrabartietal.[7],explicitlyaddresstheproblemof

6 thesemodelsdirectlyaectoperationsofthescheduler.theyareatleastpartly inspiredbythewayrealsystemsaremanagedandhowparallelapplicationsare written.thefollowingroughlyclassiesthesemodelsaccordingtovecriteria: metric. TheModelAlargevarietyofdierentmachineandschedulingmodelshave beenusedinstudiesofschedulingproblems.theconstraintsincorporatedinto goodscheduleswithrespecttothemakespanandtheweightedcompletiontime 1.PartitionSpecicationEachparalleljobisexecutedinapartitionthatconsists multiprocessor,theapplication,andtheloadofmultiprocessor[25].moreover, thesizeofthepartitionofaspecicjobmaychangeduringthelifetimeofthis ofanumberofprocessors.thesizeofsuchapartitionmaydependonthe jobinsomemodels: {Fixed.Thepartitionsizeisdenedbythesystemadministratorandcanbe {Variable.Thepartitionsizeisdeterminedatsubmissiontimeofthejob modiedonlybyreboot. architecturessuchashypercubes,trees,ormeshes.manyotherauthorsusethe {Adaptive.Thepartitionsizeisdeterminedbythescheduleratthetimethe basedonuserrequest. ofprocessorsbutcanbescheduledonanysubsetofprocessorsofthesystem. variablepartitioningparadigm,inwhicheachjobrequiresaspecicnumber {Dynamic.Thepartitionsizemaychangeduringtheexecutionofajob,to Feldmannetal.[26]haveconsideredxedpartitionsgeneratedbydierent reectchangingrequirementsandsystemload. jobisinitiated,basedonthesystemload,andtakingtheuserrequestinto Anexampleofatheoreticalstudybasedontheadaptiveapproachisthework account. processors,butcanusedierentnumbers.however,onceapartitionforajobhas beenselecteditssizecannotchangeanymore.finally,indynamicpartitioning oftureketal.[88].here,theapplicationdoesnotrequireaspecicnumberof thesizeofapartitionmaychangeatruntime.thismodelhas,forinstance, beenusedbydengetal.[11]. 2.JobFlexibilityAsalreadymentionedadvancedpartitioningmethodsmust notonlybesupportedbythemultiprocessorsystembutbytheapplicationas well.therefore,feitelsonandrudolph[25]characterizeapplicationsasfollows: {Rigidjobs.Thenumberofprocessorsassignedtoajobisspeciedexternal {Moldablejobs.Thenumberofprocessorsassignedtoajobisdetermined bythesystemschedulerwithincertainconstraintswhenthejobisrstactivated,anditusesthatmanyprocessorsthroughoutitsexecution. tothescheduler,andpreciselythatnumberofprocessorsaremadeavailable tothejobthroughoutitsexecution.

7 {Evolvingjobs.Thejobgoesthroughdierentphasesthatrequiredierent {Malleablejobs.Thenumberofprocessorsassignedtoajobmaychange numbersofprocessors,sothenumberofprocessorsallocatedmaychange duringthejob'sexecutioninresponsetothejobrequestingmoreprocessors orrelinquishingsome.eachjobisallocatedatleastthenumberofprocessors avoidtheoverheadsofreallocationateachphase). itrequiresateachpointinitsexecution(butitmaybeallocatedmoreto intentionofpromotingittooneofthelatertypes,thebetteragoodscheduler canperformwithrespecttoboththeparticularjob,andthewholeworkload. typesisused1.themoreeortdevotedtowritingtheapplicationwiththe Generally,aschedulercanstartajobsoonerifthejobismoldableoreven malleablethanifitisrigid[87]. Themannerinwhichanapplicationiswrittendetermineswhichofthefour duringthejob'sexecution,asaresultofthesystemgivingitadditional processorsorrequiringthatthejobreleasesome. applicationcodetoindicatewhereparallelismchanges[96]. phases.forsuchjobs,systemcallsareinsertedattheappropriatepointsinthe (e.g.,anypoweroftwofromfourto256).evolvingjobsarisewhenapplications gothroughdistinctphases,andtheirnaturalparallelismisdierentindierent loadincreases[25].itisgenerallynotdiculttowriteanapplicationsothatit ismoldable,andisabletoexecutewithprocessorallocationsoversomerange withthecurrentsystemload,whichdelaystheonsetofsaturationassystem Ifjobsaremoldable,thenprocessorallocationscanbeselectedinaccordance meansthatsomeprocessorsceasepickingupworkandaredeallocated[67].in applicationiswritten.onewayofattainingalimitedformofmalleabilityisby creatingasmanythreadsinajobasthelargestnumberofprocessorsthatwould everbeused,andthenusingmultiplexing(orfolding[51,38])tohavethejob tionallowsmoreprocessorstotakeworkfromthequeue,whileareduction executeonalessernumberofprocessors.alternatively,ajobcanbemademalleablebyinsertingapplicationspeciccodeatparticularsynchronizationpoints mostcases,however,itismorediculttosupportmalleabilityinthewayan canbemodiedtobemalleablerelativelyeasily.anincreasedprocessoralloca- Certainparallelapplications,suchasthosebasedonthe\workcrew"model2, processorallocationsarechangedonlyattimeswhencheckpointsaretaken. latterapproachissomewhatmoreeective,butitrequiresmoreeortfromthe torepartitionthedatainresponsetoanychangeinprocessorallocation.the forcheckpointingparalleljobs[66].hence,acombinedbenetcanbederivedif applicationwriter,aswellassignicantlymoresystemsupport.muchoftherequiredmechanismsforsupportingmalleablejobschedulingispresentinfacilities 2Inthe\workcrew"model,processorspickuprelativelysmallandindependentunits 1Sometheoreticalstudiesusedierentterminology.Forexample,LudwigandTiwari [50]speakabout\malleable"jobswhichareequivalenttomoldablejobsinour terminology. ofcomputationfromacentralqueue.

8 isprobablynotworthwhiletowriteapplicationssothattheyaremalleable. 3.LevelofPreemptionSupportedAnotherissueistheextenttowhichindividual theeortrequiredtomakethemmalleablemaybewelljustied.otherwise,it threadsorentirejobscanbepreemptedandpotentiallyrelocatedduringthe executionofajob: {NoPreemption.Onceajobisinitiateditrunstocompletionwhileholding Forstableapplicationsthatarewidelyandfrequentlyusedbymanyusers, {GangScheduling.Allactivethreadsofajobaresuspendedandresumed {MigratablePreemption.Threadsofajobmaybesuspendedononeprocessorandsubsequentlyresumedonanother. canonlyberesumedlateronthesameprocessor.thiskindofpreemption doesnotrequireanydatamovementbetweenprocessors. allitsassignedprocessorsthroughoutitsexecution. {LocalPreemption.Threadsofajobmaybepreempted,buteachthread simultaneously.gangschedulingmaybeimplementedwithorwithoutmigration. [71]usesagangschedulingmodelwithoutmigration.TheworkofDengetal. thejobmaybeeitherexplicitlywrittenoutofmemory,ormaybeimplicitly removedovertimebyaprocesssuchaspagereplacement.whetherornotthe stateofeachthreadmustbepreserved.thememorycontentsassociatedwith [11]isbaseduponmigratablepreemption. stoppedinaconsistentstate(i.e.,withoutanymessagesbeinglost),andthefull tion,morerecentlypreemptionhasalsobeentakenintoaccount.schwiegelshohn Inarealsystemthepreemptionofajobrequiresthatallthejob'sthreadsbe Whilemanytheoreticalschedulingstudiesonlyuseamodelwithoutpreemp- partonthememoryrequirementsofthepreemptingjobandthetotalamount dataofajobisremovedfrommemorywhenthejobispreempteddependsin fromoneprocessortoanother,whilepreservingallexistingcommunicationpaths tootherthreads.also,whenathreadismigrated,itsassociateddatamust ofmemoryavailable. ononlyasmallfractionofalljobs. oneprocessortoanother.insharedmemorysystems,thedatacanbetransferred acachelineorpageatatimeasitisreferenced.preemptionmayhavegreat follow.inmessagepassingsystems,thisrequiresthatthedatabecopiedfrom benetinleadingtoimprovedperformance,evenifitisusedinfrequentlyand Inadditionmigratablepreemptionneedsthemechanismofmovingathread includesatimepenaltyforeachpreemption. addresstheoverheadbyminimizingthenumberofpreemptions.inorderto 4.AmountofJobandWorkloadKnowledgeAvailableSystemsdierinthetype, compareapreemptiveschedulewithnon-preemptiveonesschwiegelshohn[71] quantity,andaccuracyofinformationavailabletoandusedbythescheduler. Preemptioninrealmachineshasanoverheadcost,e.g.Motwanietal.[55]

9 Characteristicsofindividualjobsthatareusefulinschedulinginclude(i)the totalprocessingrequirement,and(ii)thespeedupcharacteristicsofthejob. ample,workloadmeasurementsatanumberofhigh-performancecomputing pointinitsexecution). ofprocessorsbusywhenthejobisallocatedampleprocessors),andmaximum parallelism(thelargestnumberofprocessorsthatajobcanmakeuseofatany isprovidedbycharacteristicssuchasaverageparallelism(theaveragenumber Fullknowledgeofthelatterrequiresknowingtheexecutiontimeofthejobfor facilitieshaveindicatedthatthevariabilityinprocessingrequirementsamong eachnumberofprocessorsonwhichitmightbeexecuted.partialknowledge jobsisextreme,withmostjobshavingexecutiontimesofafewseconds,buta smallnumberhavingexecutiontimesofmanyhours.thecoecientofvariation3,orcv,oftheservicetimesofjobshasbeenobservedtobeinthefour preventshortjobsfrombeingdelayedbylongjobsismandatory. Workloadinformationisalsousefulinchoosingaschedulingpolicy.Forex- toseventyrangeatseveralcenters[8,64,21].thisimpliesthatamechanismto levels: {None.Nopriorknowledgeisavailableorusedinscheduling,soalljobsare {Workload.Knowledgeoftheoveralldistributionofservicetimesinthe Theknowledgeavailabletotheschedulermaybeatoneofthefollowing workloadisavailable,butnospecicknowledgeaboutindividualjobs.again, treatedthesameuponsubmission. {Job.Theexecutiontimeofthejobonanygivennumberofprocessorsis {Class.Eachsubmittedjobisassociatedwitha\class",andsomekeycharacteristicsofjobsintheclassareknown,including,forexample,estimates jobsaretreatedthesame,butpolicyattributesandparameterscanbetuned ofprocessingrequirement,maximumparallelism,averageparallelism,and possiblymoredetailedspeedupcharacteristics. Jobknowledge,whichisdenedtobeexact,isunrealisticinpractice.However,assumingomniscienceinmodelingstudiesmakesitpossibletoobtainan optimisticboundonperformancethatisnotattainableinpractice.presuming jobknowledgeinmodelingstudiessetsastandardinperformanceagainstwhich isticsarenotknownuntilthereleasetimeandtheexecutiontimerequirements ofthejobarealsonotavailablebeforethejobhasbeenexecutedtocompletion. Theyshowthatanyalgorithmwithalljobsavailableattime0canbeconverted 3Thecoecientofvariationofadistributionisitsstandarddeviationdividedbyits mean. knownexactly. tottheworkload. canbecompared. instanceshmoysetal.[78]discussedmakespanschedulingifthejobcharacter- practicallyrealizableschedulingalgorithms,whichuseclassknowledgeatmost, On-lineschedulinghasbeenaddressedmorefrequentlyinrecentyears.For

10 (preciselyorapproximately).forexample,inmanycurrentsystemsjobsare smallestdemandtypedisciplines(e.g.,\leastworkfirst"(lwf))canbeused toyieldlowaverageresponsetimesiftheresourcedemandofeachjobisknown toanalgorithmthathandlesdynamicarrivalswithacompetitivefactoratmost twicelarger. thesesystems,thisinformationcanbeusedimplicitlythroughthewayqueues time,andotherparameters.thus,eachqueuecorrespondstoajobclass.in suchinformationasthenumberofprocessorsneeded,alimitontheexecution submittedtooneofalargenumberofqueues,andthequeueselectedindicates ajobsubmissiontoestimatetheresourcerequirementsofthejob.however, Manysystemsmakenoeortatalltouseinformationthataccompanies consistentlyprovideinformationwithhighaccuracy,andalsobecausetheuser maybemotivatedtodeceivetheschedulerintentionally.thus,sourcesfrom whichtogainknowledgeaboutjobresourcerequirementsmustbebroadenedto include: mustbeusedcarefullybothbecauseitisprohibitivelydicultfortheuserto queues. areassignedto(xed)partitions,andtherelativeprioritiesassignedtothe {consideruserprovidedinformation(whilerecognizingthatitishistorically Anyinformationprovidedbytheuserrelatingtojobresourcerequirements tentiallybeusedtodetermineitsclass.thesecharacteristicsincludetheuser {measureeciencyduringexecutionandincreaseprocessorallocationsonly {keeptrackofexecutiontimeandspeedupknowledgefrompastexecutions Allidentifyingcharacteristicsassociatedwiththesubmitalofajobcanpo- onaclassbyclassbasis,andusethatinformation. estimates); forjobsthatareusingtheircurrentlyallocatedprocessorseectively; quiteunreliable,inpartbecauseusersaren'tcarefulaboutmakinggood id,theletobeexecuted,thememorysizespecied,andpossiblyothers.an Asmalldatabasecanbekepttorecordresourceconsumptionofjobsonaclass byclassbasis.thisisveryusefulparticularlyforlargejobsthatareexecuted repeatedly. 5.MemoryAllocationForhighperformanceapplications,memoryisusuallythe estimateoftheeciency[59]ortheexecutiontime[31]ofajobbeingscheduled criticalresource. canbeobtainedfromretainedstatisticsontheactualresourceusageofjobsfrom thesame(orasimilar)classthathavebeenpreviouslysubmittedandexecuted. twotypesofmemorytoconsider: memoryisrelativelydecoupledfromtheallocationofprocessors.thusthereare {DistributedMemory.Typicallyeachprocessoranditsassociatedmemory Thisisparticularlytrueinshared-memorysystems,wheretheallocationof memory. isallocatedasaunit.messagepassingisusedtoaccessdatainremote

11 Mostly,memoryrequirementshavebeenignoredinrealschedulingsystemsand arenotevenpartofthemodelintheoreticalstudies,althoughthisischanging [65,73,62,61]. {SharedMemory.Accesscosttosharedmemorycaneitherbeuniform (UMA)ornonuniform(NUMA)foralltheprocessors.WithUMA,thereis AlgorithmicMethodsTheintractabilityformanyschedulingproblemshas thepotentialformoreequitableallocationofthememoryresources.with beenwellstudied[28].examplesinthespecicareaofparalleljobscheduling anditsdatatomemories. NUMA,theperformanceissensitivetotheallocationofajobtoprocessors [70],whileothersareparticularsimplealgorithms,likelistschedulingmethods[41,95].Thelatterpromisestobeofthegreatesthelpfortheselectiononomialalgorithmsguaranteeingasmalldeviationfromtheoptimalschedule appearmoreattractive.somepolynomialalgorithmsarestillverycomplex, schedulingmethodsinrealsystems. useweightedcompletiontimeasoptimizationcriterion. includepreemptiveandnon-preemptivegangschedulingbyduandleung[13] usingmakespan,andbruno,coman,andsethi[6]andmcnaughton[53]who complexityandproduceschedulesthatareclosetotheoptimum,theyareusually notthemethodofchoiceincommercialinstallations. Althoughmanyoftheapproximationalgorithmshavealowcomputational Withtheintractabilityofmanyschedulingproblemsbeingestablished,poly- highcostsareonlyencounteredforafewjobsystemswhichmayneverbepart ofrealworkloads. problemsinceaschedulethatapproachestheworstcaseisoftenunacceptable listschedulesofnon-preemptiveparalleljobschedulesis2.inotherwords,up foraproductionschedule.forinstance,themakespanapproximationfactorfor to50%ofthenodesofamultiprocessorsystemmaybeleftidle.however,these Theworstcaseapproximationfactorisusuallyoflittlerelevancetoapractical generatesnon-preemptiveo-lineschedulesandrequirestheknowledgeofthe ofthesmartscheduleasinputforalistschedule[33].butnotethatsmart by2.thisresultwasfurtherimprovedtothefactor1.4byusingthejoborder executiontimeofalljobs.theconsiderationofmorecomplexconstraintsmay of8[72]andgiveaworstcaseexamplewithadeviationof4.5.however,applying SanDiegoSupercomputingCentergaveanaveragedeviationfromtheoptimum completiontimeschedulingofparalleljobs.theyproveanapproximationfactor thealgorithmonjobsystemsobtainedfromthetracesoftheintelparagonatthe Tureketal.[89]proposed\SMART"schedulesfortheo-linenon-preemptive againsttheoptimalscheduleoragainstschedulesgeneratedbyothermethods. makeanygeneralapproximationalgorithmimpossible[42,47]. SleatorandTarjan[80]introducedthenotionofcompetitiveanalysis.Anon-line algorithmiscompetitiveifitisguaranteedtoproducearesultthatiswithin aconstantfactoroftheoptimalresult.onlythedeviationfromtheoptimal Theevaluationofanyschedulercanbeeitherdonebycomparingitsschedule

12 schedulecandeterminewhetherthereisenoughroomforimprovementtomotivatefurtheralgorithmicresearch.unfortunately,theoptimalschedulecannot beobtainedeasily,butananalysisofanapproximationalgorithmcanuselower boundsfortheoptimalscheduletodeterminethecompetitivefactor,e.g.the toparalleljobs.aslongasnoparalleljobrequiresmorethan50%oftheprocessors,thiswillonlyincreasetheapproximationfactorfrom1.21to2[87]. ForinstanceKawaguchiandKyan'sLRFschedule[41]canbeeasilyextended Moreover,thetheoreticalanalysismaybeabletopinpointtheconditions tospecicallyhandlethosesituations. practicalapproachandhelptodeterminecriticalworkloads.iftheevaluationof realtracesrevealsthatsuchcriticalworkloadsrarelyorevenneveroccurthen theycaneitherbeignoredortheapproachcanbeenhancedwithaprocedure whichmayleadtoabadschedule.thesemethodscanalsobeappliedtoany squashedareaboundintroducedbytureketal.[89]. characteristicsofactualworkloadsthatcanbeexploitedinscheduling. approximationfactorcanbeguaranteed. ductionhigh-performancecomputingfacilitieshavebeencarriedout.theyreveal 2.2SomeSpecicStudies WorkloadCharacterizationSeveralworkloadcharacterizationstudiesofpro- However,ifjobsrequiringmoreprocessorsareallowedinaddition,noconstant patternsasthecorrespondingearlierones.hotovy[37]studiedaquitedierent system,yetfoundmanyofthesameobservationstohold.gibbons[30]also occurredfrequently,andlaterrunstendedtohavesimilarresourceconsumption parallelapplicationsareexecutedonanetworkofworkstations,concludingthat inallthreesystemsclassifyingthejobsbyuser,executionscript,andrequested degreeofparallelismledtoclassesofjobsinwhichexecutiontimevariability analyzedworkloaddatafromthecornellsiteinadditiontotwositeswhere ismuchlowerthanintheoverallworkload.thecommonconclusionisthat FeitelsonandNitzberg[21]notedthatrepeatedrunsofthesameapplication muchinformationaboutajob'sresourcerequirementscanbeuncoveredwithout demandingtheuser'scooperation. avarietyofapplications. tobeusedbyotherresearchers,leadingtoeasierandmoremeaningfulcomparisonofresults.nguyenatal.[58]havemeasuredthespeedupcharacteristicsof computing,althoughheobservedthatuserstypicallyrequestmoreprocessors environment.hefoundthatmemoryisasignicantresourceinhigh-performance thannaturallycorrespondtotheirmemoryrequirements. Jannetal.[39]haveproducedaworkloadmodelbasedonmeasurementsof Feitelson[17]studiedthememoryrequirementsofparalleljobsinaCM-5 theworkloadonthecornelltheorycentersp2machine.thismodelisintended BatchJobSchedulingInaneorttoimprovethewaycurrentschedulers orderinginordertoachievebetterpacking.lifkaetal.[49,79]havedeveloped behave,severalgroupshavemodiednqsimplementationstoallowqueuere-

13 becausetheycouldbypasstheverybigones. isfactionwasgreatlyincreasedsincesmallerjobstendedtogetthroughfaster, ofprocessorsareavailableforthem.however,thiscanmeanthatsmallerjobs sowithoutdelayingthepreviouslyscheduledjobs4.itwasfoundthatusersat- aschedulerontopofloadlevelerwiththefeaturethatthestrictfcfsorderofactivatingjobsisrelaxed.inthisscheduler,knownas\easy",jobsare scheduledinafcfsordertorunattheearliesttimethatasucientnumber maybeexecutedbeforebiggerjobsthatarrivedearlier,whenevertheycando variationofa2-dbuddysystemtodoprocessorallocationfortheintelparagon. Thread-orientedschedulingNelson,Towsley,andTantawi[57]comparefour inwhichperformancegainsareachievedbymovingawayfromstrictfcfs scheduling.wanetal.[92]alsoimplementanon-fcfsschedulerthatusesa casesinwhichparalleljobsarescheduledineitheracentralizedorde-centralized fashion,andthethreadsofajobareeitherspreadacrossallprocessorsorall Henderson[35]describesthePortableBatchSystem(PBS),anothersystem executedononeprocessor.theyfoundthatbestperformanceresultedfrom allsystemloadincreasesinordertoavoidsystemsaturation(seesevcik[77]). ciencyofparalleljobsgenerallydecreasesastheirprocessorallocationincreases, itisnecessarytodecreaseprocessorallocationstomoldablejobsastheover- otheroptions,decentralizedschedulingofsplittasksbeatcentralizedscheduling withnosplittingunderlightload,butthereverseistrueunderheavyload. DynamicallyChangingAJob'sProcessorAllocationBecausethee- centralizedschedulingandspreadingthethreadsacrossprocessors.amongthe accordingtotheirdynamicneedsledtomuchbetterperformancethaneither ZahorjanandMcCann[97]foundthatallocatingprocessorstoevolvingjobs rameterstheychose,round-robinbeatrun-to-completiononlyatquitelowsys- temloads. run-to-completionwitharigidallocationorround-robin.fortheoverheadpa- processorworkingset(pws),whichisthenumberofallocatedprocessorsfor whichtheratioofexecutiontimetoeciencyisminimized.(thepwsdiers fromtheaverageparallelismofthejobbyatmostafactoroftwo[16].)the bestofthevariantsofpwsgivesjobsatmosttheirprocessorworkingset,but underheavyloadgivesfewerandfewerprocessorstoeachjob,thusincreasing eciencyandthereforesystemcapacity. Ghosaletal.[29]proposeseveralprocessorallocationschemesbasedonthe titioningofthesystembeatsstaticpartitioningatmoderateandheavyloads. processorallocations.inalaterstudy[75],theygoontoshowthatdynamicpar- Naik,SetiaandSquillante[56]showthatdynamicpartitioningallowsmuch tigatehowparallelprocessingoverheadscauseeciencytodecreasewithlarger betterperformancethanxedpartitioning,butthatmuchofthedierencein 4Actually,EASYonlyguaranteesthattherstjobinthequeuewillnotbedelayed. Setia,Squillante,andTripathi[74]useaqueuingtheoreticmodeltoinves-

14 ing.theyndthattheformerapproachsucesunlessjobsareirregular(i.e., evolving)intheirpatternofresourceconsumption.similarly,inthecontextof oftreatingjobsasmoldabletothatofexploitingtheirmalleabilitybyfold- partitions. performancecanbeobtainedbyusingknowledgeofjobcharacteristics,andassigningnon-preemptiveprioritiestocertainjobclassesforadmissiontoxeingfoldingallowedperformancetoremainmuchbetterthanwithequipartitioning(equi)asloadincreases.padhyeanddowdy[60]comparetheeectiveness staticprocessorallocations(forwhichjobsneedonlybemoldable)ledtoperformancenearlyasgoodasthatobtainedbydynamicprocessorallocation(which quantum-basedallocationofprocessingintervals,chiangetal.[9]showedthat McCannandZahorjan[51]foundthat\eciency-preserving"schedulingus- requiresthatjobsbemalleable). byrostietal.[69]andsmirnietal.[81]. cessorstobeleftidle)untilalargernumberofprocessorsisavailable.algorithms thatleaveprocessorsidleinanticipationoffuturearrivalswerealsoinvestigated ForegoingOptimalUtilizationDowney[12]studiestheproblemofschedulinginanenvironmentwheremoldablejobsareactivatedfromanFCFSqueue, todecidewhenaparticularjobshouldbeactivated.thetradeoisbetween andruntocompletion.hesuggestshowtousepredictionsoftheexpectedqueu- startingajobsoonerwithfewerprocessorsanddelayingitsstart(causingpro- TheNeedforPreemptionAnumberofstudieshavedemonstratedthat ingtimeforawaitingtheavailabilityofdierentnumbersofprocessorsinorder underhighvariancebycomparingversionswithandwithoutpreemptionofthe productionsystems).parsonsandsevcik[64]showtheimportanceofpreemption truewhentheworkloadhasaveryhighvariability(whichisthecaseinreal samepolicy.goodsupportforshortrunningjobsisimportantbecauseitallows despitetheoverheadsofpreemption,theexibilityderivedfromtheabilityto preemptjobsallowsformuchbetterschedules. prioritytoshortrunningjobs,andthereforeapproximatestheshortest-jobfirst policy,whichisknowntoreducetheaverageresponsetime.thisisespecially forinteractivefeedback. Themostoftenquotedreasonforusingpreemptionisthattimeslicinggives importantinlargescalesystemsthatperformi/otomassstoragedevices,an fractionofthewallclocktime,butnotnecessarilysimultanously. systemsisthatitallowstheoverlapofcomputationandi/o.thisisespecially benetfromrate-equivalentscheduling,thatisallthreadsgettorunforthesame jobshaveastrongerrequirementforgangscheduling.however,allparalleljobs thatsomejobsaremoresensitivetoperturbationsthanothers,thereforesome operationthatmaytakeseveralminutestocomplete.leeetal.[45]haveshown Preemptionisalsousefultocontroltheshareofresourcesallocatedtocompetingjobs.Strideandlotteryschedulingusethenotionofticketstofairly Anotheruseofpreemptionthatisalsoknownfromconventionaluniprocessor

15 isthenproducedforeachjobcontainingtheperiodswhenthejobisscheduled allocateresources,includingcputime[90,91].eachjobgetsaproportionof thecpu,accordingtotheproportionofticketsassignedtothejob.atimeline needfornon-workconservingalgorithms.also,preemtionisneededinorderto squeezedout. torun.thatis,thetimequantumsareplacedatstridesalongthetimeline.the migrateprocessestoactivelycounterfragmentation. example,withpreemptionitisnotnecessarytoaccumulateidleprocessorsinordertorunalargejob.feitelsonandjette[20]demonstratethatthepreemptions decisions,boostingutilizationoverspace-slicingforrigidjobs,andavoidingthe timelinesfromallthejobsarepusheddownontoasingletimeline,andidletime inherentintime-slicingallowthesystemtoescapefrombadprocessorallocation ofparallelbatchjobsfailedtorunmorethanaminuteduetoreasonssuchasan Finally,inmanycomputingcentersitwasnotedthatanon-negligiblenumber Inparallelsystems,preemptionisalsousefulinreducingfragmentation.For bestartedimmediatelyaftersubmission,theninterruptedafter1minuteand incorrectlyspecieddatale.therefore,itmightbereasonablethatjobsshould uated.time-slicingismotivatedbythehighvariabilityandimperfectknowledge nallyresumedandcompletedatalatertime. Time-SlicingandSpace-SlicingSchedulingManyvariationsofscheduling algorithmsbasedontime-slicingandspace-slicinghavebeenproposedandeval- ofservicetimes,asdescribedabove,whilespace-slicingismotivatedbythegoal ofhavingprocessorsusedwithhigheciency. andrudolph[24].squillanteetal.[85]andwangetal.[94]haveanalyzeda variationofgangschedulingthatinvolvesprovidingservicecyclicallyamonga approach,butonlyatthecostoflongerresponsetimesforshortjobs.feitelson threadsinajobarescheduled(andde-scheduled)simultaneously.gangschedulingiscomparedtolocalschedulingandisfoundtobesuperiorbyfeitelson tosomepoweroftwo.theyndthatlongjobsbenetsubstantiallyfromthis setofxedpartitioncongurations,eachhavinganumberofpartitionsequal Timeslicingistypicallyimplementedbygangscheduling,thatis,allthe andrudolph[23]andhorietal.[36]analyzeamoreexiblepolicyinwhich studytheinteractionofgangschedulingandi/o,andfoundthatmanyjobs [84]providesananalysisoftheperformanceofdynamicpartitioning.Denget gangscheduling,andthereforeargueinfavoraexiblegangscheduling. al.showthatequiisoptimallycompetitive[11].dussaetal.[14]compares thereistimeslicingamongmultipleactivesetsofpartitions.leeetal.[45] maytoleratetheperturbationscausedbyi/o,thati/oboundjobssuerunder eratechargefortheoverheadoffrequentpreemptionsismade[86,48].squillante derhighvariabilityservicetimedistributions,round-robin(rr)wasfarbetter space-slicingagainstnopartitioning,andndsthatspace-partitioningpayso. Knowledge-BasedSchedulingMajumdar,EagerandBuntshowedthat,un- SeveralstudieshaverevealedthatEQUIdoesverywell,evenwhensomemod-

16 (suchasleastworkrst)werestillbetter.knowledgeoftheaverageparallelismof thanfcfs,butthatpoliciesbasedonknowledgeoftheprocessingrequirement tomakeitoperateatanear-optimalratioofexecutiontimetoeciency[16]. bycommunicationeventsweredescribedbysobalvarroandweihl[83]andby ajobmakesitpossibletoallocateeachjobanappropriatenumberofprocessors accountaswell,stillhigherthroughputscanbesustained[77].chiangetal.[8] thejobsintoframesforgangschedulingareinvestigatedbyfeitelson[18].feitelsonandrudolph[22]describeadisciplineinwhichprocessesthatcommunicate asinglepreemptionperjoballowsrun-to-completionpoliciestoapproachideal (i.e.,nooverhead)equi,andanastasiadisetal.[3]showthat,bysettingthe processorallocationofmoldablejobsbasedonsomeknownjobcharacteristics, disciplineswithlittleornopreemptioncandonearlyaswellasequi. Takingsystemloadandminimumandmaximumparallelismofeachjobinto Withtheknowledgeofhowmanyprocessorseachjobuses,policiesforpacking showthatuseofknowledgeofsomejobcharacteristicspluspermissiontouse Dusseau,Arpaci,andCuller[15]. frequentlyareidentied,anditisassuredthatthecorrespondingthreadsareall activatedatthesametime.similarschemesinwhichco-schedulingistriggered OtherFactorsinSchedulingMcCannandZahorjan[52]studiedtheschedul- systemconsistentlydoeswell.alversonetal.[2]describetheschedulingpolicy ingproblemwhereeachjobhasaminimumprocessorallocationduetoitsmem- oryrequirement.theyndthatadisciplinebasedonallocationbyabuddy byselecting(inthecompiler)atthetopofeachloopwhatdegreeofparallelism fortheteramta,whichincludesconsiderationofmemoryrequirements.brecht shouldbeusedforthatloop. leavethesejobsspacetogrow.yue[96]describesthecreationofevolvingjobs best-tscheduling,wherejobsareplacedwheretheycomeclosesttollingout apool.thisisaresultofusingamodelofevolvingjobs,whereitisbestto whereeachjobisallocatedtothepoolwiththemostavailableprocessors,beats [5]hascarriedoutanexperimentalevaluationofschedulinginsystemswhere processorsareidentiedwithclustersorpools,andintraclustermemoryaccess isfasterthaninterclusteraccess.asurprisingresultisthatworst-tscheduling, ingrigidjobsinanetworkofworkstationsenvironment.hisconclusionsinclude: variouspolicieswereimplementedinrealsystems. studiesdescribedabovehavebeencorroboratedbyexperimentalstudiesinwhich ExperimentswithParallelSchedulingManyoftheresultsofthemodeling {ActivatingjobsinLeastExpectedWorkFirst(LEWF)orderratherthan {Ifservicetimesareunknownorifonlyestimatesareavailable,then\back- Gibbonshasexperimentedwithanumberofschedulingdisciplinesforschedul- FCFSreducestheresultingaverageresponsetimebyfactorsfromtwotosix invariouscircumstances. lling"(asineasy)reducesaverageresponsetimesbyafactoroftwoor

17 {Whetherback-llingisusedornot,knowledgeofservicetimesisveryhelpful more.(ifservicetimesareknownexactly,thenback-llinghaslessrelative benet.) (particularlyifpreemptionissupported).havingjobknowledgeandusingit leadstoresponsetimesthatareafactorofthreetosixsmallerthanforthe exploitmoldableandmalleablejobs.hispositiveobservationsinclude: {Ifsomeknowledge(classorjob)isavailable,thenpreemptionismuchless caseofnoknowledge.whentheknowledgeisrestrictedtoclassknowledge theaverageresponsetimesareroughlyhalfthosewithnoknowledge. basedontheasmalldatabasethatrecordsexecutioncharacteristicsofjobs, {Ifmigratablepreemptionissupportedatlowcost,thenverygoodperformancecanbeachieved,evenifnoservicetimeknowledgeisavailable.(Also, malleabilityisnotofmuchadditionalbenet.) isneededinordertodowellbyusinglewforderforactivation. Parsonshasexperimentedwithabroaderclassofdisciplines,mostofwhich aremade(whichcanonlybecorrectedbypreemption). valuablethaninthecasewherenoknowledgeisavailableandbaddecisions {Ifonlylocalpreemptionissupported,thenclassknowledgeofservicetimes {Whenpreemptionisnotsupported,classknowledgeandLEWForderare Someadditionalobservationsonthenegativesideare: {In(typical)environmentswherethedistributionofservicetimeshasvery highvariance,lerwfdoesverywellwhensomeservicetimeknowledgeis available;otherwise,ifmalleabilitydoesn'tleadtoexcessiveoverhead,then asimplerulelikeequidoeswell. localpreemptionsupported. helpful(roughlyhalvingaverageresponsetimes),butnotasmuchaswith {Thevalueoflocalpreemptionisrestrictedbythefragmentationthatoccurs {Evenwithmoldablejobs,performanceispoorunlesspreemptionissupported,becauseifinappropriateallocationsareoccasionallymadetovery partitioningarebenecial,becausethedependenciesamongjobsarethen becausejobsmustberestartedonthesamesetofprocessorsonwhichthey previouslyran.(inthiscase,eithercleverpackingstrategiesorevenxed limited[25].) usedforeachjobischosenbytheuser,somesucientlylargepartitionacquired, istousesimpleandinexiblemechanisms.inessence,thenumberofprocessors 3RecommendationsandFutureDirections Thecurrentstate-of-the-artregardingschedulingonlarge-scaleparallelmachines longjobs,thenonlypreemptioncanremedythesituation. andthejobisruntocompletion.afewrecentsystemssupportpreemption,so thataparalleljobcanbeinterruptedandpossiblyswappedoutofmemory,but

18 manyinstallationschoosenottousethisoptionduetohighassociatedoverheads andlackofadequatei/ofacilities. beendemonstratedtobefeasiblethroughprototypeimplementationsandexperimentation. impossibletoputtheoreticalresultsintopractice,especiallyinproductionentiprocessorschedulingshouldbedone.asalways,itisnoteasyandsometimes oftheinvestigations,analysis,andsimulationsdescribedabove. state-of-the-artschedulers.therecommendationsarebasedontheconclusions Agreatdealhasbeenlearnedabouthow\intheory"multiprogrammedmul- Thissectionpresentssixrecommendationstoimprovetheperformanceof vironments.note,however,thatallofthefollowingsuggestedapproacheshave processorsassignedtoajobduringruntimewhichisdesirabletobesthandle nodes.alternatively,rate-equivalentschedulingcanbeused,meaningthatall threadsgettorunforthesamefractionofthewallclocktime,butnotnecessarily isgangscheduling,whereallthethreadsrunsimultaneouslyontheirrespective coordinatedacrossthenodesrunningthejob.onesuchformofcoordination simultanously.notethatgangschedulingimpliesrate-equivalentscheduling. Recommendation1:Providesystemsupportforparalleljobpreemption.Preemptioniscrucialtoobtainingthebestperformance.However,thisshouldbe evolvingandmalleablejobs,butthemarginalgaininperformanceisnotsubstantial.henceitisjustiedonlyifitcanbeprovidedwithlittleadditionaleort (asapartofcheckpointingprocedures,forexample). tobeevolving.sincesystemloadsvarywithtime,andusersgenerallydonot Preemptionisalsoapreconditionforchangingthenumberandidentityof Recommendation2:Writeapplicationstobemoldableand,ifitisnatural,then processorscanbeassuredatheavyload. knowwhentheysubmitajobwhattheloadconditionswillbeatthetimethe jobisactivated,itisdesirablethatjobsbemoldableratherthanrigid,sothat availableprocessorscanbefullyexploitedatlightload,butstillecientuseof tivelylongsequentialperiods),thenwritingthejobasevolving(withannotations orcommandstodynamicallyacquireandreleaseprocessors)makesitpossibleto greatlyincreasetheeciencywithwhichthejobutilizestheprocessorsassigned toitẇritingjobstobemalleableismuchmorework,andthisistypicallyjustied onlyforapplicationsthatconsumeasignicantportionofasystem'scapacity, becausetheyareeitherverylargeorinvokedveryfrequently. Ifjobsarenaturallyevolving(suchasacyclicforkjoinstructure,withrela- Recommendation3:Whensystemeciencyisofutmostimportance,thenbase processorallocationsonbothjobcharacteristicsandthecurrentloadonthesystem.jobsmakemoreecientuseoftheirassignedprocessorswhentheyhave fewerthanwhentheyhavemore.hence,astheworkloadvolumeincreases,it

19 maybenecessarytoreducethenumberofprocessorsassignedonaverageto eachjob.atlightload,processoravailabilityisnotanissue,soeachjobcanbe overallsystemcapacity,sogivingjobsasmallnumberofprocessors(evenone bility)isthemostappropriateaction.bydoingthis,thethroughputcapacityof inthelimitaslongasmemoryrequirementsdon'tprecludethisextremepossi- eciency.atheavyload,themultiprocessingoverheadmerelydetractsfromthe thesystemcanbemaximized. ableforlightloadconditions,buttheyleadtounacceptablylowprocessore- ciencyatheavyload.consideracasewheretherearenstatisticallyidentical theoptionstoeither(1)runthemoneatatimewithallpprocessors,or(2)run theminpairswithhalftheprocessorseach.boththemeanandthevarianceof theresponsetimesarelowerwiththelatterapproachunless[76]: Whenspecicprocessorallocationsareselectedbyusers,theytendtobe givenasmanyprocessorsasitcanuse,eveniftheyarenotbeingusedathigh jobstorunonpprocessors.assumingthejobsaremoldable,theschedulerhas overlyaggressiveoroptimistic.thenumbersselectedbyusersaretypicallysuit- ofjobsismoderatelylarge. submitajob,itisbestiftheyidentifyarangeofacceptableprocessorallocations, andthenleavethechoicewithinthatrangetothescheduler.thecurrentworkloadvolumecanbetakenintoaccounteitherbyjustobservingtheoccupancy Sinceuserscannotpracticallyknowtheloadonthesystematthetimethey S(P)>2 2 N+2S(P=2) Thisconditionseldomholdswheneitherthenumberofprocessorsorthenumber servicetimedistributionsbymakingaverageresponsetimeindependentofthe sorscheduling,itisknownthatrrschedulingprotectsagainsthighlyvariable servicetimedistribution(assumingpreemptionoverheadisnegligible).further, Recommendation4:Toimproveaverageresponsetimes,giveprioritytojobsthat aremostlikelytocompletesoon,usingpreemptionwhennecessary.inuniproces- overallloadasitvariesindailyorweeklycycles. ofthequeuesinwhichjobsawaitinitiation,orbytrackingsomepredictionof (FB)disciplinescanexploitthis,andyieldloweraverageresponsetimesasthe variabilityoftheservicetimedistributiongrows[10]. iftheservicetimedistributionisknowtohavehighvariability,thenfeedback havehigherpriority,thenthisisconsistentwiththeideaofusingavailableservice systemsupportorjobsaren'twrittentoexploitit,thensomeformofpreemptive schedulingbasedontime-slicing,suchasgang-scheduling,shouldbeused. torrinauniprocessorcontextinitsabilitytoschedulerelativelywelleven withnoservicetimeknowledge.ifmalleabilityisimpracticalduetolackof ofprocessorstoeachjobavailableforexecutionisoptimal.equiisanalogous exploited,theidealequidiscipline,whichattemptstoassignanequalnumber Incurrentpractice,ifqueuesforjobswithsmallerexecutiontimestendto Whennoknowledgeofservicetimesisavailableandmalleabilitycanbe

20 besttotrytoactivatethejobsinorderofincreasingexpectedremainingservice time[63].iftheservicetimesareknowntobehighlyvariable,buttheservice timesofindividualjobscannotbepredictedinadvance,thenthedisciplinethat executesthejobwithleastacquiredservicerstisbestbecauseitemulatesthe behaviorofleastexpectedremainingworkrst. betterknowledgeofjobservicetimesthanqueueidentitiesisavailable,thenitis timeknowledgetofavorthejobsthatareexpectedtocompletemostpromptly.if informationisgenerallypositivelycorrelatedwithtruth,andthatissucientto makebetterschedulingpossible.(agoodschedulingpolicywillpenalizeusers Recommendation5:Makeuseofinformationaboutjobcharacteristicsthatis formationabouttheexecutioncharacteristicsoftheirjobs,intheencodedform ofaqueueidentier.usersuppliedestimatescannotbedirectlybelieved,butthe whointentionallymisestimatethecharacteristicsofthejobstheysubmit.) eitherprovideddirectly,ormeasured,orremembered.usersalreadyprovidein- Recommendation6:DevelopNewModelsBasedonthebehaviorandshortcomingsofrealmachines,newmodelsshouldcapturerelevantaspectssuchasthe 1.dierentpreemptionpenaltycostsassociatedwithlocalpreemptionandjob 3.preventionofjobstarvationbyguaranteeingacompletiontimeforeachjob 2.arelationbetweenexecutiontimeandallocatedprocessorsformoldable, evolving,andmalleablejobs, migration, thebehaviorofnewjobswithsimilarcharacteristics. withrespecttogivingadditionalprocessors,ortakingsomeawayfromthejob. measuredwhilethejobisexecutingandthesystemcantakeappropriateaction Finally,ifsomehistoricalinformationisretained,thenobservedbehaviorof previousjobswithcertaincharacteristicscanbeusedtopredict(approximately) Assumingmalleablejobs,somejobcharacteristics(suchaseciency)canbe following: 4ThePSCHEDStandardProposal 4.pricingpoliciesthatarebasedonsomecombinationofresourceconsumption 5.cyclicloadpatternsthatmotivatedelayingsomelargejobstotimeperiods bythejob,andjobcharacteristicsthatmayormaynotbeknownatthe atthesubmissiontimeofthejob, TheoreticalresearchlikethatdescribedinSection2tendstofocusonalgorithmicsandeasilymeasurablemetrics,whileabstractingawayfromthedetails. timethejobissubmitted, Systemadministrators,ontheotherhand,cannotabstractawayfromreal-life concerns.theyarealsofacedwithunmeasurablecostsandconstraints,suchas ofloweroveralldemand(e.g.,\ohours"). interoperability(willmachinesworktogether?)andsoftwarelifetime(howsoon

21 dierentneeds,leadingtothecreationofratherelaboratesystems[4,44]. prototyope.finally,theyneedtocatertousersandadministratorswithmany andstabilityrequiredofproductionsoftwareismuchharderthanbuildinga softwarecomponents.inrecentyears,messagepassinglibrarieswerestandard- willpartsofthesystemneedtobereplaced?).moreover,achievingthematurity scheduling. 4.1Background izedthroughthempieort.similarly,thepschedproposalaimsatstan- dardizingtheinteractionsamongvariouscomponentsinvolvedinparalleljob Asaresultofsuchconcerns,thereismuchinterestinstandardizingvarious portableunixapplicationthatallowstheroutingandprocessingofbatch\jobs" inordertomaximizeutilizationofthecomputer. Deferredprocessingofworkunderthecontrolofaschedulerhasbeenafeatureof inanetwork.toencourageitsusage,theproductwaslaterputintothepublic oeredbytheproprietarysystems.thisomissionwasrectiedin1986bynasa AmesResearchCenterwhodevelopedtheNetworkQueuingSystem(NQS)asa providersandusersbecauseitdidnotincludethesophisticatedbatchfacilities mostproprietaryoperatingsystemsfromtheearliestdaysofmulti-usersystems domain. Thesupercomputingtechnicalcommitteebeganasa\BirdsOfaFeather" ThearrivaloftheUNIXsystemprovedtobeadilemmatomanyhardware ofnqs.thiseortwasnallyapprovedasaformalstandardondecember13, andprod.nonewerethoughtbehaveboththefunctionalityandacceptability decidedtousenqsasthebasisfortheposixbatchenvironmentamendment in1987.otherbatchsystemsconsideredatthetimeincludedctss,mdqs, (BOF)attheJanuary1987Usenixmeeting.Therewasenoughgeneralinterest toformasupercomputingattachmenttothe/usr/groupworkinggroups.the /usr/groupworkinggroupslaterturnedintotheieeeposixstandardeort. percomputingworkinggrouphassincebeeninactive. 1994asIEEEPOSIX1003.2d.Thestandardcommitteedecidedtopostpone addressingissuessuchasprogrammaticinterfaceandresourcecontrol.thesu- DuetothestronghardwareproviderandcustomeracceptanceofNQS,itwas standingastime,memory,orsoftwarelicenses.toruneciently,allpartsofa nounderstandingoftheneedsofparalleljobs.theonlysupportforparallelism isregarding\processors"asanotherresourceduringallocation,onthesame paralleljobneededtobescheduledtorunatthesametime.withoutsupport batchqueuesystemthatconformstotheieeestd d-1994.theproject startedinjune1993,andwasrstreleasedinjune1994[35]. PBSwasdevelopedatNASAAmesResearchCenterasasecondgeneration revertedtospaceslicingandan\alljobsruntocompletion"policy. fromthebatchqueuesystem,mostofthelargeinstallationofmppsystemshad However,bothNQSandPBSweredesignedtoscheduleserialjobs,andhave