Transcription

1 BeyondPVM3.4:WhatWe'veLearned, G.A.Geist,J.A.Kohl,P.M.Papadopoulos,andS.L.Scott What'sNext,andWhy ComputerScienceandMathematicsDivision OakRidge,TN, ,USA OakRidgeNationalLaboratory havemadepvmbotheectiveandwidespread:asimplesystemabstraction.fromahigh-performanceprogrammingpointofview,weexamine Abstract.Thispaperexploresthefoundationsandphilosophythat tionandapi,transparentheterogeneity,anddynamicsystemcongura- sisisthatpvm,mpi,andsimilarparadigmssuerfromamonolithic approachtothedistributedcomputingproblem.theapproachessimplycannotmeetthelargerangeofservicerequirementsforadvanced distributedcomputingenvironments.thenotionofageneralizedplug- controlovertheiroperatingenvironment.thisenvironmenthasthepo- InMachine(GPM)isintroducedthatallowsprogramstoexertbetter namics,andfault-tolerance.pluggablecomponents,suchasmessaging tentialtoprovidemechanismsforbetterperformance,richersystemdy- substrates,dynamically-attacheddebuggingagents,orcompletevirtual machinesthatcanbejoinedtogether,formanoperatingenvironment environmentarediscussed,andinherentchallenges,suchaseliminating thatcanbecustomizedon-the-y.generalizationsofcurrentpvmplug- themasterpvmdaemonandprovidingthepluggablesubstrate,areexins(resourcemanagers,hosters,taskers)thatleadtothisnext-generation thefeaturesthatmakepvmusefulandthosethatmakehardware-level performancediculttoachieve.thekeyconclusionfromthisanaly- 1Introduction amined. designs.thereareindeedthreekeyprinciplesthathaveguidedthedesignand majorrevisions,witheachnewreleaseusingfundamentallydierentinternal asasinglevirtualresource.thebasicpvmsystemhasgonethroughthree commoditynetworks,workstationscanbemanagedtogetherandprogrammed PVMisasuccessfulresearchprojectthatstartedwithasimpleidea:using implementation:asimpleapi,transparentheterogeneity,anddynamicsystem conguration.thesimpleapiallowsmessaging,virtualmachinecontrol,task heterogeneitymakesiteasytoconstructprogramsthatinteroperateacrossdifferentmachinearchitectures,networks,programminglanguages,andoperatincessedandcontrolledusingonlyabout60user-levellibraryroutines.transparent systems.dynamicsallowthevirtualmachinecongurationtochangeandthe control,eventnotication,eventhandlers,andamessag boxalltobeac-

2 numberoftasksthatmakeupaparallel/distributedcomputationtogrowand shrinkunderprogramcontrol.proponentsofpvmhaveexploitedthesefeatures theseboundarieshaveoftentranslatedtolimitedmessagingsemanticsanddicultiesinachievingpeakperformanceonhigh-bandwidthnetworks.thispaper thelongterm:adynamicallycongurablesystemthatallowsprogramstodene laysthegroundworkforwherewebelievedistributedcomputingshouldleadin theirperformance/featuretradeostooptimizeandcustomizetheenvironment. monolithicoperatingenvironmentintowhichuserprogramsmust\t."mpi-1, Thismaximizesperformancebyminimizingdynamicsandworksverywellin includingpvm:eachmandatesaparticularprogrammingstyleandbuildsa forexample,mandatesanspmd-stylestaticprocessmodelwithnojobcontrol. Weseeacommonthemeinallpopulardistributedcomputingparadigms andlearnedtolivewithintheboundariesthatthesystemprovides.however, staticenvironments.programsthattintothempisystemarewellservedby itsspeedandrichmessagingsemantics.pvm,ontheotherhand,allowsprogramstodynamicallychangethenumberoftasksandaddorsubtractresources. thoughmpiandpvmprovideveryusefulenvironments,someprogramssimply However,programsingeneralpayaperformancepenaltyforthisexibility.Even arenotabletondtherightmixoftoolsorarepayingforunwantedfunctionality.here,themonolithicapproachbreaksdownandamoreexiblepluggable appliedinotherareas{themachoperatingsystemisbuiltonthemicrokernel approach,linuxhaspluggablemodulestoextendfunctionality,andhorususes tocustomizetheiroperatingenvironmenttoachievetheirowncustomperformance/functionalitytradeos. generalizingtheseideastoparallel/distributedcomputing,programswillbeable a\legoblock"analogytobuildcustomnetworkprotocols.byextendingand substrateisneeded.thisideaiscertainlynotauniqueandhasbeensuccessfully ondistributedresourcesandmethodstobringbettercontrolandperformance willtransfertothenextgeneration.ournextgenerationenvironmentwillfocus toprograms,ratherthanattemptingtorewritemessagingsemantics.however, asignicantnewclassofapplicationswillbeenablediftheruntimesystemis PVMandMPIbothexhibitdesirablefeaturesandAPIsthatshouldand issimultaneouslyecient,dynamic,androbust.thenextgenerationenvironmentwillsupportvirtualmachinesthatcanroamacrossnetworkstoattachand logicallydecoupledfromthesemanticsoftheuserinterfaceandnewdynamicinteractionsaredened.thechallengeistoimplementapluggablesubstratethat enablingtechnologyforcompletelynewclassesofapplications.pvmalready mechanisms:startinghosts,startingtasks,ormanagingresources.however,if denesthreeplug-ininterfacesthatenabletaskstomodifydefaultpoliciesand detachresourcesasprogramrequirementschange.weseethisasafundamental theprogrammerwantstochangetheunderlyingmessagesubstrate,needsmessageencryption,orrequiresdierenteventnoticationproperties,atrulyherculeaneortisrequiredtomodifythepvmruntimesystem.inaddition,iftwo virtualmachinesareseparatelyinstantiated,theycanneverbemergedtogether (orasinglemachinesplitintotwo).thispaperexplorestheconceptofplug-

3 tools,andhigh-levelcontrolofthevirtualmachinethatencompassesmerging, gabilityatthreelayers{lowlevelmessaging,mid-levelresourceandanalysis splitting,andmigration. threelevels,howthesehaveinuencedtheinternaldesigndesignofpvm,and howtheyprovideasteppingstonetoamoredynamic,pluggableenvironment. ThispaperwillreectuponthepropertiesofPVM3.4inrelationtothese Oneshouldreferto[1]forcompletedetailsoftheprotocolsandinnerworkings PVMwasdesignedforheterogeneousinteroperationacrossTCP/IPnetworks. 2TheBasicPVMDesignAssumptions ofpvm3.x.theapihasthreeclassesoffunctions:messaging,taskcontrol, andmachinecontrol. ThefundamentalmessagingunitinternaltoPVMisamessagefragment,with auser-levelmessagebeingmadeupofoneormorefragments.fragmentsare 2.1MessagingAssumptionsandtheirRelationtoInternalDesign (fragmentationofanentiremessagefollowedbyasend)introducesamessage usingpvmpackbeforesendingthemessagetoitsdestination.thisserialization ityacrossmachines)requiresataskrsttocompletelyfragmentthemessage onanylength.thedefaultoperationalmode(whichguaranteesinteroperabil- usuallycompatiblewiththetcp/udplimitssetatruntime,butmaytake copy,increaseslatency,anddecreasesbandwidth.dierentmessagesemantics, likethosefoundinmpi,allowtheunderlyingsystemtopipelinethepackand sendoperationssothatlongmessagesmayoverlapthesendingofonefragment withthepackingofasubsequentone.packingdoesallowtranslationofthedata toitsxdrformat,acostlybutoftennecessarystep.italsoprovidesameansto (whichgreatlyimproveseciencyontraditionalnetworks).theinternaldesign conglomeratemanysmallerdatachunksintoasinglebuerfornetworktransfer makesasynchronousmessages(orpost/test-for-completionoperations)dicult ofpvmreectsthesepack/sendsemanticsandmakesitquitechallengingto toinclude.thecurrentpvmapihasinuencedtheinternaldesigntothepoint retrotthecodetousepipeliningandotherperformanceimprovements.pvm whereitisdiculttomapothersemanticsecientlyontopoftheexistingsystem.thelessonlearnedhereisthatthesemanticsofaparticularmessagingstyle assoonaspvmsendreturns.similartopacking,theinternalimplementation messagingalsoassumessynchronousmessaging{adatabuercanbere-used (ala'pvmormpi)mustbedecoupledfromthelow-levelinfrastructure.this doesnotimplythatthepvm(ormpi)interfaceshouldbethrownaway,but rathertheinternalimplementationneedstobemoreexibletohandleawidevarietycommunicationsemanticslike:one-sidedoperations(putsandgets),asynchronousmessaging,encryption(permessageorperlink),streams(forvideo andaudio),qualityofservicelinks,andlinkedstreams.

4 2.2PluginCommunicationModules apracticalmethodtomanagemultiplecommunicationstyles.becausemessagingisextremelyimportanttosystemperformanceandevaluation,thelowessibilityofacoresystemisessentialtoachievecriticalperformanceandgivelithicsystemcanhandleecientlyallthedesiredcommunicationstyles.exten- Asdistributedcomputinghasdeveloped,ithasbecomeclearthatnoonemono- send/receivemessagepassingisquitedierentfromone-sidedcommunication layersmustbeabletobeswappedoutfordierentscenarios.forexample, bothisautomaticallyinstalledwhennotneededbyanapplication.theineciencycomesfromthefactthatincomingmessagesneedtobecheckedamong dierentcommunicationmethodstodeterminethecorrecthandlingsequence.if aparticularmessagestyle(e.g.aput)isneverused,thenareductioninoverhead itisusuallyguaranteedbythemessagesystem.ontheotherhand,commu- failuremonitoring.ausershouldbeabletowriteadistributedapplicationand isitunnecessarytofragmentmessagesorprovidereliabletransmissionbecause canbemadebyeliminatingthisasachecked-forstyle.onmpps,forexample, semantics.low-levelperformancecanbesignicantlyaectedifsupportfor havetheruntimesystemselectwhichmethod(s)areneededfortheparticular nicatingoverawanrequiresfragmentation,timeouts/retries,andconnection run.thekeytosuccesswillbetodesignplug-incommunicationstacks(similar siretoencryptcertaincommunicationlinksonlyifthevirtualmachinespans withoutrequiringuserstodramaticallychangecode.forexample,onemayde- tothosefoundinhorus[10])thatcanbetraversedquicklyandeciently.to getoptimumperformance,itmayrequiretheusertousestrongly-typedmessaginglikempi.however,runtimepluggabilitycanstillgivesignicantadvantagestalledwithoutusercodemodication.thenextgenerationpluggablemachine severaldomains.runtimepluggabilitywouldallowanencryptedlinktobein- willhavetostrikeabetterbalanceamongperformance(ortheabilitytooptimizeperformance),exibility,andinteroperability.duetothelargebodyof themoststraightforwardgoaltoachieve. researchoncommunicationmethods,thislowestlevelofpluggabilityisprobably Thenext-generationpluggablecomputingenvironmentwillprovideafertileplatformforavarietyof\middle-level"toolsthatmanipulateandadministersystem 3Middle-LevelTools:CurrentDesign,Features,Needs resources.fundamentallydierentfromtraditionalvirtualsystems,eachuserapplicationwillcustomizeitsowncomputingenvironmentinteractively,tobring atrulypluggablesystemcanbeachieved.thissectionoverviewsthecurrent pluggablecomputingtoolsandtheirfeatures,andexploresfutureneedsalong thecourseofitsexecution.existingsystemshavealreadytakensomestepstowardthislevelofexibility,buttherearemanyissuesyettoberesolvedbefore thedesiredcomputationalandcommunicationresourcestogetherasneededover withtheaccompanyingobstacles.

5 fromthefundamentalsoftwareengineeringprinciplesofmodularityandencapsulationoffunction.ifasystemisdesignedcarefullybyseparatingconcerns to\swapout"oneimplementationdetailforanother.muchofobject-oriented andgeneralizingthehandlingofsimilaroperations,thenitisstraightforward Theconceptof\pluggable"interfacesisnotanewone,butratherstems compatibleinterfacebutdierentinternalimplementation,canbeinsertedin methodologyisbuiltonthispremise.ifan\object"hasawell-denedinterface placeoftherstobjectwithoutdisturbingtheoverallfunctioningofthesystem. andencapsulatesitsinternalstateandoperations,thenanewobject,witha machine.hence,itcurrentlyispossible,butnotsimpleforausertoprovidecustomizedauthenticatedstartupofremotepvmdaemons(pvmds).ataskercan bestartedoneachhostofthevirtualmachinetohandlethespawningoftasks there,e.g.toallowdebuggerstointerveneinatask'sexecution.installinganrm policies,includinghostaddinganddeleting,variouseventnotications,andtask inthevirtualmachineisequivalenttoacomplete\take-over"ofallscheduling informationandspawningplacement.pvm3.4alsointroducestheconceptofa \tracer"forautomaticallycollectingtraceeventmessagesfrominternallibrary instrumentation,toalleviatetheneedforafulltracingtoolsuchasxpvm[4,5] whencollectingtracesoline. PVMDtohandlecertainclassesofrequests.Subsequently,whenoneofthe registeredtooltohandlethatrequest.inthissense,certainaspectsofthepvm systemresourcesandschedulingcanbecustomizedon-the-yby\pluggingin" desiredrequestsarrivesfromauserapplication,thepvmdpassescontroltothe Inalloftheabovecases,thetool\plugsitselfin"byregisteringwitha canbestartedtocustomizetheprocedureforaddingnewhoststothevirtual instantiatedtotakeoverhandlingofcertainsystemfunctions.ahosterprogram istenceof\hoster,"\tasker,"and\resourcemanager(rm)"toolsthatcanbe PVMhassupportedthistypeofmodularitysincePVM3.3,withtheex- customizationbydesign,andhadtobespecicallyre-codedtoallowalternate computingenvironments,pvmdidnotoriginallysupportthistypeofinteractive userreplacementsforthedefaulthandlers.unfortunately,likemosttraditional pluggableinterface.whilethebrowserisrunning,a\plug-in"canberegistered externalhandlingofthedesiredoperations. occurs,thebrowserautomaticallyforwardsthedatatoaplug-intaskorthread tohandlecertaintypesofwebpagecontents.whenoneofthesecontenttypes whichthenprocessesthedataandeitherdisplaystheresultsorreturnstothe Netscape'sNavigatorwebbrowser,version3.0[9],supportsasimilartypeof browserforinlinedisplay.usingtheexplicitmimetypinginformationcarried byallwebpages,dataofanytypecanbehandledinacustomizedmanner. xedandlimitedprotocolwithinwhichtheplug-insmustoperate.theplug-ins cannottranscendtheirpredenedrangeoffunctionality.therefore,forthenextgenerationpluggablesystemtobefullygeneral,thepluggablearchitecturemust WithbothPVMandtheNetscapeNavigator,thepluggabilityisofastatic natureinthat,eventhoughtheplug-intoolscanbeaddedatruntime,thereisa alsoencompasstheveryprotocolsusedbytheplug-inelementstointeractwith

6 thesystem.thesystemmustbebuiltwithverylittleinherentfunctionality, primarilyjustasimpleinterpreterfor\self-dening"protocolsthatallowsall otheraspectsofthesystemtobegenericallyconnectedasneeded.eventhe defaultsystemhandlerswillusethesameprotocolinterfacetoattachtothe \micro-kernel,"whichessentiallyservesonlyasapointofcontact. performance.becausetheplug-inprotocolsaregeneralizedandrequireinterpretation,thecomputationrequiredtoprocessthemisincreasedandtheoverall Animportantissuesurroundingthisexibilityisthepotentialtrade-ofor typesofplug-intools,suchasdebuggers,itcouldbedisastrousfortoolswhich performanceofthesystemisdegraded.whilethismaybeacceptableforcertain istooptimizetheuseofself-deningprotocols.pvm3.4alreadyappliesan requiregreaterbandwidth,suchastracingormultimediatools.onesolution optimizationtoitsgeneralizedtracingprotocolbyonlysendingtraceeventdescriptorswiththerstoccurrenceofeachtraceevent,therebyreducingneeded wouldresultinamorebalancedpluggablesystem,withbothampleexibility provideafewstatic\high-performance"protocolsforcertainwell-denedinteractions.thesespecial-purposeprotocols,alongwiththegeneralizedprotocols, bandwidthandtracingoverhead[5].alternatively,abettersolutionmaybeto andrespectableperformance1. manytasksexecutinginparallel,severalotherissuesarise.supposethattheuser wishestointeractivelyattachatooltoasetofrunningapplicationtasks,asis donewiththecumulvssystemforinteractivevisualizationandcomputational steering[2].currently,forcumulvstobeabletoconnecttoanapplicationthe Whenthepluggableapproachisappliedtoaconcurrentapplicationwith usermustinstrumenttheapplicationsothateachtaskperiodicallypassescontrol areprocessedtoconstructconnectionswithcumulvsviewerprogramsandto toacumulvslibraryroutine.withinthisroutinethecumulvsprotocols exchangeinformationandcontroldirectives.becausetheuserapplicationtasks couldincorrectlyreceivemessagesthatarepartofthecumulvsprotocols. maynotbetightlysynchronized,raceconditionsexistwhereintheapplication Topreventthisfromhappeningtheusermustsetasideasmallportionofthe message\tagspace"topreventinterceptionofcumulvs-taggedmessagesby theapplication,andviceversa. ofpvm3.4[3].theintroductionof\messagehandlers"nowallowsprotocols, likethatrequiredforcumulvs,tobehandledtransparently.insteadofpass- ingcontroltosomeexplicitprotocolinterpreter,messagehandlerscanbede- nedtointerceptanyincomingprotocolmessagesautomatically,andprocessthe Progresshasalreadybeenmaderegardingtheseissuesinthelatestrelease realmsofcommunicationwithindistincttagspaces.asidefromamessagetag, peciallycriticaltoguaranteethatmessagetagspacesdonotoverlap.tosimplify thisproblempvm3.4usestheconceptof\context"toencapsulatethedierent messagehandlersareinvokedinternallywithoutanyuserintervention,itises- protocolinteractionstoupdatethenecessarystateinformation.becausethese eachmessageisidentiedwithasystem-generatedcontext.ataskcanonlysend 1Thisis,infact,thephilosophyunderwhichthePVMsystemwasdeveloped.

7 toolarchitectures,morefundamentalchangesarenecessarytofullyrealizethe andreceivemessageswithinitscurrentlyselectedcontext,andlikewisemessage anyuserspecicationoftagspacesforavoidingincorrectmessagedelivery. handlersaredenedtoprocessmessageswithinagivencontext.thisalleviates isrequired.however,performanceisimportantandcannotbeoverlookedentirely potentialofsuchsystems.theintegrationofgeneralizedorself-deningprotocols infavorofexibility. Thoughtheabovestrideshavebeenmadetowardmoreexiblepluggable 4VirtualMachineControl:Splitting,Merging,Migration 4.1TheRoleoftheMasterPVMDaemon Auser'svirtualmachineismadeupofauser-denedsetofcommunicatingdaemonprocessescalledPVMDs.Toexpandthevirtualmachine,anewdaemon ectthenewconguration.thepvmdesignrevolvesaroundamasterdaemon isstartedonthenewhostandcongurationparametersareexchangedtore- thatkeepsanddistributesthecurrentconguration.furthermore,newhostsare addedtothecurrentvirtualmachinebyaprotocolthatmustgothroughthe master.ifthemasterfails,thenallnon-masterdaemonsshutthemselvesdown modiesthecriticalcongurationinformation(thehosttable),severalraceconditionsareautomaticallyeliminated.thisleadstoamorestableandpredictable andtheentirevirtualmachinecollapses.becauseonlyasinglemasterkeepsand onlygrowandshrinkwhilerootedatasinglemaster.onecouldelectmultiple formmultiplevirtualmachines;andvirtualmachinescannot\move"andmay implementation:virtualmachinesstartedindependentlyofoneanothermaynot joinintoasinglevirtualmachine;asinglevirtualmachinemaynotsplitto systemformostusers.thereare,however,threesignicantconsequencesofthis couldbepromotedtothismasterpoolifanoldmasterfails.however,toachieve ofmasterscloselycoordinatetheadditionanddeletionofhosts.newdaemons coherencyofcongurationchanges,theaddition,deletionandelectionprotocols mustthemselvesbefault-tolerant.whilenotunsolvable,thisrepresentsasignificantincreaseincomplexityinhandlingthecongurationofthecurrentvirtual toexaminethetradeosofdierentsolutionscenarios. 4.2NamingandMigratingTasks PVMtaskids(ortids)arenamedrelativetoahostwhichinturnisnamed masterstoeliminatethesinglepointoffailure,butthisrequiresthatthepool machine.itisindeedachallengingproblemandwillrequireconsiderableeort plicatedduetothelargenumberofraceconditionsthatmustbesolved.systems consistentlyreectedacrossalltasksinthevirtualmachinebecomesverycom- tomigrate,thenanewtidmustbeassigned.re-assigningthetidsothatitis integer,isthephysicalhostofaparticularprocess.ifonedesiresapvmtask relativetothemasterpvmd.therefore,encodedinthetid,whichisa32bit

8 similarmannertovirtualmemory.however,theupdatingofvirtual-to-real-tid spacemappingismadeateverycall.thiscanbeimplementedecientlyina around.onelogicalsolutionistovirtualizetaskidssothatavirtual-to-real- mapsacrossanentirevirtualmachinemaybeproblematicbecauseofhostfailuresduringtheupdateprocess.migrationoftasksalsopresentsaproblemwhen likecondor[7]havewrestledwiththetasknamingproblemastheymovetasks caseofunixoperatingsystems,thelemayitselfbeanabstractionofsomeother taskshaveopenles.thelemaybeunavailableonthenewsystem,orinthe withcomplexsolutions.alargepartofwherepvmisgoingnextwillrevolve andmigratingormergingvirtualmachinespresentverychallengingproblems physicalentity(e.g.acommunicationsocket).itisclearthatmigratingtasks aroundthesetwoimportantissues. 4.3NewCapabilitiesofDynamicMachines Thecapabilityofdynamicallymergingandsplittingvirtualmachineshastheadvantageofallowinguserstogenerateroamingapplications.Whilethepluggabilityoflow-levelcommunicationprotocolsandmiddle-leveltoolsprovidepracticaple,roamingapplicationsmaydiscoverbetterresourcesandmigratetheirvirtual engineeringbenet,roamingopensupwholenewapplicationsclasses.forexam- providessomecapabilitytomigrateapplicationcomponentsorevencomplete machine(andassociatedtasks)toamoredesirableconguration.whilepvm applications,itisnoteasyorstraightforward.bymakingmigrationapluggable partofthebasefunctionality(includedonlywhenwantedbytheprogram), betterresourceutilizationcanbeachieved.furthermore,multiphaseprograms canmoreeasilybecreatedwheretheenvironmentadjuststochangingresource cessingonthelargerproblem.modelingofcomplexreal-worldproblemscould needs. componentsandthenbrieyjoinwithothervirtualmachinestocontinuepro- greatlybenetfromthiscapability.asimpleexampleistouseseparateapplicationsuitestomodelthevariousindividualairframecomponentsforanaircraft, whenrunninginanenvironmentwherethemasterlessvirtualmachinesupports migration. plicationscancreatetheirownvirtualmachinetoresolvespecicsub-problem Afurtherbenetofdynamicmergingandsplittingisthatportionsofaptions.Finally,faulttoleranceofanapplicationcanbeeasilyachievedmoreeasily withbriefapplicationjoinstointroducethevariousinteractingboundarycondi- ThenewfeaturesincludedinPVM3.4provideasteppingstonetothenext generationdistributedcomputingenvironment.assuch,pvm3.4willallow userstodevelopmuchmoreexible,dynamic,andfaulttolerantapplications. 5PVM3.4ArststeptowardsGPM theenvironment. Inaddition,feedbackfromusersofPVM3.4willhelpguidethedevelopmentof

9 tionsprovidethebiggestleapinpvmcapabilitiessincepvm3.0cameoutin 1993.Thefunctionsprovidecommunicationcontext,messagehandlers,anda tuplespacecalledmessagebox. WhiletheactualnumberofnewfunctionsinPVM3.4issmall,thesefuncmentalrequirementforparalleltoolsandapplicationsthatmustinteractwith eachother.itisalsoarequirementforthedevelopmentofsafeparallellibraries. Context,whichPVMdenesasauniquesystem-createdtag,ssentwitheach message.thematchingreceivefunctionmustmatchthecontext,destination, Theabilitytosendmessagesindierentcommunicationcontextsisafunda- andmessagetageldsforthemessagetobereceived.(wildcardsareallowedfor destinationandmessagetagbutnotforcontext).inthepast,pvmapplications 3.4therearebuiltinfunctionstocreate,set,andfreecontextvalues. hadtodivideupthemessagetagspacetomimiccontextcapabilities.withpvm thedynamicgenerationanddestructionoftasks.andbydeningthatallpvm taskshaveabasecontextbydefault,allexistingpvmapplicationscontinueto workunchanged.thecombinationofthesefeaturesallowsparalleltoolsdeveloperstocreatevisualizationandmonitoringpackagesthatcanattachtoexisting Bydeningthecontexttobesystem-wideunique,PVMcontinuestoallow predicatedontheexistenceofasimilarifnotidenticalcommunicationcontext PVMapplications,extractinformation,anddetachwithoutconcernaboutinterferingwiththeapplication. paradigmtopvm3.4. GPM'sabilitytodynamicallyplug-inmiddle-layertoolsandapplicationsis handlershasbeenraiseduptotheuserlevel.toaddamessagehandler,anapplicationtaskcalltrollingthevirtualmachine.inpvm3.4theabilitytodeneanddeletemessage PVMhasalwayshadmessagehandlersinternally,whichwereusedforcon- Thereafter,wheneveramessagearrivesatthistaskwiththespeciedsource, functionispassedtheanidsothatitmayunpackthemessageifrequired.pvm messagetag,andcommunicationcontext,thespeciedfunctionisexecuted.the handler_id=pvm_addmhf(src,tag,context,function); one-sidedcommunication,activemessages,applicationsthattriggerotherapplicationsoncertainevents,faultrecoverytoolsandschedulers,andsoon.for 3.4placesnorestrictionsonthecomplexityofthefunction.Itisfreetomake example,insteadofanerrorinsideanapplicationprintinganerrormessage, systemcallsorotherpvmcalls. theeventcouldbemadetoinvokeaparalleldebuggerfocusedontheareaof Withthefunctionalityprovidedbypvmaddmhf()itispossibletobuild handlerisinvoked. tiontodynamicallyadaptandtakeonnewfunctionalitywheneveramessage theproblem.anotherexamplewouldbeadistributeddataminingapplication searchingtasks.theexistenceofpvmaddmhf()allowstaskswithinanapplica- thatndsaninterestingcorrelationandtriggersaresponseinalltheassociated extendedtoincludetheunderlyingsystemaswellastheusertasks.onecould InGPMthisabilitytodynamicallyaddnewfunctionalitywillhavetobe

10 envisionamessagehandlerdenedinsidethevirtualmachinedaemonsthat agentscouldretrievethemoduleanduseanothermessagehandlertocausethe therequestedsoftwaremodulefromwebrepositories.thesetrusted\children" whentriggeredbytheapplicationwouldspawnointelligentagentstoseekout isoftenonmakingtheirexistenceasshortaspossible,i.e.decreaselatencyand daemontoloadthemodule,incorporatingitsnewfeatures. increasebandwidth.therearemanysituationsindistributedapplicationsseen todayinwhichprogrammingwouldbemucheasieriftherewasawaytohave persistentmessages.thisisthepurposeofthenewmessageboxfeaturein Inatypicalmessagepassingsystem,messagesaretransitiveandthefocus associatedname.copiesofthismessagecanberetrievedbyanypvmtaskthat PVM3.4.TheMessageBoxisaninternaltuplespaceinthevirtualmachine. pvmgetmboxinfo()canbeusedtondthelistofnamesactiveinthemessage knowsthename.andifthenameisunknownorchangingdynamically,then TaskscanuseregularPVMpackroutinestocreateanarbitrarymessageand BoxṪhefourfunctionsthatmakeuptheMessageBoxinPVM3.4are: thenusepvmputinfo()toplacethismessageintothemessageboxwithan index=pvm_putinfo(name,msgbuf,flag) pvm_recvinfo(name,index,flag) deletethismessage,doesthisnameallowmultipleinstancesofmessages,does Theflagdenesthepropertiesofthestoredmessage,suchas,whoisallowedto pvm_delinfo(name,index,flag) aputoverwritethemessage?theflagargumentalsoallowsextensionofthis pvm_getmboxinfo(pattern,names[],structinfo[]) messageboxes. interfaceaspvm3.4usersgiveusfeedbackonhowtheyusethefeaturesof tothelinda[6]system,thegranularityofthepvm3.4implementationisbetter suitedtolargegraineddatastorage. Whilethetuplespacecouldbeusedasadistributedsharedmemory,similar laboration.adebuggingtoolretrievesamessageleftbyaperformancemonitor resourcemonitor.anewteammemberlearnshowtoconnecttoanongoingcol- toolspontaneouslycomestolifeandndsoutwhereandhowtoconnecttoa largedistributedsimulation.aschedulingtoolretrievesinformationleftbya HerearejustafewofthemanypotentialMessageBoxuses.Avisualization ofthesecapabilitiesaredirectlyapplicabletothegpmenvironment,andsome thatindicateswhichofthethousandsoftasksismostlikelyabottleneck.many methodtohavepersistentmessageswillbeapartofthegpmdesign. boxestotheparallelvirtualmachineenvironmentallowsdeveloperstotake abigleapforwardinthecapabilitiesoftheirdistributedapplications.pvm 3.4isausefultoolforthedevelopmentofmuchmoredynamic,faulttolerant distributedapplications,butasillustratedabove,itisalsoatestbedforsoftware Theadditionofcommunicationcontexts,messagehandlers,andmessage andfeaturesthatwillmakeupthenextgenerationheterogeneousdistributed computingenvironment.

11 6Conclusions Thispaperhaspresentedthebasicconceptsforapluggablevirtualmachine. Thepluggabilitywasdescribedatthreelevels:lowlevelcommunication,middleleveltools,andvirtualmachinemerging,splitting,andmigration.Thedesign thehighesttechnicalhurdletojumpistoenableecientmigrationoftasks andphilosophyofwhatisnextweremotivatedbylessonslearnedoverthepast andvirtualmachinesbyhandlingtaskrenamingandbeingabletoconsistently astherstkeystepstobuildingthegeneralizedpluggablemachine.clearly, anditsnewfeatureslikecontext,messagehandlersandmessageboxesareseen fewyearswiththepvmenvironment.thelatestreleaseofpvmversion3.4 representthecongurationofthevirtualmachine.pluggabilityisbeingexplored References roamingapplications. toimproveperformance,providegreaterexibility,andopenupnewclassesof 1.A.Beguelin,J.Dongarra,G.A.Geist,W.Jiang,R.Manchek,V.Sunderam,PVM: 2.G.A.Geist,J.A.Kohl,P.M.Papadopoulos,\CUMULVS:ProvidingFault- 3.G.A.GeistII,J.A.Kohl,R.Manchek,P.M.Papadopoulos,\NewFeaturesof ParallelVirtualMachine,AUser'sGuideandTutorialforNetworkedParallelComputing,MITPress,Cambridge,MA, J.A.Kohl,G.A.Geist,\XPVM1.0User'sGuide,"TechnicalReportORNL/TM- Tolerance,VisualizationandSteeringofParallelApplications,"SIAM,August J.A.Kohl,G.A.Geist,\ThePVM3.4TracingFacilityandXPVM1.1,"Pro- PVM3.4,"1995EuroPVMUser'sGroupMeeting,Lyon,France,September ,OakRidgeNationalLaboratory,OakRidge,TN,April, N.CarrieroandD.Gelernter.\LindaandMessagePassing:WhatHaveWe 29),HeterogeneousProcessingMinitrackintheSoftwareTechnologyTrack,Maui, ceedingsofthe29thhawaiiinternationalconferenceonsystemsciences(hicss- 7.J.PruyneandM.Livny,\InterfacingCondorandPVMtoHarnesstheCyclesof Hawaii,January3-6,1996. Learned?"TechnicalReport984,YaleUniversityDepartmentofComputerScience, 8.MessagePassingInterfaceForum.MPI:Amessage-passinginterfacestandard."In- Sept WorkstationClusters",JournalonFutureGenerationsofComputerSystems,Vol. 12, R.VanRenesse,K.P.Birman,S.Maeis,\Horus,aFlexibleGroupCommunicationSystem",Comm.ACM,April, NetscapeNavigator3.0,NetscapeCommunicationsCorporation. ternationaljournalofsupercomputerapplicationsandhighperformancecomput- ing",internationaljournalofsupercomputerapplicationsandhighperformance Computing,Volume8,Number3/4,1994. ThisarticlewasprocessedusingtheLATEXmacropackagewithLLNCSstyle