CommunicationServices. JohnBresnahan,IanFoster,JosephInsley,BrianToonen,StevenTuecke

Transcription

1 CommunicationServices foradvancednetworkapplications JohnBresnahan,IanFoster,JosephInsley,BrianToonen,StevenTuecke MathematicsandComputerScienceDivision ArgonneNationalLaboratory Argonne,IL,U.S.A. AbstractAdvancednetworkapplicationssuchas remoteinstrumentcontrol,collaborativeenvironments,andremotei/oaredistinguishedfrom\traditional"applicationssuchasvideoconferencingby theirneedtocreatemultiple,heterogeneousows withdierentcharacteristics.forexample,asingleapplicationmayrequireremotei/oforraw datasets,sharedcontrolsforacollaborativeanalysissystem,streamingvideoforimagerendering data,andaudioforcollaboration.furthermore, eachowcanhavedierentrequirementsinterms ofreliability,networkqualityofservice,security, etc.wearguethatnewapproachestocommunicationservices,protocols,andnetworkarchitecture arerequiredbothtoprovidehigh-levelabstractions forcommonowtypesandtosupportuser-level managementofowcreationandquality.wedescribeexperienceswiththedevelopmentofsuchapplicationsandcommunicationservices. Keywords:Networkapplications,communicationlibraries,Nexus,CollaboratoryInteroperabilityFramework(CIF) 1Introduction Advancednetworkapplicationssuchasremoteinstrumentcontrol,collaborativeenvironments,andremoteI/Oaredistinguished from\traditional"networkedapplicationssuch asvideoconferencingbytheirneedtomaintainmultiple,heterogeneousowswithdifferentcharacteristics.forexample,asingle applicationmayrequireremotei/oforraw datasets,sharedcontrolsforacollaborative analysissystem,streamingvideoforimagerenderingdata,andaudioforcollaboration.furthermore,eachowcanhavedierentrequirementsintermsofreliability,networkquality ofservice,security,andsoon.forexample, inatele-immersivecollaborativeenvironment, trackinginformationneednotbepropagated reliablybutcanoftenbenetfrommulticast, whiledatabaseupdatesrequirereliablecommunicationbutcannotalwaysusemulticast capabilities.mechanismsarerequiredthatallowbothconvenientspecicationoftheseapplicationsandecientexecutioninavariety ofenvironments. Historically,suchapplicationseitherhave usedasinglelow-levelcommunicationprotocolforallows(e.g.,tcp/ip[1,2,3])orhave usedamixtureofdierent,oftenspecialized APIsfordierentows[4,5,6].Neitherapproachisideal.Therstapproachleadstoa protocolthatisgoodforsomepurposesbut lessidealforothers;inthesecond,program complexityisincreasedandportabilityishard toachieve.inbothcases,avarietyofissues relatingtothecoordinationofmultipleows (e.g.,synchronizationofaudioandvideo,prioritizationofdierentows)havetypicallynot beenaddressedatall. Webelievethatsuchapplicationsrequire moresophisticatedcommunicationservices with,ideally,thefollowingcharacteristics: AuniformAPIallowsbothhigh-level specicationofcommunicationstructure andindependentspecicationofcommunicationmechanisms.

2 Avarietyofowtypesandinteraction modelsaresupported,includingvarious typesofstreamingdata,sharedcontrols, anddatabaseupdates. Supportisprovidedforautomaticand user-managedmanipulationofowcharacteristics,suchasprivacy,integrity,compression,andnetworkquality-of-service. Supportisprovidedforthecoordinationandmanagementofensemblesof ows,enabling,forexample,programmercontrolledprioritization,synchronization, andaggregationofowsinvarioustypes ofnetwork. Integratedinstrumentationallowsuserlevelmonitoringofowqualityandnotication,forthepurposeofadaptation, ofviolationsinperformancecontracts. Ourviewsonthesetopicshavebeenshaped byourexperiencesdevelopingbothadvanced networkedapplicationsandcommunicationlibrariesdesignedtosupportsuchapplications. Inthispaper,wereviewtheseexperiences,focusingononeparticularlydemandingapplicationandthreedierentcommunicationlibraries. 2MotivatingExample Weuseasingleexampleapplicationtomotivatesomeofthediscussionthatfollows.Our chosenapplicationistypicalofanemerging classofso-called\grid"applicationsthatcouplegeographicallydistributedresourcesofvarioustypestocreatevirtualdeviceswithunique capabilities[7].inthiscase,theresources inquestionareaspecializedscienticinstrument,theadvancedphotonsource(aps)at ArgonneNationalLaboratory,usedtoprobe theinteriorstructureofmaterialsatverysmall scalesusing,inthiscase,atechniquecalled computedmicrotomography(cmt);asupercomputer,usedtoreconstruct3-dmaterial densitiesfromthesequenceof2-drawdata \slices"providedbytheinstrument;anda numberofbothhigh-endandlow-enddisplay devices,usedtosupportcollaborativeanalysisofthereconstructeddata.theseresources workinconcerttoenablequasi-real-timereconstructionandcollaborativeanalysisofaps data,sothatusersatremotesitescanbe manipulatinganddiscussingthree-dimensional imagedatajustminutesafterdatacollection begins[8]. 2.1VisualizationCapabilities Thevisualizationsystemusesaspecialized graphicsutility,thesgivolumizerlibrary,to producehigh-quality,3-dvolume-renderedimagesofthedataset.asillustratedinfigures1 and2,thisdatacanbedisplayedinmultipleways,dependingonthecapabilitiesofthe user'svisualizationenvironment: Virtualrealitydisplay:CAVEorImmersaDesk(Idesk)immersivedisplaydevicessupporthigh-quality,3-Dstereodisplay.Theusercancontrolthedisplaythroughtheuseofacontrolpanel providedwithinthevirtualenvironment. Thiscontrolpanelallowsoperationssuch asrotation,volumecropping,andassignmentofcolorandopacitytodatasetvoxels. Desktopdisplay:Ratherthanrewriting thevolumerenderertorunonlesscapable desktophardware,weuseremotelyrenderedvideoforthedesktopdisplay.the high-qualityimagesproducedbythesgibasedvolumerenderinghardwareand softwarearecaptured,compressed,and sentoverthenetworkfordisplayusing standardnetworkvideodisplaytools.a Javacontrolpanelsupportsdesktopcontroloftherenderingprocess. Webdisplay:Thesamesoftwarecan alsocaptureindividualimagesfromthe sceneandputthemonawebpage,fora verylow-endsolutionthatprovideshighqualityimages.

3 Figure1:AscreenshotoftheImmersaDesk takenduringacollaborativesessionwithtwo users. Shared-statemechanismsareusedtolink thevirtualrealityanddesktopdisplays,sothat usersatdierentlocationsandondierent systemscancooperateinthesteeringofthe volume-renderingprocess. 2.2CommunicationRequirements TheapplicationistypicalofadvancedGridapplicationsinitssimultaneoususeofmanyunderlyingcommunicationstructures: Thetransferof2-DimagesfromtheAPS tothesupercomputer,andof3-ddatasets fromthesupercomputertothevisualizationsystem,requireshigh-bandwidth(10s ofmb/stoday,gb/sormoreinthefuture),unicastcommunication. Communicationwithintheparallel3-Dreconstructionprogramrequireshighbandwidthandlow-latencycommunication,as istypicallyavailableonparallelsupercomputersthroughthemessagepassinginterface(mpi)orshared-memorylibraries. Thevideostreamusesstandard,unreliableIPmulticastprotocols(e.g.,RTP andrtcp).wecommonlyused800x600 pixelh.261video,whichrequiresapproximately300kbpsofbandwidthwhenthe Figure2:Ascreenshotofalow-resolution graphicsworkstationtakenduringacollaborativesessionwithtwousers. imageisrapidlychanging;signicantly higherresolutionisdesirable. Audiostreamsbetweenthecollaborators canalsousestandard,unreliableipmulticastprotocols.audiorequireslessbandwidththanvideobutismoresusceptible toqualitydegradationduetolostpackets. Thecommunicationbetweenthecontrol panelsofthecollaboratorsusesbothreliableandunreliablemulticastprotocols. Unreliableprotocolscanbeusedforincrementalupdatesofthepanels,forexamplewhileauserisdraggingaslideron thepanel.reliableprotocolsareusedto ensurethatallparticipantsaresynchronized,forexamplewhentheuserreleases theslideronthepaneltosetanalvalue. Hence,eveninthisrelativelysimpleapplicationweseeaneedfortensofows(ifmultiplecollaboratorsareparticipating)withwidely varyingcharacteristics.otherapplicationscan placeyetmorecomplexdemandsonacommunicationsinfrastructure.forexample,defanti andstevensidentifynineowtypesincollaborativedesignapplications[9].

4 3Nexus Theprecedingsectionoutlinesthewidevarietyofcommunicationmodalitiesthatmust besimultaneouslysupportedinanadvanced networkapplicationsuchasthecmtcollaborativeanalysisandvisualizationsystem.in general,weobservethatthelow-levelmethod usedtoachieveacommunicationcanvaryaccordingtowherecommunicationisbeingperformed,whatisbeingcommunicated,orwhen communicationisperformed[10]. Currently,developersofsuchapplications mustprogramtoavarietyofapisforthese variousows(e.g.,tcpsockets,ipmulticast, reliablemulticastlibraries,mpi)andmust knowmyriaddetailsabouteachapiinorder toachievegoodperformance(e.g.,tcpsocket buersizes).thisburdenwillonlyincreaseas theseapplicationsaddsuchfeaturesassecurity andnetworkqualityofservice. Webelievethatthesolutiontothisproblemistoallowfortheseparatespecication ofthecommunicationstructureofanapplicationandthemethodsusedtoachievethat communication.thenexuscommunicationlibrary[11,10]representsbothanambitiousexperimentinthisregardandasubstantialsoftwaresystemthathasbeenusedinmanytool developmentandapplicationprojects,rangingfromparallellanguagecompilerstohighlevelcommunicationlibrariesanddistributed performanceprolingsystems.nexusalso servesasthecommunicationcomponentofthe Globustoolkit. Nexusprovidessimple,generalwaysforexpressingcommunication,basedontheabstractionsofstartpoints,endpoints,communication links,andremoteservicerequests.theseabstractionsareabletoexpressthewidevarietyofcommunicationmodalitiesdescribed above.thenexusimplementationmapsthese abstractionsontoawidevarietyofunderlying communicationmethods. Nexusprogramsbindcommunicationstartpointsandendpointstoformcommunication links.ifmultiplestartpointsareboundtoan endpoint,incomingcommunicationsareinterleaved,inthesamemannerasmessagessent tothesamenodeinamessagepassingsystem.ifastartpointisboundtomultipleendpoints,communicationresultsinamulticast operation.astartpointcanbecopiedbetween processors,causingnewcommunicationlinks tobecreatedthatmirrorthelinksassociated withtheoriginalstartpoint.hence,startpoints canbeusedasglobalnamesforobjectsthat canbecommunicatedandusedanywhereina distributedsystem. Acommunicationlinksupportsasingle communicationoperation:anasynchronousremoteservicerequest(rsr).anrsrisapplied toastartpointbyprovidingaprocedurename andadatabuer.foreachendpointlinked tothestartpoint,thersrtransfersthedata buertotheaddressspaceinwhichtheendpointislocatedandremotelyinvokesthespeciedprocedure,passingtheendpointandthe databuerasarguments.alocaladdresscan beassociatedwithanendpoint,inwhichcase startpointsassociatedwiththeendpointcanbe thoughtofas\globalpointers"tothataddress. Anadvantageofthestartpointconstructin adistributedcomputingenvironmentisthat thestartpointcanbeusedtoencapsulatenot onlyinformationaboutwherecommunication shouldbeperformed,butalsohowtocommunicate.dierentcommunicationmethodscan beassociatedwithdierentcommunication links,withselectionbeingeitherautomaticor userguided.thecommunicationmethodscurrentlysupportedbynexusarelistedintable1. Inaddition,amessagetransform,orlter, canbeappliedtoeachcommunicationlink. Thisfeatureallowsoperationssuchascompression,encryption,andprolingtobespecied andperformedonaper-linkbasis. OurexperienceisthattheNexusabstractionscapturenicelynumerouscommunication structuresandmapcleanlyontoavarietyof underlyingprotocolsandcapabilities(e.g.,securityandqualityofservice).theonelimitationofwhichweareawarerelatestosupport formulticastcommunication.thenexusapi forcreatingstartpointsandendpointsiscurrentlybettersuitedforthecreationofunicast

5 Table1:CommunicationmethodssupportedbyNexus Name Description Local Reliableorderedunicastwithinasingleprocess SysV Reliableorderedunicastbetweenprocesses onthesamecomputer,viasystemvsharedmemory MPI/MPL/INXReliableorderedunicastbetweenprocesseson dierentnodesofasingledistributed-memorycomputer, vialow-levelcommunicationlibraries TCP Reliableorderedunicast UDP Unreliable,unorderedororderedunicast IPmulticastUnreliable,unorderedororderedmulticast XTP Reliable,source-orderedmulticast Totem Reliable,totallyorderedmulticast communicationthanformulticastcommunication.inparticular,thereiscurrentlynowayto directlybindastartpointtomulticastgroup. Instead,onemustrstcreateanendpointthat isboundtothemulticastgroup,andthenbind astartpointtothatendpoint.thiscanbeannoyingforprocessesthatonlywanttosend toaparticularmulticastgroup.thisproblem canbecorrectedbyaddingthecommunication linkmanagementtotheapiandthenallowing startpointsandendpointstodirectlybindto thecommunicationlink.therefore,multicast communicationwouldbesetupbycreatinga communicationlinkwithmulticastproperties andbybindingoneormorestartpointsand endpointstothatcommunicationlink. 4CIFCommLibrary WhileNexusdemonstratesthatauniforminterfacecanbeconstructedforavarietyofprotocolsandmessaginglibraries,thisinterface (whichwasoriginallydesignedforusebycompilers)istoolowlevelforallbutthemost expertprogrammer.hence,inamorerecent projectwehavedevelopedahigher-levelinterfacethatmakesthesameprotocolsavailable inamoreconvenientform.thisinterface,developedaspartofthedoe2000collaboratory InteroperabilityFramework(CIF)project,is termedcifcomm. TheCIFCommdesignemploysobjectorientedconceptsasameansofencapsulating protocoldetails.theinterfaceconsistsofthree coreclasses:abstractconnectionandlistener classes,andafactoryclasstoinstantiatethem. Theabstractconnectionclassprovidesa simpleinterfaceforsendingandreceivingmessages.itisfromthisclassthatallprotocolspecicconnectionclassesarederived.as theclassnameandcapabilitiesimply,eachof theprotocol-specicimplementationsprovide aconnection-oriented,message-passingstyle viewofthecommunicationirrespectiveofthe underlyingprotocol.hence,applicationscan switchbetweendierentprotocolssimplyby instantiatingadierentclass. Theabstractlistenerclassallowstraditional client-serverapplicationstoimplementserversidefunctionalityusingcifcomm.oncea classhasbeeninstantiated,thelistenerwaits forconnectionrequestsfromremoteconnectionobjects.theseconnectionrequestsare transformedintolocalconnectionobjectswhen theapplicationrequeststhenextincomingconnectionfromthelistener. Inreality,anapplicationneverinstantiates aprotocol-specicconnectionorlistenerclass. Instead,itmakesarequesttothefactoryclass, whichperformstheinstantiationonitsbehalf.tofacilitateprotocolindependenceinthe factory,allrequestsaremadeusingurlsin whichtherstcomponentspeciestheproto-

6 coltobeused.thisprotocolinformationis usedtoinstantiatethecorrectconnectionor listenerclass,whichisthenpassedtheremainderoftheurl. Atpresent,bothC++andJavabindings havebeenimplementedforthecifcomminterface,supportingtcp,udp,ipmulticast, andtotem.inaddition,xtpissupportedin thec++implementationandwillsoonbesupportedinjavaaswell.withtheseprotocols, theapplicationhasthefullcrossproductof reliable/unreliableandunicast/multicastcommunicationavailabletoit. Todate,CIFCommhasbeenusedintwo applications:amulti-usercameracontroller systemdevelopedbydebagarwalatlawrence BerkeleyNationalLaboratoryandtheCIF SharedStatelibrary(describedbelow),afundamentalpieceoftheCMTapplication. 5CIFSharedStateLibrary Collaborativeapplicationsrequiremechanisms formaintainingandsynchronizingupdatesto theshareddataelementsthatrepresentsthe stateoftheworldinwhichcollaborationoccurs.forexample,inthecmtdataanalysis systemthissharedstateincludesthevarious controlsfortheremotevisualizationsystem: pointofview,colormap,andsoforth.wehave usedcifcommtoimplementashared-state abstractionlibrary,cifsharedstate,which wasthenusedtoimplementthecmtcollaborativedataanalysissystem. TheSharedStatecomponentofCIFallows forsharedcontrolofabstractstatesincollaborativespaceacrossmultipleplatforms.aninitialimpetusforthecreationoftheshared-state librarywastoallowforsharedcontrolof\widgets"acrossdierentcomputerarchitectures andlanguages.(othersystems,inparticular NCSA'sHabanero,supportashared-stateabstraction,butonlywithinaJavaframework.) Ifsharedcontrolofsliders,buttons,andother arbitrarycomponentscouldbeestablished,a graphicalprogramrunningonahigh-endresourcecouldbecontrolledremotelyfroma simpler,moreaccessiblecomputer.thecmt collaborativevisualizationapplicationusesthe CIFSharedStateLibrarytodojustthat. TheCIFSharedStatelibraryisanobjectorientedAPIwithbothC++andJavaimplementationsthatallowsforsharedcontrol ofabstractstates.(ajavaimplementation isprovidedforportabilityandac++implementationforuseonhigh-endplatformsandin C-basedapplications;acommonNexus-based wireprotocolallowsforinteroperability.)the abstractstatescanbeimplementedasgui components(sliders,buttons,toggles)ormore simplyasarraysofdataprimitives(integers, oatingpointnumbers,bytes).tocreatea sharedstate,theuserneedsonlytoprovidea mechanismforpackingandunpackingitscurrentvaluesintoacifsharedstate\serial" objectviaconvenientmethodsprovidedbythe API. Theuseofshared-stateinformationrather thancollectivecontrolfunctionsasourbasicprimitiveprovedextremelyeectiveinthe CMTapplication.Wewereabletocreateunorthodoxvisualcomponentsthatprovidedno controltotheuserbutwereusedtodisplay usefulinformation,suchashistogramgraphs, colorbandwidthltercurves,andimagesof alloftheuserscurrentlyparticipatinginthe collaborativesession.thislayerofabstraction betweenshareddataandvisualcontrolalsoallowedustocoupledierentvisualcomponent packageswiththemessagingstructure:ajavabasedcontrolguifordesktopclientsandaset of3-dwidgetsforuseinthecave. 6Conclusions Emergingnetworkedapplicationsinvolvemultipleowswithdierentandtime-varyingrequirementsforlow-levelprotocols,security, performance,andsoon.wehavearguedthat thecommunicationservicesthatweprovide tosupporttheseapplicationsneedtorecognizethisfactandprovideexplicitsupportboth fortheseparatespecicationofcommunicationowandcommunicationmethodandfor

7 themanagementofensemblesofowsinan integratedfashion.wehavedescribedthree softwaresystemsthatwehavedevelopedto addresstherstoftheseconcerns,namely, thenexuscommunicationlibraryandthecif CommandCIFSharedStatelibraries.Applicationexperienceswiththesesystemsindicate thattheseparatespecicationofcommunicationsstructureandmethodisindeeddesirable. Infuturework,wewilladdresstheassociation ofquality-of-serviceattributeswithowsand themanagementofowensembles. Acknowledgments Wegratefullyacknowledgethemanycolleagues whohavecontributedtothedevelopmentof Nexus,theCIFlibraries,andtheCMTapplication,inparticularGregorvonLaszewskiand SteveWangatArgonne;CarlKesselmanand MeiSuatUSC/ISI;DebAgarwalatLBNL; andbrucemahatsnl/ca.thisworkwas supportedinpartbythemathematical,information,andcomputationalsciencesdivision subprogramoftheoceofcomputationaland TechnologyResearch,U.S.DOE,underContractW Eng-38;byDARPAundercontractN C-8523;andbyNSF. References [1]C.ShawandM.Green.TheMR toolkitpeerspackageandenvironment.in ProceedingsoftheIEEEVirtualReality AnnualInternationalSymposium.IEEE ComputerSocietyPress,1993. [2]K.Birman.Theprocessgroupapproach toreliabledistributedcomputing.communicationsoftheacm,36(12):37{53, [3]C.CarlssonandO.Hagsand.DIVEamulti-uservirtualrealitysystem.In ProceedingsoftheIEEEVirtualReality AnnualInternationalSymposium.IEEE ComputerSocietyPress,1993. [4]J.Mandeville,J.Furness,andT.Kawahata.Greenspace:Creatingadistributed virtualenvironmentforglobalapplications.inproceedingsoftheieeenetworkedvirtualrealityworkshop.ieee ComputerSocietyPress,1995. [5]M.Roussos,A.Johnson,J.Leigh, C.Valsilakis,C.Barnes,andT.Moher. NICE:Combiningconstructionism,narrative,andcollaborationinavirtuallearningenvironment.ComputerGraphics, 31(3):62{63,August1997. [6]M.MacedoniaandM.Zyda.Ataxonomyfornetworkedvirtualenvironments. InProceedingsofthe1995Workshopon NetworkedRealities [7]I.FosterandC.Kesselman,editors.The Grid:BlueprintforaFutureComputing Infrastructure.MorganKaufmannPublishers,1999. [8]G.vonLaszewski,I.Foster,J.Insley,J.Bresnahan,C.KesselmanM.Su, M.Thiebaux,M.Rivers,I.McNulty, B.Tieman,andS.Wang.Real-timeanalysis,visualization,andsteeringofmicrotomographyexperimentsatphotonsources. InProceedingsoftheNinthSIAMConferenceonParallelProcessingforScientic Computing.SIAM,1999. [9]T.DeFantiandR.Stevens.Teleimmersion.In[7],pages131{156. [10]I.Foster,J.Geisler,C.Kesselman,and S.Tuecke.Managingmultiplecommunicationmethodsinhigh-performance networkedcomputingsystems.journal ofparallelanddistributedcomputing, 40:35{48,1997. [11]I.Foster,C.Kesselman,andS.Tuecke. TheNexusapproachtointegratingmultithreadingandcommunication.JournalofParallelandDistributedComputing,37:70{82,1996.