ADistributedApplicationsManagementTestbed. MichaelJ.Katchabaw,StephenL.Howard, AndrewD.Marshall,andMichaelA.Bauer. TheUniversityofWesternOntario

Transcription

1 EvaluatingtheCostsofManagement: ADistributedApplicationsManagementTestbed MichaelJ.Katchabaw,StephenL.Howard, AndrewD.Marshall,andMichaelA.Bauer DepartmentofComputerScience TheUniversityofWesternOntario Abstract Intoday'sdistributedcomputingenvironments,usersaremakingincreasingdemandson thesystems,networks,andapplicationsthey use.usersarecomingtoexpectperformance, faulttolerance,conguration,security,andaccountingservicescomparabletothoseontraditionalcentralizedsystems.meetingtheseexpectationsimpliestheneedformanagement. Whilesomesuccesseshavebeenrealizedinthe areasofsystemandnetworkmanagement,only preliminaryinvestigationshavebeenconducted onmanagementofdistributedapplications. Atthisstage,littleisknownofthecostsassociatedwiththistypeofmanagement.To assessthesecosts,itisnecessarytocollect dataabouttheperformanceandoperationof distributedapplicationsandtheirmanagement systems.furthermore,itisimportanttocollectthisdataundervaryingconstraints,congurations,andmanagementassumptions.inordertodothis,wehavedevelopedadistributed applicationsmanagementtestbed.wepresent ourinitialrequirementsforthetestbed,itsdesign,andaprototypeimplementation.wedescribesomepreliminaryresultsandanalysis obtainedthroughitsuse.finally,weidentify itslimitsbasedonourexperiencetodateand outlineourplanstoevolvethetestbedtoaccommodatefurtherexperimentation. ThisresearchworkissupportedbytheIBMCentreforAdvancedStudiesandtheNaturalSciencesand EngineeringResearchCouncilofCanada.TheIBM contactforthispaperisjacobslonim,centreforadvancedstudies,2g/894,1150eglintonavenueeast, NorthYork,Ontario,CanadaM3C1H7. 1Introduction Withtheincreasedcomplexityofdistributed systemscomesagreaterneedforautomated toolstoassistinthemanagementofnetworks, systemresources,andapplicationprocesses. Whilesignicantexperiencehasbeengainedin theareasofnetworkandsystemmanagement, littlehasbeenaccomplishedattheapplication level. Thefocusofmuchofourongoingresearch hasbeenintheareaofdistributedapplications management[2,3].wehavedevelopedand continuetouseapplicationsmanagementsystemprototypesbasedonbothosiandsnmp protocols.wehavebecomeinterestedinthe practicalissuesofapplicationsmanagement thatmayimpairitsacceptance;issuessuch asthecostofmanagement(forexample,the performanceoverhead)andhowmanagement canbedoneeciently.thispaperdescribesa projectmotivatedbythisinterest. Overheadassociatedwithmanagementof distributedapplicationsisunavoidable.instrumentationcodeaddedtoeachoftheapplicationcomponentsinsupportofruntimemonitoringandcontrolwillaecttheapplication's performancedirectlybecauseoftheextrainstructionsintheexecutionpath.inaddition, themanagementsystemconsistsofcommunicatingprocessesthatcompeteforthesame memory,cpu,andi/oresourcesthatareused bytheapplication. Abasicquestionwearetryingtoanswer is:whatisthepricewepayinoverallapplicationperformanceforthismanagementand whatstepscanbetakentominimizeit?to 1

2 addressthisproblem,itisnecessarytomonitor thebehaviorofmanagementsystemsandtheir managedapplicationsundervariousworkloads andcongurations.workhasbeguntoidentify thesecosts[1]. Oncethecostsareidentiedandunderstood, wecanbeginminimizingthem.thereare manyparametersinthedesignanddeploymentofthemanagementsystemthatpotentiallyinuencethecostofmanagement;forexample,controllingcomponentdeployment,togglingfunctionsonando,adjustingthreshold valuestocontroleventgenerationconditions andrates,conguringevent-lteringmechanismstoreducethetracow,controllingthe quantityandnatureofinformationineventreports,andsoon. Toaddressthisissue,wehavedevelopeda distributedapplicationsmanagementtestbed thatisindependentofthemanagementsystemanditsmanagedapplications.thetestbed isatoolthatenablesexperimentstoobserve andanalyzetheperformanceofbothmanagementsystemcomponents(suchasmanagers andagents)andmanagedapplicationcomponents(suchasinstrumentedprocesses).the resultsofsuchexperimentsprovideinsightinto howmanagementcanbeperformedmosteectively. Inthispaper,wepresentasetofrequirementsforthedistributedapplicationsmanagementtestbedanddescribeatestbeddesign thatmeetstheserequirements.wediscussthe testbedprototypeandsomepreliminaryresults obtainedinanosimanagementenvironment. Thepaperconcludeswithaprojectstatusreport,identifyingthelimitsofourexperienceto dateandourplansforfuturework. 2Requirements Thegoalwastodevelopamanagementsystem testbedthatcanbeusedtoassesstheimpactof managementonsystemresourcesandonmanagedapplications.weidentiedthefollowing keyrequirementsforthetestbed: Thetestbedshouldallowanexperimentto becontrolled.thisincludestheabilityto startandstopanexperiment,tolaunch andterminatecomponentsofthemanagementsystem,andtolaunchandterminate componentsofthemanagedapplication. Thetestbedshouldbeexible,allowingrecongurationforavarietyofexperiments. Itmustmaintaincongurationinformationaboutthemanagementsystemand managedapplications. Thetestbedshouldprovideawaytocollectdataaboutthebehaviorandperformanceofthemanagementsystemand managedapplications. Simplestatisticalanalysisofcollecteddata shouldbeprovidedbythetestbed. Higher-levelapplicationsshouldbeableto usethetestbedthroughawell-denedapplicationprogramminginterface(api). Thetestbeditselfshouldhaveminimal interferencewiththeperformanceofthe managementsystemandthemanagedenvironment.thetestbedmustoperateoutsidetheexecutionpathofthesystembeing tested;itmustnotrequireintrusiveinstrumentationorcodemodication.allinformationmustbegatheredatarm'slength throughtheuseofoperatingsystemservicesandexistingobservablebehavior. Toasgreatadegreeaspossible,the testbeddesignshouldbeindependentof thesystembeingtested.itshouldnot makeassumptionsaboutthetypeofmanagementsystemorthemanagedenvironment. 3TestbedArchitecture Wedevelopedagenericarchitecturetomeet ourkeyrequirements.inthissection,wedescribethekeycomponentsofthearchitecture andaninterfaceforaccessingtestbedservices. 3.1ComponentOverview Thedistributedapplicationmanagement testbedarchitectureshowninfigure1isthe basisforourresearchandprototypedevelopment.thisarchitectureconsistsofthetestbed 2

3 Testbed Applications Experiment Drivers Report Generators Visualization Tools... Interactions Testbed Service Interface Configuration Control Figure1:DistributedApplicationsManagementTestbedArchitecture Monitoring Analysis Distributed Applications Management Testbed itself,managedapplicationsandmanagement Manipulations systemswhoseperformanceistobeanalyzed, and andtestbedapplicationsthatmakeuseofthe Monitored Data testbed.thesecomponentsaredescribedbelow. Management Managed System 3.1.1TestbedComponents Application Environment Environment providedbythetestbedareaccessedbytestbed nents. TestbedServiceInterface:Theservices applicationsthroughthetestbedserviceinterface.thisinterfacecoordinatesactivitiesbilityinclude: onthecongurationofthemanagementsystem andmanagedapplications.areasofresponsi- Theservicesprovidedthroughthisinterfaceare nentisresponsibleformaintaininginformation discussedindetailinsection3.2. Conguration:Thecongurationcompo- Thetestbedconsistsofasetofcorecompo- andreportstotheappropriatecomponents. withinthetestbedbyroutingrequests,replies, Processrelationships whichapplicationclientsreporttowhichapplication whichmanagementcomponentsreportto whichothermanagementcomponents. servers,whichapplicationprocessesreport Processdeployment whichprocessesare executingonwhichhosts. 3 towhichmanagementcomponents,and

4 Workloadpopulation whatkindofinteractionsoccurbetweenthecomponent processes,whatinformationispassedin eachinteraction,andhowoftentheseinteractionsoccur. Control:Thecontrolcomponentprovidesa basicserviceforcontrollingexperiments.it usesdeploymentandrelationshipinformation fromthecongurationcomponentandprovides operationsto: Startanexperimentandallocatethere- sourcesrequired,basedonthedenedcon- guration. Finishanexperimentanddeallocateexperimentresources. Launchmanagedapplicationandmanagementsystemprocessesaccordingtotheir conguration. Terminatemanagedapplicationandmanagementsystemcomponentprocesses. Monitoring:Themonitoringcomponent collectsthefollowinginformationaboutthe managementsystemanditsmanagedapplications: Operationalstatusinformationaboutprocesses. Timinginformation,includingelapsed times,cputimes,andsoon. Resourceutilization,includingCPUutilization,memoryutilization,diskutilization,andnetworkutilization. Analysis:Usingcongurationinformation anddatacollectedfromexperiments,theanalysiscomponentcancompute: Basicmeasures,suchastotals,maximum values,minimumvalues,means,medians, andmodes. Variabilitymeasures,suchasstandarddeviation,samplevariance,andskewness. Thesefunctionscanbeappliedtoindividual processesorgroupsofprocesses.moresophisticatedanalyses,suchasregressionanalysis,can beperformedbyhigherleveltestbedapplications ManagedApplicationandManagementSystemEnvironments Themanagedapplicationandmanagement systemenvironmentsshowninfigure1consistofthehardwareandsoftwareresources andservicesusedbycomponentprocessesduringexecution.thisincludescomputingnodes, networks,operatingsystemservices,anddistributedprocessingservices,suchasnameservices,repositoryservices,andtransactionservices. Byinteractingwiththeirenvironments, ratherthanthecomponentprocessesthemselves,thetestbedcancongure,control,and monitortheprocesseswithminimalimpacton theiroperation.forexample,resourceutilizationstatisticscanbeobtainedthroughoperatingsystemservicesinsteadofqueryingindividualprocesses,whichwouldaddunwantedload totheprocesses TestbedApplications Usersaccesstheservicesofthetestbedthrough externaltestbedapplications.theseapplicationsinvokethetestbedoperationsthroughthe testbed'sserviceinterface.examplesofsuch applications,areshowninfigure1,andexplainedbelow. ExperimentDrivers:Carryingoutanexperimentusingthetestbedinvolvesselecting thedesiredconguration,executingtheexperiments,andanalyzingthecollectedresults. Eventhesimplestperformanceanalysisre- quiresseveralexperimentswithdierentcon- gurationstoberepeatedoverextendedperiodsoftime.repeatingtheexperimentsmanuallywouldbeextremelytedious.anexperimentdriverfacilitatesandautomatesthisprocess. ReportGenerators:Reportgenerationapplicationscanbeusedtoaggregatethedata 4

5 fromexperiments,performhigh-levelanalysis, andpresenttheresults. VisualizationTools:Experimentscanbe conguredandcontrolledmoreintuitivelyby manipulatingvisualabstractionsofthecomponentsinvolved.datacanbepresented moreconciselyandunderstandablyingraphicalforms. 3.2TestbedServices Tosupportthedevelopmentoftestbedapplications,wedenedaconsolidatedinterfacefor accessingthefourtestbedservices.thissectiondescribesthisinterfaceandtheoperations supportedbyeachservice CongurationService Thecongurationserviceprovidesoperations todeploymanagementsystemandmanaged applicationcomponents(i.e.processes),setrelationshipsbetweenthesecomponents,andset workloadsforthesecomponents.theseoperationsare: deploycomponent:deploysanapplication componentofacertaintypeonaspecied host.thisoperationdoesnotstartthecomponent,butconguresittobestartedwhenthe launchcomponentsoperationisinvoked(see Section3.2.2).Whenacomponentisdeployed, itisassignedauniqueidentierusedforreferencingthecomponentuntilitisremovedusing undeploycomponent. deploycomponent( [in:]componenttype_ttype, [in:]host_thost, [out:]componentid_tid, redeploycomponent:changesthehoston whichacomponentruns.thecomponent's identierandthenewhostarespecied. redeploycomponent( [in:]componentid_tid, [in:]host_thost, undeploycomponent:removesagiven componentfromthesetofdeployedcomponents.oncethisisdone,onlythe collectcomponentdataoperationcanbeused toreferencetheundeployedcomponent.all relationshipspreviouslydenedforthiscomponent(bysetcomponentrelationship)areremoved. undeploycomponent( [in:]componentid_tid, setcomponentrelationship: Establishesandconguresarelationshipbetween two previouslydeployedcomponents.thisrelationship canbe:client-to-server,peer-to-peer,applicationcomponent-to-managementcomponent,or managementcomponent-to-managementcomponent.arelationshipremainsvaliduntileitherremovecomponentrelationshipor undeploycomponentisinvoked. setcomponentrelationship( [in:]reltype_trelationship, [in:]componentid_tcomponent_1, [in:]componentid_tcomponent_2, removecomponentrelationship: Removesarelationshippreviouslyestablished bysetcomponentrelationship. removecomponentrelationship( [in:]reltype_trelationship, [in:]componentid_tcomponent_1, [in:]componentid_tcomponent_2, populatecomponentworkload: Setsaworkload(asequenceofservicerequests)forthegivencomponent.This workloadispopulatedaccordingtoatemplateprovidedintheoperationinvocation. ThepopulatedworkloadisuseduntileitherpopulateComponentWorkloadisinvoked again,orthecomponentisremovedusing undeploycomponent. populatecomponentworkload( 5

6 [in:]componentid_tcomponent, [in:]workloadtemplate_ttemplate, 3.2.2ControlServices Controloperationsareprovidedbythecontrol servicetostartandnishexperimentsandto startandterminatemanagementsystemand managedapplicationcomponentprocesses. startexperiment:notiesthetestbedthat anewexperimentisstarting.theappropriate resourcesareallocatedandmutualexclusionto thetestbedisprovided,sincetwoconcurrent experimentscouldhaveundesirableeectson oneanother.anidentierfortheexperiment isreturnedasaresult. startexperiment( [out:]experimentid_tid, nishexperiment:notiesthetestbedthat anexperimenthasnishedandthatpreviously allocatedresourcescanbereleased. finishexperiment( launchcomponents:startsasetofcomponentsusingthecongurationspeciedpreviouslyusingcongurationoperations.componentsarereferencedthroughtheircomponent identiers. launchcomponents( [in:]componentset_tcomponent_list, terminatecomponents:terminatesa givensetofcomponentsthathadbeenstarted usinglaunchcomponents.componentsarereferencedthroughtheircomponentidentiers. terminatecomponents( [in:]componentset_tcomponent_list, 3.2.3MonitoringService Monitoringoperationscollectdataonmanagementsystemandmanagedapplicationcomponents,bothduringandafterexperiments.A singleoperationisdened. collectcomponentdata:collectsthespeciedmonitoreddataonthegivensetofcomponents.ifacomponentisexecuting,thereturneddatareectsthecurrentstateofthe component.otherwise,thereturneddatare- ectsthestateofthecomponentattermination. collectcomponentdata( [in:]componentset_tcomponent_list, [in:]dataspecifier_tspecifier, [out:]data_tdata, 3.2.4AnalysisService Theanalysisserviceanalyzesdatacollected frommanagementsystemandmanagedappli- cationcomponents.asingleoperationisde- ned. performanalyticalfunction: Performsthespeciedanalyticalfunctionon thedataprovided.samplefunctionscompute totals,maximumvalues,minimumvalues,medians,modes,means,andothersimplestatisticalfunctions. performanalyticalfunction( [in:]functionspec_tanalytical_fn, [in:]data_tdata1,data2,data3,... [out:]data_tresult, 4PerformanceMetrics Todeterminethecostofmanaging,wecompareapplicationperformancewithandwithout management.weuseasetofperformancemetrics,whichweclassifyasresponsivenessmetrics,productivitymetrics,utilizationmetrics, anderrororfailuremetrics[8]. 6

7 4.1ResponsivenessMetrics Theresponsivenessofadistributedapplication isthetimebetweenanoperationinvocation andthecompletionofthatoperation.bycomparingtheresponsivenessofapplicationswith andwithoutmanagement,wecaneectively computetheimpactofmanagement.threeresponsivenessmetricsareused: Theresponsetimeforasingleapplication request. Thestart-to-completiontimeforaworkload(thatis,a\standard"batchofapplicationrequests). Processinitializationtime,measuredfrom thetimetheprocessstartsexecutionuntil itisreadytoperformusefulwork. 4.2ProductivityMetrics Productivity(orthroughput)metricsprovide measuresoftheamountofworkcompletedper unitoftime.thesemetricsrelatebothtothe rateatwhichapplicationoperationsareperformedandtotheamountofdatapassedbetweenapplicationcomponentswithinagiven periodoftime.thefollowingproductivitymetricsaredened: Foraserverprocess,thenumberofapplicationrequestsprocessedpersecond. Thenumberofbatchesofapplicationrequestsprocessedpersecond. Theamountofdatatransferredbetween processespersecond. 4.3UtilizationMetrics Utilizationmetricsprovidemeasuresofsystem resourceconsumptionand,assuch,areonly indirectmeasuresofanapplication'sperformance.nevertheless,itisusefultoobtainutilizationdataonarangeofsystemresourcesto giveabetterunderstandingofthetruecostof management.thefollowingutilizationmetrics areidentied: CPUutilization Memoryutilization Diskutilization I/Outilization Networkutilization Measurementsoftheseresourcescanbecollectedeitherforindividualservices,oroverthe lifespanofaprocess.dependingontheanalysisrequired,bothmeasurementscanbeuseful. 4.4ErrororFailureMetrics Anerrorisdenedasasituationinwhichan applicationoperationcannotbecompletedcorrectly.afailureisdenedasasituationin whichanapplicationoperationcannotbecompletedatall.whileerrorsandfailurestend tobeapplication-specic,theymustbeunderstoodinordertoperformanalysiscorrectly. Foreachapplication-specicclassoferroror failure,wecanmeasuremetricssuchas: Theprobabilityofanerrororfailureoccurringoneachoperationinvocation. Themeantimebetweenerrorsorfailures. The\down-time"duringwhichthemanagedapplicationisunavailableduetofailure. Theseverityoftheerrororfailure. 5Parameters Aecting Performance Theperformancemetricspresentedintheprevioussectionidentiedwhatwewanttoknow aboutperformance.thesearetheobservable outputsoftheexperiments.wenowfocus ontheparametersthatinuenceperformance. Theseparameterscorrespondtotheinputsto theexperiments. Identifyingtheperformanceparametersis criticaltotheplanning,design,andexecutionofexperimentsfordeterminingthecostsof management.theseparametersgenerallyfall intooneoftwocategories systemparameters orworkloadparameters. 7

8 5.1SystemParameters Systemparametersarethoserelatedtohardware,theoperatingsystem,middlewareservices,andsoon.Theyareoutsidetheapplicationanditsmanagementsystem,andfroman experimentationpointofview,theyareconsideredtobestatic.theseparametersinclude: SystemLevelParameters: Numberofsystemsinvolved Architectureofeachsystem Speedofeachsystem'sprocessor(s) Speedofeachsystem'scoprocessors Amountandspeedofsystemmemory Speedofdisksoneachsystem Useofhardwarecachesoneachsystem NetworkLevelParameters: Speedofthenetwork Networkreliability Networkprotocolsuse. Networktopology/conguration(subnets, routers,bridges,etc.) OperatingSystemandMiddlewareParameters: Operatingsystemoverhead Eectsofsoftwarecaches Middlewareoverhead Nature/speedofotherservices(le,time, name,etc.) 5.2WorkloadParameters Workloadparametersrepresentthevariableaspectsintheexperiments;factorsthatcanbe controlledorcanbevariedfromoneexperiment tothenext.weincludecharacteristicsofthe applicationanditsmanagementsystemaswell asaspectsofsystemstatethatareaectedby theworkloadimposedbytheapplicationand itsmanagementsystem.theseparametersinclude: LoadParameters: Systemload(numberofusers,number/typeofprocesses,etc.) Networkcongestion Resourceutilization(memory,disk,network,etc.) ManagementSystemParameters: Presence/absenceofmanagement Typeofmanagementsystem Managementsystemconguration Managementfunctionsandservices Underlyinginfrastructureandcommunication Numberofprocesses ManagedApplicationParameters: Applicationtype Processtypes(client,server,peer) Resourcerequirements Communicationmethods Conguration(numberofprocesses,locations,etc.) Requesttypes(includingratesanddistribution) Instrumentationtype ManagementParameters: Managementservicesused Number,rateanddistributionofservice requests Whethermanagementusespolling(pull style)orevent-driven(pushstyle)communication Requesttype(informationaccessorapplicationcontrol) 8

9 6Prototype Implementation Inthissection,wedescribeaprototypemanagementsystemtestbedbasedonthegeneric architecturepresentedinsection3.thisprototypeiscurrentlybeingusedtoconductexperimentstoinvestigatethecostsofdistributed applicationsmanagement. 6.1TargetEnvironment Theprototypewasdevelopedinalaboratory settingwithadozenhostsofvariousarchitectures,includingibmrs/6000,mipsrc2030, SunSPARCstation,Sun3/60,andSequent Symmetry,spreadovertwosubnets.Thissettingprovidesdiversityinprocessors,memory, deviceconguration,avorsoftheunixoperatingsystem(aix,risc/os,sunos,and DYNIX/ptx),communicationinfrastructure andmiddleware(sockets,oncrpc,dce, andcorba).therefore,alargenumberofthe systemparametersidentiedinsection5.1as xedperformancefactorsarepresent. 6.2Implementation Tomeettheexibilityandportabilityrequirementsdemandedbytheheterogeneityofthe environment,weimplementedthetestbedas astructuredcollectionofunixshellscripts. Sincethetestbedoperatesoutsidethemanagementsystemanditsmanagedapplications, mostofitsinteractionsarewiththeoperating system.theeaseofaccesstosystemservices andutilities,andthefactthattheperformance ofthetestbedisoflittleconcern,madeshell scriptsanobviouschoice. Thetestbedarchitecturewaspreservedby structuringeachserviceasacollectionof scriptsprovidingtherequiredoperations.the interfacetothetestbedisdenedbyscript namesandtheirarguments.testbedapplicationsthatinvokethesescriptscanbewritten inavarietyoflanguages(forexample,shell scripts,c,tcl,perl,orjava). Thecongurationservicestorescongurationinformationasasetofatles.This informationismanipulatedwhentheappropriatecongurationoperations(giveninsection3.2.1)areinvoked.workloadspopulated bythepopulatecomponentworkloadcongurationoperationarestoredaseitherdatales orscripts,dependingonthetypeofcomponent forwhichaworkloadisgenerated. Controlserviceoperationsaccesscongurationlesandworkloadscripts.Forexample, thestartexperimentoperationneedstoknow whichprocessestostart,wheretostartthem, andwhatworkloadstoassign. ThecollectComponentDatamonitoringserviceoperationcollectsspecieddataquerying localandremotesystemsusingunixsystem utilities.thisdataisstoredinmonitoreddata les. Finally,analysisserviceoperationsarecarriedoutbyscriptsthatprocessmonitoreddata. 6.3ManagementSystemand ManagedApplications Forinitialtestbedvalidationandexperimentation,weadaptedamanagementsystemand managedapplicationfromourpreviouswork [9].Themanagementsystem,basedonthe OSIManagementFramework[4,7],hadbeen enabledtomanagedceapplications.several DCEapplicationshavebeendevelopedandinstrumentedformanagementbythissystem ManagementSystem IntheOSIManagementFramework(seeFigure2),realresources(applicationprocesses) aremodeledbymanagedobjects.managers (alsocalledmanagementapplications),which existtoenforcemanagementpolicyonbehalf ofhumanmanagers,viewthemanagedsystem asacollectionofthesemanagedobjectsand areunawareoftherealresourceshiddenbehind thelayerofabstraction.therearemanyadvantagestothisapproach.forexample,componentgranularitycanbedened(onemanagedobjectmightmodelacollectionofreal resourcesasasingleentity). Amanagementagentholdsresponsibilityfor aparticularsetofmanagedobjects.theagent processesmanagers'requestsanddeliversevent reportstomanagers.theagentinteractswith therealresourcestoupdatemanagedobjects' states,carryoutcontrollingoperations,accept 9

10 Managed Objects Agent Manager (CMIP) Event Reports Operations Management Notifications Operations Management Real Resources Figure2:OSIManagementFramework eventnotications,andsoon,tofullthemanagementobligationsimposedonit. Communicationbetweenthemanagerand agentisdenedbytheosicommonmanagementinformationservice(cmis)[5]and thecommonmanagementinformationprotocol(cmip)[6].communicationbetweenthe agentandtherealresources(applicationprocesses)fallsoutsidetheosistandards;inour experiments,weuseddceremoteprocedure Calls(RPC)[10] ManagedApplications Inourpreviouswork[9],severalDCEapplicationsweredevelopedandinstrumentedfor managementintheosimanagementenvironment.theseapplicationsfollowaclientservermodelwithvaryingdegreesofcomplexity.themostinteresting,fromthetestbedvalidationpointofview,isadistributedtime-table schedulingapplication,whichusesadb/2 repositoryforstoringscheduleinformation.althoughconceptuallysimple,thisapplication provedtobeofadequatecomplexitytofacilitateourperformanceexperiments. Intheexperiments,thetestbedwasused tocontrolthedeploymentandworkloadof theapplicationandtocontrolitsoperation. Toenhancetheusefulnessoftheexperiments, Motif-basedclientswerereplacedwithscriptdrivenclientssowecouldcreateandapplyarticialapplicationworkload.Whilethe testbedcanprovideusefulresultsonnaturallyoccurringworkloads,wefeltthatusingcontrollableclientswouldyieldmoreaccurateresults andastrongerbasisforvalidation. 6.4TestbedApplications Forinitialtestingandexperimentation,abasicexperimentdriverandasimplereportgeneratorweredeveloped.Workhasalsobegun onavisualizationapplication.theexperiment driverusesscriptstoguidetheexecutionofseveralexperimentsinsequence.tocompleteeach phaseofanexperiment,thedrivermakesuse oftheappropriateservicescriptsprovidedby thetestbed.eachexperimentconsistsofeight phases: 1.Startingtheexperimentandgainingsole accesstothetestbed. 2.Conguringthetestbed,managementsys- tem,andmanagedapplicationappropri- 10

11 ately,dependingonthegoalsoftheexperiment. 3.Startingthemanagementsystemand managedapplicationcomponentsusedin theexperiment. 4.Monitoringthestartedcomponentsand collectingtherequiredinformation. 5.Terminatingthecomponentsandcollectingthestateinformationneeded. 6.Analyzingthecollecteddataasnecessary. 7.Finishingtheexperiment. 8.Generatingtablessummarizingtheresults oftheexperiment. Usingtheexperimentdriverandreportgenerator,severalexperimentshavebeenperformedtodate.Theresultsofthesearepresentedinthenextsection. 7Experience Aninitialsetofexperimentswasconductedto verifytheoperationofthesystemandidentify areasforimprovement.basedonthesuccessof thesetests,morerigorousandcomplexexperimentswereplannedandexecuted.theresults oftheseexperimentsprovedtobequiteinteresting,demonstratingtheutilityandexibility ofthetestbed. Inourexperimentation,wesetouttodeterminetheinuenceofasetofparameterson applicationperformance(asmeasuredbythe metricsdescribedinsection4).theparametersselectedincluded:thepresenceofmanagement,whethermanagementwaspush-drivenor pull-driven,thefrequencyanddistributionof managementmessages(eitherrequestsornoti- cations),andthefrequencyanddistribution ofapplicationrequests.wecarriedoutseveral experiments,varyingtheparametervaluesin eachexperiment. Toassessthesignicanceoftheresults, weusedlinearregressiontechniquestodeterminethedegreetowhichthevariationin computedmetricscouldbeattributedtoeach parameter parameterscausinglargervariationbeingconsideredstrongerinuencesonapplicationperformance.thefollowingresults wereobtainedfromourexperiments1: Managementcausesasignicantincrease inanapplication'suseofsystemresources suchascpu,memory,andnetworkbandwidth. Inexperimentswhichcomparedthepresenceofmanagementtoseveralotherapplicationparameters,managementwasfound toaccountfor79.6%to99.9%ofthevariationinresourceutilization(dependingon theresource).onaverage,eachapplicationrequestrequired42.4%(0.05seconds) morecputimeand31.1%moresocket communication(20messages).theapplicationwasnotcomputationallyintensive, sotheratioofcpuusewithmanagement enabledtocpuusewithmanagementdisabledisarticiallyhigh,andshouldbe consideredaworst-casemeasure.asignicantincreaseinthenumberofcontext switchesandpagefaultswasalsoobserved overthelifeofthemanagedprocesses.the maximummemoryrequiredbytheprocesswasincreasedbyanaverageof7.4% (70pages),whiletheminimummemory requirementwasincreasedanaverageof 16.7%(150pages). Managementdoesnotcauseasignicant increaseintheresponsetimeofapplication requests. Managementaccountsforonly1%ofthe variationintheresponsivenessoftheapplication;theremainingvariationwasexplainedbychangestootherapplication workloadparameters.onaverage,this meansa3.4%(0.51seconds)dierenceper request.ontheotherhand,someapplicationworkloadvariations(suchasthemean delaybetweenapplicationrequests)cause adierenceof31.9%(4.2seconds). 1Wecautionthereaderthattheseresultswereobtainedinaparticularenvironmentandwithaparticular managementsystem.whileweanticipatethendings toberepresentative,wecannotmakearmclaimto generality. 11

12 Fromtheseresults,wecanconcludethat, givensucientsystemresources,management canbepracticalinlightofitslowimpactonapplicationresponsiveness.however,ifresources arescarce,oraheavysystemloadiscausing resourcecontention,managementcansignicantlyinuenceapplicationperformance. Withfurtherexperimentation,weobserved anumberofsecondaryresultsthatgaveadditionalinsightintowhatfactorscontributeto thecostofmanagement: Thetimeintervalbetweenmanagement messagesaccountsforthelargestvariation inresourceutilization(68.5%to92.2%, dependingontheresource),comparedto othermanagementfactors.thisisparticularlyinteresting,sincethenumberof managementmessagesbatchedtogether (sentinrapidsuccession)appearedtobe muchlesssignicant.inotherwords,it ismoreecient,fromaresourceutilizationperspective,togeneratemanagement messagesinbatcheswithlongerintervals betweenthem,asopposedtoevenlydistributingthemessagesacrosstime. Onthecontrary,itismoreecient,from aresponsivenessperspectivetoevenlydistributemessagesacrosstime. Equallysignicantwithrespecttoresponsivenesswaswhethermanagement waspush-drivenorpull-driven.onaverage,applicationrequestsunderpushdrivenmanagementwere4.62%(0.70seconds)faster. Push-drivenmanagementwasalsosuperiortopull-drivenmanagementinterms ofresourceutilization. Fromthesesecondaryresults,wecanconcludethatfromaresourceutilizationpointof view,itisbettertobatchmanagementrequests ratherthandistributethemevenlyacrosstime, whereastheconverseistrueforresponsiveness. Wealsoconcludethatpush-driveneventcommunicationisgenerallysuperior. Mostexperimentscarriedoutthusfarhave focussedonworkloadparameters.theseexperimentshavebeenconductedusingonly RS/6000machines(duetotheapplication'sdependenceonDCEandDB/2,whichwehave installedonlyonthosemachines).morework needstobedonetomeasuretheeectsofdifferentsystemparameters. 8ConcludingRemarks Theworkwedescribehereispartofanongoingstructuredattackonthedistributedapplicationsmanagementproblem.Therearemany complexissuesyettoberesolved.webelieve thatidentifyingandunderstandingthecosts associatedwithmanagementmakesiteasierto addresstheseissuesandcontributestotheacceptanceofmanagingdistributedapplications. Theresearchdetailedinthispaperfocusses ontheuseofadistributedapplicationsmanagementtestbedtoinvestigatethecostsof management.werstpresentedacoresetof requirementsforthetestbed,andthenwedescribedadesignsatisfyingtheserequirements. Usingaprototypetestbedbasedonthisdesign, wehavebeenabletocarryoutanumberofexperimentsandarebeginningtorealizeuseful results. Weplantocontinueexperimentationtovalidatethesepreliminaryresultsandtogenerate newresults.weneedtolookatdierentmanagementsystemsandapplythesetoavarietyof applicationtypes,makingfulluseoftheheterogeneityofthetestbedenvironment.moreadvancedtestbedapplicationsareneededforbetterreportingandvisualization.wealsoneed toinvestigatethepotentialforusingtestbed resultsinthesimulationandmodelingofthe managementofdistributedapplications. Throughthiswork,wehopetogaininsight intoboththecostofmanagement,andthefactorscontributingtoit.withthisexperience, wecanpursueoptimizationofdistributedapplicationsmanagementsystemsinaneortto minimizetheircosts. Acknowledgements Theauthorswouldliketothankthestudents ofthecomputerscience612classattheuniversityofwesternontariofortheirassistance withinitialtestbedexperimentation. 12

13 AbouttheAuthors MichaelJ.KatchabawisaPh.D.studentintheDepartmentofComputerScience attheuniversityofwesternontario.hisresearchinterestsincludedistributedcomputing, networkmanagement,anddistributedmultimedia.mr.katchabawcanbereachedat StephenL.HowardisaPh.D.studentin thedepartmentofcomputerscienceatthe UniversityofWesternOntario.Hisresearch interestsincludedistributedcomputing,operatingsystems,andobject-orientedmanagementmodels.mr.howardcanbereachedat AndrewD.MarshallisaResearchAssociatewiththeMANDASProjectandaPh.D. studentinthedepartmentofcomputerscienceattheuniversityofwesternontario.his researchinterestsincludedistributedcomputing,applicationsmanagement,andsoftwarereengineering.mr.marshallcanbereachedat MichaelA.BauerisaProfessorinthe DepartmentofComputerScienceandSenior DirectorofInformationTechnologyServicesat theuniversityofwesternontario.dr.bauer receivedaph.d.incomputersciencefromthe UniversityofToronto.Hisresearchinterests includedistributedcomputing,distributeddirectories,andsoftwareengineering.dr.bauer References [1]H.Abdu,H.Lutyya,andM.Bauer. InvestigatingMonitoringCongurations. Proceedingsofthe1996ACMSymposium onappliedcomputing,pages366{373, February1996. [2]J.W.Hong,M.J.Katchabaw,M.A. Bauer,andH.Lutyya.DistributedApplicationsManagementUsingtheOSIManagementFramework.TechnicalReport #448,Dept.ofComputerScience,UniversityofWesternOntario,London,Canada, January1995. [3]J.W.Hong,M.J.Katchabaw,M.A. Bauer,andH.Lutyya.Modelingand ManagementofDistributedApplications andservicesusingtheosimanagement Framework.ProceedingsoftheInternationalConferenceonComputerCommunication,Seoul,Korea,August1995. [4]ISO.InformationProcessingSystems -OpenSystemsInterconnection-Basic ReferenceModel-Part4:Management Framework.InternationalOrganization forstandardization,internationalstandard7498-4,1991. [5]ISO.InformationProcessingSystems- OpenSystemsInterconnection-Common ManagementInformationServiceDenition.InternationalOrganizationforStandardization,InternationalStandard9595, [6]ISO.InformationProcessingSystems- OpenSystemsInterconnection-Common ManagementInformationProtocolSpeci- cation.internationalorganizationfor Standardization,InternationalStandard ,1991. [7]ISO.InformationProcessingSystems- OpenSystemsInterconnection-Systems ManagementOverview.InternationalOrganizationforStandardization,InternationalStandard10040,1991. [8]R.Jain.TheArtofComputerSystems PerformanceAnalysis.WileyProfessional Computing,1991. [9]M.J.Katchabaw,S.L.Howard,H.L.Lut- yya,andm.a.bauer.ecientmanagementdataacquisitionandrun-time ControlofDCEApplicationsUsingthe OSIManagementFramework.Proceedings ofthesecondinternationalieeeworkshoponsystemsmanagement,toronto, Canada,June1996. [10]JohnShirley.GuidetoWritingDCEApplications.O'Reilly,July