User Program. System call stubs kernel instrumentation. Operating System Kernel. Network monitoring. RPC Device Driver XDR UDP/TCP IP

Transcription

1 Acomparisonofsystemmonitoringmethods,passivenetwork monitoringandkernelinstrumentation A.W.Moore,A.J.McGregory&J.W.Breenz Abstract Thispaperpresentsthecomparisonoftwomethodsof systemmonitoring,passivenetworkmonitoringand kernelinstrumentation.thecomparisonismadeon thebasisofpassivenetworkmonitoringbeingused asareplacementforkernelinstrumentationinsome situations.despitethefactthatthepassivenetwork monitoringtechniqueisshowntoperformpoorlyas adirectreplacementforkernelinstrumentation,this paperindicatestheareaswherepassivenetworkmonitoringcouldbeusedtothegreatestadvantageand presentsmethodsbywhichthediscrepanciesbetween resultsofthetwotechniquescouldbeminimised. 1 Introduction Theuseoflesystemmonitoringingeneral,and comprehensivekernelmonitoringtechniquesinparticular,havelaidthecriticalgroundworkforthedevelopmentandrenementformanyoperatingsystems.kernelinstrumentationhasthepotentialto giveanexactrecordofwhatoccurredinthekernel ofasystemand,asaresult,iscommonlyusedwhen highprecisionisrequired. Ithasbeenused,instudiessuchasOusterhoutet al.[20],smith[28],mummertandsatyanarayanan[17,29]andbakeretal.[2],torecordinformation aboutanoperatingsystemanditslesystems.the resultsandconclusionsofthesestudieshavethen beenusedforstudiesoftopicssuchascacheissues DepartmentofRoboticsandDigitalTechnology,MonashUniversity,Clayton,Victoria3168,Australia(andrew.moore@rdt.monash.edu.au). ydepartment of ComputerScience,WaikatoUniversity,PrivateBag3105,Hamilton,NewZealand(T.McGregor@cs.waikato.ac.nz). zdepartmentofroboticsanddigitaltechnology,monashuniversity,clayton,victoria3168,australia(j.breen@rdt.monash.edu.au). andsimulationmodels[32,7,8,33],andinthedesignprocessofnewsystems[11,19,13,10,21]. However,whilethereisalargevarietyofsystemsincommonuse,asimilarlywidevarietyofcomprehensivestudiesisnotevident,whichcanbeattributedtothedicultiesinperformingsuchstudies. Comprehensivestudiesusingkernelinstrumentation haveanumberofdrawbacks,asseeninthefollowing list(adaptedfrommoguletal.[16]): codewhichistoresideinthekernelisdicult towriteanddebug, kernelsource-codeisnotalwaysavailable, thekernelmustberecompiledandthemachine rebootedeachtimeanerrorisfound, errorsinthekernelcodearelikelytocause systemcrashes, functionally-independentkernelmodulesmay havecomplexinteractionsoversharedresources, kernel-codedebuggingcannotbedoneduring normalmachineoperation;specicdevelopmenttimemustbescheduled,resultingininconvenienceforuserssharingthesystemand oddworkhoursforsystemprogrammers, commonlyadditionalloadisintroducedonto themonitoredsystem, sophisticateddebuggingandmonitoringfacilitiessuchasthoseavailablefordevelopinguserlevelprogramsmaynotbeavailableforkernel code. Kernelinstrumentationforle-systemmonitoringtakestheformofcodeinsertedatthesystemcallinterfaceorattheinternalinterfacebetween

2 system-callandle-systemoperations.suchmethodssuggestanalternativetechniqueformonitoring acomputer'sle-systemactivitieswherethecommunicationschannelbetweenamachineanditsdisk drives,andinparticularbetweenadisklessclient anditsdiskserver,ispassivelymonitored.blaze usedthistechniquewithhissnooper/rpcspysoftware[3],passivelymonitoringtracbetweennetworkfilesystem(nfs)[24]clientsandservers,and predictingtheoperationstheclientsperformedto causethoseoperations. Fullkernelinstrumentationisusedcommonlyin systemmonitoringbut,bydenition,itinvolvesthe modicationoftheoperating-systemsource-codefor themachineinquestion.passivenetworkmonitoring canbeapreferredchoiceoverkernelinstrumentationforcertainsystem-monitoringwork,particularly ifthesource-codeisnotavailable.passivenetwork monitoringalsohasotheradvantages,including: resultsfromthemachinesbeingmonitoredcan becollectedindependentlyofthosemachines, nomodicationsarerequiredtotheoperation ofthemonitoredsystems, thecollectionofdatawithpassivenetwork monitoringdoesnotimpactonthemachines beingmonitored,and theabilityexiststomonitormultiplemachines simultaneouslyonanetwork. Thisnalpointisimportantbecausedistributed systemsaregrowinginpopularityandasignicant numberofcomputersincommonusearepartofa distributedsystem,ifonlythroughthedistribution andsharingofles.comprehensivestudiesofdistributedsystemsingeneralanddistributedlesystemsinparticulararerelativelyrare.themainreasonforthisisthatthecomplexityofcollectingdata isgreatlyexacerbatedbytheneedtocollectitsimultaneouslyfromalargenumberofmachines.kernel instrumentationwouldrequiremodicationofany numberofdierentcomputerswithdierentoperatingsystemsrunningondierenthardware.there wouldalsobetheissuesoftheloadimposedinthe actualcollectionofdataonorfromeachclientand theimmensetaskofpost-processingthedatafrom thedierentmachines. Thesedisadvantagesdonotexistwhenusingpassivenetworkmonitoringofthedatachannelbetween clientsandaserver.dataaboutallactiveclientscan becollectedsimultaneouslyand,ifadistributedle systemssuchasnfsisinuse,thedatacollected fromthenetworkisindependentoftheoperating systemormachinearchitecture[15,14]. 2 rpcspy/nfstrace Ifrpcspy/nfstracearetobeusedasareplacement forkernelinstrumentation,thetechniquemustbe abletodelivertoaresearcherdatasimilartothat generatedbythekernelinstrumentationtechniques. rpcspy/nfstracehastwodistinctcomponentsfor achievingthis. rpcspyinteractswiththeethernet-interfacefacilitiesofthemonitoringmachineandcollectspackets traversingthenetworktowhichitisconnected.the packetsareconvertedintotherequestorreplypart ofannfsremoteprocedurecall(rpc).eachrequestandreplyisthenmatchedtogether,dataofinterestareextractedandatransactionrecordismade alongwithatimestampofwhenthetransactionwas completed. Thesecondcomponent,nfstrace,usesanheuristicbasedontheoperationofNFStomakean estimationofthedurationofale'sopen-closesession(thetimebetweenwhenaleisrstopened, readfromand/orwrittentoandthenclosed)which causedthenfstransactionsrecordedbyrpcspyto occur.nfstracecreatesrecordsofleopen-close sessionsitestimateshaveoccurred(and,thus,to havegeneratedthenfstransactionsseen).this estimationrelies,inpart,onconsistencyinnfsimplementations. Forexample,foreveryopensystemcall(independentofwhethertheleistobereadtoand/orwrittenfromorjustaccessed)anNFSgetattr transactionisgenerated.however,open-closesessionstoreadorwritedatahandletheactualdata insignicantlydierentwaysalthoughitshouldbe notedthatthewritetransactioncaseiseasierto handlebecausethecachedoesnothaveasdominant eectonthewriteoperations.asaresult,muchof thespecial-casehandlingnfstracemustdoapplies onlytonfsreadtransactions. rpcspy/nfstracetogethergenerateanestimationoftheopen-closesessionsthatakernelinstrumentationsystemcouldrecorddirectly.figure1

3 User Program System call stubs kernel instrumentation Local Filesystem Block Cache NFS Filesystem Figure1:Thedataowbetweenauserprogramand RPC annfslesystem.instrumentationpointsforker- Device Driver XDR nelinstrumentation(snooper)andnetworkmonitor- ing(rpcspy)areindicated.thisdiagramcompares UDP/TCP IP Operating System Kernel thedierenceintheinformationavailabletoeach network system.inparticular,oneinstrumentationpoint, Network monitoring snooper,isbeforethecacheandtheother,rpcspy, isafterthecache. showshowtheinstrumentationpointsforeachofthe twosystemsdierinthecomponentsofthesystem availabletothemand,thus,theinformationaccessibletothem. suchasthesystemcallsoccurringonthecomputer, whilepassivenetworkmonitoringmustinterpretthe whichsystemoperationshaveoccurred.rpcspy transactionsbetweenclientandservertoestimate Kernelinstrumentationisabletorecordevents curredbetweenclientandserver. serversideoftheclientcache-thetracthatoc- /nfstraceresultsrecordwhathappenedonthe 3 Previousworkusingrpcspy hasbeenusedinseveraldierentworkstoaidinthe Therpcspy/nfstraceimplementationbyBlaze rpcspy/nfstracetoanalyseclient-serverlesystem tracandmadeuseofinformationaboutthetrac congurationofexistingsystems.anderson[1]used developmentofnewsystemsandasanaidtothe Regularly-read,static,read-onlylesweremovedto tobetterutiliselocaldisksintheclientsthemselves. thelocaldisk,onapartiallyautomaticbasis,takingintoaccounteachle'sutilisationbyapartic- ularworkstation.blazeusesresultscollectedusing rpcspy/nfstraceinanumberofworkstojustify versionofnfstracetocollectresultsusedinapaper thedesignofawidearealesystem[4,5].finally, tojustifyaparticularlesystemdesign[34]. Dahlinetal.[9]usesrpcspyandapartially-modied sumptionthatcomplicationsintroducedbythere- sultsofrpcspy/nfstracearenegligible.thisis validforstudiesbaseduponthetracbetweena clientandserverwhichtakeintoaccounttheeects Eachofthesestudieshasbeenmadewiththeas- oftheclientcachesuchasdahlinetal.[9].however,otherstudiescanbeatriskforassumingthat rpcspy/nfstraceissuchaperfectreplacementfor traditionaltechniques. softwarethattheeectsofpacketlossshouldbe oftherpcspy/nfstracesoftwarethatpeculiarities quantied.additionally,henotesinhisdescription oftheheuristicsofnfstrace(anditsoriginalimplementation)needtobeevaluatedmorecompletely. beenwithoutanyharddataontheaccuracyofthe implementationandonlyapassingappreciationof areaswheretheimplementationisinaccurateand thereasonsforthoseinaccuracies.thefollowingsec- Previously,theuseofrpcspy/nfstracehas Blaze[4]notesinhisdescriptionoftherpcspy datafromthesamesourceandthendiscusstheresultstionspresentacomparisonoftwosystemsrecording Therpcspy/nfstracetoolsdependheavilyonthe 4abilityofthenetworkinterfaceofthemachineon Systematicerrorinrpcspy whichtheyarebeingruntocapturealltracpass- notlosedatawhenutilisationislowbutdataloss withnetworkutilisation.thenetworkinterfacewill ingthroughthenetwork.packet-lossbythenet- workinterfacedoesnothavealinearrelationship willincreaseasutilisationincreasestoapointbeyondwhichitwillbeunabletoacceptanyfurther dataitcanprocesswillattenoutnomatterwhat theutilisationbeyondthatpoint. datalossofrpcspyandtocalibratethenetworkinterfacerpcspyuses.toperformthesetestssatisfactorily,anetworkanalysercapableoffull-utilisation increaseinthedata-transferrate.theamountof Astudywasperformedtoquantifythepotential

4 measurementsonethernetwasrequired.ahewlett PackardInternetAdvisorModelJ2522Aswasused bothtomakemeasurementsandtogeneratearticialloadsonthenetwork.thepacketfiltermech- operatingsystem. anismusedwasinadecstation3100runningul- trix4.3aandthenitmechanismusedwasinasol- bournesc2000(amachinecompatiblewiththesun Sparcstation2)runningaSunOS4.1.2compatible articially,usedthetracbreakdownintable1 whichwasbasedonananalysisofthenetwork'sregulartraccontentcollectedoverseveral24-hourperiods. Testsofrpcspy,wherethenetworkwasloaded congurationoptions.inparticular,thesizeofthe canbeset.thenitmechanisminsunosdoesnot packetbuer,wherepacketsprocessedbypacket filterareplacedforcollectionbytheuserprocess, ThepacketfilterfacilityofUltrixoerssome oerthiscongurability.thedefaultconguration Ethernetnetworkutilisation.Itisapparentthatnot percentageofunprocessedethernetpacketsversus arecomparedinfigure2.thisgureshowsthe packetfilterinadditiontothenitmechanisms andanoptimum(largestcongurablebuersize)for onlyarethecharacteristicsofthenitmechanism packetfiltermechanismshowednolossuntilover 15%utilisation,astagebywhichNITmechanismloss poorbeyond10%utilisationbutthatthepacket leveloflossuntilutilisationwascloseto50%.the wascloseto25%. filtermechanismdidnotdemonstratethesame ofprocessingoverheadontheclient,whichisimposedbytheneedofrpcspytomatchrpctransactions,andthepacket-losscharacteristicsoftheethernetinterfacewhichrpcspyisusing.figure3shows Asignicantissueinrpcspyisthecombination thenumberofnfstransactionsversusethernetutilisation.theethernetutilisationinthesetestsisal- processinagiventimeperiod.thehewlettpackard mostpurelynfstrac.byusingnfstracexclu- sivelyweareabletoestablishthemaximumnumber ofnfstransactionseachrpcspysystemisableto guration. testthepacketfilterwasleftinthedefaultcon- transactionsthatoccurredoverthistime.forthis testequipmentrecordedtheactualnumberofnfs mumnumberofpacketsitcanprocess.thenit- Thetestshowsthateachsystemhasamaxi- Total packets unprocessed (%) Packetfilter (default) Packetfilter (optim.) NIT suspacketlossforvariousworkstationethernetin- terfaces.packetfilterdefaultandoptim(um)are Figure2:AcomparisonofEthernetutilisationver twocongurationsoftheethernetpacketcapturefacilityofthedigitaldecstation,nitistheethernet 10 capturefacilityinsunmicrosystem'ssunos Utilization (%) urationpacketfilter-ultrixcombinationappears tobelimitedtoprocessingapproximately260nfs NFStransactionspersecond.Thedefaultcong- transactionspersecond.itisimportanttonote SunOSsystemislimitedtoprocessingabout175 thiswasastress-testingofrpcspyandthatsuch NFSloadswerenotacharacteristicofthenetwork guresintable1wecanseethat36%ofthetotalethernettracisfromnfs.however,itwould notbetruetosayofthis36%thathalfthenumber ofnfsethernetpacketsisanapproximatecount ofcompletenfstransactions.suchasimplicationwouldnotallowfortherebeingincompletenfs transactions(thelossoftherequestorreplyina transaction),norwoulditallowfornfstransactions (transactionswherethedatapayloadrequiredtwo thatrequiredmorethanonepairofnetworkpackets towhichthesemachineswereconnected.fromthe thetestnetworkoperatingat12%utilisationcould meanlessthan72transactionspersecondinamixed rpcspydoesnotneedasmuchprocessortimeasif ithadhadacompletenfstransaction.asaresult, ormoreethernetpackets).ineachofthesecases loadwithavarietyofnfstracratherthanthe200 indicates. transactionspersecondthatthefigure3stress-test forcertainbutitcouldresultfromlimitationsin thehardwareofthenetworkinterfaceand/orinthe Theexactcauseofsuchdatalossisnotknown

5 InternetProtocol(IP)67UDP Protocoltype Sub-protocolTypesofpacketPacketsize(avg.) TCP NFS (all) % NovellNetware(IPX) Table1:AbreakdownofthetracmixtureusedfortestingrpcspyresponsetoEthernetutilisation areactuallyclosetojust5%ofthenetworkcapacity) supportsthisobservationwithamaximumloadover andthenetworkonwhichmeasurementsweretaken thatmostethernetloadsarewellbelow50%(and isationover24hoursofcloserto1.1%. 24hoursofnogreaterthan18%andanaverageutil- Figure3:ThenumberofNFStransactionsversus yserrecordingnopacketlossisalsogiven. EthernetutilisationfortheNITandpacketfilterrpcspy/nfstraceandakernelinstrumentation 5Thefollowingresultspresentacomparisonbetween Comparison softwareofthepacketcollectionandlteringmechanism.thischaracteristicisunfortunate.itisduringsnooperpackage.itwasimplementedoriginallyby asubstituteforkernelinstrumentation. sumptionthatrpcspy/nfstracewouldbeusedas capturemechanisms.resultsfromanetworkanal-technique.intheseresultstherehasbeenanas- acrossadistributedlesystemwillpotentiallybe highest.becausethereispotentialforrpcspybased thetimewhenthenetworkisbusiestthatutilisation SiebenmannandZhou[27]forUltrixversion3.3. Snooperisasetofkernelinstrumentationroutines Thekernelinstrumentationthispaperusedisthe toolstolosedataabouttransactionsatbusytimes, studiessuchaslesharing,asituationthatwouldblockoperations,processexecutionandtermination, functionsincludinglogicalleoperations,physical- bemorelikelytooccuratbusiertimes,wouldbeetc.thesnooperpackageisbaseduponthepackage forrecordinginformationaboutanumberofkernel aectedadversely. inturn,sharesitsancestrywiththepackageusedby ofthesamenamedescribedinzhouetal.[35]which, offasterworkstationswithfasterhardwarenetwork implementationperformanceineitherthenetwork interfaces.however,thismaynotbesolvedaseasily iftheproblemisdueprincipallytopoorsoftware SuchdrawbackscouldbeovercomebytheuseOusterhoutetal.[20]toperformtheirstudyofthe UNIX4.2BSDlesystem. nicantonlyaboveabout10%utilisationforthe packetcapturemechanism(nit/packetfilter)or packetfiltermechanism.boggsetal.[6]comment rpcspy. ditionalpost-processingcodewasrequiredforthe dedbyanopenandaclosesystemcall.nfstrace open-closesessions,i.e.setsofletransactionsboun- performsthatfunctionforrpcspy/nfstracebutad- Theresultsfromeachsystemwereprocessedinto Whilethischaracteristiclossdoesexist,itissig-dataproducedbysnoopetracehavebeendonebyusingsimultaneoustraces ofasinglemachineovera24-hourperiod.thetrace ofthismachinewasperformedfrom11:00a.m.monday,12thofdecember,1994,until11:00a.m.thefol- Thecomparisonsofsnooperandrpcspy/nfs- NFS transactions recorded per second NFS transactions recorded versus ethernet utilization packetfilter NIT Hardware Monitor Ethernet utilization (%)

6 lowingday.themachinetracedwasadigitaldecstation3100conguredwith20mbytesofmemory, runningultrix4.3a.thismachinewascongured withalocaldiskforvirtualmemoryswapactivities. Therpcspytracewasrecordedtoanadditionallocal disksoasnottoperturbtheresultswithextraneousnetworkactivity.duringthe24-hourperiod,a lossof1.5%oftotalethernettracwasrecorded. Basedonthegraphsofpacketlossversusutilization (gure2)andupontheaverageethernetpacketutilization(table1)thisgivesanapproximatelossof 0.6%ofNFStransactionsfromthetotalrecorded trace. Thenfstracepost-processingtoolusesaheuristicwhichincorporatesatimeouttodeterminehow longanopen-closesessionwilllast.thevalueisuserselectablebutthedefaultvalueof135secondswas usedthroughouttheanalysisdescribedherein. 5.1Excludeddata Alltransactionsassociatedwiththereadingofexecutablelesrecordedbyeitherthesnooperorrpc spysystemswereremovedfromthetracedatabefore processing.thiswasdonetoavoidproblemsassociatedwithashortcominginthesnooperinstrumentation,nottherpcspy/nfstracesystem.records pertainingtothesnoopertraceleitselfwereremovedfromtheoutputrecordsduringtheprocessing stage. Whiletheremovalofallexecutiontransactions mayseemtochangetheresultspresented,theremainingdatastillpermitasatisfactorycomparison ofthetwomonitoringsystems.theamountofpotentialcomparison-errorwhichwouldbeintroduced duetotheinclusionofincompleteexecutionrecords bysnooperwasnotjustied.additionally,lesystemtracresultingfromtheloadingofexecutable leswasexcludedfrompreviousstudiessuchas Ousterhoutetal.[20]andBakeretal.[2]duetosimilarproblemsintheloggingofexecutableletrac. 5.2Systemtrac Thecharacteristicsofthetotalle-systemcommunicationstracarecommonly-usedmeasurements.In thecaseofdisklessworkstations,themeasurements areimportantforinsuringthatthenetworkshave adequatetransportcapacityandthattheserversof disklessworkstationshaveadequateservicecapacity. Inanysortofworkstationsuchvaluesdenetherequiredcapacityfordiskinterfaces,aswellasbeing usedincacheandbusdesign[35,21,20,2,23]. Acomparisonofcommunicationstractoand fromthelesystematthelogicallevelandofthe communicationstracattherpcspynetworklevel arenotstrictlycomparablebecauseeachsetofmeasurementswasmadeonadierentsideofthecache. However,oneoftheobjectivesofnfstracewasto estimateoperationsthatoccurredattheuserlevelby analysingthedatacommunicationstracbetween clientandserverandthetransactionsusedbythe clienttoensurethecontentsofthecacheareup todate.asaresult,whilerpcspy/nfstracecannotgenerateinformationonexactlywhatdatawere transferredbetweentheuserprogramsandthele system(includingthenfsle-systemroutinesand theblockcache),itcancalculatetheexactamount ofdatatransferredbythenfslesystembetween NFSclientandserver. Table2givesasummaryofresultsforthecomparisonperiod.Itisapparentimmediatelythatthere isamajordierenceinthevaluenfstraceestimates forthetotaldatatransferredwhencomparedwith snooper.theydierbyafactorof1:7.fromthese resultsitisequallyapparentthatoverthecourseof along-termanalysis(24hours)theresultsforpeak valuesandwritedataarecomparableforthetwo systems. Peakvaluesdisplaythischaracteristicbecause theytypicallyinvolveamountsofdatathataretoo largeortoovolatiletobesuitableforlongtermstorageinthecache[20,2,28]-thischaracteristicis independentoftheparticularloadamachineisunder[18].asaresult,thesimilaritybetweentransferreddata,particularlypeakvalues,wouldremain acrossanysampletaken.incomparison,valuesfor thetotalquantityofdatatransferredovertimeis notsimilar.thedierencebetweensnooperread averagesandnfstracevaluesisnotsurprising.the clientcachewilleliminatesuccessivenfstransactionsforreadingdatafromthenfsserverand,as aresult,nfstracecannotrecordthedatatransfer thatoccurredatthelogicallevel. Figure4showsplotsofdatatransferredovertime asrecordedbysnooperandrpcspy/nfstrace. Higherlevelsofdatatransfer,particularlysignicantwritingactivity,between7a.m.and11a.m.

7 Totaldatatransferred Averagedatatransferred measurement Particular interval length 86,644,53046,967,724 snooper (bytes) nfstrace Peakdatatransferred 10seconds 10,028 (bytes) Averagedataread Peakdataread Averagedatawritten 7,468 5,048,320 3,914,935 5,436 Peakdatawritten 2,560 2,590 Averagedatatransferred Peakdatatransferred 10minutes19,028,55017,015,414 5,120, ,698 5,048, ,165 2,846 Averagedataread Peakdataread Averagedatawritten 10,427, , ,595 7,144, ,387 Table2:Thetotaldatatransferredforthesystem.Peakandaveragevaluesfor10secondand10minute intervalsarealsogiven. Peakdatawritten 8,600,705 9,289, ,777 thereadingandwritingoflargeimageles. (byanotherresearcher)onthismachinerequiring isduetothetestingofimageencodingalgorithms mentationandthatavailabletonfstrace.peri- dierencebetweendatagainedfromsnooperinstruodicaccessesbyautomaticjobsaccountfortheregularcommunicationstracloggedduringthe19:00 Thegraphofread-datashowsanexampleoftheble3liststhedierentlesystemstheclientaccessed lesystemcontainsexecutablelesforthesystem, to07:00period.becausethiscommunicationstraf-overthetraceperiodandthetaskseachlesystem storetracedatalocally,andalocal,swapdisk.ta- etc.thedecstationanalysedinthisstudydidnot cinvolvestheregularexecutionofprograms,com-servedmonlywithlittleotherle-systemactivity,thecacheandfromeachlesystemcanbeusedtoassistin makingle-system-congurationdecisions.suchde- haveanylocallesystems,apartfromthatusedto oftheclientholdsallthenecessarysoftwareandassociateddatales.theresultisthatapproximately 300Kbytesoflogicaldataarereadeach30minutes period. read-activitybetweenclientandserveroverthesame atthesnooperlevelbutrpcspyrecordsnegligiblecisionscanincludewhichlesystemsgenerateso Abreakdownofthetypeofdatatransferredto ordsofrawdatatransferisbecausenfstransactions donotcontainsignicantinformationaboutblocks readdataavailabletorpcspyaboutdatatransfers readfromthecacheoftheclient.theonlyspecic Thereasonnfstraceisnotasaccurateforrec-ble4. muchservertracthatitwouldbebetterforthem tobeattachedlocallytothemachineandhowwidely lesystem'scommunicationstracisgiveninta- aparticularlesystemisused.abreakdownofeach theserver'sdisks. 5.3Filesystemtransactions thatoccuriswhendataarereadbytheclientfromnoteisaverylargepercentageofoperationsasso- ciatedwiththe/partition.thelargenumberof Itisimportanttonotethatatthesystem-call isticbreakdownofthesetransactions.ofparticular level,asrecordedbysnooper,thereisacharacter- transactionsonthispartitionwillhavebeenpotentiallycompoundedbecausethe/tmpand/var/tmp directoriesresidedonthe/lesystems./tmpand typicallyforaparticularpurpose.forexample,one Aswithmostunixsystems,eachlesystemisused/var/tmpcanpotentiallycarryalargepercentageof operationsbecausetemporarylesaretraditionally lesystemcontainstheusers'directories,anothercreatedinthisdirectorystructure[31,22]. resultsfromthetwomonitoringmethods.notable exceptionsaretracinvolvingthe/partitionand Table4showsamoderatesimilaritybetweenthe

8 Read data transferred over day (10 minute samples) Bytes transferred 10 Mbyte 8 Mbyte 6 Mbyte 4 Mbyte 2 Mbyte snooper read nfstrace read 0 Mbyte 11:00 13:00 15:00 17:00 19:00 21:00 23:00 01:00 03:00 05:00 07:00 09:00 11:00 Time of day 10 Mbyte snooper write nfstrace write 8 Mbyte 6 Mbyte Figure4:ReadandWritetransfersasrecordedbykernelinstrumentation(snooper)andnetworkmonitoring(nfstrace).Aquiescentsystemfrom19:00until7:00,themachineisbusyduringthedaylight 4 Mbyte hours. 2 Mbyte 0 Mbyte 11:00 13:00 15:00 17:00 21:00 23:00 01:00 03:00 05:00 07:00 09:00 11:00 Time of day FileSystem FunctionandContents Bytes transferred Write data transferred over day (10 minute samples) / /var/spool/mailcontainseachusers'maille. rootlesystem,alsoincludes/varand/tmp.top-levellesystem /usr/local containsstandardsoftwaredistribution,inadditiontolibrariesandincludelesforthecurrentsystem. containingtemporarydirectoriesandloggingdirectories. containslocallyinstalledsoftware. /usr2 /packages Table3:Abreakdownofthelesystemsofthestudyandtheirrespectivefunctions. homedirectoriesforagroupofusers. containscommercialsoftwarepackagesandcollectionsofprojectspecic data(inthiscaseimagedata).

9 FileSystem/31,736,478(36.63) snooper total rpcspy/nfstrace /var/spool/mail /usr/local 2,941,480(03.39) 4,385,788(05.06) 5,863,351(11.59) /packages /usr238,660,513(44.62)35,251,413(69.66) 1,455,692(01.68) 7,464,579(08.62) 1,446,003(02.86) 3,142,239(06.21) read 3,934,663(07.78) 965,364(01.91) /var/spool/mail /usr/local /27,267,823(42.26) 2,941,480(04.56) 3,836,074(05.94) 2,853,302(12.73) /packages /usr221,561,247(33.41)10,895,621(48.63) 1,455,692(02.26) 7,464,579(11.57) 1,446,003(06.45) 2,311,208(10.32) write 3,934,663(17.56) 965,364(04.31) Table4:Totaldata,readdataandwritedatatransferredperlesystemasmeasuredbysnooperand /var/spool/mail /usr217,099,266(77.31)24,355,792(86.38) / 4,468,655(20.20) 549,714(02.49) 3,010,049(10.68) rpcspy/nfstrace. 831,031(02.95) read-tracingeneral.whiledierencesbetween valuesforreadbetweensnoopermeasurementsand particularlypronounced.thisdierenceislikelyto thoseofrpcspycanbeexplainedasresultingfrom clientandserver,thereadtracforthe/partitionis thecachemechanismlteringreadrequestsbetween resultfromahighusageofsystemleslocatedinthe ofsuchlesinclude/etc/passwd:thelistofusers spondingcacheentriesalwaysbeingvalid.examples /etcdirectorybeingaccessed,resultinginthecorre-fromtheserverandthenmayremaininthelocal cachewhilebeingusedrepeatedly.finally,during thesource-codeles,librarieswillbereadonlyonce server(andseenbyrpcspy)butnotnecessarilyread fromthenfsserver.duringstage2,inadditionto onthisclient. lelistingthelesystemsthatshouldbemounted thesystemsknowntothisclientand/etc/mount:a abletouseasystem,/etc/hosts:astatictableofrunstherawimagestreamcouldhaveremainedin stage3,whileletransactionsrelatingtotheloadingoftheexecutableleitselfhavebeenremoved, thisprogramtakesasinputarawimagestreamand outputsanencodedimagestream.onconsecutive ofreadandwritedatafor/usr2isareectionof thevolatilenatureoflesonthislesystem.in particular,softwareforimageencodingwasbeing developedandacycleof Thenotabledierenceintherecordedquantities/usr2asmeasuredwiththesnoopersystembutthe tracalreadygreatlyfavoursthewrite-tracfor localcache. Itshouldbenotedthattheratioofread-to-write 1.editprogram forboththe/usr2and/var/spool/maillesystemscanbeattributedtotheblockcacheneeding writetracrecordedbyeachmonitoringsystem Signicantdierencesbetweentheamountof cacheactivities,lteringtrac,increasethisratio. existed.thisdevelopmentcycle,duringstage1,resultedinsource-codelesbeingwrittentothenfs 3.runprogram totransferdatatoandfromthelesysteminblocksizedpieces.theresultofthisisthatamodication wholeblock(8kbytesfortheselesystems). ofonebyteinalewillresultinthewritingofa 2.compileprogram percentageoflogicalletrac,blockcachingseems itiesonthe/lesystemareresponsibleforalarge Fromthisbreakdownitisclearthat,whileactiv-

10 toreducethequantityofdatatransferredbyafactorofupto6.bycomparison,the/usr2lesystem isresponsibleforahigherquantityofdatatransfer and,inthedevelopmentandbalancingoflesystems,itwouldbeimportanttoestablishwhether thisisatransientconditionoraregulartrendfor communicationstracforthatparticularlesystem. 5.4Systemusers Table5presentsseveralvaluesrelatedtothenumber ofactiveusersonthesystemandtheamountoftraf- cgeneratedbythem.suchtabulationshavebeen madeinanumberofpreviousstudiesandareuseful intheestimationoftheloadausermayimposeon asystemaswellastheworst-casescenariosforthis loadṫhedierencesintable5forthenumberofusers aremostlikelytheresultofsnooperrecordingthe realuserid(uid)associatedwitheachlogicaloperationandrpcspyrecordingtheeectiveuserid associatedeachnfstransaction.thisdierence comesaboutbecauseprogramssuchasinetd(the internetservicedaemon)performoperationsasone userandspawnprogramsthatwillrunasanother user.theresultisthatcountsofactiveusersmade throughrpcspy/nfstraceusuallydierbyavalue ofonewhencomparedwiththeactiveusercount fromsnooper. Average-data-utilisedperuserindicatesthat cache-hitratesare,onceagain,absorbingasubstantialquantityofcommunicationsthatwouldhaveoccurredbetweeneachuserandthelesystem.itis interestingtonotethatthemaximumvaluesrecordedbyeachsystemarealmostidentical.thisis mostlikelyduetothetransferoflargeamountsof data,causingtheclient'scachetobequicklyoverrun withnewdata.asaresult,onlyaminimalamount ofdataiscachedduringthistime. 5.5Files Aslesarethecommonunitofdataaccessedon alesystem,informationabouttherangeofles accessed,aswellastheworkingsizeofthoseles, enablesdeveloperstodeterminethenecessarysize oflecaches,toestablishcommonworking-setsizes andtoquantifyotherrelatedmeasurements. Ashasbeenmentionedearlier,thedierencein theaveragelesizeforthe/lesystemwaspredictable.thiswillprincipallybearesultofalarge numberofsmall,system-relatedlesnotrequiring accessfromthenfssystem.thedierencesinother valueswillhaveresultedfromthecachingof,andrepeatedaccessesto,activeles(eveniftheseleswere activeforonlyashortperiodoftime).inthiscontext,anactiveleisonewhichisaccessedoneor moretimes. Table7liststhenumberofdierentlesrecorded atthesnooper,rpcspyandnfstracelevels.atthe rpcspylevel,thisisacountofeverylethathada readorwritenfsoperationperformedonit.the lteringcharacteristicofthecacheisobviouswhen comparingthenumberoflesthathadlogicaloperationsperformedonthematthesnooperlevelwith thenumberoflesforwhichdatawasreadfromor writtentoattherpcspylevel.largerdierencesfor the/lesystemwillhavebeenasaresultofaccesses tothelargenumberofregularly-accessedsystemles locatedthere.theseleswouldbeaccessedoften andbemodiedinfrequentlyandwould,asaresult, havealongcachelife. Theresultsinthistableshowanareawhere theestimationmethodusedbynfstracecangeneratediscrepancies.nfstracemustestimatetrac toandfromlesthathavenotcausedanyrpcspy readorwritetransactions.withtheexception of/var/spool/mail,nfstracemustestimateadditionaloperationsforlesoneachofthevele systems.nfstracehasestimatedextraneousoperationsonlesof/usr2andunderestimatedthese operationsfortheotherlesystems,inparticular the/lesystem. Therulebaseunderwhichnfstraceoperates estimatesoperationsonlesfromacombination ofnfsread,write,setattrandgetattrtransactions.theestimatesofleswhichdidnotinvolvenfsreadorwritetransactionswouldhave resultedfromsetattrorgetattroperations.by usinggetattrtransactionsalone,thereispotential fornfstracetoconfusegetattrtransactionscaused bysuchoperationsasgettingadirectorylistingwith thosetransactionsbeingusedtovalidatethecontentsoftheclientcache. Incomparison,thegraphofFigure5,anormalisedcumulativedistributionofthenumberofles ofeachsize,showsthattheestimationcalculatedby

11 intervallength Numberofactiveusers 10minute snooper 4 rpcspy 10second Totalbytestransferredperactiveuser 10minute6,342,8505,477, Maximum Average 10second5,120,0005,048, ,535 11, ,820 datatransferredperactiveuserinthoseintervals. Table5:Themaximumandaveragenumberofactiveusersovergivenintervalsandthetotalquantityof 18,404 /var/spool/mail Filesystem/ snoopernfstrace /usr/local 267, ,123 43,378 10, , , ,417 Table6:Acomparisonoftheaveragesizeforles /packages1,316,180 /usr2 42, ,371 46,067 12,310 nfstracecompareswellwiththeresultsofsnooper. Thetwosignicantdierencesbetweentheresults accessedoneachparticularlesystem. ofnfstraceandthoseofsnooperwhichleadto disparitiesinthegraphareforthenumberofzerolengthlesandthenumberofleswhichwereapproximately700bytesinlength. Figure5:Cumulativedistributionofnumberofdifferentlesaccessedversuslesize.Fromthisgraph thanagivensizehavebeenaccessed.forexample cessedare1kbytesinsizeorsmaller.note:thele ateaccurateestimationsofaccessestovariouszero- lengthlesandcreatesrecordsofmanymoreac- cessesthanactuallyhappened.thismaymostlikely beduetonfstracebeingunabletodierentiatebetweengetattrtransactionsfordirectoriesandthosesizeaxisislogarithmic. resultingfromtheopeningofazero-lengthle.in Intherstcase,nfstraceisnotabletogener-bothtechniquessuggestthatover150ofthelesac- wecandeducethenumberoftimesdierentlesless evaluatesmanyofthe700-byteleaccessesasbeing thesecond,relatedcase,nfstracehasunderestimatedthenumberofaccessestovariousleswhichenoughlife-spanbetweenlecreation,thewriting absorbingthesmall-letransactions. wereapproximately700bytesinlength.inadditionandreadingofdata,andledeletion,nonfsread orwritetransactionsmayoccurduringtheopenclosesession.asaresultnfstraceisnoteasily ashortlife-span. abletorecorddatatransferoperationsonleswith zero-bytelesaccessesbecauseoftheblockcache tothereasonsabove,itispossiblethatnfstrace problemtonfstrace.thisisbecausegivenashort Fileswithashortlife-spancanalsopresentanumberofdierentlesaccessedbythesystemdur- sistentwiththehypothesisthatnfstracewasun- abletoevaluatecorrectlythenumberofaccessesto Thefollowingtable,7,givesabreakdownofthe ingthemeasurementperiod.thesevaluesarecon- zerolengthles.theaveragelesizefor/would Number of files Cumulative distribution of the number of files of each file size snooper nfstrace K 10K 100K 1M Size of file (bytes)

12 stronglyconrmthis,althoughthe/packagesresultsruncountertothis.thisstrongcounter-examplecouldbeduetotheunusualnatureofleson thatparticularlesystem.wenotealsothatnfstraceresultscountonelessleforthatlesystem; asinglelargelewouldhavemodiedthisaverage considerably. Whiletherearenotabledierencesineachof Tables6and7,theresultsfromthem,inadditiontothoseofFigure5showthatnfstracewas abletogiveresultsbroadlycomparablewiththose ofsnooper. 5.6Fileopen-closesessions Theopen-closesessionofaparticularleisaconcept aroundwhichanumberofmeasurementsarebased. Anumberofstudieshaveusedsuchmeasurements; examplesincludelesharing,leutilisationandvariouscachestudies[20,2,12,13,25]. Suchopen-closesessionmeasurementsincludethe lengthoftimeaparticularleisopen,theamount ofdataaccessedinthattime,theamountofdatapotentiallyaccessed(thesizeoftheleopened),what sortofopen-closesessionwasinvolved,whetherthe lewasopenedforreadand/orwriteoperations,etc. Thenumberofopen-closesessionsaswellasa breakdownoftherelativetypes,aretabulatedin Table8.TheimplementationofNFSunderUltrix includesthesynchronouswritingofmodieddata blockstothelesystematthecloseofale.this meansthatnfstracecanpotentiallymisswriteoperationsonlesthatultimatelyleavethelewith zerolength,forexamplesomesortoftemporaryle. nfstracewillnotbeabletogenerateresultsfor readsonlesthatoccurinclosesuccession(where thecachecontentsarestillvalid).additionally, nfstracemaynotcorrectlyinterpretgetattrnfs transactionsusedtovalidatethecache.theresult isnfstracewilleithermisssomeopen-closesessions altogether,incorrectlyinterpretnfstransactionsas notbeinganopen-closesession,orincorrectlyconsiderthatthenfstransactionsfromtwoormore separateopen-closesessionarefromthesameopenclosesession. Thelargernumberofwritesrecordedbynfstracewillcertainlyincludetheread-writeoperations snooperrecorded.nfstraceisunabletodetect read-writesessionsandwouldconsidereachofsuch operationsasaseparatereadandwritesession.null open-closesessions,wherenodataaretransferred andtheleissimplyclosed,wouldnotbeableto bedetectedbynfstrace.instead,nfstraceinterpretsanyleopen,ifthatweretheonlyoperation onaparticularle,tobeareadingofanunknown amountofdatafromtheclientcache. Becausethebordersbetweenreadandwriteoperationscannotbedeterminedaccurately,nfstrace willtendtocollectsuccessiveopen-closesessionstogether,interpretingthemasone,longer,open-close session.asaresultofthis,theaveragedurationof theopen-closesessionsreportedbynfstracemay behigherthanthedurationsreportedbysnooper. Tables9and10,recordtheopen-closesessions onatypeofopen-closeoperationperlesystembasis,andbylesystemperoperation.firstly,table9showsthefulleectofthecacheltering,combinedwithnfstraceincorrectlyinterpretinginformationavailable,causingopen-closesessionstobe removed.thisisespeciallythecaseforthe/lesystem.theresultsfor/var/spool/mailareagood exampleofwherenfstracehasmisinterpretedthe NFSgetattrtransactionsasopen-closesessionsbecaus lesareoftencheckedfornewmailresultingingetattrtransactions.Bywayofcomparison, abetterresultisgivenforthe/packageslesystem.filesfromthislesystemareunlikelytobe abletobekeptincacheforlongperiods.theresult isthatnfstraceisabletogiveabetterresultfor open-closesessionsbecausethenfstransactionsfor thislesystemweremorecomplete. Becausethecacheisremovingtheneedforalarge numberofthereadoperationstoresultinnfstransactions,theread:writeratioisclosertounityfor theresultsofnfstracethantheresultsofsnooper. Whilethisratioisexpected,evendesirable,forthe measurementsofdatatransferred,thesevaluesare incorrectforopen-closesessionsresultinginhigher guresforaveragedatatransferredpersessionand incorrectinformationaboutthecharacteristicsofthe sessions. However,whiletheratiosofthevarioustypesof open-closesessionsproducedbynfstracearenot particularlyclosetothoseofrecordedbysnooper, addingtheguresfornullsessionstothereadopenclosesessionsimprovesthecomparisonforalllesystemsexceptfor/. ForTable10,allwritevaluesareincreasedby

13 /var/spool/mail FileSystem /111(24.89) 10(02.24) snooper 68(17.13) 8(02.02) rpcspy 98(22.37) nfstrace /usr/local /packages /usr2269(60.31)269(67,76)278(63.47) 49(10.99) 3(00.67) 4(00.90) 46(11.59) 3(00.76) 48(10.96) 8(01.83) Table7:Abreakdownperle-systemofthetotalnumberofdierentlesaccessedduringthetraceperiod. Total (00.68) Thevaluesinparenthesesareeachcountasapercentageofthetotalnumberofles. read-writeentries readentries7442(88.07)1749(68.51) 557(06.59) 35(00.41) snooper 804(31.49) nfstrace nullentries 416(04.92) - Table8:Thecountofopen-closesessionseachmonitoringsysteminterprets.Additionally,abreakdown Total oftheseopen-closesessionsintoread-only,write-only,read-writeandnullopen-closesessionsisshown.a nullsessioniswherenodataarereadfromorwrittentothele(althoughthelewasopened).valuesin parenthesesarethepercentageofthetotalnumberofleseachtyperepresents. FileSystemsessiontype /read read-write snooper (90.33)818 (04.98)476 nfstrace 298 (00.49) (04.20) (63.21) (36.79) /var/spool/mailread /usrread (73.21) (26.79) 61(100.00) write null (40.91) (50.00) (09.09) 75 7 (91.46) (08.54) /usr/localread /usr2read write 146(100.00)100(100.00) (74.52)686 (20.29)321 (68.12) (31.88) Table9:Abreakdownoftheopen-closesessionsoneachlesystembytypeofopen-closesession.Values /packagesread null 51 9(100.00) (05.20) 9(100.00) - - inparenthesesareeachtypeofoperationasapercentageoftheopen-closesessionsonthatlesystem.

14 errorwillpartlybebecausenfstraceinterpretsthe thatleastwoseparatewriteevents.additionally, nfstracecanincorrectlyinterpretmultiplewrites creationofanyleandanysubsequentwritingto nfstrace,particularlyinthecaseof/usr2.this nfstracecaninterpretmultiplewritesintothesame locationinaleasmultipleopen-closesessionson Becausenfstraceinterpretsanaccesstotherst tothesameleasconsecutiveopen-closesessions. thevieditor.viusesloglesthatcheck-pointthe thatle.asanexample,thissituationcanarisewith byteofaleasthestartofanewopen-closesession, ofwritesmadetotherstblock,therstblockmay editoperationsastheyoccuronthele,sovican becontinuallywritingsmallchangestothelogle. beingwrittentotheserverandifthereareanumber bewrittentotheserverseveraltimes.eachtimethe Thesecollectionsofsmallwriteswillresultinblocks rstblockiswrittennfstracecouldpotentiallymisinterpretthewritingofdataasseparateopen-close willbeoverwhelmedbythequantityofotheropenclosesessions(readsessionsinparticular).however, foropen-closesessionswritingtoale,theseextra incomparisonwithvaluesforallopen-closesessions, actualnumberofextrasessionsisquitesmalland, sessionsonthelogle.itisworthnotingthatthe open-closesessionscanbesignicant. havebeenmadebynfstracesoastoproducean rithmsusedbynfstrace.whilesomeassumptions open-closesessionrecord,thisparticularsituation mayberesolvedwithamoresophisticatednfstrace Someoftheseproblemsarearesultofthealgo- algorithm. leisinuse.this,inturn,isimportantincalculatingtheamountoftimelesaresharedbetween usersand,inadistributedlesystem,betweensystantindeterminingtheamountoftimeaparticular Thedurationofanopen-closesessionisimportems.Figure7showsthatdurationofopen-closesessionsrecordedbyrpcspywillbelongerthanthose recordedbysnooper.thelongeropen-closesessions thatcausethedierencesinaveragedurationsare ofseparateopen-closesessionsbeinginterpretedas likelytobearesultoftransactionsthatarepart partofthesameopen-closesession. session.thesesituationsarerepresentedgraphically bedierentfromtheaveragelengthoftheopen-close NFStracmeansthatleadandlagtimes(timesin whichtheleisopenbutnooperationoccurs)will Additionally,thecalculationofdurationfrom % of open-close operations Cumulative percentage of open-close operations against duration snooper nfstrace 60 Figure7:Normalisedcumulativedistributionofthe 50 numberofopen-closesessionsversustheduration open-closesessionsforagivennumberofthosesessions.forexample,thesnoopertechniquerecords 20 that70%ofthesessionshaveadurationofabout Fromthisgraphwecandeducethelongestofthe 10 1ms 10ms 100ms 1s 10s 1min 10min 1Hr 10Hr Duration logarithmic. infigure6.theguresshowthattheblockoperationsuponwhichnfstrace'srecordwillbebased operationsinanopen-closesession. sferrateasmeasuredbysnooper,asperfigure4, withtheamountofdatanfstraceestimateswaspotentiallyavailabletothesystem(thesumofthesizes oflesaccessed).whilenotdirectlycomparable,it Figure8graphesacomparisonofthedata-tran- 100millisecondsorless.Note:thedurationaxisis maynotcorrespondwiththelogicalopenandclose andserver. accuracythetrendsofdatatransferbetweenclient isworthnotingthatthesumofthesizesoflesis abletogiveenoughinformationtoestimatewithfair ferred.itisimportanttonotethatonereasonthat open-closesessionsversustheamountofdatatrans- nfstracedierssosignicantlywithsnooperisthat nfstracewasunabletodetectthelargepercentage Figure9showsacumulativedistributionof 1Kbytewastransferred.Additionally,snooperresultsestimatethatfewerthan500oftheopen-close sessionstransferredoneorzerobytes,whereasnfstraceresultsestimatethosecircumstancesexisted ofopen-closesessionsduringwhichapproximately 80,750,900and1,100bytesisbecausethoseles largenumberofsessionstransferringapproximately formorethan1,000ofthesessionsitrecorded. Aprimaryreasonnfstracedoesnotrecordthe

15 FileSystem read /var/spool/mail sessiontype / snooper (00.24) (86.20)818(46.77) (01.65) 75(04.29) 61(03.49) nfstrace /usr/local /packages /usr (00.12) (01.96)100(05.72) /var/spool/mail 354 (63.55)476(59.20) (09.82)686(39.22) /usr (00.72) (35.73)321(39.93) 7(00.87) 9(00.51) read-write null / (100.00) 45 (10.82) (71.63) Table10:Abreakdownoftheopen-closesessionsofeachtype,breakdownisbythelesystemofthele. /var/spool/mail /usr (05.29) (12.26) - Valuesinparenthesesareeachlesystem'soperationsasapercentageoftheopen-closesessionsofthat type. - Open Open Open Read Read Block Read Block Read Block Read Read Close Block Read Read Open Write Block Write Read Close Block Read Write Close Close Block Write transactioneachblockaccesswillinvoke. Figure6:Severalopen-closesessionsasgeneratedbynfstracearecomparedwiththeactualopen-close sessionthatoccurred.theopen-closesessiongeneratedbynfstracedependsheavilyonthetypeofnfs Case 1 Case 2 Case 3 real nfstrace

16 Bytes transferred 20 Mbyte 15 Mbyte 10 Mbyte 5 Mbyte Data over day (10 minute samples) snooper read (data transfered) nfstrace read (filesize) 0 Mbyte 12 Mbyte 10 Mbyte 11:00 13:00 15:00 17:00 19:00 21:00 23:00 01:00 03:00 05:00 07:00 09:00 11:00 Time of day 8 Mbyte 6 Mbyte Figure8:Thesegraphscomparethetransferratemeasuredwithsnooper,tothetotalamountofdata 4 Mbyte 2 Mbyte ofthetrace. nfstracehascalculatedtheclienthashadaccesstoineachlefromwhichithasreaddata.asalow-order approximation,thesevaluesarecomparablegivingthesamecharacteristicsfordatautilisationovertime 0 Mbyte 11:00 13:00 15:00 17:00 19:00 21:00 23:00 01:00 03:00 05:00 07:00 09:00 11:00 Time of day Bytes transferred Data over day (10 minute samples) snooper write (data transferred) nfstrace write (filesize)

17 Cumulative number of open-close operations against data transferred snooper nfstrace Cumulative number of open-close operations against file size snooper nfstrace Number of open-close operations Figure9:Cumulativedistributionofthenumberof eachopen-closesession.fromthisgraphwecandeducetheamountofdatatransferredperopen-close open-closesessionsversusthedatatransferredfor sessionforagivennumberofthosesessions.for example,thesnoopertechniquerecordsthatover K 10K 100K 1M Data transferred (bytes) areinthecacheandnodataaretransferredbetween Note:thedatatransferredaxisislogarithmic. 7,000sessionstransferabout1,100bytesofdata. tualnumberofsessionsisgreatlyreduced). ofsnooperforothertransfervalues(eveniftheac- serverandclient.thisreasoningisstrengthenedby thefactthatnfstracegivestrendssimilartothose infigure10haveresultedfromnfstracebeingunabletointerpretfrequentaccessestolesofacertain Thedierencesbetweensnooperandnfstrace 1,100bytesinsize.Accessesofsuchlesaccount length,inparticular,leswhichare80,750,900and recordinganopen-closesessionforthem.thisresultsinanexaggerationinthegraphsforthenumberofopen-closesessionsforcommondata-transfeerbatedbytheinabilityofnfstracetorecordmany oftheopen-closesessionsinwhichnodatatransfer ismade. 5.7Lossesduetorpcspy Duringthisstudy,therecordingofallEthernettraf- ifweassumearatioofnfstonon-nfstracat cbytherpcspymachinewasnotpossible(aloss thetotalnfstransactionsfromtherecordedtrace, sionsforregularly-accessedlesbutnfstraceisnot foralargepercentageoftheoverallopen-closeses- andle-sizevalues.thissituationisprobablyexac- of1.5%wasrecorded).thisimpliesalossof0.6%of Number of open-close operations Figure10:Cumulativedistributionofthenumberof 4000 open-closesessionsversusthesizeoftheleaccessed 3000 ineachopen-closesession.fromthisgraphwecan sions.forexample,thesnoopertechniquerecords deducethemaximumsizeoflesopenedforeach open-closesessionforagivennumberofthoseses K 10K 100K 1M File (bytes) thatover7,000ofthesessionsaccesslescontaining islogarithmic. thesameratioaswasrecordedduringthetestingof lessthan1,100bytesofdata.note:thelesizeaxis sourceofpotentialerror,thisdatalossisovershadowedbytheerrorsintroducedbycertainaspectsof theoperationofnfstrace.whilethiserrorshould notbediscounted,itcanbeconsideredtohavelow 6overallsignicanceintheresults. Theprecedingresultsshowthat,whilethetwosets ofresultsarenotdirectlycomparable,nfstraceis ComparisonSummary rpcspynetworkpacketcapturemechanism.whilea abletomakearstorderapproximationofanumberofvaluestraditionallymeasuredbysystemssuch machineorthequantityofdatawritten.additionciselybynfstraceinthecurrentversion,couldpotentiallygiveaccurateenoughresultstobeabletberofcircumstancesincludingmeasuringthenum- replacesystemssuchassnooperoutrightinanum- assnoopersuchasthetotali/otransferredbya ally,otherestimatedvalues,whileestimatedimpre- comparedwithresultsfromsnooperrelatetothe crepanciesintheinterpretationbynfstracewhen oflesizecomparedwithlesaccessed.mostdisberofactiveuserspermachineorthedistribution

18 identicationofopen-closesessions.minimisation oftheseerrorswouldimprovetheestimationofboth open-closesessiondurationanddata-sizeresults. Anumberoftheresultscollectedbynfstrace arenotcomparablewiththosecollectedbysnooper, e.g.theamountofdatatransferred.whilevalues forthemaximumdatatransferredandwriteoperationscanbecomparable,valuesaectedbysignicantcaching(e.g.readingofdata,particularlysmall amountsrepeatedlyfromthesamele)willdiersignicantly. Inadditiontomeasurementswhichcanbecompared,theuniquenatureofbothsnooperandrpcspymeansthateachhasaccesstodierenttypes ofinformation.snooperisideallysuitedtorecordinginformationaboutprocesses,anareafromwhich networkmonitorsareunabletoretrieveinformation. Ontheotherhand,nfstraceisideallysuitedtocollectinginformationaboutallmachinesonaparticularnetworkincluding,forexample,allthetrac foraparticularserver.thesedierencesmeaneach techniquehasaroletofullbutthereiscertainly potentialfornetworkmonitoringtobeabletomake measurementsforwhichkernelinstrumentationhas traditionallybeenusedinthepast. Additionally,itisworthpointingoutthatthe informationrpcspygeneratesandthatnfstracein turnuses,isnotinerror.thedierencesbetween nfstraceoutputandthatofsnooperoccurbecause nfstraceattemptstoestimatetheoperationsonthe usersideofthecachefromtheoperationsthatoccur onthelesystemsideofthecache.improvements intheperformanceofnfstracewouldcomeabout fromimprovementsinthisestimationprocess. 7 rpcspy/nfstraceproblems Fornfstracetobeamoreusefultool,theaccuracy ofitsestimationsneedstobeimproved.therearea numberofareaswherenfstraceeithermakeserrors ordoesnothaveenoughinformationwithwhichto work. nfstraceproblemstobeaddressed: 1.nfstracetreatsthecreationofaleastwo separateopen-closesessions. 2.Underestimationofthenumberofopen-close sessions.thisalsomeansnfstracecanoverestimatethedatatransferredperopen-close session,particularlyinthecaseofwrites. 3.nfstraceisunabletoobservelogicaldata transfer. 4.nfstracehasnorecordofopen-closesessions thattransfernodataatthelogicallevel. 5.nfstracehasnorecordofopen-closesessions thatbothreadandwritedata. 6.Thenfstracemethodusedforsummationof readoperationsandwriteoperationscanresult intransferreddatanotbeingcounted. 7.Themethodusedforestimatingthepurposeof annfsgetattrtransactionissimplistic. 8.nfstracedoesnotestimatethecontentsofa clientcache.asaresultnfstracewillassume lesincachearebeingaccessedwhenthisis notthecase. 9.nfstraceisunabletodetectshortopen-close sessions. Toalargeextenttheseproblemsarealsoaresult ofnfsnotmakingenoughinformationavailablefor nfstracetobeabletoestimatetheoperationsthat areoccurring.thelackofdatasuppliedbynfsalso meansnfstraceactsasalterremovingshort,consecutive,open-closereadsessions.suchoperations areabsorbedbythecacheandasaresultne-grain sporadicoperationsaremissed. 8 Improvingrpcspyandnfstrace Improvementstorpcspywillbeachievedbyusingahigh-speedmachinewithahigh-speed,lowlossnetworkinterfacetobededicatedtothetaskof datacollection.theimprovementstonfstracecan notbestatedquiteasconcisely.smallerchangesto nfstraceinclude: addingtheabilitytointerpretothersignicant NFStransactionssuchascreate, usingasimpleratiomultipliertoobtainanestimateofdatatransfersatthelogicallevel,

19 modicationofnfstracetokeepinformation aboutletruncationgivingtheabilitytointerpretlere-writeevents separatelyrecordingdatareadfromandwrittentotheserverforallopen-closesessions, recordinginformationonwhichblocksofale havebeenaccessed,and interpretingnfsgetattrtransactionsthat immediatelyfollowalebeingreadorwritten asanotheropen-closesession. Whilesomeofthesechanges,suchasthelast itemlisted,wouldneedtobetestedtoensurethe resultingextrarecordswerecorrect,othersinthelist wouldgiveimmediateimprovementintheabilitiesof nfstrace. Moresignicantchangestonfstraceinclude pre-loadinginformationaboutprogramsthat causestatsystemcallssuchasls, buildablockcachesimulatorintonfstrace Inordertopre-loadinformationaboutcommonly-usedprogramsthatcausestatsystemcalls,it maybenecessarytoprolethesystempriortoany signicanttracingactivity.inmostsystems,commonly-usedprogramssuchaslscouldbeexpected togeneratepotentialproblemsandcouldbeadded bydefault.however,theneedtodoaprolingoperationwouldnotonlyincreasethecomplexityof passivenetworkmonitoringbutmightalsonegate anyadvantageofnetworkmonitoringbypotentially requiringaccesstothemachinebeingmonitored. Anotheralternative,oraddition,topre-loadedcon- gurationinformationisfornfstracetocharacteriseprogramssuchaslsasitprocessesthenfstransactiondata.nfstracewouldlocatelstype programsbynotingprogramswhich,onceexecuted, causedclustersofnfslookupandgetattrtransactions,typicallyforlessharingthesamesubdirectory.inthisway,nfstracewouldbesimultaneouslyprocessingthedataandgainingenough informationtolocateprogramscausingextraneous NFSgetattrtransactionsthusimprovingthepredictionoflstypeprogramsduringthecourseofthe runṫheincorporationofablock-cachesimulator intonfstraceoersthebestpotentialforincreasing theaccuracyofnfstrace.unfortunately,several signicantitemsofinformationwouldbeneededto recreateaccuratelytheblockcacheofaclient.these wouldincludethecachesizeontheclient,thenumberofcacheentriesandthesizeofthedatablocksbeingtransferredbetweenclientandserver.additionally,theprogrammingandtestingofacachesimulatorisnotasimpletaskandbecauseofresourcesused (memory,etc.)wouldpotentiallymeannfstrace couldnotberunsimultaneouslywithrpcspywhich isapreferredoperatingmode(inordertoreduce outputdata). Theadditionofthesimulatorwouldmeanthat nfstracewouldbeattemptingtomodelaparticulartypeofblockcache.whilethereisacommonancestryforthemethodusedbyblockcaches inunixanditsderivatives,therearenotabledifferences.theintroductionofsuchfacilitiesasthe demand-pagingofexecutablesaswellassubtlechangesinthecachesystemmeansthebehaviourofthe cachesofsystemsbeingmonitoredwilldier,sometimesdramatically.theresultisthatnfstracemay berequiredtoincorporatemodelsforseveraldierentblock-cachesystems.whilethiswouldaddto thecomplexityofnfstrace,thecommonancestry ofblockcachesmeansmuchofthecodeusedineach simulatorwouldbecommontoall.itisconceivable thatsuchannfstracecouldreadaconguration lecontaininginformationonwhichcachemethod eachclientwasusing.withoutappropriatecongurationinformation,nfstracecouldassumeaparticularmodel,perhapsthemostcommoncachemethod usedortheworst-casesimulatormodel. Suchapre-loadedcongurationlewouldalso containinformationaboutnfsparameterssuchas cacheandattributetimeouts,therebyincreasingthe accuracyofthesimulator.thisinformation,ona le-systembyle-systembasis,couldalsogiveinformationaboutthecharacteristicsofaccesstoale system,e.g.maillesystemscanpotentiallycause open-closesessionstobegeneratedwhennonewas, andsoon. Ablock-cachesimulatorwouldincreasetheaccuracyoftheopen-closesessionpredictionsnfstrace makesandallownfstracetobeusedforotherpurposes.nfstracehasthepotentialtosimultaneously simulatethecachesofallthemachinesonanetwork soitcouldbeusedtostudyinteractionsbetweenthe cachesofdierentmachines.forexample,suchafacilitywouldenableacomprehensivestudyofblock

20 sharingamongnfsclients. Anextensiontonfstracewouldenableittokeep trackofinformationaboutthedirectorysystemsin adistributedlesystem.modicationstodirectory informationarewrittensynchronouslybacktothe serverasthemodicationstakeplacebutthedirectoryinformationitselfiscachedontheclients.becausechangesarewrittensynchronously,itispossiblefornfstracetomaintainanaccuratesimulationofthestateofthelesystem.additionally, nfstracecouldincorporateadirectory-namecache simulatorinthesamestyleasablock-cachesimulatorandbeabletosimulatethecontentsofthiscache amongmanyclients.asinthecaseofablock-cache simulator,adirectory-namecachesimulatorwould enablenfstracetobeusedtostudyinteractionsbetweenthecachesoftheclientsandtrackthehistory ofchangestothelesystem.theuseofsuchamodi- cationmayenableafollow-upstudytoshirriand Ousterhout'sworkonnameandattributecaching ([26]). Manyofthelimitationsinnfstrace,indeed,the veryneedfornfstracetohavetoestimateopenclosesessions,arecausedbythefactthatthisinformationaboutopenorcloseisnottransmittedinthe NFSprotocol.Otherdistributed-systemprotocols, suchassprite[19]andtheandrewfilesystem[12], transmitinformationrelatedtothestateoflesin thedistributedlesystem.ifnfstracewasmodi- edtoworkwithsuchastate-orientateddistributed system,theaccuracyofnfstraceoutputcouldpotentiallybeashighasafullkernelinstrumentation trace.thepotentialforaccuraterpcspy/nfstrace analysisofdistributedsystemsshouldalsoholdtrue foranydistributedlesystemthattransmitsenough stateinformationacrossthenetwork.thismethod evenhasthepotentialtoworkontheoreticaldistributedlesystems,suchasxfs[36,9],whichdepartfromacentralleservermodelcompletely.it isconceivablethatduringthedevelopmentofsuch monitoringsystems,methodsbasedonthepassive monitoringofnetworktracwouldbecomeaprimarytoolforassistinginthedevelopmentandultimatelythemanagementofsuchsystems. Anothertechniqueforincreasingtheaccuracyof nfstraceistoaddsimulatedstateoperationsto NFS.Thiswouldinvolvemodifyingthekernelof eachclienttooutputextranfstransactionsforsystemcallssuchasopen,closeandseek.itwould notbenecessaryfortheservertoactonoreven acknowledgethesecallsbutthetransmissionofthe extrainformationthroughthenetworkwouldpotentiallygivenfstraceenoughinformationtobeable toestablishwhenleswereopenedandclosed.of course,suchmodicationsarecontrarytomanyof theconceptsofpassivenetworkmonitoring,requiringmodicationstoperhapsmanyclientmachines. However,thistechniquewouldmaintainthebenet thatthecollectionofthetracedatawouldbeindependentoftheserverandclients.itwouldimpose noextraworkloaddirectlyuponthem.thismethod ofaddingadditionalinformationtothecommunicationstracbetweenclientandserver,forthepurposesofmonitoring,wasusedinbakeretal.[2]as oneofanumberofmodicationstheymadetocollectdatafortheirwork. Distributedcomputersystemsdonotconsist solelyofdistributedlesystems.systemsinthe styleofsprite[19]andamoeba[30]enablethemigrationofprocessesamongcpuelements(typically acpuelementisacomputerworkstation).amonitoringmethodforsuchasystemmightinvolvemonitoringthenetwork'sinterconnectingprocessingelementsandtrackingthemovementoftheprocessesin thesamewaythatnfstracemonitorsthemovement ofledataamongworkstations.inthisway,passive networkmonitoringhaspossibleapplicationsinareasotherthanjustthemonitoringofdistributedle systems. 9 Summarycomments Systemmonitoringhasasignicantroleinthedevelopmentofcomputersystems. Acommonmethodofmonitoringsystemsisto usefullkernelinstrumentation,involvingthemodi- cationofthesource-codefortheoperatingsystem ofthemachine.passivenetworkmonitoringcanbea preferredchoiceoverkernelinstrumentationforcertainsystemmonitoringwork,particularlywherethe source-codeoftheoperatingsystemisnotavailable. Additionally,otheradvantagesofpassivenetwork monitoringinclude: anindependenceofthecollectionofresults fromthemachinesbeingmonitoredonthenetwork, theabilitytomonitorsimultaneouslymultiple machinesonanetwork;thepassivenetwork