MeasuringWebPerformanceintheWideArea 111CummingtonSt,Boston,MA2215 PaulBarfordandMarkCrovella ComputerScienceDepartment BostonUniversity fbarford,crovellag@cs.bu.edu BU-CS-99-4 April26,1999 andcontinuedgrowthmakethisadicultquestiontoanswer.wedescribethewideareaweb whyusersoftenexperiencelongdelaysindocumentretrieval.theinternet'ssize,complexity, OneofthemostvexingquestionsfacingresearchersinterestedintheWorldWideWebis Abstract Measurementproject(WAWM)whichusesaninfrastructuredistributedacrosstheInternetto infrastructurearepresentedinthispaper. tracgeneratortocreaterepresentativeworkloadsonservers,andbothactiveandpassivetools studywebperformance.theinfrastructureenablessimultaneousmeasurementsofwebclient tomeasureperformancecharacteristics.initialresultsbasedonaprototypeinstallationofthe performance,networkperformanceandwebserverperformance.theinfrastructureusesaweb TheWorldWideWebisafascinatingexampleofadistributedsystemonanimmensescale.Its 1largescaleandincreasinglyimportantroleinsocietymakeitanimportantobjectofstudyand Introduction evaluation.atthesametime,sincethewebisnotcentrallyplannedorcongured,manybasic questionsaboutitsnaturearestillopen. preciselyaspossible,whilerecognizingthatitdoesnotadmittoasinglesimpleanswer. whataretherootcausesoflongresponsetimesintheweb?ourgoalistoanswerthisquestionas Thequestionthatmotivatesourworkisabasicone:whyistheWebsoslow?Moreprecisely: measurement.toanswerthisquestion,tworesearchneedsmustbeaddressed: 1.Theneedforintegratedserver/networkmeasurement.Mostrelatedresearchtodatehas AttemptingtoanswerthisquestionimmediatelyexposesgapsintheresearchtodateonInternet 2.Theneedtounderstandtherelationshipbetweenactiveandpassivemeasurementsinthe focusedonmeasuringserversandnetworksinisolation;theinteractionsbetweenthetwo, Internet.Activemeasurementsareeasilyunderstood,butdonotclearlypredictactualWeb especiallyinawide-areasetting,arenotwellunderstood. interpret. performance;passivewebmeasurementscanreectactualperformancebutcanbehardto 1

intendedtomeetthesetwoneeds.theprojectemploysbothnetworkandservermeasurement, andaswellusesbothpassiveandactivemeasurementapproaches.ourgoalinthispaperisto describetheproject,toexplainitsplaceinthecontextofinternetmeasurementresearch,andto Thispaperdescribesaprojectframework,calledWideAreaWebMeasurement(WAWM), demonstratetheutilityofourapproachusingsamplesofourearlymeasurements. taneously;ontakingconcurrentmeasurementsatboththeclientandtheserverendsofsample connections;andonvaryingserverloadsoastoallowcontrolledexplorationofawiderangeof server/networkinteractions.forexample,byvaryingserverloadandrunningexperimentsatdifferenttimesofday,wecanexplorethefourcasesinwhichtheserverandthewide-areanetwork Ourmeasurementarchitectureisbasedonmeasuringserverstateandnetworkstatesimul- issuesandactasaplatformfordevelopingtestinfrastructureandexpertise.itconsistsofaweb areindependentlyeitherheavilyorlightlyloaded(inqualitativeterms). theserver(evenuptoanoverloadedcondition).asingleo-siteclient,locatedacrosstheinternet serverrunningatoursite,alongwithanumberoflocalclientscapableofgeneratingvaryingloadon Thetestapparatusdecribedinthispaperisasmall-scaleprototypeintendedtoexposedesign (15hops)fromus,generatestestconnectionstoourserver.Passivepackettracemeasurementsare takenbothattheo-siteclientandattheserver.activemeasurementsoftheconnectionpathare alsocollectedduringeachtransferusingpoissonping(poip)[3]whichsendsudppacketsalong thesamepathastheconnectionandmeasuresthetime-averagednetworkstate(packetdelayand assessmentofwide-areawebperformance.nonethelesstheyyieldsomeimmediateinsightsabout deploy(asdescribedinsection3),itissucienttoyieldsurprisingandinformativeresults. lossrate).althoughthisapparatusisareducedversionofthefull-scalesystemwewilleventually andpassivenetworkmeasurement. thebenetsofintegratedserver/networkmeasurement,andabouttherelationshipbetweenactive TheresultswepresentinSection4areonlysamplesanddonotyetrepresentathorough serverloadistogenerateasignicantgapbetweenthetransmissionofconnectionsetuppacketsand therstdatapacketowingfromtheserver.whilethisgapcanbeunderstoodasaconsequence thenetwork.weshowthatforourexperimentalsetup,oneofthemostnoticeableeectsofhigh Forexample,wendthatserverloadhasdistinctiveeectsonthepatternofpacketowthrough ofsocketandleoperations,itisnotablethatfortypical(i.e.,short)transfers(2kb)thegap loaded(i.e.,packetlossratesarehigh),itisnotuncommonforaheavilyloadedservertoshow generallydominatestransfertime. bettermeanresponsetimethanalightlyloadedserver.ourmeasurementssuggestthatthismay bebecauseheavilyloadedserversoftenshowlowerpacketlossrates,andsincepacketlosseshave Inaddition,weshowanevenmoresurprisingeectofserverload.Whenthenetworkisheavily TCPconnections;thiseectmaybeduetothefactthatTCPpacketsowinaburstierpatternthan thatpoipmeasurementsofpacketdelayaregenerallylowerthanthoseexperiencedbypacketsin dramaticeectsontransferlatency,thiscanreducemeanresponsetime. dopoippackets.wealsoshowthatpoipmeasurementsofpacketlossarenotstronglypredictive Finally,wealsoexploretherelationshipbetweenactiveandpassivemeasurements.Weshow inthepresenceofloss. ofpacketlossexperiencedbytcpconnections,whichmaybeduetothefeedbacknatureoftcp themotivatingquestion:whyisthewebsoslow? thatalthoughtheanswersarenotyetinhand,ourapproachappearstoshowpromiseforaddressing Theseresultsdemonstratetheutilityofintegratednetwork/servermeasurement.Weconclude 2

2TheWAWMprojectisbasedonalargeanddiversebodyofpriorwork.Inparticular,werelyon resultsfromwebcharacterizationstudies,passiveandactivenetworkmeasurementstudies,time RelatedWork workwiththegoalsofthewawmproject. ofthestudiesfromeachoftheseareaswhichservedasthebasisforourworkandcontrastthat calibrationstudies,andwideareanetworkperformancestudies.inthissectionwepresentselections Alargebodyofworkhasdevelopedoverthepastveyearsthatpresentsmeasurementsand andperformancesuchasclientbehavior,serverbehavior,proxybehaviorornetworkbehavior. characterizationsoftheweb.thisworkhastypicallybeenfocusedonspecicaspectsofbehavior 2.1WebPerformanceCharacterization Clientstudiessuchas[2,24,25,33,39,72](aswellasclientproxystudies[2,16,26,31,42,45, 71,78,79])provideinformationonthebehaviorofusersacrossthemanyserversthattheymight accessduringbrowsingsessions.thisworkhasbeencriticalinbuildingthetoolsthatareusedto sametime.priorstudieshaveonlyhadaccesstoeithertheclientoraproxyortheserver. priorwebstudiesandthewawmprojectisthatwewillmonitorclientandserveractivityatthe simulateuseractivityinthewawmproject.themostsignicantdierencebetweenallofthe provideuswithmethodologiesformeasuringperformanceinboththeuserspaceandsystemspace. provideinsightintoserverbehaviorundervariousloadsandversionsofhttp.theyarefocusedon eithertheanalysisofhttplogsorondetailedmeasurementsofserverperformance.thesestudies Webserverbehaviorhasalsobeenstudiedextensively[5,6,11,14,15,32,47,55].Thesestudies aspossibleoratsomepredeterminedrate.workloadgeneratorssuchas[46,29,77]replayweb serverlogsorasummaryofserverlogs.weusethesurgeworkloadgenerator[1]togenerate generatorssuchas[1,9,23,22,59,75,76]areallbasedonmakingrepeatedrequestsasquickly ManytoolshavebeendevelopedforgeneratingworkloadsonWebservers.Syntheticworkload previousworkloadgeneratorsbecauseitincorporatesawiderangeorworkloadcharacteristicsthat representativelocalloadsonourserver.surgeisasyntheticworkloadgeneratorwhichisunlike areimportanttomanydierentaspectsofserverperformance. measurementsinoneplace(somewherenearanendpointofanhttptransaction)and,thus, tions.theshortfallofthesestudieswhenitcomestoanalyzinglatencyisthattheyonlytake showtheperformanceeectsofbothhttpprotocolinteractionandtcppacketlevelinterac- StudiesofthenetworkeectsofWebtracinclude[7,8,21,3,38,43,54,56].Thesestudies 2.2PassiveandActiveNetworkMeasurements cannotprovideinsightintothepacketdelaysinbothdirectionsofatransaction. Thetaskofassessinggeneralnetworkperformancehasbeenwellstudied.Agoodsourceofgeneral andthosewhichuseactivetechniques.passivetechniquestypicallyconsistofgatheringpacket informationoncurrentmeasurementtoolsandprojectscanbefoundat[73].generaltechniques leveldataatsometappointinthenetwork.examplesofpassivemonitorsincludecoral[58]and forstudyingnetworkperformancecanbesplitintothosewhichusepassivemonitoringtechniques in[17,18],andthecharacterizationoftheself-similarnatureoftracwasdevelopedfrompacket tracesin[41].anotherpassivemeasurementplatformhasbeenproposedin[44].thissystem, asinglepoint.forexample,earlypackettracestudiesofapplicationlevelinteractionsweredone tcpdump[74].agreatdealofinformationcanbeextractedfrompassivemonitoring,evenatonly calledwindmill,doesonlineanalysisofdatathathasbeencollectedpassively.oneofthebenets 3

passivemonitoringhaslimitationsintermsoftheamountofdatawhichisgatheredduringtests (typicallyverylarge)andthedicultyofrealtimeanalysisofthedata. ofpassivetechniquesisthattheirusedoesnotperturbthesystembeingmeasured.however, thenetworkandthenmeasuretheresponsetothatstimulus.activemeasurementtoolscanbe characteristicsbetweenhosts. usedtomeasurebulkthroughput,pathcharacteristics,packetdelay,packetloss,andbandwidth Activemeasurementtoolsinjectsomekindofstimulus(apacketorseriesofpackets)into thenetworkaccordingtoapoissonprocess.themotivationforthismethodisthatsamplingat intervalsdeterminedbyapoissonprocesswillresultinobservationsthatmatchthetime-averaged wecurrentlyusepoissonping(poip)[3]inthewawmproject.poipinjectsudppacketsinto Thereareanumberoftoolswhichcanbeusedtomeasureend-to-enddelayandpacketloss; stateofthesystem(assumingitisinastationarystate).thereareanumberofstudiesthathave usedsimilaractivemeasurementstostudypacketlossincluding[12,8]andtodrivemodelsoftcp approach. behaviorincluding[49,6].thesestudiesprovidebackgroundthatguidesouractivemeasurement \TracerouteReno"isatoolusedtomeasurebulkTCPthroughputcapacityoveranInternetpath. oute[36]isanactivemeasurementtoolthatgatherssimplepathcharacteristics.treno[48]or widthalongtheend-to-endpathincludingpathchar[37],bprobe[19],andnettimer[4].tracer- Anumberofactivemeasurementtoolshavebeendevelopedthatcanbeusedtomeasureband- projectwecompareactiveandpassivemeasurementsinordertounderstandhowactivemeasurementsofnetworkconditionscorrelatetoperformanceasseenattheclientandattheserver. datatheyinjectintothenetwork,theycanalsoperturbthenetwork'sperformance.inthewawm Theseactivemeasurementscangiveinsightintonetworkconditions,butdependingonhowmuch beendonebypaxsoninaseriesofstudiesreportedin[61,62,63,65].thesestudiesareastarting pointforthewawmprojectfrombothmeasurementandanalysisperspectives. Someofthemostextensiveworkusingbothpassiveandactivemeasurementtechniqueshas 2.3TimeCalibration Measurementsofend-to-enddelayinourarchitecturearebasedonhavingnearlysynchronized clocksontheclientandtheserver.thedicultiesofsynchronizingclocksinawidelydistributed Additionalstudiesofclocksynchronizationproblemsinclude[57,66].Paxsonpointsoutin[66] clientsandservers.ntpassuresthatclocksaresynchronizedontheorderoftensofmilliseconds. ofthenetworktimeprotocol(ntp)daemonwhichweuseinwawmtosynchronizetheclockson environmenthavebeenwellstudied,especiallybymills[52,53].thatworkledtothedevelopment thattheuseofntpprovidesclosesynchronizationofclocksbutthatchecksforskewshouldstillbe madewhenanalyzingdata.theparticularresultsreportedinthispaperarenotstronglysensitive 2.4WideAreaNetworkPerformanceMeasurement externaltimesource(gps). toclockskew,howeverinfutureworkweplantoeithercorrectforclockskeworuseasynchronized thepresenttime.noneproposetodothetypeofworkthatisbeingproposedforwawm;however Thereareanumberofwideareameasurementprojectswhichareeitherproposedorunderwayat therearesomesimilarities.theprojectswhicharethemostcloselyrelatedtowawmare: CooperativeAssociationforInternetDataAnalysis(CAIDA)[28].CAIDA'smissionisto \addressproblemsofinternettracmeasurementandperformanceandofinter-provider communicationandcooperationwithintheinternetserviceindustry." 4

TheNationalInternetMeasurementInfrastructure(NIMI)project[3].TheNIMIproject's IETF'sInternetProtocolPerformanceMetrics(IPPM)workinggroup[51,67].TheIPPM Internet." missionisto\createanarchitectureforfacilitatingameasurementinfrastructureforthe NLANR'sMeasurementandOperationsAnalysisTeam(MOATS)project[27].TheMOAT's workinggroup'smissionisto\developasetofstandardmetricsthatcanbeappliedtothe quality,performance,andreliabilityofinternetdatadeliveryservices." TheInternetPerformanceMeasurementandAnalysisproject(IPMA)[5].TheIPMA's missionisto\createanetworkanalysisinfrastructuretoderiveabetterunderstandingof missionisto\studytheperformanceofnetworksandnetworkingprotocolsinlocalandwideareanetworks." systemicservicemodelsandmetricsoftheinternet." KeynoteSystems,Inc.[34].KeynotePerspectiveisbilledasa\globalreal-timeservice" whichtheyuseto\continuallymeasuretheperformanceofpopularwebsitesandinternet TheSurveyorProject[69].EortsintheSurveyorprojectcenteronusingPoissonping-like backboneprovidersandregularlypublishtheresults(intheformofaperformanceindex)..." EachoftheseprojectsattemptstomeasureoranalyzesomeaspectofInternetperformance.The WAWMprojectissimilartotheaboveprojectsinthatitusesawideareameasurementinfrastructureandthattherearesignicantdatamanagementandanalysistasks.TheWAWMproject isdierentfromtheseforanumberofreasons.first,wawmprojectfocusesonauserlevelapplication theweb.mostoftheprojectslistedabovefocusongeneralnetworkmeasurement, tests,measurementsofuserlevelcharacteristicswillbemadeatthesametime.second,wawm proposestoinstallthesystemswhichwillactasserversintheinfrastructure.thisapproachfacili- monitoringandanalysis.althoughwawmproposestotakenetworklevelmeasurementsduring measurementsofone-waydelayandlossalongpathsbetweenanumberofremotesites. haveperformancemonitoringtoolswhichwillenabledetailedsystemmeasurementsduringtests. ofmanagement.third,thesystemswhichwillbeactingasserversinthewawmtestswillalso tateseaseandconsistencyofsoftwareinstallation,controlovertheenvironment,security,andease intofourcategories:measurementarchitectureandtools,managementtools,testingprotocol,and 3InthissectionwedescribethetoolsandmethodsusedintheWAWMproject.Webreakthemdown WAWMProjectOverview thespecicimplementationsusedintheresultsreportedhere. 3.1MeasurementArchitecture analysismethods.ineachcategorywerstdescribethelong-termprojectgoals,thenwedescribe 3.1.1GeneralArchitecture usedtoservedocumentstotheclients.serversarealsoconguredwithperformancemonitoring toolssothatdetailedmeasurementsofcpu,memoryandnetworkcharacteristicscanbemade ThegeneralarchitectureoftheWAWMhardwarecomponentsisshowninFigure1.Ontheleftare componentsthatarelocatedatoursite.therstsystemisconguredasthewebserverandisonly 5

Server Under Test LAN Load Generator(s) Figure1:GeneralArchitectureofWAWMHardwareComponents Active Passive Measurement Distributed Clients with thedurationofatest.thisisimportantsincewecannotexpectremotesystemstogenerate duringtests.thesecondandthirdsystemsareconguredaslocalloadgenerators.theyrunthe Surgeworkloadgeneratorwhichisusedtoadjusttheamountofloadontheserverthroughout Active Passive Measurement delays,packetlosses,pathroutes,andpathcharacteristicssuchasbottlenecklinkspeed.another systemisusedtotakebothpassivemeasurements(e.g.,packettraces)andactivemeasurements enoughloadtosaturatetheserverwhenwewanttoruntestsunderheavyserverload.thefourth system(notshown)isusedasthecontrolsystemwhichinitiatestestsandgathersdatafromthe infrastructure.allofthesystemsintheserverclusterareconnectedtothelocalinternetservice (e.g.,usingnetworkprobepackets).usingthesetoolsweexpecttomeasure(atminimum)packet istodistributethesesystemswidelyacrosstheinternetinanattempttoexploreawiderangeof pathtypes.theremotesystemsmakerequestsforlesfromtheserverduringtests.theyarealso provider(isp)suchthattheserverclusternetworktracisnotabottleneck. conguredwithbothpassiveandactivenetworkmeasurementtools.requestsaregeneratedby Ontherightsideofthegurearesystemsthatarelocatedremotelyfromoursite.Ourgoal clientsusingamodiedversionofsurge.thismodiedversioncanbeconguredtoruninits standardmodeortomakerepeatedrequestsforasingleleinmuchthesamewayaswebstone remotesiteswewouldliketohaveinthecompletewawm.thatstudyutilized37siteswhich [75]ẆeareinuencedbyPaxson'sNetworkProbeDaemonstudy[65]intermsofthenumberof enabledmeasurementofover1,distinctinternetpaths.selectionoftheadditionalsiteswillbe basedongeographiclocationandinternetconnectiontypewithheterogeneityasthegoal.atleast 3.1.2InitialPrototype oneadditionalserversitewillalsobeincludedtodiversifymeasurements. atbostonuniversity(bu)andconsistedofsixpcsconnectedviaswitched1mbpsethernet arealltakenfromthisprototypesystem.inourprototype,asingleserverclusterwaslocated issuesandactasaplatformfordevelopingtestinfrastructureexpertise.theresultsinthispaper Wehaveimplementedaprototypeofthisarchitectureinordertoexposemeasurementanddesign PentiumProCPUsand128MBofRAMeach.EachsystemranLinuxv2..3.Theserverwas conguredwithapache1.3.[68].traceroutev1.4a5andtcpdumpv3.4wereusedtomeasure routecharacteristicsandgatherpackettracesrespectively.xntpdv3-5.93wasusedtosynchronize whichisconnectedtothebuispviaa1mbpsbridge.thepcswereconguredwith2mhz timestampsbetweentheserverclusterandtheremotesystem.oneofthepcsintheservercluster 6

settoonesecondandwithapacketsizeof256bytes.thesystemsetupasthewebserverwas alsoconguredwithwebmonitor[4]tomeasuredetailedserverperformancecharacteristicsinthe wasconguredtorunbothtoandfromtheserverclusterwiththemeanmeasurementinterval wassetupasthetimeserverforalltests.thexntpdupdateintervalwassetto64seconds.poip CPU,memoryandnetworkcomponentsoftheserver. wererunundereitherloworhighlocalloadlevels.loadlevelsinsurgearedenedintermsof setuptomakerequestsforalesetwhichconsistedof2distinctles.thetotalsizeofthisdata setwasabout5mb;thustheentiredatasetcouldbecachedintheserver'sram.experiments ThelocalloadgeneratingsystemsintheserverclusterwereconguredwithSurgewhichwas arepresentedin[11].usinghttp/1.,weused4uesforlowloadand52uesforhighload. userequivalents(ues).characterizationsofloadunderarangeofuesforthesamepcsystems andranlinuxv2..35.itwasconnectedinthelocalareaviaswitched1mbpsethernetandthen ofdenver(du).thissystemwasconguredwitha333mhzpentiumprocpuand64mbofram totheduispviaa1mbpsbridge.thesystemwassetupwiththesamenetworkmeasurement TheprototypeinstallationusedonlyasingleremotesitewhichwaslocatedattheUniversity network:1kblesmeasureessentiallyonlytheconnectionsetuptime;2kblesaretypical-sized were1kb,2kband5kb.theselesizeswerechosentodemonstratearangeofbehaviorinthe andtimesynchronizationtoolsasthesystemsintheservercluster.themodiedversionofsurge Webtransfersandtypicallyinvolveabout6or7roundtrips,soaredominatedbyTCP'sslow-start wassetuptomakerepeatedrequestsforoneofthreelesontheserver.thethreelesizesselected 3.2ManagementArchitecture behavior;5kblesallowustoobservethemodeinwhichmostpacketsaresentintheweb, namelyintcp'scongestionavoidancemode. Inordertoruntests,downloaddataandmanagesystemswhicharepartoftheWAWMinfrastructure,asecuremanagementplatformisrequiredonallWAWMsystems.Weareevaluating third-partymanagementplatformsincludingthenimiplatformandinsoft'svitalagent[35].both providetheessentialmanagementfeaturesandexibilitynecessaryforwawmaswellasbuilt-in networkperformancemeasurementtools. designchoices.therefore,inordertogaininsightintotherelevantissues,wedevelopedourown managementtoolfortheprototypestudy.thetooliswritteninperlandisbasedonusingsecure Shell[7](SSHv1.2.26wasusedintheprototypetests).Thistoolprovedtobesucientforthe Themanagementplatformisanareaofthesysteminwhichpracticalmatterscandictate prototypetests. whichwawmtestsarerun.atthispointintimeasimplelestructureisadequatetostorethe data.eachtestruninourprototypestudywithonlyasingleclientresultedinabout9mbof byotherresearchers.thewawmdatarepositoryismaintainedatthesamecentralsitefrom Thedatacollectediscatalogedandmaintainedinsuchawayastofacilitateanalysisandreuse compressedresultsdata. 3.3.1GeneralArchitecture 3.3TestingProtocol AWAWMtestisaperiodoftimeoverwhichaspecicsetofclientsystemsmakerequeststoa specicserver.specicationoftestsrequiresdeterminingthefollowing: 1.Theserverclusterwhichwillbeusedinthetest. 7

2.Thelocalloadwhichwillbeplacedontheserver. 3.Thesetofremoteclientswhichwillmakerequestsduringatest.Theinuenceofdistance 4.Thetimeofdaythetestwillberun.DuetothediurnalcycleofInternettracinNorth willallinuencetheoutcomeofmeasurements. fromtheserver,internetconnectiontypeandthenumberofclientsparticipatinginatest 5.Thedurationofthetest.Thetestmustberunlongenoughtocollectarepresentativesample America,testsrunduringthedaytypicallyexperiencemorepronouncedinuencefromcross ofdataforthegivennetworkandserverconditions. tracthanthoserunlateatnight. 6.Thescheduleforactivenetworkmeasurementsduringtests.Sinceactivemeasurementscould 3.3.2InitialPrototype perturbhttpperformance,thisschedulemustbeselectedsuchthattheresultingdatagives aclearpictureofthenetwork'sstatewhilenotinuencingthehttptests. Thetestingprotocolusedintheprototyperanonesetoftestsduringtheday(between11:amand 7:pm)andonesetoftestsatnight(between9:3pmand5:am)foreachoftendays.Initiating testsatthesetimesprovidedavarietyofnetworkconditionsforourdata.eachtestbeganby alwayssucienttotransferatleast4ofthe5kblesunderheavynetworkloadconditions routesduringthetests.eachtestwasrunfor15minutes;weobservedthatthisdurationwas runningtraceroutetoandfromtheserverwhichgaveusanindicationofthecurrentend-to-end and4696ofthe1kblesunderlightnetworkloadconditions.poipwasrunbothtoandfrom wererunduringeachtestset.theyconsistedofmakingrepeatedrequestsforasinglele(1kb, 2KBor5KB)underbothlowandhighserverloads. theserversimultaneouslywiththehttprequestsforthedurationofeachtest.sixseparatetests 3.4.1GeneralArchitecture 3.4AnalysisProtocol Oncetestdataiscollected,itisreducedandanalyzed.Automationoftheanalysisprocessis Ourmodelisbasedontcpanaly[64].Inadditiontostatisticalapproaches,weintendtodevelop necessaryduetothelargeamountofdatathatisgatheredduringeachtest.wehavebegunthe methodsfordetailedanalysisofindividualletransfers.detailedanalysismayrequireavisual developmentofatoolcalledtcpevalwhichwillperformthesetasksonthepackettracedata. \coordinationtool"muchlikethetooldevelopedin[13]. 3.4.2InitialPrototype datagatheredthroughactiveandpassivemeasurements. Wealsousenatalie[3]toanalyzePoiptracesandintendtodevelopmethodstocorrelatethe Atthehighestlevelwecurrentlytabulatesummarystatisticsanddistributionalstatisticsofle ontimelinediagrams(e.g.,figures3,5,and7)becausetheycanrepresentthetimingofeventsat transferdelays,packetdelays,packetlossandserverperformance.tcpevalalsogeneratesanumber bothendsofaconnection. exchangesbetweenaclientandtheserverwithinasinglehttptransaction.inthispaperwefocus ofgraphicalaidstointerpretation,includingtimelinediagramswhichillustrateindividualpacket 8

Sizes FileServer LoadTrnsfdLatency Table1:Filetransferlatencystatisticsunderlightnetworkload Files MeanStd.Dev. Latency%pktlossmeandelaypktloss tcpdump tcpdumppoip%poipmean 5KB 2KB 1KB low 4696 1521 365.276.678 2.552.58.273.283..46.5.56. delay.44.45 5KB 2KB 1KB high 768 572 223 1.256 1.662 4.125 1.171 1.239 1.584.1..45.49.62.1.47.43.44 4 SampleResults.45 Inthissectionwereportoninitialresultsobtainedusingtheprototypeapparatusdescribedin measurementsshowedthattheroutetoandfromtheremotesitewasasymmetricbutfairlystable forallofthetests(13hopsfrombutoduand15hopsfromdutobu).whiletherewere Section3.Thetestingprotocolcalledfortraceroutetoberunbeforeandaftereachtest.These occasionalroutechangesbetweensetsoftest,wemeasuredonlyoneroutechangeduringoneof thetests. period,wepresentresultsofdatasetswhicharetypicalofallofthedatathatwascollected. networkloadandlesizesusedinthetests.whilemeasurementsweretakenoveratwoweek 4.1Measurementsofletransferlatency Firstwepresentperformancemeasurementsforletransfersovercombinationsofserverload, 4.1.1Performanceunderlightnetworkload Thedetailsoftheletransferlatencyunderlightnetworkloadforallthreelescanbeseenin Table1.Thetableshowsthatthebusyserveraddsapproximatelyonesecondtotheaveragele transferlatencyforthe1kband2kblesandover1.5secondstotheaverage5kbletransfer latency.meanpacketdelayandlosscharacteristicsunderlightnetworkloadasmeasuredbyboth PoipandtcpdumpcanbeseeninTable1.Delayandlossmeasurementsarefortheoutboundpath fromtheservertotheclient.thesenetworkmeasurementsshowthatthenetworkconditionswere fairlystablethroughoutthemeasurementperiod. relativelyconstantamountforeachlesize.theseeectscanbeseeinthecumulativedistribution forallthreelesizes.underheavyserverload,thevariabilityincreasesslightly.theprincipal eectofserverloadwhennetworkloadislowwastoincreasetheaverageletransfertimebya Underlightnetworkloadandlightserverload,theletransferlatencyhadverylowvariability percentageoftransfers,latencycanincreaseover6timeswhenserverloadishigh. function(cdf)diagramforthe2kbleinfigure2.thisdiagramalsoshowsthatforasmall networkmeasurementsisthroughtheuseoftcptimelineplots.thetimelinesfortypicaltransfers ofthe2kblecanbeseeninfigure3.intheseplots(aswellastherestofthetimelineplots inthissection),theclientisonthelefthandsideandtheserverisontheright.thetimeline Anotherwaytoanalyzeletransfersandtheeectsofvaryingserverloadforthelightlyloaded plotshighlightthedelayincurredforallletransferswhentheserverisbusy.afterthetcp 9

1.8.6 low server load high server load Figure2:CDFforletransferlatencyunderlightnetworkloadforthe2KBle.4.2 1 2 3 4 5 6 2 2 1.5 1.5 lowserverload highserverload 1 1.5.5 connectionsetupsequence,thereisroughlyaoneseconddelaybeforetherstdatapacketissent bytheserver.thisdelayisduetotheneedfortherequestfromtheclienttogetuptouserspace Figure3:Timelinediagramoftypical2KBletransfersunderlightnetworkload informationfromthecdfofletransferlatencyleadsustoconcludethatifthenetworkislightly loaded,themaximumadditionaldelaythatmosttransfersmightexperiencewhenaccessingour case,thisisnearlyinstantaneouswhentheserverislightlyloaded.thisdatacombinedwiththe andthenfortheresponsetogetbackdowntothenetwork.ascanbeseeninthelightlyloaded busyserverisaboutonesecond. Whenpacketdelaysandlossesincrease,transfercharacteristicschange.Thedetailsofthele transferlatencyunderheavynetworkloadforallthreelescanbeseenintable2.thenetwork 4.1.2Performanceunderheavynetworkload surementperiod.however,thesemeasurementsdonotcorrelatecloselywiththemeandelayand measurementsfrompoipindicatethatnetworkconditionswerefairlystablethroughoutthemea- networkloadtoheavynetworkload.theincreaseinpacketdelaysisanindicationofthelevelof lossvaluesextractedfromthetcpdumptraces.weexplorethisingreaterdetailinsection4.2. Packettransferdelaysextractedfromthetcpdumptracesincreasebetween13%and48%fromlight alsoshowsthatunderheavyserverloads,themeantransferlatencyincreasesinarangefrom1.3to increasesinarangefrom2.2to5timesthemeanlatencywhenthenetworkislightlyloaded.it congestioninthenetwork.thetableshowsthatunderlightserverloads,themeantransferlatency 1

Sizes FileServer LoadTrnsfdLatency Table2:Filetransferlatencystatisticsunderheavynetworkload Files MeanStd.Dev. Latency%pktlossmeandelaypktloss tcpdump tcpdumppoip%poipmean 5KB 2KB 1KB low 1543 656 7 12.977.728 1.53 1.373 1.827 4.51 4.2 3.8 3.2.52.56.83 5. 5.9 delay.51 5KB 2KB 1KB high 61 444 11 1.598 2.127 8.297 1.329 2.169 2.993 2.7 1.9 1.5.51.56.85 6.4 4.3 5.8.53.52.5.51 1.9.8.7 low server load.6 high server load.5 Figure4:CDFforletransferlatencyunderheavynetworkloadforthe2KBle.4.3.2.1 whentheserverloadishighversuswhenitislow.thiseectisexploredinmoredetailinsection tonoteisthattransferlatencyofthe5kbleunderheavynetworkloadsactuallydecreases 2timesthemeanlatencywhenthenetworkislightlyloaded.Perhapsthemostinterestingstatistic 1 2 3 4 5 6 takenduringthedaytimeperiodsforalldatasetsshowedhigherpacketlossanddelaycharacteristicsaswouldbeexpected.underheavynetworkload,letransferlatencyhadgenerallyhigher delayandhighervariabilityforallthreelesizesregardlessofserverload.theseeectscanbe Filetransferdelaysarestronglyinuencedbybothcongestionandpacketloss.Measurements 4.1.4. infigure2,thekneeinthecdfoccuredaroundthe9thpercentile,infigure4,thekneeismuch seenbycomparingfigure2withthecorrespondinggureforheavynetworkload(figure4).while lowerindicatingthatamuchlargerfractionoftransfersexperiencedseveredelays. timesversusthelightlyloadednetworktimelines,whichisanindicationofthelevelofcongestion beseeninfigure5.asinthelightlyloadednetworkcase,thedelayinsendingtherstdata packetfromtheserverisevident.thesediagramsshowageneralelongationinpackettransfer Thetimelinesfortypicaltransfersofthe2KBlewhenthenetworkisheavilyloadedcan inthenetworkduringthemeasurement. 4.1.3EectsofPacketLoss Measurementsmadeduringhighnetworkloadshowtheeectsofbothnetworkcongestionand packetloss.inthissectionwespecicallyexaminetheeectsofpacketlossonletransferlatency. 11

2 2 1.5 1.5 lowserverload highserverload 1 1.5.5 Table3:Filetransferstatisticsunderheavynetworkloadforleswithnodroppedpacketsorat leastonedroppedpacketfor5kble Figure5:Timelinediagramoftypical2KBletransfersunderheavynetworkload Sizes FileServerFilesw/o Load pktlosslatency MeanStd.Dev.Fileswith Latency pktlosslatency MeanStd.Dev. 5KB 2KB 1KB low 135 351.326.784 n/a.62.66 n/a 238 35 7 12.976 2.936 2.331 Latency 2.541 2.428 5KB 2KB 1KB high 536 314 6 3.947 1.337 1.869 1.53.763 1.956 14 13 74 8.548 3.491 2.752 2.548 4.51 2.771 2.511 Thedetailsoftheeectsofpacketlossunderheavynetworkloadandhighserverloadcanbeseen isalsomuchlowerthanthatoflestransferredwithloss. transferredwithoutpacketloss.asmightbeexpected,variabilityforlestransferredwithoutloss Filetransfersforleswhichloseatleastonepacketarebetween1.3and7.8timesslowerthanles intable3.thistableshowsthedierencesbetweenlestransferredwithandwithoutpacketloss. 2KBle.Ascanbeseenfromthegure,packetlosscansignicantlyincreasebothmeanand variabilityofletransferdelay.whencomparingthisgurewithfigure4itcanbeseenthat almostalllongtransferdelaysareduetopacketdrops.onenotablefeatureofthegureisthe Figure6comparestheimpactofpacketlossonletransferlatencyforthe1KBleandthe secondtimeoutperiodforthere-sendingofalostsynpacketinthelinuxtcpimplementation. cuspataboutthe4%levelforthe1kble.thisisduetolostsynpackets.thereisathree torestartslowstartinordertotransfertheremainingpackets.theoveralleectofthislossisthe Figure7.Thisgureshowstheeectofasinglelostpacketduringthedatatransferphaseofthe HTTPtransactionunderlightserverloadandheavynetworkload.ThepacketlosswillcauseTCP Thetimelinesfortransfersofatypical2KBlewithandwithoutpacketlosscanbeseenin additionofabout.5secondstotheletransferlatency. 12

1.9.8.7.6.5 Figure6:CDFforletransferlatencyunderheavynetworkloadandheavyserverloadforles.4 withandwithoutpacketloss.3 1K without drops 2K without drops.2 1K with drops 2K with drops.1 1 2 3 4 5 6 2 2 1.5 1.5 1 1 Figure7:Timelinediagramofa2KBletransfersunderheavynetworkloadandlowserverload withoutpacketloss withpacketloss.5.5 13

1.8 high server load low server load.6 Figure8:Filetransferlatencyforthe5KBletransfersunderheavynetworkload.4.2 5 1 15 2 ever,closerexaminationofthedatafromalltendatasetsshowsaninterestingphenomenon.for Initialeectsofserverloadontransferlatencieshavebeenshownintheprecedingsections.How- 4.1.4Eectsofserverload whenserverloadisincreasedduringthetests.furthermore,inthreeoutofthosesevencases,le transfersofthe5kble,insevenoutoftenofthedatasets,packetlossactuallygoesdown andthenetworkactivitycouldchangeenoughduringthistimetocauseperformancetoimprove. caseswhichisthesameasthedataintable2.thisresultcanpossiblybeattributedtooneoftwo things.first,theexperimentsforthe5kbletransferarerunapproximately45minutesapart latencydecreaseswhenserverloadisincreased.thiseectcanbeseeninfigure8foroneofthese otherpossibleexplanationisthattheheavyloadontheserverpreventsitfromproducingasmany However,activemeasurementsindicatethatthenetworkwasfairlyconsistentacrossalltests.An- burstsofpacketsasitmightnormallyifitwerelightlyloaded.weintendtofurtherinvestigate 4.2Activeversuspassivemeasurementsofnetworkconditions thiseectunderawiderrangeofconditionsandconclusivelydetermineitscause. thetcpstreamsthemselvesoftendisagree.thisdierenceisimportantbecauseitsuggeststhat Aswasseenintheprevioussection,valuesfordelayandlossfromPoipmeasurementsandfrom takenfromtcpstreams.infigure9weshowthecdfsofpacketdelayexperiencedbytwosetsof nections.inthissectionweexplorethesedierencesandtheirimplications. measurementsderivedfrompoipmaynotbeindicativeoftheexpectedperformanceoftcpcon- tolightlyloadednetworkconditions;theplotontherightcorrespondstoheavilyloadednetwork 5KBTCPowsandbyPoipduringthesametwotimeintervals.Theplotontheleftcorresponds OurrstobservationisthatPoip'smeasurementsofpacketdelayaretypicallylowerthanthose conditions.typicaldierencesbetweenthemeanvaluesreportedbypoipandthoseexperiencedby TCPpacketsareontheorderof1ms.NotethatthePoipmeasurementsweremadeconcurrently withthetcpconnections,sotheyaremeasuringthesamenetworkstate. thattcpoftentransmitspacketsinburstsontheorderofthecongestionwindowsize,thenwaits processisconsistentwiththenotionthattcp'spacketstreamisburstier(i.e.,interarrivalshave ahighercoecientofvariation)thanthatofapoissonprocess.thisseemsclearwhenconsidering ThehigherdelayexperiencedbyTCPpacketscomparedtopacketsarrivingviaaPoisson foracknowledgments.thestartofthenextburstisdeterminedbyroundtriptimeandgenerally occurssometimeafterthecurrentbursthasbeensent,leadingtoahighcoecientofvariationin 14

1.8.6.6 Poip high SL.5 Poip low SL Figure9:CDFofpacketdelayfromservertoclientduringtwo5KBletests(left:lightlyloaded.4 Pkts high SL.4 Pkts low SL network;right:heavilyloadednetwork;sl=serverload)..3.2.2.1.2.4.6.8.1.2.4.6.8.1 1.9.8.7 Poip high SL Poip low SL Pkts high SL Pkts low SL 1 % Packet loss from Poip 8 6 Figure1:ScatterplotofpacketlossrateinTCPvs.Poip. 4 2 related.figure1showsascatterplotoflossrateexperiencedbytcp(onthexaxis)versusthat interpacketspacing. OursecondobservationisthatpacketlossasexperiencedbyTCPandbyPoiparenotstrongly 2 4 6 8 % Packet from tcpdump experiencedbypoipduringthesametimeframe(ontheyaxis)forallofour5kbexperiments. Ifthetwomeasurementswerestronglyrelated,pointswouldtendtooccuraroundtheliney=x (whichisplottedforreference);theydonot.ingeneral,itseemsthatpoipseesahigherpacket lossestriggerfeedbackbehaviorintcp.uponencounteringpacketloss,tcpoftenentersastate lossratethantcp,althoughthisisnotuniversallytrue. inwhichnopacketsowforsometime,afterwhichitrecoversandpacketsbegintoowagainat areducedrate(forexample,seefigure7,rightside).thisbehaviormeansthattcp'sviewofthe Thereasonforthislackofcorrelationmaybethat unlikethecasewithpacketdelays packet fewerlosseventsthanareseenbyapoissonprocess(whichisindependentofpacketlossevents). networkisnotindependentofpacketlossevents;asaresult,itmaybethattcptendstoobserve ItseemsclearinanycasethatTCP'sfeedbackbehaviormeansthatitsviewofnetworklossesare conditionsasexperiencedbytcp.1asregardspacketdelays,theburstynatureoftcpows notstronglycorrelatedwiththatofatime-averageview. networkstate.itseemsthatpoiphasonlylimitedvalueinpredictingthenatureofnetwork 1Tobefair,itisimportanttonotethatPoipwasnotdesignedtoassessTCP'sviewofthenetwork,andthatit ThesetwoobservationshaveimplicationsfortheuseofPoipandsimilartoolstomeasure 15

meansthatpacketstypicallyseelargerqueuesthanthetime-averagedconditions.withrespectto packetloss,itseemsthatthefeedbacknatureoftcp'scongestionavoidancemakesitveryhard topredicttcp'sactualpacketlossesusinganopen-looptoollikepoip. 5InthispaperwehavedescribedtheWAWMprojectandarguedthatitrepresentsanapproach thatisbothnovelanduseful.itsnoveltyarisesfromitstreatmentoftheserverandnetworkas Conclusion behaviorandnetworkbehavior,andtoidentifycasesinwhichthetwodonotinteroperatewell. anintegratedsystem.itsutilitycomesfromtheabilitytoexploretheinteractionbetweenserver wehaveshownthatinmanyofourexperiments,serversunderhighloadsueredsignicantlyless canyieldinformativemeasurements.forexample,usingitwehaveshownthat(forourserver)the maineectofserverloadontypicaltransfersistodelaytherstdatapacketsent.inaddition Wehavedescribedasmall-scaleimplementationoftheprojectarchitectureandshownthatit packetlossthanthoseunderlowload.incomparingpacketdelayandlossesintcpconnections tomeasurementsobtainedwithpoipwendthatthereareconsiderable(andunderstandable) dierences.theseresultsareonlyinitiallooksatourdataandrequireconrmationinawider rangeofsettings;howevertheyaresuggestiveofinterestingeects. toexpandthemeasurementapparatusuntilitallowsustoassessrepresentativebehaviorofthe infrastructuretoaddressthisquestionwillrequireprogressalongtwodimensions.first,weneed Web.Thismeansthatweneedtoaddmoreremoteclients(frommorelocations),andconsidera Ourmotivatingquestioninthisprojectis:WhyistheWebsoslow?TousetheWAWM widerrangeofserverandplatforms(e.g.,iisrunningonwindowsnt). methodsthatoperateontracesofindividualtransfersandassesstherelativeimpactofserver hopetoobtainamorepreciseunderstandofthecausesoftransferlatencyintheweb. delay,networkdelay,packetloss,etc.ontransferlatency.usingthesemoredetailedanalyseswe Theseconddimensionisthatofanalyticmethods.Weintendtodevelopcharacterization studyaswellasprovidinginputontheprojectdesign.theauthorswouldalsoliketothanklars paper.theauthorswouldalsoliketothankvernpaxsonforprovidingpoissonpingforuseinthis Denverforaccesstothesystemwhichwasusedastheremoteclientforthetestspresentedinthis Acknowledgements.TheauthorswouldliketothankDavidMartinfromtheUniversityof Kellogg-Stedmanforhishelpinsettingupsystemswhichwereusedintheproject. References [1]WebBench2..http://www.zdnet.com/zdbop/webbench/webbench.html. [2]G.Abdulla,E.Fox,andM.Abrams.Shareduserbehaviorontheworldwideweb.InWebNet97,Toronto, [4]J.Almeida,V.Almeida,andD.Yates.Measuringthebehaviorofaworldwidewebserver.InProceedingsof [3]A.Adams,J.Mahdavi,M.Mathis,andV.Paxson.Creatingascalablearchitectureforinternetmeasurement. http:==www:psc:edu=mahdavi=nimipaper=nimi:html,1998. Canada,October1997. [5]V.Almeida,A.Bestavros,M.Crovella,andA.deOliveira.CharacterizingreferencelocalityintheWWW. theseventhifipconferenceonhighperformancenetworking(hpn),whiteplains,ny,april1997. hasmanyusesotherthantheonethatweareconsidering. pages92{13,december1996. InProceedingsof1996InternationalConferenceonParallelandDistributedInformationSystems(PDIS'96), 16

[6]M.ArlittandC.Williamson.Internetwebservers:workloadcharacterizationandperformanceimplications. IEEE/ACMTransactionsonNetworking,5(5):631{645,October1997. [7]H.Balakrishnan,V.Padmanabhan,S.Seshan,M.Stemm,andR.Katz.Tcpbehaviorofabusyinternetserver: Analysisandimprovements.InProceedingsofIEEEINFOCOM'98,SanFrancisco,CA,March1998. [8]H.Balakrishnan,S.Seshan,M.Stemm,andR.Katz.Analyzingstabilityinwide-areanetworkperformance.In ProceedingsofACMSIGMETRICS'97,Seattle,WA,June1997. [9]G.BangaandP.Druschel.Measuringthecapacityofawebserver.InProceedingsoftheUSENIXAnnual TechnicalConference,Monterey,CA,December1997. [1]P.BarfordandM.Crovella.Generatingrepresentativeworkloadsfornetworkandserverperformanceevaluation. InProceedingsofACMSIGMETRICS'98,pages151{16,Madison,WI,June1998. [11]P.BarfordandM.Crovella.Aperformanceevaluationofhypertexttransferprotocols.InToappearin ProceedingsofACMSIGMETRICS'99,Atlanta,GA,May1999. [12]J.Bolot.End-to-endpacketdelayandlossbehaviorintheinternet.InProceedingsofACMSIGCOMM'93, SanFrancisco,Setpember1993. [13]L.Brakmo,S.O'Malley,andL.Peterson.Tcpvegas:Newtechniquesforcongestiondetectionandavoidance. InProceedingsofACMSIGMETRICS'96,Philadelphia,PA,May1996. [14]H.BraunandK.Clay.Webtraccharacterization:Anassessmentoftheimpactofcachingdocumentsfrom ncsa'swebserver.inproceedingsofthesecondinternationalwwwconference,chicago,il,october1994. [15]T.Bray.Measuringtheweb.InFifthInternationalWorldWideWebConference,Paris,France,May1996. [16]L.Breslau,P.Cao,L.Fan,G.Phillips,andS.Shenker.Webcachingandzipf-likedistributions:Evidenceand implications.inproceedingsofieeeinfocom'99,newyork,ny,march1999. [17]R.Caceres.Measurementsofwideareainternettrac.TechnicalReportUCB/CSD89/55,ComputerScience Department,UniversityofCalifornia,Berkeley,1989. [18]R.Caceres,P.Danzig,S.Jamin,andD.Mitzel.Characteristicsofwide-areatcp/ipconversations.InProceedings ofacmsigcomm'91,september1991. [19]R.CarterandM.Crovella.Measuringbottlenecklinkspeedinpacket-switchednetworks.InProceedingsof Performance'96,Lausanne,Switzerland,October1996. [2]L.CatledgeandJ.Pitkow.Characterizingbrowsingstrategiesintheworldwideweb.ComputerNetworksand ISDNSystems,26(6):165{173,1995. [21]S.Cheng,K.Lai,andM.Baker.AnalysisofHTTP/1.1onaWirelessNetwork.Technicalreport,Stanford University,1998. [22]HTTPClient.http://www.innovation.ch/java/HTTPClient/. [23]TheStandardPerformanceEvaluationCorporation.Specweb96.http://www.specbench.org/org/web96/. [24]M.CrovellaandA.Bestavros.Self-similarityinworldwidewebtrac:Evidenceandpossiblecauses.InProceedingsofthe1996ACMSIGMETRICSInternationalConferenceonMeasurementandModelingofComputer Systems,Philadelphia,PA,May1996. [25]C.Cunha,A.Bestavros,andM.Crovella.Characteristicsofwwwclient-basedtraces.TechnicalReportTR-95-1,BostonUniversityDepartmentofComputerScience,April1995. [26]A.Feldmann,R.Caceres,F.Douglas,andM.Rabinovich.Performanceofwebproxycachinginhetrogeneous bandwidthenvironments.inproceedingsofieeeinfocom'99,newyork,ny,march1999. [27]NationalLaboratoryforAppliedNetworkResearch.http://www.nlanr.net,1998. [28]CooperativeAssociationforInternetDataAnalysis.http://www.caida.org,1998. [29]H.Frystyk-Nielsen.Libwww.http://www.w3.org/Library/. [3]H.Frystyk-Nielsen,J.Gettys,A.Baird-Smith,E.Prud'hommeaux,H.Wium-Lie,andC.Lilley.Network performanceeectsofhttp/1.1,css1andpng.inproceedingsofacmsigcomm'97,cannes,france, Setpember1997. [31]S.Glassman.Acachingrelayfortheworldwideweb.ComputerNetworksandISDNSystems,27(2),1994. [32]S.GribbleandE.Brewer.Systemdesignissuesforinternetmiddlewareservices:Deductionsfromalargeclient trace.inproceedingsofthe1997usenixsymposiumoninternettechnologiesandsystems(usits'97), Monterey,CA,December1997. 17

[33]B.Huberman,P.Pirolli,J.Pitkow,andR.Lukose.Strongregularitiesinworldwidewebsurng.Science, 28:95{97,1998. [34]KeynoteSystemsInc.http://www.keynote.com,1998. [35]INSoft.http://www.vitalsigns.com,1998. [36]V.Jacobson.traceroute.ftp://ftp.ee.lbl.gov/traceroute.tar.Z,1989. [37]V.Jacobson.pathchar.ftp://ftp.ee.lbl.gov/pathchar/msri-talk.ps.gz,1997. [38]S.KhaunteandJ.Limb.Statisticalcharacterizationofaworldwidewebbrowsingsession.TechnicalReport GIT-CC-97-17,CollegeofComputing,GeorgiaInstituteofTechnology,1997. [39]T.Kwan,R.McGrath,andD.Reed.UseraccesspatternstoNCSA'sWWWserver.TechnicalReportUIUCDCS- R-95-1934,UniversityofIllinois,DepartmentofComputerScience,February1995. [4]K.LaiandM.Baker.Measuringbandwidth.InProceedingsofIEEEINFOCOM'99,NewYork,NY,March 1999. [41]W.Leland,M.Taqqu,W.Willinger,andD.Wilson.Ontheself-similarnatureofethernettrac(extended version).ieee/acmtransactionsonnetworking,pages2:1{15,1994. [42]B.LiuandE.Fox.Webtraclatency:Characteristicsandimplications.InWebNet98,Orlando,FL,November 1998. [43]B.Mah.AnempiricalmodelofHTTPnetworktrac.InProceedingsofINFOCOM'97,Kobe,Japan,April 1997. [44]R.MalanandF.Jahanian.Anextensibleprobearchitecturefornetworkprotocolperformancemeasurement. InProceedingsofACMSIGCOMM'98,Vancouver,Canada,September1998. [45]C.Maltzahn,K.Richardson,andD.Grunwald.Performanceissuesofenterpriselevelwebproxies.InProceedings ofacmsigmetrics'97,seattle,wa,june1997. [46]S.Manley,M.Courage,andM.Seltzer.Aself-scalingandself-conguringbenchmarkforwebservers.Harvard University,1997. [47]S.ManleyandM.Seltzer.Webfactsandfantasy.InProceedingsofthe1997USENIXSymposiumonInternet TechnologiesandSystems,Monterey,CA,December1997. [48]M.Mathis.Empiricalbulktransfercapacity.ftp://ftp.advanced.org/pub/IPPM/treno.txt,July1997. [49]M.Mathis,J.Semke,J.Mahdavi,andT.Ott.Themacroscopicbehaviorofthetcpcongestionavoidance algorithm.computercommunicationsreview,27(3),july1997. [5]InternetPerformanceMeasurementandAnalysisProjecct.http://nic.merit.edu/ipma/,1998. [51]InternetProtocolPerformanceMetrics.http://www.advanced.org/ippm/index.html,1998. [52]D.Mills.Networktimeprotocol(version3):Specication,implementationandanalysis.TechnicalReportRFC 135,NetworkInformationCenter,SRIInternational,MenloPark,CA,1992. [53]D.Mills.Improvedalgorithmsforsynchronizingcomputernetworkclocks.IEEE/ACMTransactionsonNetworking,3(3):245{254,June1998. [54]J.Mogul.Thecaseforpersistent-connectionHTTP.TechnicalReportWRL95/4,DECWesternResearch Laboratory,PaloAlto,CA,1995. [55]J.Mogul.Networkbehaviorofabusywebserveranditsclients.TechnicalReportWRL95/5,DECWestern ResearchLaboratory,PaloAlto,CA,1995. [56]J.Mogul,F.Douglis,A.Feldmann,andB.Krishnamurthy.Potentialbenetsofdeltaencodinganddata compressionforhttp.inproceedingsofacmsigcomm'97,cannes,france,setpember1997. [57]S.Moon,P.Skelley,andD.Towsley.Estimationandremovalofclockskewfromnetworkdelaymeasurements. InProceedingsofIEEEINFOCOM'99,NewYork,NY,March1999. [58]Coral:Passivenetworktracmonitoringandstatisticscollection.http:==www:caida:org=tools=coral. [59]UniversityofMinnesota.Gstoneversion1.http://web66.coled.umn.edu/gstone/info.html. [6]J.Padhye,V.Firoiu,D.Towsley,andJ.Kurose.Modelingtcpthroughput:Asimplemodelanditsempirical validation.inproceedingsofacmsigcomm'98,vancouver,canada,setpember1998. [61]V.Paxon.Fast,approximatesynthesisoffractionalgaussiannoiseforgeneratingself-similarnetworktrac. ComputerCommunicationsReview,27(5):5{18,October1997. 18

[62]V.Paxson.Empirically-derivedanalyticmodelsofwide-areatcpconnections.IEEE/ACMTransactionson [64]V.Paxson.Automatedpackettraceanalysisoftcpimplementations.InProceedingsofACMSIGCOMM'97, [63]V.Paxson.End-to-endroutingbehaviorintheinternet.InProceedingsofACMSIGCOMM'96,PaloAlto,CA, August1996. Networking,pages316{336,August1994. [65]V.Paxson.End-to-endinternetpacketdynamics.InProceedingsofACMSIGCOMM'97,Cannes,France, [66]V.Paxson.Oncalibratingmeasurementsofpackettransittimes.InProceedingsofACMSIGMETRICS'98, Cannes,France,September1997. [67]V.Paxson,G.Almes,J.Mahdavi,andM.Mathis.FrameworkforIPperformancemetrics,RFC233. pages11{21,madison,wi,june1998. [7]SSHCommunicationsSecurity.http:==www:ssh:fi,1999. [69]TheSurveyorProject.http:==www:advanced:org=csg ippm=,1998. [68]ApacheHTTPServerProject.http://www.apache.org,1998. ftp://ftp.isi.edu/in-notes/rfc233.txt,1998. [72]L.Tauscher.Evaluatinghistorymechanisms:anempiricalstudyofreusepatternsinworldwidewebnavigation. [71]N.Smith.Whatcanarchivesoertheworldwideweb.InTheFirstInternationalWorldWideWebConference, Geneva,Switzerland,May1994. [75]G.TrentandM.Sake.Webstone:Therstgenerationinhttpserverbenchmarking,February1995.Silicon [73]CAIDAMeasurementToolTaxonomy.http:==www:caida:org=tools=taxonomy. [74]tcpdump.http:==ftp:ee:lbl:gov=tcpdump:tar:Z. Master'sthesis,DepartmentofComputerScience,UniversityofCalgary,Alberta,Canada,1996. [76]WebCompare.http://webcompare.iworld.com/. [78]R.Wooster.Optimizingresponsetime,ratherthanhitrates,ofwwwproxycaches.Master'sthesis,Virginia [77]Webjamma.http://www.cs.vt.edu/chitra/webjamma.html. GraphicsWhitePaper. [79]R.WoosterandM.Abrams.Proxycachingthatestimatespageloaddelays.InSixthFirstInternationalWorld [8]M.Yajnik,S.Moon,J.Kurose,andD.Towsley.Measurementandmodelingoftemporaldependenceinpacket WideWebConference,SantaClara,California,1997. PolytechnicInstituteandStateUniversity,Blacksbury,Virginia,1996. loss.inproceedingsofieeeinfocom'99,newyork,ny,march1999. 19