Transcription

1 MeasuringWebPerformanceintheWideArea 111CummingtonSt,Boston,MA2215 PaulBarfordandMarkCrovella ComputerScienceDepartment BostonUniversity BU-CS-99-4 April26,1999 andcontinuedgrowthmakethisadicultquestiontoanswer.wedescribethewideareaweb whyusersoftenexperiencelongdelaysindocumentretrieval.theinternet'ssize,complexity, OneofthemostvexingquestionsfacingresearchersinterestedintheWorldWideWebis Abstract Measurementproject(WAWM)whichusesaninfrastructuredistributedacrosstheInternetto infrastructurearepresentedinthispaper. tracgeneratortocreaterepresentativeworkloadsonservers,andbothactiveandpassivetools studywebperformance.theinfrastructureenablessimultaneousmeasurementsofwebclient tomeasureperformancecharacteristics.initialresultsbasedonaprototypeinstallationofthe performance,networkperformanceandwebserverperformance.theinfrastructureusesaweb TheWorldWideWebisafascinatingexampleofadistributedsystemonanimmensescale.Its 1largescaleandincreasinglyimportantroleinsocietymakeitanimportantobjectofstudyand Introduction evaluation.atthesametime,sincethewebisnotcentrallyplannedorcongured,manybasic questionsaboutitsnaturearestillopen. preciselyaspossible,whilerecognizingthatitdoesnotadmittoasinglesimpleanswer. whataretherootcausesoflongresponsetimesintheweb?ourgoalistoanswerthisquestionas Thequestionthatmotivatesourworkisabasicone:whyistheWebsoslow?Moreprecisely: measurement.toanswerthisquestion,tworesearchneedsmustbeaddressed: 1.Theneedforintegratedserver/networkmeasurement.Mostrelatedresearchtodatehas AttemptingtoanswerthisquestionimmediatelyexposesgapsintheresearchtodateonInternet 2.Theneedtounderstandtherelationshipbetweenactiveandpassivemeasurementsinthe focusedonmeasuringserversandnetworksinisolation;theinteractionsbetweenthetwo, Internet.Activemeasurementsareeasilyunderstood,butdonotclearlypredictactualWeb especiallyinawide-areasetting,arenotwellunderstood. interpret. performance;passivewebmeasurementscanreectactualperformancebutcanbehardto 1

2 intendedtomeetthesetwoneeds.theprojectemploysbothnetworkandservermeasurement, andaswellusesbothpassiveandactivemeasurementapproaches.ourgoalinthispaperisto describetheproject,toexplainitsplaceinthecontextofinternetmeasurementresearch,andto Thispaperdescribesaprojectframework,calledWideAreaWebMeasurement(WAWM), demonstratetheutilityofourapproachusingsamplesofourearlymeasurements. taneously;ontakingconcurrentmeasurementsatboththeclientandtheserverendsofsample connections;andonvaryingserverloadsoastoallowcontrolledexplorationofawiderangeof server/networkinteractions.forexample,byvaryingserverloadandrunningexperimentsatdifferenttimesofday,wecanexplorethefourcasesinwhichtheserverandthewide-areanetwork Ourmeasurementarchitectureisbasedonmeasuringserverstateandnetworkstatesimul- issuesandactasaplatformfordevelopingtestinfrastructureandexpertise.itconsistsofaweb areindependentlyeitherheavilyorlightlyloaded(inqualitativeterms). theserver(evenuptoanoverloadedcondition).asingleo-siteclient,locatedacrosstheinternet serverrunningatoursite,alongwithanumberoflocalclientscapableofgeneratingvaryingloadon Thetestapparatusdecribedinthispaperisasmall-scaleprototypeintendedtoexposedesign (15hops)fromus,generatestestconnectionstoourserver.Passivepackettracemeasurementsare takenbothattheo-siteclientandattheserver.activemeasurementsoftheconnectionpathare alsocollectedduringeachtransferusingpoissonping(poip)[3]whichsendsudppacketsalong thesamepathastheconnectionandmeasuresthetime-averagednetworkstate(packetdelayand assessmentofwide-areawebperformance.nonethelesstheyyieldsomeimmediateinsightsabout deploy(asdescribedinsection3),itissucienttoyieldsurprisingandinformativeresults. lossrate).althoughthisapparatusisareducedversionofthefull-scalesystemwewilleventually andpassivenetworkmeasurement. thebenetsofintegratedserver/networkmeasurement,andabouttherelationshipbetweenactive TheresultswepresentinSection4areonlysamplesanddonotyetrepresentathorough serverloadistogenerateasignicantgapbetweenthetransmissionofconnectionsetuppacketsand therstdatapacketowingfromtheserver.whilethisgapcanbeunderstoodasaconsequence thenetwork.weshowthatforourexperimentalsetup,oneofthemostnoticeableeectsofhigh Forexample,wendthatserverloadhasdistinctiveeectsonthepatternofpacketowthrough ofsocketandleoperations,itisnotablethatfortypical(i.e.,short)transfers(2kb)thegap loaded(i.e.,packetlossratesarehigh),itisnotuncommonforaheavilyloadedservertoshow generallydominatestransfertime. bettermeanresponsetimethanalightlyloadedserver.ourmeasurementssuggestthatthismay bebecauseheavilyloadedserversoftenshowlowerpacketlossrates,andsincepacketlosseshave Inaddition,weshowanevenmoresurprisingeectofserverload.Whenthenetworkisheavily TCPconnections;thiseectmaybeduetothefactthatTCPpacketsowinaburstierpatternthan thatpoipmeasurementsofpacketdelayaregenerallylowerthanthoseexperiencedbypacketsin dramaticeectsontransferlatency,thiscanreducemeanresponsetime. dopoippackets.wealsoshowthatpoipmeasurementsofpacketlossarenotstronglypredictive Finally,wealsoexploretherelationshipbetweenactiveandpassivemeasurements.Weshow inthepresenceofloss. ofpacketlossexperiencedbytcpconnections,whichmaybeduetothefeedbacknatureoftcp themotivatingquestion:whyisthewebsoslow? thatalthoughtheanswersarenotyetinhand,ourapproachappearstoshowpromiseforaddressing Theseresultsdemonstratetheutilityofintegratednetwork/servermeasurement.Weconclude 2

3 2TheWAWMprojectisbasedonalargeanddiversebodyofpriorwork.Inparticular,werelyon resultsfromwebcharacterizationstudies,passiveandactivenetworkmeasurementstudies,time RelatedWork workwiththegoalsofthewawmproject. ofthestudiesfromeachoftheseareaswhichservedasthebasisforourworkandcontrastthat calibrationstudies,andwideareanetworkperformancestudies.inthissectionwepresentselections Alargebodyofworkhasdevelopedoverthepastveyearsthatpresentsmeasurementsand andperformancesuchasclientbehavior,serverbehavior,proxybehaviorornetworkbehavior. characterizationsoftheweb.thisworkhastypicallybeenfocusedonspecicaspectsofbehavior 2.1WebPerformanceCharacterization Clientstudiessuchas[2,24,25,33,39,72](aswellasclientproxystudies[2,16,26,31,42,45, 71,78,79])provideinformationonthebehaviorofusersacrossthemanyserversthattheymight accessduringbrowsingsessions.thisworkhasbeencriticalinbuildingthetoolsthatareusedto sametime.priorstudieshaveonlyhadaccesstoeithertheclientoraproxyortheserver. priorwebstudiesandthewawmprojectisthatwewillmonitorclientandserveractivityatthe simulateuseractivityinthewawmproject.themostsignicantdierencebetweenallofthe provideuswithmethodologiesformeasuringperformanceinboththeuserspaceandsystemspace. provideinsightintoserverbehaviorundervariousloadsandversionsofhttp.theyarefocusedon eithertheanalysisofhttplogsorondetailedmeasurementsofserverperformance.thesestudies Webserverbehaviorhasalsobeenstudiedextensively[5,6,11,14,15,32,47,55].Thesestudies aspossibleoratsomepredeterminedrate.workloadgeneratorssuchas[46,29,77]replayweb serverlogsorasummaryofserverlogs.weusethesurgeworkloadgenerator[1]togenerate generatorssuchas[1,9,23,22,59,75,76]areallbasedonmakingrepeatedrequestsasquickly ManytoolshavebeendevelopedforgeneratingworkloadsonWebservers.Syntheticworkload previousworkloadgeneratorsbecauseitincorporatesawiderangeorworkloadcharacteristicsthat representativelocalloadsonourserver.surgeisasyntheticworkloadgeneratorwhichisunlike areimportanttomanydierentaspectsofserverperformance. measurementsinoneplace(somewherenearanendpointofanhttptransaction)and,thus, tions.theshortfallofthesestudieswhenitcomestoanalyzinglatencyisthattheyonlytake showtheperformanceeectsofbothhttpprotocolinteractionandtcppacketlevelinterac- StudiesofthenetworkeectsofWebtracinclude[7,8,21,3,38,43,54,56].Thesestudies 2.2PassiveandActiveNetworkMeasurements cannotprovideinsightintothepacketdelaysinbothdirectionsofatransaction. Thetaskofassessinggeneralnetworkperformancehasbeenwellstudied.Agoodsourceofgeneral andthosewhichuseactivetechniques.passivetechniquestypicallyconsistofgatheringpacket informationoncurrentmeasurementtoolsandprojectscanbefoundat[73].generaltechniques leveldataatsometappointinthenetwork.examplesofpassivemonitorsincludecoral[58]and forstudyingnetworkperformancecanbesplitintothosewhichusepassivemonitoringtechniques in[17,18],andthecharacterizationoftheself-similarnatureoftracwasdevelopedfrompacket tracesin[41].anotherpassivemeasurementplatformhasbeenproposedin[44].thissystem, asinglepoint.forexample,earlypackettracestudiesofapplicationlevelinteractionsweredone tcpdump[74].agreatdealofinformationcanbeextractedfrompassivemonitoring,evenatonly calledwindmill,doesonlineanalysisofdatathathasbeencollectedpassively.oneofthebenets 3

4 passivemonitoringhaslimitationsintermsoftheamountofdatawhichisgatheredduringtests (typicallyverylarge)andthedicultyofrealtimeanalysisofthedata. ofpassivetechniquesisthattheirusedoesnotperturbthesystembeingmeasured.however, thenetworkandthenmeasuretheresponsetothatstimulus.activemeasurementtoolscanbe characteristicsbetweenhosts. usedtomeasurebulkthroughput,pathcharacteristics,packetdelay,packetloss,andbandwidth Activemeasurementtoolsinjectsomekindofstimulus(apacketorseriesofpackets)into thenetworkaccordingtoapoissonprocess.themotivationforthismethodisthatsamplingat intervalsdeterminedbyapoissonprocesswillresultinobservationsthatmatchthetime-averaged wecurrentlyusepoissonping(poip)[3]inthewawmproject.poipinjectsudppacketsinto Thereareanumberoftoolswhichcanbeusedtomeasureend-to-enddelayandpacketloss; stateofthesystem(assumingitisinastationarystate).thereareanumberofstudiesthathave usedsimilaractivemeasurementstostudypacketlossincluding[12,8]andtodrivemodelsoftcp approach. behaviorincluding[49,6].thesestudiesprovidebackgroundthatguidesouractivemeasurement \TracerouteReno"isatoolusedtomeasurebulkTCPthroughputcapacityoveranInternetpath. oute[36]isanactivemeasurementtoolthatgatherssimplepathcharacteristics.treno[48]or widthalongtheend-to-endpathincludingpathchar[37],bprobe[19],andnettimer[4].tracer- Anumberofactivemeasurementtoolshavebeendevelopedthatcanbeusedtomeasureband- projectwecompareactiveandpassivemeasurementsinordertounderstandhowactivemeasurementsofnetworkconditionscorrelatetoperformanceasseenattheclientandattheserver. datatheyinjectintothenetwork,theycanalsoperturbthenetwork'sperformance.inthewawm Theseactivemeasurementscangiveinsightintonetworkconditions,butdependingonhowmuch beendonebypaxsoninaseriesofstudiesreportedin[61,62,63,65].thesestudiesareastarting pointforthewawmprojectfrombothmeasurementandanalysisperspectives. Someofthemostextensiveworkusingbothpassiveandactivemeasurementtechniqueshas 2.3TimeCalibration Measurementsofend-to-enddelayinourarchitecturearebasedonhavingnearlysynchronized clocksontheclientandtheserver.thedicultiesofsynchronizingclocksinawidelydistributed Additionalstudiesofclocksynchronizationproblemsinclude[57,66].Paxsonpointsoutin[66] clientsandservers.ntpassuresthatclocksaresynchronizedontheorderoftensofmilliseconds. ofthenetworktimeprotocol(ntp)daemonwhichweuseinwawmtosynchronizetheclockson environmenthavebeenwellstudied,especiallybymills[52,53].thatworkledtothedevelopment thattheuseofntpprovidesclosesynchronizationofclocksbutthatchecksforskewshouldstillbe madewhenanalyzingdata.theparticularresultsreportedinthispaperarenotstronglysensitive 2.4WideAreaNetworkPerformanceMeasurement externaltimesource(gps). toclockskew,howeverinfutureworkweplantoeithercorrectforclockskeworuseasynchronized thepresenttime.noneproposetodothetypeofworkthatisbeingproposedforwawm;however Thereareanumberofwideareameasurementprojectswhichareeitherproposedorunderwayat therearesomesimilarities.theprojectswhicharethemostcloselyrelatedtowawmare: CooperativeAssociationforInternetDataAnalysis(CAIDA)[28].CAIDA'smissionisto \addressproblemsofinternettracmeasurementandperformanceandofinter-provider communicationandcooperationwithintheinternetserviceindustry." 4

5 TheNationalInternetMeasurementInfrastructure(NIMI)project[3].TheNIMIproject's IETF'sInternetProtocolPerformanceMetrics(IPPM)workinggroup[51,67].TheIPPM Internet." missionisto\createanarchitectureforfacilitatingameasurementinfrastructureforthe NLANR'sMeasurementandOperationsAnalysisTeam(MOATS)project[27].TheMOAT's workinggroup'smissionisto\developasetofstandardmetricsthatcanbeappliedtothe quality,performance,andreliabilityofinternetdatadeliveryservices." TheInternetPerformanceMeasurementandAnalysisproject(IPMA)[5].TheIPMA's missionisto\createanetworkanalysisinfrastructuretoderiveabetterunderstandingof missionisto\studytheperformanceofnetworksandnetworkingprotocolsinlocalandwideareanetworks." systemicservicemodelsandmetricsoftheinternet." KeynoteSystems,Inc.[34].KeynotePerspectiveisbilledasa\globalreal-timeservice" whichtheyuseto\continuallymeasuretheperformanceofpopularwebsitesandinternet TheSurveyorProject[69].EortsintheSurveyorprojectcenteronusingPoissonping-like backboneprovidersandregularlypublishtheresults(intheformofaperformanceindex)..." EachoftheseprojectsattemptstomeasureoranalyzesomeaspectofInternetperformance.The WAWMprojectissimilartotheaboveprojectsinthatitusesawideareameasurementinfrastructureandthattherearesignicantdatamanagementandanalysistasks.TheWAWMproject isdierentfromtheseforanumberofreasons.first,wawmprojectfocusesonauserlevelapplication theweb.mostoftheprojectslistedabovefocusongeneralnetworkmeasurement, tests,measurementsofuserlevelcharacteristicswillbemadeatthesametime.second,wawm proposestoinstallthesystemswhichwillactasserversintheinfrastructure.thisapproachfacili- monitoringandanalysis.althoughwawmproposestotakenetworklevelmeasurementsduring measurementsofone-waydelayandlossalongpathsbetweenanumberofremotesites. haveperformancemonitoringtoolswhichwillenabledetailedsystemmeasurementsduringtests. ofmanagement.third,thesystemswhichwillbeactingasserversinthewawmtestswillalso tateseaseandconsistencyofsoftwareinstallation,controlovertheenvironment,security,andease intofourcategories:measurementarchitectureandtools,managementtools,testingprotocol,and 3InthissectionwedescribethetoolsandmethodsusedintheWAWMproject.Webreakthemdown WAWMProjectOverview thespecicimplementationsusedintheresultsreportedhere. 3.1MeasurementArchitecture analysismethods.ineachcategorywerstdescribethelong-termprojectgoals,thenwedescribe 3.1.1GeneralArchitecture usedtoservedocumentstotheclients.serversarealsoconguredwithperformancemonitoring toolssothatdetailedmeasurementsofcpu,memoryandnetworkcharacteristicscanbemade ThegeneralarchitectureoftheWAWMhardwarecomponentsisshowninFigure1.Ontheleftare componentsthatarelocatedatoursite.therstsystemisconguredasthewebserverandisonly 5

6 Server Under Test LAN Load Generator(s) Figure1:GeneralArchitectureofWAWMHardwareComponents Active Passive Measurement Distributed Clients with thedurationofatest.thisisimportantsincewecannotexpectremotesystemstogenerate duringtests.thesecondandthirdsystemsareconguredaslocalloadgenerators.theyrunthe Surgeworkloadgeneratorwhichisusedtoadjusttheamountofloadontheserverthroughout Active Passive Measurement delays,packetlosses,pathroutes,andpathcharacteristicssuchasbottlenecklinkspeed.another systemisusedtotakebothpassivemeasurements(e.g.,packettraces)andactivemeasurements enoughloadtosaturatetheserverwhenwewanttoruntestsunderheavyserverload.thefourth system(notshown)isusedasthecontrolsystemwhichinitiatestestsandgathersdatafromthe infrastructure.allofthesystemsintheserverclusterareconnectedtothelocalinternetservice (e.g.,usingnetworkprobepackets).usingthesetoolsweexpecttomeasure(atminimum)packet istodistributethesesystemswidelyacrosstheinternetinanattempttoexploreawiderangeof pathtypes.theremotesystemsmakerequestsforlesfromtheserverduringtests.theyarealso provider(isp)suchthattheserverclusternetworktracisnotabottleneck. conguredwithbothpassiveandactivenetworkmeasurementtools.requestsaregeneratedby Ontherightsideofthegurearesystemsthatarelocatedremotelyfromoursite.Ourgoal clientsusingamodiedversionofsurge.thismodiedversioncanbeconguredtoruninits standardmodeortomakerepeatedrequestsforasingleleinmuchthesamewayaswebstone remotesiteswewouldliketohaveinthecompletewawm.thatstudyutilized37siteswhich [75]ẆeareinuencedbyPaxson'sNetworkProbeDaemonstudy[65]intermsofthenumberof enabledmeasurementofover1,distinctinternetpaths.selectionoftheadditionalsiteswillbe basedongeographiclocationandinternetconnectiontypewithheterogeneityasthegoal.atleast 3.1.2InitialPrototype oneadditionalserversitewillalsobeincludedtodiversifymeasurements. atbostonuniversity(bu)andconsistedofsixpcsconnectedviaswitched1mbpsethernet arealltakenfromthisprototypesystem.inourprototype,asingleserverclusterwaslocated issuesandactasaplatformfordevelopingtestinfrastructureexpertise.theresultsinthispaper Wehaveimplementedaprototypeofthisarchitectureinordertoexposemeasurementanddesign PentiumProCPUsand128MBofRAMeach.EachsystemranLinuxv2..3.Theserverwas conguredwithapache1.3.[68].traceroutev1.4a5andtcpdumpv3.4wereusedtomeasure routecharacteristicsandgatherpackettracesrespectively.xntpdv3-5.93wasusedtosynchronize whichisconnectedtothebuispviaa1mbpsbridge.thepcswereconguredwith2mhz timestampsbetweentheserverclusterandtheremotesystem.oneofthepcsintheservercluster 6

7 settoonesecondandwithapacketsizeof256bytes.thesystemsetupasthewebserverwas alsoconguredwithwebmonitor[4]tomeasuredetailedserverperformancecharacteristicsinthe wasconguredtorunbothtoandfromtheserverclusterwiththemeanmeasurementinterval wassetupasthetimeserverforalltests.thexntpdupdateintervalwassetto64seconds.poip CPU,memoryandnetworkcomponentsoftheserver. wererunundereitherloworhighlocalloadlevels.loadlevelsinsurgearedenedintermsof setuptomakerequestsforalesetwhichconsistedof2distinctles.thetotalsizeofthisdata setwasabout5mb;thustheentiredatasetcouldbecachedintheserver'sram.experiments ThelocalloadgeneratingsystemsintheserverclusterwereconguredwithSurgewhichwas arepresentedin[11].usinghttp/1.,weused4uesforlowloadand52uesforhighload. userequivalents(ues).characterizationsofloadunderarangeofuesforthesamepcsystems andranlinuxv2..35.itwasconnectedinthelocalareaviaswitched1mbpsethernetandthen ofdenver(du).thissystemwasconguredwitha333mhzpentiumprocpuand64mbofram totheduispviaa1mbpsbridge.thesystemwassetupwiththesamenetworkmeasurement TheprototypeinstallationusedonlyasingleremotesitewhichwaslocatedattheUniversity network:1kblesmeasureessentiallyonlytheconnectionsetuptime;2kblesaretypical-sized were1kb,2kband5kb.theselesizeswerechosentodemonstratearangeofbehaviorinthe andtimesynchronizationtoolsasthesystemsintheservercluster.themodiedversionofsurge Webtransfersandtypicallyinvolveabout6or7roundtrips,soaredominatedbyTCP'sslow-start wassetuptomakerepeatedrequestsforoneofthreelesontheserver.thethreelesizesselected 3.2ManagementArchitecture behavior;5kblesallowustoobservethemodeinwhichmostpacketsaresentintheweb, namelyintcp'scongestionavoidancemode. Inordertoruntests,downloaddataandmanagesystemswhicharepartoftheWAWMinfrastructure,asecuremanagementplatformisrequiredonallWAWMsystems.Weareevaluating third-partymanagementplatformsincludingthenimiplatformandinsoft'svitalagent[35].both providetheessentialmanagementfeaturesandexibilitynecessaryforwawmaswellasbuilt-in networkperformancemeasurementtools. designchoices.therefore,inordertogaininsightintotherelevantissues,wedevelopedourown managementtoolfortheprototypestudy.thetooliswritteninperlandisbasedonusingsecure Shell[7](SSHv1.2.26wasusedintheprototypetests).Thistoolprovedtobesucientforthe Themanagementplatformisanareaofthesysteminwhichpracticalmatterscandictate prototypetests. whichwawmtestsarerun.atthispointintimeasimplelestructureisadequatetostorethe data.eachtestruninourprototypestudywithonlyasingleclientresultedinabout9mbof byotherresearchers.thewawmdatarepositoryismaintainedatthesamecentralsitefrom Thedatacollectediscatalogedandmaintainedinsuchawayastofacilitateanalysisandreuse compressedresultsdata GeneralArchitecture 3.3TestingProtocol AWAWMtestisaperiodoftimeoverwhichaspecicsetofclientsystemsmakerequeststoa specicserver.specicationoftestsrequiresdeterminingthefollowing: 1.Theserverclusterwhichwillbeusedinthetest. 7

8 2.Thelocalloadwhichwillbeplacedontheserver. 3.Thesetofremoteclientswhichwillmakerequestsduringatest.Theinuenceofdistance 4.Thetimeofdaythetestwillberun.DuetothediurnalcycleofInternettracinNorth willallinuencetheoutcomeofmeasurements. fromtheserver,internetconnectiontypeandthenumberofclientsparticipatinginatest 5.Thedurationofthetest.Thetestmustberunlongenoughtocollectarepresentativesample America,testsrunduringthedaytypicallyexperiencemorepronouncedinuencefromcross ofdataforthegivennetworkandserverconditions. tracthanthoserunlateatnight. 6.Thescheduleforactivenetworkmeasurementsduringtests.Sinceactivemeasurementscould 3.3.2InitialPrototype perturbhttpperformance,thisschedulemustbeselectedsuchthattheresultingdatagives aclearpictureofthenetwork'sstatewhilenotinuencingthehttptests. Thetestingprotocolusedintheprototyperanonesetoftestsduringtheday(between11:amand 7:pm)andonesetoftestsatnight(between9:3pmand5:am)foreachoftendays.Initiating testsatthesetimesprovidedavarietyofnetworkconditionsforourdata.eachtestbeganby alwayssucienttotransferatleast4ofthe5kblesunderheavynetworkloadconditions routesduringthetests.eachtestwasrunfor15minutes;weobservedthatthisdurationwas runningtraceroutetoandfromtheserverwhichgaveusanindicationofthecurrentend-to-end and4696ofthe1kblesunderlightnetworkloadconditions.poipwasrunbothtoandfrom wererunduringeachtestset.theyconsistedofmakingrepeatedrequestsforasinglele(1kb, 2KBor5KB)underbothlowandhighserverloads. theserversimultaneouslywiththehttprequestsforthedurationofeachtest.sixseparatetests 3.4.1GeneralArchitecture 3.4AnalysisProtocol Oncetestdataiscollected,itisreducedandanalyzed.Automationoftheanalysisprocessis Ourmodelisbasedontcpanaly[64].Inadditiontostatisticalapproaches,weintendtodevelop necessaryduetothelargeamountofdatathatisgatheredduringeachtest.wehavebegunthe methodsfordetailedanalysisofindividualletransfers.detailedanalysismayrequireavisual developmentofatoolcalledtcpevalwhichwillperformthesetasksonthepackettracedata. \coordinationtool"muchlikethetooldevelopedin[13] InitialPrototype datagatheredthroughactiveandpassivemeasurements. Wealsousenatalie[3]toanalyzePoiptracesandintendtodevelopmethodstocorrelatethe Atthehighestlevelwecurrentlytabulatesummarystatisticsanddistributionalstatisticsofle ontimelinediagrams(e.g.,figures3,5,and7)becausetheycanrepresentthetimingofeventsat transferdelays,packetdelays,packetlossandserverperformance.tcpevalalsogeneratesanumber bothendsofaconnection. exchangesbetweenaclientandtheserverwithinasinglehttptransaction.inthispaperwefocus ofgraphicalaidstointerpretation,includingtimelinediagramswhichillustrateindividualpacket 8

9 Sizes FileServer LoadTrnsfdLatency Table1:Filetransferlatencystatisticsunderlightnetworkload Files MeanStd.Dev. Latency%pktlossmeandelaypktloss tcpdump tcpdumppoip%poipmean 5KB 2KB 1KB low delay KB 2KB 1KB high SampleResults.45 Inthissectionwereportoninitialresultsobtainedusingtheprototypeapparatusdescribedin measurementsshowedthattheroutetoandfromtheremotesitewasasymmetricbutfairlystable forallofthetests(13hopsfrombutoduand15hopsfromdutobu).whiletherewere Section3.Thetestingprotocolcalledfortraceroutetoberunbeforeandaftereachtest.These occasionalroutechangesbetweensetsoftest,wemeasuredonlyoneroutechangeduringoneof thetests. period,wepresentresultsofdatasetswhicharetypicalofallofthedatathatwascollected. networkloadandlesizesusedinthetests.whilemeasurementsweretakenoveratwoweek 4.1Measurementsofletransferlatency Firstwepresentperformancemeasurementsforletransfersovercombinationsofserverload, 4.1.1Performanceunderlightnetworkload Thedetailsoftheletransferlatencyunderlightnetworkloadforallthreelescanbeseenin Table1.Thetableshowsthatthebusyserveraddsapproximatelyonesecondtotheaveragele transferlatencyforthe1kband2kblesandover1.5secondstotheaverage5kbletransfer latency.meanpacketdelayandlosscharacteristicsunderlightnetworkloadasmeasuredbyboth PoipandtcpdumpcanbeseeninTable1.Delayandlossmeasurementsarefortheoutboundpath fromtheservertotheclient.thesenetworkmeasurementsshowthatthenetworkconditionswere fairlystablethroughoutthemeasurementperiod. relativelyconstantamountforeachlesize.theseeectscanbeseeinthecumulativedistribution forallthreelesizes.underheavyserverload,thevariabilityincreasesslightly.theprincipal eectofserverloadwhennetworkloadislowwastoincreasetheaverageletransfertimebya Underlightnetworkloadandlightserverload,theletransferlatencyhadverylowvariability percentageoftransfers,latencycanincreaseover6timeswhenserverloadishigh. function(cdf)diagramforthe2kbleinfigure2.thisdiagramalsoshowsthatforasmall networkmeasurementsisthroughtheuseoftcptimelineplots.thetimelinesfortypicaltransfers ofthe2kblecanbeseeninfigure3.intheseplots(aswellastherestofthetimelineplots inthissection),theclientisonthelefthandsideandtheserverisontheright.thetimeline Anotherwaytoanalyzeletransfersandtheeectsofvaryingserverloadforthelightlyloaded plotshighlightthedelayincurredforallletransferswhentheserverisbusy.afterthetcp 9

10 1.8.6 low server load high server load Figure2:CDFforletransferlatencyunderlightnetworkloadforthe2KBle lowserverload highserverload connectionsetupsequence,thereisroughlyaoneseconddelaybeforetherstdatapacketissent bytheserver.thisdelayisduetotheneedfortherequestfromtheclienttogetuptouserspace Figure3:Timelinediagramoftypical2KBletransfersunderlightnetworkload informationfromthecdfofletransferlatencyleadsustoconcludethatifthenetworkislightly loaded,themaximumadditionaldelaythatmosttransfersmightexperiencewhenaccessingour case,thisisnearlyinstantaneouswhentheserverislightlyloaded.thisdatacombinedwiththe andthenfortheresponsetogetbackdowntothenetwork.ascanbeseeninthelightlyloaded busyserverisaboutonesecond. Whenpacketdelaysandlossesincrease,transfercharacteristicschange.Thedetailsofthele transferlatencyunderheavynetworkloadforallthreelescanbeseenintable2.thenetwork 4.1.2Performanceunderheavynetworkload surementperiod.however,thesemeasurementsdonotcorrelatecloselywiththemeandelayand measurementsfrompoipindicatethatnetworkconditionswerefairlystablethroughoutthemea- networkloadtoheavynetworkload.theincreaseinpacketdelaysisanindicationofthelevelof lossvaluesextractedfromthetcpdumptraces.weexplorethisingreaterdetailinsection4.2. Packettransferdelaysextractedfromthetcpdumptracesincreasebetween13%and48%fromlight alsoshowsthatunderheavyserverloads,themeantransferlatencyincreasesinarangefrom1.3to increasesinarangefrom2.2to5timesthemeanlatencywhenthenetworkislightlyloaded.it congestioninthenetwork.thetableshowsthatunderlightserverloads,themeantransferlatency 1

11 Sizes FileServer LoadTrnsfdLatency Table2:Filetransferlatencystatisticsunderheavynetworkload Files MeanStd.Dev. Latency%pktlossmeandelaypktloss tcpdump tcpdumppoip%poipmean 5KB 2KB 1KB low delay.51 5KB 2KB 1KB high low server load.6 high server load.5 Figure4:CDFforletransferlatencyunderheavynetworkloadforthe2KBle whentheserverloadishighversuswhenitislow.thiseectisexploredinmoredetailinsection tonoteisthattransferlatencyofthe5kbleunderheavynetworkloadsactuallydecreases 2timesthemeanlatencywhenthenetworkislightlyloaded.Perhapsthemostinterestingstatistic takenduringthedaytimeperiodsforalldatasetsshowedhigherpacketlossanddelaycharacteristicsaswouldbeexpected.underheavynetworkload,letransferlatencyhadgenerallyhigher delayandhighervariabilityforallthreelesizesregardlessofserverload.theseeectscanbe Filetransferdelaysarestronglyinuencedbybothcongestionandpacketloss.Measurements infigure2,thekneeinthecdfoccuredaroundthe9thpercentile,infigure4,thekneeismuch seenbycomparingfigure2withthecorrespondinggureforheavynetworkload(figure4).while lowerindicatingthatamuchlargerfractionoftransfersexperiencedseveredelays. timesversusthelightlyloadednetworktimelines,whichisanindicationofthelevelofcongestion beseeninfigure5.asinthelightlyloadednetworkcase,thedelayinsendingtherstdata packetfromtheserverisevident.thesediagramsshowageneralelongationinpackettransfer Thetimelinesfortypicaltransfersofthe2KBlewhenthenetworkisheavilyloadedcan inthenetworkduringthemeasurement EectsofPacketLoss Measurementsmadeduringhighnetworkloadshowtheeectsofbothnetworkcongestionand packetloss.inthissectionwespecicallyexaminetheeectsofpacketlossonletransferlatency. 11

12 lowserverload highserverload Table3:Filetransferstatisticsunderheavynetworkloadforleswithnodroppedpacketsorat leastonedroppedpacketfor5kble Figure5:Timelinediagramoftypical2KBletransfersunderheavynetworkload Sizes FileServerFilesw/o Load pktlosslatency MeanStd.Dev.Fileswith Latency pktlosslatency MeanStd.Dev. 5KB 2KB 1KB low n/a n/a Latency KB 2KB 1KB high Thedetailsoftheeectsofpacketlossunderheavynetworkloadandhighserverloadcanbeseen isalsomuchlowerthanthatoflestransferredwithloss. transferredwithoutpacketloss.asmightbeexpected,variabilityforlestransferredwithoutloss Filetransfersforleswhichloseatleastonepacketarebetween1.3and7.8timesslowerthanles intable3.thistableshowsthedierencesbetweenlestransferredwithandwithoutpacketloss. 2KBle.Ascanbeseenfromthegure,packetlosscansignicantlyincreasebothmeanand variabilityofletransferdelay.whencomparingthisgurewithfigure4itcanbeseenthat almostalllongtransferdelaysareduetopacketdrops.onenotablefeatureofthegureisthe Figure6comparestheimpactofpacketlossonletransferlatencyforthe1KBleandthe secondtimeoutperiodforthere-sendingofalostsynpacketinthelinuxtcpimplementation. cuspataboutthe4%levelforthe1kble.thisisduetolostsynpackets.thereisathree torestartslowstartinordertotransfertheremainingpackets.theoveralleectofthislossisthe Figure7.Thisgureshowstheeectofasinglelostpacketduringthedatatransferphaseofthe HTTPtransactionunderlightserverloadandheavynetworkload.ThepacketlosswillcauseTCP Thetimelinesfortransfersofatypical2KBlewithandwithoutpacketlosscanbeseenin additionofabout.5secondstotheletransferlatency. 12

13 Figure6:CDFforletransferlatencyunderheavynetworkloadandheavyserverloadforles.4 withandwithoutpacketloss.3 1K without drops 2K without drops.2 1K with drops 2K with drops Figure7:Timelinediagramofa2KBletransfersunderheavynetworkloadandlowserverload withoutpacketloss withpacketloss

14 1.8 high server load low server load.6 Figure8:Filetransferlatencyforthe5KBletransfersunderheavynetworkload ever,closerexaminationofthedatafromalltendatasetsshowsaninterestingphenomenon.for Initialeectsofserverloadontransferlatencieshavebeenshownintheprecedingsections.How Eectsofserverload whenserverloadisincreasedduringthetests.furthermore,inthreeoutofthosesevencases,le transfersofthe5kble,insevenoutoftenofthedatasets,packetlossactuallygoesdown andthenetworkactivitycouldchangeenoughduringthistimetocauseperformancetoimprove. caseswhichisthesameasthedataintable2.thisresultcanpossiblybeattributedtooneoftwo things.first,theexperimentsforthe5kbletransferarerunapproximately45minutesapart latencydecreaseswhenserverloadisincreased.thiseectcanbeseeninfigure8foroneofthese otherpossibleexplanationisthattheheavyloadontheserverpreventsitfromproducingasmany However,activemeasurementsindicatethatthenetworkwasfairlyconsistentacrossalltests.An- burstsofpacketsasitmightnormallyifitwerelightlyloaded.weintendtofurtherinvestigate 4.2Activeversuspassivemeasurementsofnetworkconditions thiseectunderawiderrangeofconditionsandconclusivelydetermineitscause. thetcpstreamsthemselvesoftendisagree.thisdierenceisimportantbecauseitsuggeststhat Aswasseenintheprevioussection,valuesfordelayandlossfromPoipmeasurementsandfrom takenfromtcpstreams.infigure9weshowthecdfsofpacketdelayexperiencedbytwosetsof nections.inthissectionweexplorethesedierencesandtheirimplications. measurementsderivedfrompoipmaynotbeindicativeoftheexpectedperformanceoftcpcon- tolightlyloadednetworkconditions;theplotontherightcorrespondstoheavilyloadednetwork 5KBTCPowsandbyPoipduringthesametwotimeintervals.Theplotontheleftcorresponds OurrstobservationisthatPoip'smeasurementsofpacketdelayaretypicallylowerthanthose conditions.typicaldierencesbetweenthemeanvaluesreportedbypoipandthoseexperiencedby TCPpacketsareontheorderof1ms.NotethatthePoipmeasurementsweremadeconcurrently withthetcpconnections,sotheyaremeasuringthesamenetworkstate. thattcpoftentransmitspacketsinburstsontheorderofthecongestionwindowsize,thenwaits processisconsistentwiththenotionthattcp'spacketstreamisburstier(i.e.,interarrivalshave ahighercoecientofvariation)thanthatofapoissonprocess.thisseemsclearwhenconsidering ThehigherdelayexperiencedbyTCPpacketscomparedtopacketsarrivingviaaPoisson foracknowledgments.thestartofthenextburstisdeterminedbyroundtriptimeandgenerally occurssometimeafterthecurrentbursthasbeensent,leadingtoahighcoecientofvariationin 14

15 Poip high SL.5 Poip low SL Figure9:CDFofpacketdelayfromservertoclientduringtwo5KBletests(left:lightlyloaded.4 Pkts high SL.4 Pkts low SL network;right:heavilyloadednetwork;sl=serverload) Poip high SL Poip low SL Pkts high SL Pkts low SL 1 % Packet loss from Poip 8 6 Figure1:ScatterplotofpacketlossrateinTCPvs.Poip. 4 2 related.figure1showsascatterplotoflossrateexperiencedbytcp(onthexaxis)versusthat interpacketspacing. OursecondobservationisthatpacketlossasexperiencedbyTCPandbyPoiparenotstrongly % Packet from tcpdump experiencedbypoipduringthesametimeframe(ontheyaxis)forallofour5kbexperiments. Ifthetwomeasurementswerestronglyrelated,pointswouldtendtooccuraroundtheliney=x (whichisplottedforreference);theydonot.ingeneral,itseemsthatpoipseesahigherpacket lossestriggerfeedbackbehaviorintcp.uponencounteringpacketloss,tcpoftenentersastate lossratethantcp,althoughthisisnotuniversallytrue. inwhichnopacketsowforsometime,afterwhichitrecoversandpacketsbegintoowagainat areducedrate(forexample,seefigure7,rightside).thisbehaviormeansthattcp'sviewofthe Thereasonforthislackofcorrelationmaybethat unlikethecasewithpacketdelays packet fewerlosseventsthanareseenbyapoissonprocess(whichisindependentofpacketlossevents). networkisnotindependentofpacketlossevents;asaresult,itmaybethattcptendstoobserve ItseemsclearinanycasethatTCP'sfeedbackbehaviormeansthatitsviewofnetworklossesare conditionsasexperiencedbytcp.1asregardspacketdelays,theburstynatureoftcpows notstronglycorrelatedwiththatofatime-averageview. networkstate.itseemsthatpoiphasonlylimitedvalueinpredictingthenatureofnetwork 1Tobefair,itisimportanttonotethatPoipwasnotdesignedtoassessTCP'sviewofthenetwork,andthatit ThesetwoobservationshaveimplicationsfortheuseofPoipandsimilartoolstomeasure 15

16 meansthatpacketstypicallyseelargerqueuesthanthetime-averagedconditions.withrespectto packetloss,itseemsthatthefeedbacknatureoftcp'scongestionavoidancemakesitveryhard topredicttcp'sactualpacketlossesusinganopen-looptoollikepoip. 5InthispaperwehavedescribedtheWAWMprojectandarguedthatitrepresentsanapproach thatisbothnovelanduseful.itsnoveltyarisesfromitstreatmentoftheserverandnetworkas Conclusion behaviorandnetworkbehavior,andtoidentifycasesinwhichthetwodonotinteroperatewell. anintegratedsystem.itsutilitycomesfromtheabilitytoexploretheinteractionbetweenserver wehaveshownthatinmanyofourexperiments,serversunderhighloadsueredsignicantlyless canyieldinformativemeasurements.forexample,usingitwehaveshownthat(forourserver)the maineectofserverloadontypicaltransfersistodelaytherstdatapacketsent.inaddition Wehavedescribedasmall-scaleimplementationoftheprojectarchitectureandshownthatit packetlossthanthoseunderlowload.incomparingpacketdelayandlossesintcpconnections tomeasurementsobtainedwithpoipwendthatthereareconsiderable(andunderstandable) dierences.theseresultsareonlyinitiallooksatourdataandrequireconrmationinawider rangeofsettings;howevertheyaresuggestiveofinterestingeects. toexpandthemeasurementapparatusuntilitallowsustoassessrepresentativebehaviorofthe infrastructuretoaddressthisquestionwillrequireprogressalongtwodimensions.first,weneed Web.Thismeansthatweneedtoaddmoreremoteclients(frommorelocations),andconsidera Ourmotivatingquestioninthisprojectis:WhyistheWebsoslow?TousetheWAWM widerrangeofserverandplatforms(e.g.,iisrunningonwindowsnt). methodsthatoperateontracesofindividualtransfersandassesstherelativeimpactofserver hopetoobtainamorepreciseunderstandofthecausesoftransferlatencyintheweb. delay,networkdelay,packetloss,etc.ontransferlatency.usingthesemoredetailedanalyseswe Theseconddimensionisthatofanalyticmethods.Weintendtodevelopcharacterization studyaswellasprovidinginputontheprojectdesign.theauthorswouldalsoliketothanklars paper.theauthorswouldalsoliketothankvernpaxsonforprovidingpoissonpingforuseinthis Denverforaccesstothesystemwhichwasusedastheremoteclientforthetestspresentedinthis Acknowledgements.TheauthorswouldliketothankDavidMartinfromtheUniversityof Kellogg-Stedmanforhishelpinsettingupsystemswhichwereusedintheproject. References [1]WebBench2.. [2]G.Abdulla,E.Fox,andM.Abrams.Shareduserbehaviorontheworldwideweb.InWebNet97,Toronto, [4]J.Almeida,V.Almeida,andD.Yates.Measuringthebehaviorofaworldwidewebserver.InProceedingsof [3]A.Adams,J.Mahdavi,M.Mathis,andV.Paxson.Creatingascalablearchitectureforinternetmeasurement. Canada,October1997. [5]V.Almeida,A.Bestavros,M.Crovella,andA.deOliveira.CharacterizingreferencelocalityintheWWW. theseventhifipconferenceonhighperformancenetworking(hpn),whiteplains,ny,april1997. hasmanyusesotherthantheonethatweareconsidering. pages92{13,december1996. InProceedingsof1996InternationalConferenceonParallelandDistributedInformationSystems(PDIS'96), 16

17 [6]M.ArlittandC.Williamson.Internetwebservers:workloadcharacterizationandperformanceimplications. IEEE/ACMTransactionsonNetworking,5(5):631{645,October1997. [7]H.Balakrishnan,V.Padmanabhan,S.Seshan,M.Stemm,andR.Katz.Tcpbehaviorofabusyinternetserver: Analysisandimprovements.InProceedingsofIEEEINFOCOM'98,SanFrancisco,CA,March1998. [8]H.Balakrishnan,S.Seshan,M.Stemm,andR.Katz.Analyzingstabilityinwide-areanetworkperformance.In ProceedingsofACMSIGMETRICS'97,Seattle,WA,June1997. [9]G.BangaandP.Druschel.Measuringthecapacityofawebserver.InProceedingsoftheUSENIXAnnual TechnicalConference,Monterey,CA,December1997. [1]P.BarfordandM.Crovella.Generatingrepresentativeworkloadsfornetworkandserverperformanceevaluation. InProceedingsofACMSIGMETRICS'98,pages151{16,Madison,WI,June1998. [11]P.BarfordandM.Crovella.Aperformanceevaluationofhypertexttransferprotocols.InToappearin ProceedingsofACMSIGMETRICS'99,Atlanta,GA,May1999. [12]J.Bolot.End-to-endpacketdelayandlossbehaviorintheinternet.InProceedingsofACMSIGCOMM'93, SanFrancisco,Setpember1993. [13]L.Brakmo,S.O'Malley,andL.Peterson.Tcpvegas:Newtechniquesforcongestiondetectionandavoidance. InProceedingsofACMSIGMETRICS'96,Philadelphia,PA,May1996. [14]H.BraunandK.Clay.Webtraccharacterization:Anassessmentoftheimpactofcachingdocumentsfrom ncsa'swebserver.inproceedingsofthesecondinternationalwwwconference,chicago,il,october1994. [15]T.Bray.Measuringtheweb.InFifthInternationalWorldWideWebConference,Paris,France,May1996. [16]L.Breslau,P.Cao,L.Fan,G.Phillips,andS.Shenker.Webcachingandzipf-likedistributions:Evidenceand implications.inproceedingsofieeeinfocom'99,newyork,ny,march1999. [17]R.Caceres.Measurementsofwideareainternettrac.TechnicalReportUCB/CSD89/55,ComputerScience Department,UniversityofCalifornia,Berkeley,1989. [18]R.Caceres,P.Danzig,S.Jamin,andD.Mitzel.Characteristicsofwide-areatcp/ipconversations.InProceedings ofacmsigcomm'91,september1991. [19]R.CarterandM.Crovella.Measuringbottlenecklinkspeedinpacket-switchednetworks.InProceedingsof Performance'96,Lausanne,Switzerland,October1996. [2]L.CatledgeandJ.Pitkow.Characterizingbrowsingstrategiesintheworldwideweb.ComputerNetworksand ISDNSystems,26(6):165{173,1995. [21]S.Cheng,K.Lai,andM.Baker.AnalysisofHTTP/1.1onaWirelessNetwork.Technicalreport,Stanford University,1998. [22]HTTPClient. [23]TheStandardPerformanceEvaluationCorporation.Specweb96. [24]M.CrovellaandA.Bestavros.Self-similarityinworldwidewebtrac:Evidenceandpossiblecauses.InProceedingsofthe1996ACMSIGMETRICSInternationalConferenceonMeasurementandModelingofComputer Systems,Philadelphia,PA,May1996. [25]C.Cunha,A.Bestavros,andM.Crovella.Characteristicsofwwwclient-basedtraces.TechnicalReportTR-95-1,BostonUniversityDepartmentofComputerScience,April1995. [26]A.Feldmann,R.Caceres,F.Douglas,andM.Rabinovich.Performanceofwebproxycachinginhetrogeneous bandwidthenvironments.inproceedingsofieeeinfocom'99,newyork,ny,march1999. [27]NationalLaboratoryforAppliedNetworkResearch. [28]CooperativeAssociationforInternetDataAnalysis. [29]H.Frystyk-Nielsen.Libwww. [3]H.Frystyk-Nielsen,J.Gettys,A.Baird-Smith,E.Prud'hommeaux,H.Wium-Lie,andC.Lilley.Network performanceeectsofhttp/1.1,css1andpng.inproceedingsofacmsigcomm'97,cannes,france, Setpember1997. [31]S.Glassman.Acachingrelayfortheworldwideweb.ComputerNetworksandISDNSystems,27(2),1994. [32]S.GribbleandE.Brewer.Systemdesignissuesforinternetmiddlewareservices:Deductionsfromalargeclient trace.inproceedingsofthe1997usenixsymposiumoninternettechnologiesandsystems(usits'97), Monterey,CA,December

18 [33]B.Huberman,P.Pirolli,J.Pitkow,andR.Lukose.Strongregularitiesinworldwidewebsurng.Science, 28:95{97,1998. [34]KeynoteSystemsInc. [35]INSoft. [36]V.Jacobson.traceroute.ftp://ftp.ee.lbl.gov/traceroute.tar.Z,1989. [37]V.Jacobson.pathchar.ftp://ftp.ee.lbl.gov/pathchar/msri-talk.ps.gz,1997. [38]S.KhaunteandJ.Limb.Statisticalcharacterizationofaworldwidewebbrowsingsession.TechnicalReport GIT-CC-97-17,CollegeofComputing,GeorgiaInstituteofTechnology,1997. [39]T.Kwan,R.McGrath,andD.Reed.UseraccesspatternstoNCSA'sWWWserver.TechnicalReportUIUCDCS- R ,UniversityofIllinois,DepartmentofComputerScience,February1995. [4]K.LaiandM.Baker.Measuringbandwidth.InProceedingsofIEEEINFOCOM'99,NewYork,NY,March [41]W.Leland,M.Taqqu,W.Willinger,andD.Wilson.Ontheself-similarnatureofethernettrac(extended version).ieee/acmtransactionsonnetworking,pages2:1{15,1994. [42]B.LiuandE.Fox.Webtraclatency:Characteristicsandimplications.InWebNet98,Orlando,FL,November [43]B.Mah.AnempiricalmodelofHTTPnetworktrac.InProceedingsofINFOCOM'97,Kobe,Japan,April [44]R.MalanandF.Jahanian.Anextensibleprobearchitecturefornetworkprotocolperformancemeasurement. InProceedingsofACMSIGCOMM'98,Vancouver,Canada,September1998. [45]C.Maltzahn,K.Richardson,andD.Grunwald.Performanceissuesofenterpriselevelwebproxies.InProceedings ofacmsigmetrics'97,seattle,wa,june1997. [46]S.Manley,M.Courage,andM.Seltzer.Aself-scalingandself-conguringbenchmarkforwebservers.Harvard University,1997. [47]S.ManleyandM.Seltzer.Webfactsandfantasy.InProceedingsofthe1997USENIXSymposiumonInternet TechnologiesandSystems,Monterey,CA,December1997. [48]M.Mathis.Empiricalbulktransfercapacity.ftp://ftp.advanced.org/pub/IPPM/treno.txt,July1997. [49]M.Mathis,J.Semke,J.Mahdavi,andT.Ott.Themacroscopicbehaviorofthetcpcongestionavoidance algorithm.computercommunicationsreview,27(3),july1997. [5]InternetPerformanceMeasurementandAnalysisProjecct. [51]InternetProtocolPerformanceMetrics. [52]D.Mills.Networktimeprotocol(version3):Specication,implementationandanalysis.TechnicalReportRFC 135,NetworkInformationCenter,SRIInternational,MenloPark,CA,1992. [53]D.Mills.Improvedalgorithmsforsynchronizingcomputernetworkclocks.IEEE/ACMTransactionsonNetworking,3(3):245{254,June1998. [54]J.Mogul.Thecaseforpersistent-connectionHTTP.TechnicalReportWRL95/4,DECWesternResearch Laboratory,PaloAlto,CA,1995. [55]J.Mogul.Networkbehaviorofabusywebserveranditsclients.TechnicalReportWRL95/5,DECWestern ResearchLaboratory,PaloAlto,CA,1995. [56]J.Mogul,F.Douglis,A.Feldmann,andB.Krishnamurthy.Potentialbenetsofdeltaencodinganddata compressionforhttp.inproceedingsofacmsigcomm'97,cannes,france,setpember1997. [57]S.Moon,P.Skelley,andD.Towsley.Estimationandremovalofclockskewfromnetworkdelaymeasurements. InProceedingsofIEEEINFOCOM'99,NewYork,NY,March1999. [58]Coral:Passivenetworktracmonitoringandstatisticscollection. [59]UniversityofMinnesota.Gstoneversion1. [6]J.Padhye,V.Firoiu,D.Towsley,andJ.Kurose.Modelingtcpthroughput:Asimplemodelanditsempirical validation.inproceedingsofacmsigcomm'98,vancouver,canada,setpember1998. [61]V.Paxon.Fast,approximatesynthesisoffractionalgaussiannoiseforgeneratingself-similarnetworktrac. ComputerCommunicationsReview,27(5):5{18,October

19 [62]V.Paxson.Empirically-derivedanalyticmodelsofwide-areatcpconnections.IEEE/ACMTransactionson [64]V.Paxson.Automatedpackettraceanalysisoftcpimplementations.InProceedingsofACMSIGCOMM'97, [63]V.Paxson.End-to-endroutingbehaviorintheinternet.InProceedingsofACMSIGCOMM'96,PaloAlto,CA, August1996. Networking,pages316{336,August1994. [65]V.Paxson.End-to-endinternetpacketdynamics.InProceedingsofACMSIGCOMM'97,Cannes,France, [66]V.Paxson.Oncalibratingmeasurementsofpackettransittimes.InProceedingsofACMSIGMETRICS'98, Cannes,France,September1997. [67]V.Paxson,G.Almes,J.Mahdavi,andM.Mathis.FrameworkforIPperformancemetrics,RFC233. pages11{21,madison,wi,june1998. [7]SSHCommunicationsSecurity. [69]TheSurveyorProject. ippm=,1998. [68]ApacheHTTPServerProject. ftp://ftp.isi.edu/in-notes/rfc233.txt,1998. [72]L.Tauscher.Evaluatinghistorymechanisms:anempiricalstudyofreusepatternsinworldwidewebnavigation. [71]N.Smith.Whatcanarchivesoertheworldwideweb.InTheFirstInternationalWorldWideWebConference, Geneva,Switzerland,May1994. [75]G.TrentandM.Sake.Webstone:Therstgenerationinhttpserverbenchmarking,February1995.Silicon [73]CAIDAMeasurementToolTaxonomy. [74]tcpdump. Master'sthesis,DepartmentofComputerScience,UniversityofCalgary,Alberta,Canada,1996. [76]WebCompare. [78]R.Wooster.Optimizingresponsetime,ratherthanhitrates,ofwwwproxycaches.Master'sthesis,Virginia [77]Webjamma. GraphicsWhitePaper. [79]R.WoosterandM.Abrams.Proxycachingthatestimatespageloaddelays.InSixthFirstInternationalWorld [8]M.Yajnik,S.Moon,J.Kurose,andD.Towsley.Measurementandmodelingoftemporaldependenceinpacket WideWebConference,SantaClara,California,1997. PolytechnicInstituteandStateUniversity,Blacksbury,Virginia,1996. loss.inproceedingsofieeeinfocom'99,newyork,ny,march