StrategicPlan FY20143FY2023 ESnet March1,2013 ThisworkwassupportedbytheDirector,OfficeofScience,OfficeofAdvanced ScientificComputingResearchoftheU.S.DepartmentofEnergyunderContractNo.DECAC02C05CH11231.
Table&of&Contents& 1.&Facility&Overview&...&1 2.&Mission,&Goals,&Drivers,&and&Strategic&Connections&...&4 2.1VisionandMissionStatements...4 2.2FacilityStrategicGoals...4 2.3ConnectionswithDOEandASCRStrategicPlans...4 2.4ScienceDrivers...5 3.&Facility&S.W.O.T.&and&Dependencies&...&7 3.1,3.2Strengths,Weaknesses,Opportunities,Threats...7 3.3Risks,Dependencies,andCriticalGaps...8 4.&Science&and&Community&Needs&...&9 4.1UserCommunityDescription...9 4.2UserCommunityEngagement...10 4.3RequirementsGathering...11 5.&Center&Operational&Priorities&...&12 5.1CenterPriorities...12 5.2InterCCenterandInterCFacilityCollaboration...14 5.3ConnectionswithBerkeleyLabPlans...14 6.&Center&Strategy&for&the&Future&...&15 6.1MajorInitiatives...15 Initiative(1:(increase(partnership,(engagement,(and(outreach(...(16 Initiative(2:(real:time(discovery,(correlation,(and(visualization(tools(...(18 Initiative(3:(next:generation(network(testbed(...(19 Initiative(4:(intelligent(network(services(...(21 Initiative(5:(increase(worldwide(deployment(of(best:practice(architectures(...(23 6.2SystemsSoftwareandLibraries...24 6.3TechnologyStrategy...24 6.4Power,Space,Cooling...26 6.5SummaryRoadmapandTimeline...27 7.&Human&Resources&...&27 7.1CurrentWorkforce...27 7.2FutureWorkforce...28 7.3ObstaclesandStrategies...29 8.&Conclusion&...&29 Appendix&1:&Process&for&Developing&this&Strategic&Plan&...&31 Appendix&2:&ESnet&Traffic&Growth&Projections&...&32 Appendix&3:&ESnet&Budget&Scenarios&and&Analysis&...&33
1.&Facility&Overview& TheEnergySciencesNetwork(ESnet)istheDepartmentofEnergy shighcperformance networkingfacility,engineeredandoptimizedforlargecscalescience. FundedbytheOfficeofScience(SC)andmanagedbyBerkeleyLab,ESnetinterconnectsthe entirenationallaboratorysystem,includingitssupercomputercentersanduserfacilities enablingtensofthousandsofscientiststotransferdata,accessremoteresources,and collaborateproductively. BecausenetworkrequirementsforlargeCscalesciencearedemanding,ESnetallocates significantefforttoappliedresearch,development,andinnovation.currentlythefastest sciencenetworkintheworld,esnetisregardedasaninfluentialleaderamongitsglobal peers.thefoundationforthissuccessisanunparalleledteamofengineersandother contributors,eachfocusedonthemissionofenablingandacceleratingscientificdiscovery. Inthefacility smostrecentoperationalassessmentreport,reviewerscommented: The entirestaffconscientiouslyandcontinuallyleadtheirfield. ESnetwasfoundedin1986,soonafterthecreationoftheglobalInternet.Since1990, ESnet straffichasincreasedbyafactorof10every48months,roughlydoublethegrowth rateofthecommercialinternet. 1 ESnet Accepted Traffic: Jan 1990 - Jan 2013 (Log Scale) 100.0000 10.0000 1.0000 0.1000 Petabytes 0.0100 0.0010 0.0001 0.0000 Jan 2013 Jul 2012 Jan 2012 Jul 2011 Jan 2011 Jul 2010 Jan 2010 Jul 2009 Jan 2009 Jul 2008 Jan 2008 Jul 2007 Jan 2007 Jul 2006 Jan 2006 Jul 2005 Jan 2005 Jul 2004 Jan 2004 Jul 2003 Jan 2003 Jul 2002 Jan 2002 Jul 2001 Jan 2001 Jul 2000 Jan 2000 Jul 1999 Jan 1999 Jul 1998 Jan 1998 Jul 1997 Jan 1997 Jul 1996 Jan 1996 Jul 1995 Jan 1995 Jul 1994 Jan 1994 Jul 1993 Jan 1993 Jul 1992 Jan 1992 Jul 1991 Jan 1991 Jul 1990 Jan 1990 Figure1:ESnettrafficaccepted,PB/month,since1990. 1 IEEE802.3 IndustryConnectionsEthernetBandwidthAssessmentAdHocReport, http://www.ieee802.org/3/ad_hoc/bwa/bwa_report.pdf. Month 1
ESnetservesmorethan40DOEsites,connectingthemtoover100researchand commercialnetworksworldwide.therichnessofesnet sglobalconnectivityismotivated bythefactthat80%ofitstrafficoriginatesorterminatesoutsidethenationallaboratory complex.thispatterninturnreflectsthecollaborative,increasinglyinternationalnatureof scientificresearch.esnetservesotherdoeofficesinadditiontosc,includingnnsa,eere, IN,andEM. 2 Inlate2012,ESnetcompletedthemostimportantupgradeinitshistory.With$62Mof ARRAfunding,thefacilityacquiredrightstotheunderlyingspectralcapacityona nationwidefootprintoffibercopticcable;builttheworld sfirstcontinentalcscale100 GigabitCperCsecond(100Gbps)networkinpartnershipwithInternet2;andtransitioned productiontraffictothenewinfrastructure.asaresultofthisinvestment,esnetcannow supportthedatacmobilityneedsofdistributedsciencemoreflexiblyandcostceffectively thaneverbefore. ESnet Topology, January 2013 SEAT 100 10 PNNL 100 LBNL JGI 10 10 100 10 10 100 SUNN 100 10 SNLL 100 LLNL 10 10 10 Salt Lake 100 AMES 1 100 STAR 100 10 100 10 10 ANL 100 100 EQCH CLEV 100 10 10 100 100 10 PPPL GFDL PU Physics JLAB 10 10 BNL LOSA SDSC 10 10 1 LASV ALBQ 100 LANL SNLA 100 100 10 100 100 100 SUNN 100 ESnet managed 100G routers 10 ESnet managed 10G router 10 100 Site managed routers LOSA ESnet optical node locations (only some are shown) ESnet optical transport nodes (only some are shown) commercial peering points LBNL LLNL ESnet PoP/hub locations R&E network peering locations Major Office of Science (SC) sites Major non-sc DOE sites Routed IP 100 Gb/s Routed IP 4 X 10 Gb/s 3 rd party 10Gb/s Express / metro 100 Gb/s Express / metro 10G Express multi path 10G Lab supplied links Other links Tail circuits Figure2:CurrentESnettopology,after100Gupgradein2012. ESnet sarchitectureandcapabilitiesaretailoredforitsusers,whoseneedsarederived throughaformalrequirementscgatheringprocess.thefacilitysponsorsperiodic requirementsreviewsforeachofthesixscprogramoffices.thesereviewsfocuson scientificprojectsandworkflows,ratherthantechnology.byfirstseekingtounderstand howdisciplinesandcollaborationsproduceknowledge,esnetobtainsmoreusefuland accuraterequirementsinformation.thisapproachhasbeenhighlysuccessful,servingasa modelforotherfacilities. 2 Thebusinessmodelfortheserelationshipsistransitioningfromincremental(cost:recoverytocost:sharing. 100 2
Toforgestrongtieswithitscommunityandusers,ESnetinvestsconsiderableenergyin education,outreach,andadvocacy.inatypicalyear,esnetstaffparticipateinmorethan 100communityandstakeholdermeetings 3 andgives60c90technicalpresentations, includingkeynotes,tutorials,workshops,andinformationalsessionstohelpdoescientists anditprofessionalsmakebetteruseofhighcperformancenetworks.theorganization recentlyhiredaleadforpartnershipsandoutreach,andwillincreaseactivityinthisarea significantly. ESnetcurrentlyhas37careerstaff,plus2.5partCtime,term,contractor,orstudent employees;studentnumbersincreaseby5c6insummermonths.internet,systems,and softwareengineersmakeupnearly80%ofstaff.esnet sannualbudgetwas$34.5infy12, witharequestof$32minfy13.inresponsetorecentretirements,aswellaschronic undercstaffing,theorganizationplanstoadd4c6ftesinthecomingyear. ItisimportanttounderstandthatESnetdiffersfromacommercialproviderofInternet services.massivesciencedataflowsrequiredifferentnetworkcapabilitiesthansmaller flowsgeneratedbyemail,video,andwebbrowsing.asanetworkcarefullyoptimizedfor largecscalescience,esnet scapabilitiesinclude: anengineeringemphasisonlosslesstransport(seesection5.1below) bandwidthguarantees,spanningmultiplenetworkdomains adistributedperformancemonitoringplatform ateamofexpertsforrapiddiagnosisofmulticdomaindatatransportissues webcbasedvisualizationtoolsformonitoringthehealthofsciencedataflows veryhighcspeed(nx100g)datatransport,scalingcostceffectively globalleadershipindevelopingstandardsandapplicationstosupportdatac intensivescienceflows globaladvocacyfornetworkandsecurityarchitecturestoenablehighcthroughput dataexchange extensivescientificoutreachandrequirementsgatheringactivity aconsistentrecordofarchitecturalinnovation Thestructureofmodernsciencenowpresumestheavailabilityofreliable,highCbandwidth, featurecrichnetworksforinterconnectinginstrumentsandcollaboratorsglobally. 4 Inthis context,esnetservesasavital circulatorysystem foralldoefacilities,andforevery missionspacewithinintheofficeofscience. 3 Someoftheseoccurusingvideoconferencingservices,whichESnetoperatesandprovidesfortheentireSC community. 4 The Large Hadron Collider at CERN may have been the first major experiment for which advanced networking was a design( premise, but it certainly won t be the last. Dozens of facilities currently under constructionworldwidemakeexactlythesameassumption. 3
2.&Mission,&Goals,&Drivers,&and&Strategic&Connections& 2.1&Vision&and&Mission&Statements& ESnet smission Toenableandacceleratescientificdiscoverybydeliveringunparallelednetwork infrastructure,capabilities,andtools. ESnet svision 1. Scientificprogresswillbecompletelyunconstrainedbythephysicallocationof instruments,people,computationalresources,ordata. 2. Collaborationsateveryscale,ineverydomain,willhavetheinformationandtools theyneedtoachievemaximumbenefitfromscientificfacilities,globalnetworks,and emergingnetworkcapabilities. 3. ESnetwillfosterthepartnershipsandpioneerthetechnologiesnecessarytoensure thatthesetransformationsoccur. & 2.2&Facility&Strategic&Goals& 1. EnsurethatESnetcontinuestoprovidehighlyreliabledatatransportcapabilities, optimizedfortherequirementsoflargecscalescience,andscalingtoaccommodate theexponentialtrafficgrowthofdoesciencemissions. 2. Inresponsetothechallengesposedbythescientificdatarevolution,developthe abilitytoassessandrapidlyimprovehowcollaborationsmakeuseofhighc performancenetworking. 3. Pioneer,develop,andsustainthearchitectures,protocols,andsoftwarenecessaryto achievethefacilityvision. 4. InfluenceDOEfacilities,labs,campuses,andnetworksaroundtheworldtooptimize theirinfrastructuresforendctocenddatamovement. & 2.3&Connections&with&DOE&and&ASCR&Strategic&Plans& ThemissionoftheDepartmentofEnergy,asarticulatedinitsMay2011strategicplan,isto ensureamerica ssecurityandprosperitybyaddressingitsenergy,environmental,and nuclearchallengesthroughtransformativescienceandtechnologysolutions.because ESnetserveseveryprogramofficeandnearlyeverymajorfacilityinDOE includingallsc userfacilities,majorsccollaborators,andnnsalabsandsites it(broadly(advances(and( 4
5 sustains(the(entire(spectrum(of(doe(missions.formanydoeactivities,includingthoseinsc andthennsa,esnetoffersvitalenablingcapabilities. 5 ESnet smissionistightlycoupledwiththatofsc,whichhasthetwinobjectivesofdirect supportforscientificresearch,andsupportfordevelopment,construction,andoperation ofopencaccessscientificuserfacilities.esnetensuresthatitsstrategiesarealignedwith thoseofscbysponsoringongoingnetworkrequirementsreviewsforeachscprogram office.thesereviewsaredescribedinsection4.3.1,below. ThemissionoftheAdvancedScientificComputingResearch(ASCR)programistodiscover, develop,anddeploycomputationalandnetworkingcapabilitiestoanalyze,model, simulate,andpredictcomplexphenomenaimportanttothedepartmentofenergy(doe). AsthescientificworkflowsinitiatedandsupportedbySCbecomemoredataCintensive, ESnet scapabilitiesbecomemorestrategicforsc.anyinitiativetocreateadatacservices facilityforsc,eitherintegratedacrossmultiplesitesorconcentratedinone,willrelyon ESnetforadvancednetworkingcapabilitiestosupportactivitiessuchasdatareplication, datasharing,remoteaccesstostorage,andbulkdatatransfer. 2.4&Science&Drivers& ESnet smethodforassessingsciencedriversforscisdescribedinsection4.3,below. Severalconsistentthemeshaveemergedfromthisprocess. Exponentialdatagrowth.Newinstrumentsandcomputingsystemscontinuetogenerate exponentiallygrowingdatasets.moore slawistransformingphysicsdetectorsandimagec basedsensors,anddatagrowthingenomicsexceedsmoore slawbyaconsiderable margin.therelentlessgrowthindataassociatedwithsccfundedsciencemissionswill requireesnettoscaleitscapacityaggressively,whilemaintainingtheservicequality requiredforhighcthroughputworkflows. Datamobility.Largedatasetsaremovedtofacilitieswheretheycanbeanalyzed;moved againforsecondaryanalysis;andeventuallysharedwithcollaborators.thispattern appliesbroadly whetherthecontextislhccomputation,proteincrystallography,or climatemodeling.datamobility theplacementofdatawhereitcanbeproductively analyzed isahallmarkofdatacintensivescience,andtheneedforefficientdatamobility motivatesmuchofesnet sstrategicplanningactivity. Centralityofcomputingandnetworking.Computinghasbecomeessentialinvirtually everyareaofsciencefundedbydoe.fromextremecscalesimulationstohighcthroughput analysisofinstrumentdata,computationisubiquitous.ascrcomputationalcentersare thereforekeycomponentsinthedoesciencecomplex,andesnetprovidesthehighc performanceservicestomakethesefacilitiesavailableandproductive. 5 ESnetisnotregardedasa criticalinfrastructure fornnsa,andisnotmanaged,engineered,orfundedfor useasalifecsafetyornationalsecuritysystem.
6 Globalcollaboration.Sciencecollaborationsaregrowinginscopeandscale. CollaborationsinHighEnergyPhysics,NuclearPhysics,andEarthSciencescancomprise hundredsoreventhousandsofcollaborators.thesecollaborationsarebasedonthe assumptionthattheservicesprovidedbyesnetandtheothernetworksoftheworldare consistentandreliable,andreachtothefulllengthandbreadthofthecollaboration includinginstruments,facilities,andparticipatinginstitutions. Afewscientificworkflowpatternsmeritspecialdiscussion. Lightsources.Theevolutionofbeamlinedetectorsatphotonandneutronsourcesis followingacompoundedmoore sclawcurve.theareaofsensorsisincreasingexponentially (512x512,1024x1024,4096x4096),alongwithreadoutrate(0.5Hz,100Hz,1kHz, 1MHz).Eachdetector andtheremaybedozensoftheseinasinglefacility isconnected toadataacquisitiondeviceinsideacomputer,whereadevicedriverconvertsincoming datastreamstofiles(forexample,tiffimages)forscientificanalysis.asaresult,highc performancecomputingresourceswillsoonberequiredforanalyzingbeamlinedata.data ratesfromcurrentcgenerationdetectorsareontheorderof1.5gbps,withratesof8gbps expectedinthenext18months.asconsequenceofthisgrowth,highcspeeddatatransfer betweenlightsourcesandneutronsources(als,aps,lcls,nslscii,sns,ssrl,andso forth)andhpcfacilities(alcf,nersc,olcf)willbecomecriticaltothescientificprocess ofphotonandneutronfacilities. Fusionscience.Experimentalfusionhasaworkflowthatfollowsaregulardutycycle.The instrument(forexample,atokamak)isruninpulsedmode:plasmaiscreatedforashort periodoftime,duringwhichdataiscollectedfromthousandsofsensors.foraperiodof roughly15minutes,datafromthepulseisanalyzed,andtheconfigurationforthenext pulseisdeterminedbasedonthatanalysis.thisdutycycleplacessignificantdemandson thenetworkforconsistentandreliableperformance.thistypeofrequirementstandsin contrasttothemorecommonneedtomovea10tbdatasetinaperiodofhours,for example.asmoreadvancedfusioninstrumentshavebeendeveloped(eastinchina, KSTARinSouthKorea,andthefutureITERinFrance),pulsetimesaregrowinglonger,data volumesareincreasing,andthedistributednatureoftheexperimentalteamscontinuesto expand.peaktranscatlanticdatatransferratesforitercouldreach400gbps. LHCscience.AnalysisworkflowsforALICE,ATLAS,andCMSinvolvetheprocessingof billionsofindividualevents.detectorsinthelhcatcerngenerateeventdatafrom particlecollisions,andeventsareanalyzedthoughagloballydistributedcollaboration involvingthousandsofscientistsonmultiplecontinents.thelhcexperimentsrelyheavily ontheautomateddistributionoflargevolumesofdata;thereissimplytoomuchdata(both intermsofnumberofeventsanddatavolume)toaccomplishtherequiredanalysison humantimescaleswithoutautomatingdatadistribution.manyofthenetworkcapabilities ESnethaspioneered(forexample,bandwidthreservations)wereinitiallydevelopedto servetheneedsoflhcexperiments. JGIworkflows.MostofthecomputingfortheJointGenomeInstitute(JGI)facilityisnow performedatnersc.theuseofnerscresourcesbyjgihasevolvedovertime,butakey componenthasbeenesnet sdedicatedbandwidthservice,basedonitsoscarssoftware. OSCARSmakesJGIsubnetsavailablefromwithintheNERSCdatacenter,andsupports
directaccesstonersccomputationandstorage(forexample,filesystemmounts)byjgi resources.oscarsenablesroutinefacilityoperations,largecscaletransactionssuchasbulk datamigration,andthe coupling ofgeographicallyremoteresources. EarthSystemGrid.TheCoupledModelIntercomparisonProject/Phase5(CMIP5)usesthe EarthSystemGrid(ESG)Federationforinteroperablediscovery,distribution,andanalysis oflargeandcomplexdatasets.therearetwoprimaryworkflows:datareplicationanddata distribution.thedatareplicationworkflowinvolvesthetransferofmanydatasets betweenlargestoragesystemsatclimatefacilities.datasetsreplicatedaretypicallyinhigh demand,andthereplicationprocessspreadsloadgeographicallyacrossmultipledata centers.thedistributionworkflowinvolvesthediscoveryanddownloadofspecificdata setsbyscientistsforanalysis.individualdataflowsaresmaller,buttherearemanysets downloadedandthedownloadscanstillbeterabytesinsize.bothworkflowsassumethe abilitytotransferdatasetsfromtenstohundredsofterabytesinsize reliably around theglobe. 3.&Facility&S.W.O.T.&and&Dependencies&& 3.1,&3.2&Strengths,&Weaknesses,&Opportunities,&Threats& Strengths worldcclassstaff highcapacityforinnovation cultureofurgency globalreputation newopticalanddarkcfiberassets strongrelationships(withprogram Manager,Internet2,Labs) Weaknesses understaffed,oversubscribed internalprocessesneed modernizationandattention recruitmentisslow,becausefacilityis selectiveandcompetitioniskeen Opportunities increaseproductivityofscientific userfacilitiesthroughoutreach promoteworldcwideadoptionof bestcpracticearchitecturesfordata mobility transformnetworkfromblackcbox infrastructuretointelligent, programmableinstrument provideservicesandcapabilitiesfor noncdoefederalsciencepartners Threats talentcpoachingfromlarge,wellc fundedcompanies budgetcutslimitopportunitiesto advancemission travelrestrictionsimpairabilityto leadandpromoteglobalchange operationalactivitiescrowdout innovation&strategicthinking 7
8 3.3&Risks,&Dependencies,&and&Critical&Gaps& FacilityRisks 1. ESnetwillnotbefundedtodeliverthenetworkcapacityrequiredtomatchSC s exponentialgrowthindemand. Asexplainedinsection5.1below,ESnetmustbefreeofpacketlossinorderto avoiddelayinglargescientificdataflowsbyafactorof50ormore.thankstoa $62MARRAinvestmentinnewopticalinfrastructure,ESnetnowoperatesa platformthatcanscaletomeetexponentialgrowthrequirementsofscforthenext 5C7years,basedoncurrenttrafficprojections.However,accommodatingthis growthisnotfree itrequirespurchasingopticalandroutinggearforevery additional100gbpsopticalwaveaddedtothenetwork;thesepurchasesinturn drivedatacentercostsrelatedtopower,spaceandcooling,aswellasadditional effortcosts.thecriticalbudgetaryquestionforesnetiswhetherthedeclining costccurvefor100gbpscomponentswilloffsettheexponentialriseinbandwidth requiredbysc.becauseitwouldmeanpredictingthefutureofamarket,thereisno waytoanswerthisquestiondefinitively.however,thereareplausiblescenariosin whichesnet sfundingwouldneedtoincreasesignificantly(bymanymillionsof dollars)inordertoaccommodategrowth.nowthatthetransitionto100gbps technologyisfinished,esnetwillworkactivelywithitsopticalvendorstobuild reasonablecostmodelsforthenextfiveyears,andwillbrieftheascrfacilities DirectorandESnetProgramManagerwhentheseprojectionsarecomplete. 2. ESnetwillnotcontinuetoattracttheworld sbestinternetengineeringtalent,or elselosecriticalteammemberstoindustry. ThefoundationforESnet ssuccessisanunparalleledteamofinternetengineersand othercontributors.staffhavebeenrecruitedfromgoogle,oracle,sun,wolfram Research,Level3Networks,Internet2,UCBerkeley,NCSA,NERSC,ALS,JGI,BNL, LBNL,AmesLab,LLNL,andmanyotherprominentorganizations.Thefacilityis regardedasoneofthebestplacestoworkinnetworking,andjobsatesnetare coveted,inpartbecause talentattractstalent. ThegreatestthreattoESnet stalent poolisprobablyanextendedperiodofventureccapitalinvestmentinnetworking startcups;aperiodofsubstantialgrowthforhighlyccapitalizedorganizationsthat ownbignetworks(apple,facebook,amazon,twitter);oradeclineinascrfunding thatresultsinunderfundingthegrowthofthefacility.sinceanyofthesescenariosis possible,esnetleadershiptakesgreatcaretomotivateexistingemployees,hire wisely,refreshthetalentpool,andperformsuccessionplanning.thegeneralissue ofretentionisdiscussedinmoredetailinsection7.3below.
DependenciesandNeeds 1. ESnetdependsonexternalinvestments. Becausenetworkingisan endctocend problem,esnetdependsonmanyother organizations(networks,labs,universities,andscfacilities)tomakeinfrastructure andengineeringinvestmentsrequiredtoadvancescmissions.tociteanobvious example,ascrsupercomputingfacilitiesmustinvestinequipmentforoptimizing widecareadatatransportinorderforascrtotakemaximumadvantageofitsnew 100Gbpsnationalnetwork. 6 Similarinfrastructureinvestmentsneedtohappen worldwideinorderforglobalnetworkstoenableandacceleratediscovery.reduced investmentsininfrastructurebylabs,campuses,andpeeruniversitieswillprevent maximumutilizationofwidecareacapabilitiescarefullyplannedandengineeredby ESnet.Theoutcomehereisnotsimplyafunctionoffunding;it srelated,perhaps morefundamentally,tohoweffectivelyesnetcanleadtheworld snetworks towardsasharedarchitecturalvision.seestrategicinitiative#4(insection6.1, below)forfurtherdiscussion. 2. ESnetdependsoncontinuedinnovationfromnetworkingvendors. ESnetisgrowingroughlytwiceasfastasthecommercialInternet.SpecialARRA fundingenabledtheorganizationtodeploythefirst100gbpsnetworkatcontinental scaleintheworld,butthisupgradehappened justintime fromtheperspectiveof DOEscience.Inthefuture,ESnetwillcontinuetobeamongtheearliestadoptersof newnetworkingtechnology,andwilldependoncontinuedinnovationatevery networklayer.disruptiveinnovationsinopticalandpacketroutingtechnology(for example,terabitwavesorsiliconcphotonicintegration)havethepotentialtomakeit cheapertobuildhighcperformancenetworksatscale.esnetdependsonvendors investinginsuchinnovation,inordertohaveahopeofscalingeconomically.it wouldbesuboptimalfordoe,andcertainlymoreexpensive,ifesnet straffic growthrequiredittodivergetoofarfromcommerciallyavailablearchitectures;this dynamicisfamiliarintheexascaleprogram. 4.&Science&and&Community&Needs& 4.1&User&Community&Description& Asdiscussedinsection5.1below,it smorecomplicatedtoformulateadefinitionof user foranetworkingfacilitythanacomputingcenter,asaresultoftheinternet sdifferent servicemodel(variable,(best:effort,(and(ad:hoc,ratherthanfixed(and(reserved).insome sense,everysccfundedscientistisanesnetuser,becausethenetworkingfacilityis fundamental,interconnectingallothers,andprovidingaccessnoncdoecollaboratorsas well.unfortunately,thislineofreasoningdoesnotproduceasatisfactoryorwellcbounded conceptof user forcomparisonwithotherscfacilities.afterconsultingwithesnet s 6 Fortunately,allthreecomputingfacilitiesaredoingexactlythat. 9
ProgramManager,thefacilityhasdevelopedtheusercategoriesdescribedbelow.These categoriesencompassjustasmallsubsetofdoecfundedscientistswhobenefitfromesnet: thosewhohavenetworkresearchobjectives;andthosewithhighdatavolume,networkc intensivescienceworkflowrequirementswhohaverequesteddedicatedbandwidth services.proposedtechniquesformeasuringesnet simpactonscientistsoutsidethissmall subsetofusersaredescribedinsection6.1. 1)OnTSiteUsers.ThesearenetworkresearchersgrantedphysicalaccesstoESnet sdarkc fiberornetworktestbedresources,basedonthenatureoftheirresearchgoals.these projectsarepeerreviewed,asdescribedinsection6.1. 2)RemoteUsers.Thiscategoryincludesuserswhoreceiveallocationsofdedicated bandwidthcapacityonesnetitself,anduserswhoaregrantedaccesstoesnettestbed resources. 3)DataUsers.Theseareresearcherswhodownloadorotherwisegainlegitimateaccessto datasetscollectedbyesnetaspartofanapprovedandpeercreviewedresearchproject. (ESnetcollectsagreatdealofmonitoringdata,relatedtoutilization,energyconsumption, bulkdatatransfer,routing,ipv6,flowdynamics,performancemonitoring.muchofthisdata isofinteresttonetworkresearchers,andalldataiseitherpubliclyavailable,oravailablein anonymizedformtoapprovedresearchers.) 4.2&User&Community&Engagement& AlthoughafewlargeCscalecollaborationssuchasATLAShavethestaffandresources necessarytomakeexcellentuseofadvancednetworkingcapabilities,mostscientific collaborationsaresmallerandneedassistancetoachievethesamelevelofbenefit.asan overarchinggoal,esnetwantstoensurethatitsservicesarewellunderstood,accessible, andeasytouseforanyscientistorcollaboration. Becausethisgoalissocriticalforaddressingthe extremedata challengesconfrontingthe OfficeofScience,ESnethasdefinedtwostrategicinitiativesintheareaofusercommunity engagement,partnership,andoutreach(seesection6.1,whichalsodescribesesnet s historicalcommunityengagementactivities).thegoalforalloutreachactivity indeedthe goalforeverythingesnetdoes istoachieveitsvisionofaworldinwhichscientific progressiscompletelyunconstrainedbythephysicallocationofinstruments,people, computationalresources,ordata. ESnet snewoutreachinitiativeswillsupplementengagementactivitiesthatarealready vigorous.inatypicalyear,esnetstaffparticipateinmorethan100communityand stakeholdermeetings,givingalmostasmanytechnicaltalks,keynotes,tutorials,and webinars.esnetalsoformallyparticipatesinmultiplecommunityevents,includingthe JointTechsworkshop(themostimportantglobalgatheringofresearchandeducation networkers,whichitcocsponsors);theesnetsitecoordinators Committee(ESCC) meeting;thelargehadroncollideropticalprivatenetwork(lhcopn)andopennetwork Environment(LCHONE)meetings;theUSATLASThroughputWorkingGroup;theNational UserFacilityOrganization(NUFO);theGreenTouchconsortium;TERENA(TheTransC 10
EuropeanResearchandEducationNetworkingAssociation)conferences;variousInternet2 meetings;theopengridform(ogf)standardsbody;thegloballambdaintegrated Facility(GLIF);andtheOpenScienceGrid(OSG),amongothers. 4.3&Requirements&Gathering& 4.3.1(Data(Gathering( ESnet sarchitectureandcapabilitiesaretailoredforitsusers,whoseneedsarederived throughaformalrequirementscgatheringprocess.thesetwicecyearlyreviewsofnetwork requirementsfocusoneachofthesixprogramofficesfundedbysc:advancedscientific ComputingResearch,BasicEnergySciences,BiologicalandEnvironmentalResearch, FusionEnergySciences,HighEnergyPhysics,andNuclearPhysics.Thepurposeofthese reviewsistocharacterizenearcterm,mediumcterm,andlongctermnetworkrequirements forthesciencesupportedbyeachprogramoffice.reviewsfocusonscientificcase(studies, ratherthaninformationtechnology.byfirstunderstandinghowdisciplinesand collaborationsproduceknowledge,esnetcanderivenetworkrequirementsmore accurately.thishighlysuccessfulapproachhasservedasamodelforotherresearchand educationorganizations,andforatleastoneotherdoefacility(nersc). Duringeachreview,participantstalkthroughwrittencasestudiespreparedinadvance, ratherthanpresentingslides.aroundtableformatispreferred,andanyoneintheroom canaskclarifyingquestions.oftenthemostusefulconversationisopportunistic,when researchersindifferentcollaborationshearandrespondtoeachothers processcofcscience explanations. ESnetalsosponsors(orparticipatesin)requirementsassessmentoutsidetheregularcycle ofscprogramofficereviews.recentexamplesincludeorganizingaworkshoptoexplore networkrequirementsforthebelleciiexperiment;andcocchairingtheefforttogather10c yearnetworkrequirementsforthecommunityplanning( Snowmass )processwithinthe HighEnergyPhysicscommunity. 4.3.2(Data(Forecasting( ESnet sprimarybasisformodelingaggregatefuturedatarequirementsisaconsistent23c yearhistoricalpattern:since1990,totaltrafficacceptedbythefacilityhasincreased tenfoldevery48months(seefigure1).esnethasnoreasontobelievethisrateofincrease willchange,becausetheunderlyingexponentialtechnologydriverscontinuetooperate.in addition,esnetusestheprogrammaticreviewsdescribedabovetodiscover,document,and monitoranynewrequirements. Whileoverallbackboneutilizationwilllikelycontinueonthecurrentgrowthcurve,transC Atlantic,transCPacific,andcommercialpeeringsmaybeondifferentpaths,andcould exhibitstepcfunctionchanges.forexample,thebelleciiexperimentinjapanisprojectinga 30GbpsincreaseintransCPacificnetworkuseby2017;theinternationalITERfacilityin FrancewillcauseasubstantialincreaseintransCAtlantictraffic;andfutureimprovements oflhcluminositywilldriveevenhighertranscatlanticdatarates.esnetalsousesits 11
12 requirementsreviewprocesstoidentifycapabilitiesneededbyscienceexperimentsthat arenotobviousfromaggregatetrafficvolumestatistics.theseinclude,forexample,the abilitytoprovideguaranteedbandwidthasaservicethroughprogrammaticapisforgrid middlewarecomponents;ortheabilitytosupportinternationalnonctcp,fixeddatacrate protocolssuchasrdmanolaterthan2013.bycombiningextrapolationsbasedon historicaldatawithcapabilityrequirementsexpressedbythesciencecollaborations,esnet canbuildandrunahighcperformancenetworkcapableofprovidingboththecapacityand thecapabilityrequiredtosupportdoe sdatacintensivescienceportfolio. 5.&Center&Operational&Priorities& 5.1&Center&Priorities& BeforediscussingoperationalprioritiesforESnet,it shelpfultonoteanimportant distinctionbetweenascr scomputingandnetworkingfacilities. ThebenefitsofcomputingfacilitieshavetraditionallyaccruedtoalimitednumberofPIs whoreceivedfiniteallocationsofascarceresource,throughaprocessofcompetitivepeer review.theserviceallocationhasbeenfixedandreserved. 7 Othermodelsareavailable, however.forexample,usersatnerscwhoexplorelargedatasetsthroughwebcbased ScienceGatewaysreceivevariable,bestTeffort,adThoc(services.Althoughmostservices availableatcomputingfacilitiesstillfallintothefixed(and(reservedcategory,theadventof richwebinterfacesforexploringdatamaycontributetoaparallelriseindemandfor variable,(best:effort,(ad:hoc(services.neitherservicemodelissuperiortotheother,and bothadvancetheascrmission.however,it smorecomplicatedtoformulateadefinitionof user forthesecondservicecategory. InthecaseofESnet,thetwomodelsevolvedintheoppositeorder.Whenitwasfounded 26yearsago,ESnetquitenaturallyinheritedtheservicemodelforthenewlylaunched globalinternet,whichwasvariable,(best:effort,(and(ad(hoc.infact,thismodelwasakey innovationoftheinternetarchitecture,distinguishingitsharplyfromcircuitcreservation systems(mostnotablypublictelephony)thatcamebefore.overtime,thesciencedata flowscarriedbyesnetbegantodivergefromcommodityinternetflowsintheirscaleand servicerequirements.startingtenyearsago,esnetpioneered,developed,anddeployeda fixed(and(reservednetworkingmodelinresponsetothechangingnatureofitstraffic.the softwareesnetdevelopedforthispurposeisnowusedby40networksworldwide,andthe fixed(and(reserved(servicemodelhasbeenadoptedbyevenmoreorganizations.today, roughlyhalfofesnet strafficishandledthroughthefixed(and(reservedmodel,andit s reasonabletobelievetheratiowillgrowovertime. Withthiscontextinmind,itshouldbeeasiertounderstandESnet soperationalpriorities. 7 Notethespecialdefinitionofreserved.AfixedallocationisreservedforanamedPI,buttheallocationdoes notguaranteethatanyparticularjobwillstartorcompleteatafixedtimet;thisisnot,inotherwords, anythinglikeahotelordinnerreservation.
First,thefacilitymustallocatesufficientresourcestoaccommodateexponentialgrowthin demandforbothservicemodels.ifesnetfailstostayaheadofthedemandforboth services,theresultisundesirablepacket(loss.asmallrateofpacketloss(lessthan.01%)in anationalcscalenetworkcauseshugelydisproportionateeffects(>50xreduction)ondata transferspeedsforlargeflows,duetocharacteristicsoftcp,thedominantinternet transportprotocol. 8 Forthisreason,ESnetisdesignedtobelossless,totheextentpossible. 9 CommercialnetworkssuchasComcastorAT&Tdonothavethesamedesignobjectives, becausetheflowstheysupporthavemuchlowerthroughputandtravelmuchshorter distancesthandoescienceflows.ifesnetfailedtomeettheexponentialbandwidth demandsofdoescience,ordidsoinawaythatintroducedpacketloss,theuserexperience fordoescientistswouldresembletheexperienceofwatchingyoutubeinahighly congestedwirelessenvironment.suchanoutcomewouldbedetrimentalforthe productivityofeverycollaborationfundedbydoe,bothinsideandoutsidethenationallab system,andwouldsignificantlyimpairthescmission. Second,ESnetmustdedicateenoughofitsunderlyingnetworkcapacitytomeetthe demandforfixed(and(reservedservice.massivesciencedataflowsarebetterserved, generally,whenmigratedfromthevariable(tothefixed(and(reservedmodel.(this generalizationislessvalidinawellcprovisioned,losslessnetworksuchasesnet,butthe fixed(and(reservedmodelisstillimportantformeetingresiliencyandsecurityrequirements ofmanysciencecollaborations,andfordeliveringnetworkservicesthatarenotcompatible withabestceffortroutedarchitecture.)networkreservationsareafiniteresource,andit s possibleto oversubscribe anetworkpath.uptothispoint,esnethasaddedcapacity rapidlyenoughtomeetallreservationrequests;further,utilizationofnetwork reservationshasbeenhigh.butbecausethesetrendsmaynotcontinue,esnetwillestablish ausergrouptoprovideinputandguidanceonthequestionofallocationstrategiesforall Facilityresources. Inadditiontonetworkcapacity,ESnethasotherresourcesthatrequireallocation.The mostimportantoftheseisaccesstoanationalcscale,multiclayernetworktestbed(inmany respectsuniqueintheworld),designedtopromotetheresearchandinnovationnecessary foradvancingthemissionsofesnetandascr.theesnettestbedisavailabletoalldoec fundednetworkresearchers,andacceptsproposalseverysixmonthsfromresearchersin industry,government,andacademia.proposalsarepeercreviewedbyacommitteeoflab, university,andindustrynetworkingexperts.thisreviewincludesassessmentsof operationalreadinessinadditiontomerit,althoughproposalsbyprojectsfundedbydoe ASCRarereviewedforoperationalreadinessonly.Thearchitectureandservicemodelfor 8 Inarecenttest,ESnetmeasuredtheeffectsofa0.0046%loss(1packetoutof22000)ondatatransferrates atvariousdistances.atdoecrelevantdistances(forexample,nersctoanl)anddoecrelevanttransferrates (10Gbps), this tiny rate of loss decreased data transfer times by 80x, an effect that renders the network virtuallyuselessforthepurposesoflargecscalescience. 9 CreatingalossCfreenetworkrequirescarefulengineering,andacombinationoftactics:theremustbe adequatenetworkcapacitytobeginwith,deepandproperlycconfiguredpacketbuffersincorenetwork components,andatestinfrastructurethatcanautomaticallyandcontinuallyverifythehealth(andimplicitly theproperconfiguration)ofallequipment.forthislatterpurpose,esnetcocdevelopedtheperfsonar softwaretoolkit,whichisnowwidelyusedintheresearchnetworkingcommunity. 13
theesnettestbedarecurrentlyevolvingtoaccommodateresourcesatotherascrfacilities andlabs,andindeedoutsidethedoecomplex(seesection6.1);thisnewmodelwillbe definedinareportduetoascrattheendofapril. OnefinaloperationalpriorityforESnetishighavailability.Theimportanceofthis requirementhighlightsanotherrespectinwhichnetworkingandcomputingfacilities differ.esnetnormallyachievesfourcninesofavailabilitytoendsites(99.99%),alevelof servicewhichrequiresgreatengineeringeffort,operationaldiscipline,andanappropriate leveloffundingtoachieve.therequirementforhighavailabilityisaconsequencethe criticalroleinternetservicesnowplayinenablingandacceleratingmodernscience.evena 30Csecondoutageofnetworkconnectivitytoanationallabisconspicuousanddisruptive. Therecent$62MARRAinvestmenthasgreatlyimprovedtheunderlyingresiliencyofESnet andthenationallabs(especiallyattheopticallayer),justasithasreducedthecostof addingnetworkcapacity. 5.2&InterZCenter&and&InterZFacility&Collaboration& Becausenetworkingisan endctocendproblem, bettercollaborationwithascrcomputing centersanduserfacilitiesisamajorthemeofthisstrategicplan.approximatelyhalfofthe initiativesdescribedinsection6haveincreasedcollaborationasaprimaryobjectiveor beneficialsideeffect. Oneyearago,ESnethiredaLeadforPartnershipandOutreach,thefirststaffmember entirelydedicatedtothefunctionofoutreachintheorganization shistory.theoutreach teamwillgrowby1.5fteinthecomingyear,andwillinitiallyprioritizeengagementwith userfacilities;formoredetail,seesection6.1. 5.3&Connections&with&Berkeley&Lab&Plans& ThebroadeststrategicgoalforBerkeleyLabisCarbonCycle2.0,designedtoaccelerate thedevelopmentofacarboncneutralglobalenergysystem.thisgoalissupportedbythree strategicpillars:nanoscience,biology,andcomputing.theconnectionbetweenesnetand overallberkeleylabstrategyismostobviousinthe computingpillar,whichincludes threecomponents:simulation/modeling,extremedata,andesnetitself.eachofthese componentsiscriticalinthepursuitofbroaderlabstrategy,especiallyasdoescience entersaneweraof extremecscale dataandcomputation.facilitiessuchasthealsand(in thefuture)thenglsrequireevermoresubstantialcomputingandnetworkingresources fortheirdiscoveryprocesses,andasthathappens,thestrategicimportanceofberkeley Lab scomputingpillarincreases.finally,esnetplanstosupportandenablenersc snew datainitiative(seenerscstrategicobjective#2)byprovidingthenecessarynetwork capacity,capabilities,andarchitecturalconsultation. 10 10 ESnet would, of course, eagerly provide the same capabilities to any SC Facility (ASCR or otherwise) requestingthem,andwillseektodoexactlythataspartofitsnewoutreachinitiativedescribedinsection6.1 below. 14
6.&Center&Strategy&for&the&Future& ThecorestrategicgoalforESnetisdeliveryofhighlyCreliabledatatransportcapabilities, optimizedfortherequirementsoflargecscalescience,andcontinuallyscalingto accommodatetheexponentialtrafficgrowthofdoesciencemissions.ifesnetfailsto achievethiscentralgoal,thefacilitycannotsucceedinitsmission.havingsaidthat,the coregoalisnecessarybutnotsufficient.inaddition,esnetneedstodevelopthecapacityto: assessandrapidlyimprovehowcollaborationsuseadvancednetworking;and influencedoefacilities,labs,campuses,andnetworksaroundtheworldto optimizetheirinfrastructuresforendctocenddatamovement. Alloftheseactivitiesareenabledbyanoverarchingstrategyforinnovation:ESnetwill pioneer,develop,andsustainthearchitectures,protocols,andsoftwarenecessaryto achieveitsvision.becauseesnet sgrowthissorapid,andtherequirementsofitsscience trafficsodifferentfromthoseofcommercialinternetflows,aninnovationstrategyiskeyto thefacility ssuccess,andinturntothesuccessofscsciencemissions. 6.1&Major&Initiatives& Allinitiativesdiscussedinthissectionareorganizedandgroupedwithintherubricof ESnet sfourstrategicgoals,describedinsections2.2and6,andhighlightedwithinblue boxes.theinitiativesareselected(astheascrguidancedirects)tofocusonatimecscaleof thenextseveralyears.limitationsofspacerequireconsiderablecompressionintheir description. StrategicGoal Ensure(that(ESnet(continues(to(provide(highly(reliable(data(transport( capabilities,(optimized(for(the(requirements(of(large:scale(science,(and(scaling( to(accommodate(the(exponential(traffic(growth(of(doe(science(missions.( Onanygivenday,alargefractionofESnetstaff(fromInternetengineerstodeployment specialists;toolsdeveloperstosystemsadministrators)isbusypursuingthiscentralgoal. Supportingactivitiesvarywidely,andincludetechnologyevaluation,vendorengagement, appliedresearch,growthmodeling,testandprototyping,financialanalysis,procurement, anddeployment. Althoughthisdocumentdoesnotdescribeaninitiativeinsupportoftheabove goal,section5.1articulatesthejustificationformaintainingalosslessnetworkcore; section3.3describestherisksoffailure;andsection6.3discussesthetechnical challenges,inflectionpoints,andlikelystrategiesassociatedwithcontinuingtoscaleesnet capacitytomeetscmissionneeds. 15
StrategicGoal In(response(to(the(challenges(posed(by(the(scientific(data(revolution,( develop(the(ability(to(assess(and(rapidly(improve(how(collaborations(use( high:performance(network(capabilities.( Initiative&1:&increase&partnership,&engagement,&and&outreach&& Goal,Motivation,andStrategyforProducingChange AworldCclassnetworkisoflittleuse,ifcollaborationsandfacilitiesdon thavethe informationtheyneedtobenefitfromitscapabilities.theinitiativedescribedbelowaims toincreaseandtrackesnet sactivitiesinpartnership,engagement,andoutreach witha goalofmaximizingtheabilityofsccfundedscientiststouseesnetinacceleratingtheir discoveryworkflows.thisoutreachinitiativeisparticularlyurgent,sinceesnet sprogram requirementsreviewsandinteractionswithfacilitystaffclearlyindicatealoomingdatac intensitychallenge(foranalysisof extremedata asanesnetdriver,seesections2.3and 4.2above). Inthepast,theESnetoutreachefforthasbeenconfinedtothreenarrowactivities: gatheringrequirementsfromscofficesonaregularbasis workingwithindividualpisonanadchocbasis engagingwithlabnetworkingstaff,throughregularmeetingsoftheesnetsite CoordinatorsCommittee(ESCC)aswellasmoreinformally Eachoftheseactivitiesisimportant,andeachwillcontinue.Butasacoordinatedstrategy, theylackanymethodformakingcontactwithavitalsetofesnetconstituents:user facilitiesandscientificcollaborations.theinitiativedescribedbelowprovidesawayof closingthisgap,bydevelopingoutreachtechniquestailoredspecificallyfortheneedsof facilitiesandcollaborations.theinitiativerepresentsanimportant andforesnet, unprecedented expansionofactivityforthepurposeofdirectengagement. Thestrategyforproducingthischangehastwoforks: creationofrapidcresponse,datacmobility tigerteams implementationandtracking(usingstatecofcthecartcustomerrelationship managementtools)ofpartnership,engagement,andoutreachactivitiestargeted specificallyforfacilitiesandcollaborations DataCmobilitytigerteamswillbeflexibleincomposition,butwillalwaysincludeESnet engineering,outreach,andprojectmanagementpersonnel,whowillworkinpartnership withstafffromrelevantfacilities,scientificcollaborations,andlabs.theobjectiveofeach tigerteamwillbeclearlydefinedandnarrowlyscoped,withtheaimofsolvingasignificant datacmobilityproblem.theterm tigerteam canhavenegativeconnotationswithindoe, buttheintenthereistoassemble: 16
undomesticatedanduninhibitedtechnicalspecialists,selectedfortheir experience,energy,andimagination,andassignedtotrackdownrelentlessly everypossiblesourceoffault. 11 Thoughthetigerteammodelisreactive,andwillnotscalebroadly,ithasverysignificant advantagesintheshortandmediumterm.first,itgeneratessuccessfulcasestudies,aswell asproductiverelationships,whichcanbereusedinbroaderoutreachefforts.second,it producesvalueforscientistswhoseproblemsaresolved;therehavebeentworecent examplesofsuchactivityinvolvingals,nerscandesnetstaffinthepastyear.finally,this techniquewasdiscussedatthemostrecentesnetsitecoordinatorsmeeting,andwas broadlyendorsedbylabrepresentatives. Thesecondforkofthisinitiative(implementation(and(tracking(of(partnership,(engagement,( and(outreach(activities)ismorestrategicinnature,becauseitpromisestodeliverascalable andcomprehensiveoutreachsolution.esnettookthestepofhiringaleadforpartnerships andoutreachoverayearago,andthepartnershipsleadhaspresentedheroutreach strategytoesnet sprogrammanagerandberkeleylab saldforcomputingsciences.the primarygoalsofthiseffortareto: buildawarenessofesnetincommunitiesthathavenotmadeoptimaluseof advancednetworking(especiallysmalltomidcsizedcollaborations) targetfacilitiesusedbythesecollaborations;partnerwithusercsupportand outreachstafftocreatenewcommunicationschannelsforinformationand possibilitiesforeducation developasetofearlyadopters(throughthetigerteamprocess,forexample)toact asadvocatesforthesecommunities partnerwithcomplementarytoolproviderssuchasglobusonline,toexpandthe reachandimpactofesnetoutreach addcontentonwww.es.netdedicatedtotheneedsofscientificusers monitortheimpactofoutreachat(the(network(layer(bybenchmarkingand measuringtrafficinandoutoffacilitiesovertime implementcustomerrelationshipmanagement(crm)technologytocaptureuser data,gaugeoutreacheffectiveness,measuresatisfaction,andtargetspecific communitieswithfocusedandcustomizedinformation inthelongerterm,ensurethateverydoeuserfacilityhasthetools,knowledgeand resourcesitneedstoinformscientistsaboutesnetcapabilitiesanddirectusersto ESnetwhenappropriateforproblemsolvingandsupport RequiredResources Themostimportantresourcesrequiredforthisinitiativearehuman.ESnethashireda LeadforPartnershipsandOutreach;isrecruitingforahalfCtimeOutreachCoordinator;and willrecruitforatechnicallyorientedoutreachengineerinthenexttwomonths,tosupport tigerteamandotheractivities.inaddition,approximately.5fteisallocatedfromthe NetworkEngineeringGrouptoOutreachonasteadyCstatebasis.It slikelythatthe 11 J.R.Dempsey,W.A.Davis,A.S.Crossfield,andWalterC.Williams,"ProgramManagementinDesignand Development." 17
PartnershipsandOutreachteamwillgrowovertime,andmaydoubleagaininthenextfew years,assumingoutreacheffortsaredemonstrablyvaluable.thenotionofpartnershipalso impliescommitmentfromotherorganizations;oneexampleistherecentlyannounced partnershipbetweenesnetandglobusonline.althoughesnetdoesnothavethepowerto commitfundsorresourcesofotherorganizations,thefacilitywillaimtomakeoutreach partnerships(suchastigerteamsfordatamobility)sousefulandattractivethatoutside organizationshaveacompellingmotivationformakingthesecommitmentswithesnet. Risks TheprimaryriskofthisinitiativeisfailuretoadvanceESnet sstrategicgoalofimproving thewayscientificcollaborationsuseadvancednetworkcapabilities.thecustomer relationshipmanagementtoolsdescribedabovewillreducethisrisk. StrategicGoal Pioneer,(develop,(and(sustain(the(architectures,(protocols,(and(software( necessary(to(achieve(our(vision. & Initiative&2:&realZtime&discovery,&correlation,&and&visualization&tools& Goal,Motivation,andStrategyforProducingChange Aperennialfrustration(anddeeptechnicalchallenge)innetworkingisthedifficultyof obtainingusefulinformationabout bitsonthewire.itistrivialtocountthosebits,but muchhardertoextractmeaning forexample,toassesstherealctimeimpactofhighc performancenetworksonaparticularglobalresearchcollaboration.intheinitiative describedbelow,esnetaimstoleadtheworldindevelopingdiscoveryandvisualization toolstoidentifysignificantnetworkflowsinrealtime,correlatethemwithscientific activity,makeaccuratedeductions,andperformusefulinterventions. Forthepasttwoyears,ESnethasbuiltthecapacitytodevelopworldCclasstools.This initiativebeganwiththecreationofadedicatedesnettoolsteam,currentlyagroupof two:anexperiencednetworkandsoftwareengineer,plusanexperiencedwebdesigner anddeveloper.thisteamhasbeenfocusedonthefunctionalityofesnet scustomerportal, MyESnet(http://my.es.net).ThisportalprovidesfeatureCrichtoolscustomizedfor networkingstaff(andnetworkcsavvyscientists)ateachesnetsite,andalsodesignedfor usebyprogrammanagementwithinascr.myesnetiscurrentlythemostpolishedand usefuloperationalinterfaceforanyresearchnetworkintheworld.however,this functionalityisonlythebeginningofwhatthetoolsteamcanaccomplish. NowthatESnethasdevelopedthenecessaryskills(dataanalyticsandvisualizationfor networkcbasedinformation),itcanbuildtoolsofgreaterimpactforscientific collaborations.thefoundationforthisnewactivityiscreationofaglobaldatabasewith communityccuratedmetadatadocumentingcyberinfrastructureresourcesrelevantto science:supercomputers,clusters,datarepositories,massstoragesystems,archival storage,datatransfernotes,sciencegateways,performancemonitoringnodes,and 18
networkserviceinterfaces.inthewordsofonememberofthetoolsteam,thedatabase willbea who swhoofipaddresses. Buildingonthisproject,theToolsTeamwoulduse nearlyrealctimenetworkflowinformationtocreatecorrelationandvisualizationengines ofrelevancetosccollaborations,orofoperationalrelevancetoesnet.theopportunities forextractingrealctimeintelligencefromnetworkinclude: diagnosingperformanceproblemwithaccuracyandprecision recommendingcorrectiveaction,againwithaccuracyandprecision droppinglargeflowintoopticalcircuitsforthepurposeofmaximizingutilizationof expensiveinfrastructure applyingappropriatesecuritypolicywhennecessary analyzingtrafficwithfinecgraineddiscriminationofscienceimpact identifyingopportunitiesforspecificfacilities,orspecificcollaborations,tomake betteruseofesnetresources RequiredResources ThesizeoftheESnetToolsTeamwillneedtogrowfromtwoFTEtofouratminimumin ordertopursuetheopportunitiesdescribedabove,butthisincreaseismorethanjustified bytheprospectofobtainingthefunctionalitydescribed.developingthesetoolswillalso requireconsultationandcollaborationwiththenetworkresearchcommunity;ascrc fundedresearchershave,forexample,alreadystudiedstatisticalmethodsforidentifying probablebigdataflowsbasedonhistoricaldatatransferrecords. Risks TheprimaryriskassociatedwiththisinitiativeisunderCestimatingtheeffortrequiredto achievethegoalsdescribed,andinparticularunderestimatingthedifficultyofestablishing acommunityccuratedrepositoryforinformationaboutglobalcyberinfrastructure resources.carefulscopingandwatchfulmanagementwillmitigatebothrisks. & Initiative&3:&nextZgeneration&network&testbed& Goal,Motivation,andStrategyforProducingChange Fortheforeseeablefuture,ESnetwillcontinuetobeamongtheearliestadoptersof networktechnologyintheworld theresultofunrelentingexponentialgrowth,combined withdemandingservicerequirementsforsciencedataflows.forpreciselythesame reasons,theorganization sfocusonappliedresearch,development,andinnovation(and collaborationwithascrcfundednetworkresearchers)willcontinueandincrease. Duringmostofitshistory,ESnethasnothadaccesstoarealistic,nationalCscaleplatform forresearchandinnovation.however,thefacility s$62marracfundedadvanced NetworkingInitiative(ANI)includeddedicatedfundstosupportanationalnetwork testbed,andthattestbedprovedtobeextremelyusefulforesnetandthebroadernetwork researchcommunity. 12 Thegoalofthetestbedwastoprovidearealisticenvironmentfor researchersandtheprivatesectortoexperimentwithnewtechnologies,protocols,and applicationsat100gbps.thetopicsexploredincludednonctcptransport;multiclayerand 12 AcloseoutreportfortheANITestbedwassubmittedtoDOEASCRinDecember2012. 19
20 multicdomainarchitectures;protocolscaling;componenttestingforhighperformance; protectionandrecovery;automatedflowclassification;andhighcthroughputmiddleware. BasedonthesuccessandutilityoftheoriginalANItestbed,ESnethasproposedandASCR hasagreedtocontinuefundingit,althoughthearchitectureandservicemodelwillevolve considerably.infact,ascrhasrequested(andesnetispreparing)areportdescribingits futurevisionforanetworktestbed,dueinapril2013.table1brieflysummarizesmajor differencesbetweentheoriginalandfuturetestbedarchitecturesandservicemodels. ANITestbed FutureTestbed Whocanuseit? DOE,.edu,industry. DOE,.edu,industry. Possibletopologies? Fixedandconstrainedby availabilityofphysical circuits. Programmableforeach collaboration,througha combinationofdedicated 100GbpswavesandOSCARS circuitsonthesparecapacityof thenew100gbpsesnetfootprint. Attachedresources? Onlythoseprovidedby ESnet. Anyresourcesprovidedona temporaryorpermanentbasisby ESnet,DOEsupercomputing centers,labs,userfacilities, universities,exchangepoints,or othernetworktestbeds. BareCmetalaccessto networkcomponents andsystems? Yes. Yes(onESnetCmanaged resources). OpenFlowcapability Yes,with1GbpsNEC switches. Yes,withmultiCvendor deploymentonnationalfootprint; opticalopenflowpossible. Connectivityto othernationalcscale testbedssuchasgeni, andthoseineurope? No. Yes,withpeeringattheStarlight exchangepointinchicago. Accessto13,000Cmile darkfiberfootprintfor disruptiveoptical research? Yes,butresearchersmust supplyopticalcomponents andcoverassociatedcosts. Yes,butresearchersmustsupply opticalcomponentsandcover associatedcosts. Table1.MajordifferencesbetweenANITestbedandFutureNetworkTestbed. RequiredResources TheESnetstaffrequirementforthetestbedisonly1FTE,whichincludes.4FTEfor systemssupport/upgrades,.4ftetoprovidesupporttoexperimenters,.1ftefornetwork engineeringsupport,and.1ftetomanagetheproposalprocessandothermanagement
21 tasks.thetestbeditselfrequiresadedicated100gbpswave,andaminimumof40gbps sparecapacitytoseveralmajordoelabstoperformrelevantexperiments.openflow componentsrequire10gbpsofsparecapacitybetweenanumberofesnethubs.the purposeofthetestbedistoperformresearchonnextcgenerationarchitectures,sotestbed hostswillneedtobeupgraded/replacedevery2c3years.finally,a100gbpspeeringatan openexchangepointinchicagoisneededtofederatethetestbedwithothernetworks. Risks Manyoftheoperationalrisksofmanagingatestbedforthenetworkresearchcommunity havealreadybeenmitigatedinthenewplan;inparticular,theriskofinexperienced researchersrequiringdisproportionateassistancewillbeaddressedbyanoperational readinessreviewforallproposals. Initiative&4:&intelligent&network&services& Goal,Motivation,andStrategyforProducingChange ThepoweroftheInternetislargelyattributabletothesuccessofwellCdocumentedand widelydeployednetworkprotocols.someofthese,liketcp/ip,areindispensableforany Internetcommunication.Others,likeHTTP,arevitalforparticularapplicationsorservices. Currently,thereisawidelypublicizedefforttobuildnewprotocolsforinteractingwith networkcomponentsdirectly throughsimple,openinterfaces.thismodeliscalled softwarecdefinednetworking (SDN),andOpenFlowisthemostwellCknownSDNprotocol. IfthepromiseofOpenFlowcanberealized,itwillbepossibletoprogramcomplex,multiC vendornetworkssuchasesnet,optimizingthemforspecificuses,suchassciencedata transport.forexample,openflowcouldbeusedtodetectscienceflowsthatrequire specialhandling,andmovethemintodedicatedopticalpaths whichistosay,openflow couldplayanenablingroleininitiative2forrealctimetools.asexplainedinsection5.1 above,scienceapplicationsdesignedformovinglargedatasetscanbepoorlyservedbythe currentinternetarchitecture(specifically,bythecombinationofstatisticallycmultiplexed, bestceffortippacketdeliveryandtcp,whichishighlysensitivetopacketloss). Althoughanyfreshwaveoftechnologygenerateshypeandhyperbolethatmustbe discounted,esnetfirmlybelievesthattheinternetispassingthroughaninflectionpointof programmability,andthatopenflowandrelatedprotocolsholdgreatpromisein advancingthefacility svisionofscientificdiscoveryunconstrainedbygeography.certain scientificcommunities(notablyhighcenergyphysics,buttheremaybeothers)seem willingandabletoembraceintelligentnetworkservicessoon,forthepurposeofensuring moreconsistentandpredictabledatatransferrates. 13 Theinitiativeproposedhereisthreefold.First,ESnetwillcontinuetoleadprotocol standardizationeffortswhichadvancethecauseofdoescience,andparticularlyinthe areaofbuildinginteroperableinterfacesformanagingendctocendbandwidthreservations 13 ESnetrecentlyparticipatedinaworkshoponthistopic,whichbroughttogethertechnologistsfrom nationalresearchnetworkswithapplicationandmiddlewaredevelopersfromthelhcexperiments.
(soccalleddedicatedcircuits),whicharedescribedinsection5.1above. 14 Inpursuitofthis goal,esnet schieftechnologisthasbeencocchairingtheefforttostandardizethenetwork ServicesInterface(NSI),anSDNprotocolthroughwhichapplicationsandmiddleware componentscanrequestdedicatedcircuitsacrossmultipleadministrativedomains and eventually,getperformanceinformation,configureoverlays,andobtainotherservices. Inadditiontoleadingstandardizationworktosupportitssciencemission,ESnetwill prototypeanddeploysoftwareinterfacesinpartnershipwithatleasttwohighly distributedcollaborations,verylikelyatlasandkbase,thoughothersarepossible. InitiallyusingthenextCgenerationtestbeddescribedinInitiative3,ESnetwillexposewebC servicesbasedsdninterfacesforeachcollaboration smiddlewarestack,withafocuson dedicatedcircuits. Finally,ESnetwillexpandtheintelligentnetworkservices(includingNSIandOpenFlow) supportedbyitsoscarssoftware theleadingnetworkmiddlewareforbuildingand managingdedicatednetworkcircuits. RequiredResources PursuitofthisinitiativewillrequireoneortwoadditionalsoftwaredeveloperFTEs,overa periodoftwoyears.inaddition,esnetwillworkcloselywithothernetworksandprojects attemptingsimilarprojects,especiallythensfcfundedanseinitiative(piharveynewman, Caltech). Risks Principalriskswiththisinitiative:effortcouldbewasted;networksmightnotbeevolving intoprogrammablesystems;orifso,innovationsmightaddminimalvaluewithincommon scientificdataworkflows.suchrisksareendemictoallactivitiesthatinvolveresearchand innovation.thetechnicalconsensusofnumerousesnetengineersisthatsdnparadigms aretransformative,helpfultolargecscalescience,andheretostay. StrategicGoal Influence(DOE(facilities,(labs,(campuses,(and(networks(around(the(world(to( optimize(their(infrastructures(for(end:to:end(data(movement.( 14 Thisisnotunexpected,sinceESnetwasapioneerinthisarea,andusesdedicatedcircuitstocarryabout halfofitsdataflows. 22
Initiative&5:&increase&worldwide&deployment&of&bestZpractice&architectures& Goal,Motivation,andStrategyforProducingChange Considerthat: 1. 80%ofESnet strafficleavesthedoecomplex. 2. Theremaining20%crossesmultipleadministrativeboundaries(forexample:ALSC >LBLnetC>ESnetC>ANLC>ALCF),becausenetworkingisanendCtoCendservice. ThesetwofactshavewideCrangingconsequences,andtheysuggestanotherwayinwhich ASCR snetworkingandcomputingfacilitiesdiffer.esnet ssuccessistiedtothehealthofits peerorganizations(globalandcampusnetworks).thismeansthatesnetcouldattractthe beststaff,developthebestservices,andfosterthemostvisionaryinnovation butallthat wouldbeminimallyusefultodoeiftheorganizationdidnotalsoleadtheglobal networkingcommunitystrongly(and(effectively.(esnet sobligationforgloballeadershipcan bederivedfromdoe sglobaltrafficflows. TheinitiativedescribedbelowisanefforttoincreaseworldwidedeploymentofbestC practiceendctocendarchitectures,includingsciencedmz, 15 performancemonitoring,and guaranteedbandwidthservices.widedeploymentofthesearchitectureswouldbenefit manyscientificcommunities,butwouldespeciallyhelpclimate,genomic,andphoton science whereesnet srequirementsworkshopshaverevealedthataninabilitytotransfer datahampersproductivity. ESnethasspentsignificanteffortcreatingaknowledgebaseforthenetworkingcommunity (http://fasterdata.es.net);thissitereceivesmorehitseachweekthanesnet shomepage. Recently,thefacilityalsohelpedinitiatewhatmaybeatippingpointindeploymentof sciencecfriendlycampusarchitecturesintheunitedstates:nsfpublishedanimportantcall forproposals( CCCNIE )referencingesnet ssciencedmzarchitectureanditsknowledgec base,andisfundingdozensofcampusestoupgradetheirinfrastructuresaccordingly. Thisisaverywelcomedevelopment,butESnetneedstocontinueitsefforttopromotebest practices,especiallyineuropeandasia andalsointheunitedstates,amongcampuses andregionalopticalnetworksthathavenotyetbeenfundedorinspiredtomakechanges. ThisinitiativedovetailswithInitiative1( increasepartnership,engagement,and outreach )andwillinvolveseveralofthesamestaff,butit streatedseparatelyinthisplan becausethescopeislargelytechnical,andthenaturalaudienceisnetworkoperatorsand fundingorganizations,notsciencefacilitiesandusers. InordertoincreaseworldwidedeploymentofbestCpracticearchitecturesfordatamobility, ESnetwillpursueafourCpartstrategy: 1. Collaboratewithfundingagencies,promotingEuropeanandAsianequivalentsof therecentnsfcccnieproposals. 15 ScienceDMZisacyberinfrastructuredesignpatternpioneeredbyESnetthatoptimizescampus infrastructureforhighcspeeddatatransfer. 23
2. Enhancethefasterdataknowledgebasewithvideos,additionalhowCto documentation,andmoredetaileddiscussionofsecuritymodelsfordatacintensive science. 3. ParticipateinatleastonetigerCteamactivityasdescribedinInitiative1,to documenttheclearperformanceadvantagesthatresultinsciencedmz implementation. 4. ContinuetoserveasglobaladvocatesandthoughtCleadersbykeynoting conferences,organizingpanels,sponsoringwebinars,andpublishingcontenton ScienceDMZandrelatedtopics. RequiredResources Inordertostepupactivityinthisarea,itwillprobablybenecessarytotransitionan existingnetworkengineerintotherolealmostfullctime,andbackfillforhisorherposition inthenetworkengineeringgroup. Risks TheprimaryriskisfailuretoimpactasubstantialnumberofDOECrelevantnetworksand facilities,whichinturnwouldperpetuatethestatusquoforsmallercollaborationsand nonchepscience.therisksofnotattemptingthiseffortaregreaterthantherisksof attemptingit. 6.2&Systems&Software&and&Libraries& Thissectionwasprobablyintendedprimarilyforcomputingcenters.It simportanttonote, however,thattheabilityofglobalresearchnetworkstoadvancethescmissiondependson softwaretools,middleware,andapplications forwhichtherearefewreliablefundingor sustainabilitymodels.themostcriticalexamplesforesnetareoscars(toenablenetwork reservations)andperfsonar(fornetworkdiagnostics),bothopencsourceprojectsheavily supportedbythefacility,andeachrequiringabout1fteforongoingsupport.ingeneral, supportforusefulbutinherentlynonccommercialsoftwaretoolsishardtosecureoutside oflarge,wellcfundedcollaborations.softwarewillcontinuetoplayanimportantroleinthe facility sfuture,asintelligentnetworkservicesdriveintegrationofscienceapplications withnetworkmiddleware. ESnetiskeenlyinterestedinthenewfundingmodel 16 beingpursuedforglobusonline, anotherimportantcommunitytoolfordatamobility.butevenifthe freemium model sustainsthisparticularproject,thatmodelismoreappropriateforclientcfacingsoftware thannetworkmiddleware,whichhasamuchsmallerdirectconstituency. 6.3&Technology&Strategy& ESnet stechnologystrategyisdrivenbythegoalsofscalingtoaccommodateexponential growth;maintainingalosslessnetworkinfrastructure;andmakingintelligentuseof availablenetworkresources,asexplainedinsection5.1and6.1 16 https://www.globusonline.org/news/globusconlinecannouncescbigcdatacsharingcservice/ 24
25 The$62MARRAinvestmentenabledESnettoobtain,forthefirsttime,accesstoterabitsof opticalcapacityinanationwidefibertransportsystem,whichissharedwithinternet2(the researchandeducationsysteminterconnectingusuniversities).withoutthisinvestment, exponentialtrafficgrowthwouldcertainlyhaveleadtounsustainableyearlycostsfor ESnet.TheupgradefromESnet4toESnet5wasthemostimportanttechnologytransitionin thefacility shistory,andit sunlikelyatransitionofsimilarmagnitudewilloccuragain before2019. SmoothScaling Betweennowand2019,ESnetwilldeployopticaltransponders,routers,andother networkcomponentstomakeincrementaluseofavailableopticalspectrum,asthescience driversdescribedinsection2.4continuetopushanexponentialdemandcurve.inthenear term,esnetwillleverageitsnewopticalplatformtoaddcapacityjustintime,andatacost thatisproportionaltotherawcostofopticaltransponders.thesecostswillfollowa downwardpricecurve;whethertheshapeofthatcurvecompletelyoffsetstheexponential growthintrafficisaquestionofgreatimportanceforesnet sbudget,anditshouldbe notedthatadditionalopticalwavescreateothercostsaswell(seesection3.3onboth counts).foraroughchronologyofupgradesthatcanbeanticipatednow,withcurrent knowledgeaboutgrowthpatternsandroadmapsofopticalvendors,seesection6.5below. DisruptiveChange Disruptiveimprovementsintechnologyarepossible andwouldbewelcome,ifaffordable. OneclassofbreakthroughswouldbetheadventofultraChighCspeed(400Gbpsor1Tbps), longcreachopticalcomponents.eventhoughsuchspeedshavebeendemonstratedinthe lab,deploymentofthesetechnologieswillbeexpensivebecauseofoperationallimitations regardingdistance,noncitugridspectralboundaries,andcomplexmodulationformats. Leadingopticalresearcherspredictadeclineintherateofgrowthofnetworkspeedsdue totheselimitations,unlessnewarchitecturesthatinvolveopticalparallelism(forexample, multiccorefibers)areadopted.muchmorelikely,ifnotcertain,isthereleaseof200gbps components,whichesnetwilladoptassoonastheyarecostceffective. ESnetpartnerswithvendors,othernetworks,andresearcherstotest,influenceand supportproductinnovationsthatpromisetodrivedowncosts,deliverneededcapabilities forscience,orproducedisruptivechange.inthatspirit,esnetismonitoringseveralmajor areasofresearchinhighcperformancenetworking.theseinclude: 1. Gridless(Optical(Infrastructure( Tosupport400Gbps+capacityperwavelength,opticalinfrastructurewillrequireflexible allocationofspectralcapacity,ratherthanbeinglimitingittotheitucspecified,50ghz spacingsupportedbyallequipmenttoday.spectralblockingthroughfragmentationisone majorconcernthatwillneedtobeaddressedbeforesuchtechnologiesareadopted. 2. Packet:optical(integration( Manyvendorsarepursingtheintegrationofopticalandroutingcomponents,aprocessthat isalmostcertaintoresultinproductsthatcanbepurchasedintimetoactivateesnet sdark fiberfootprintin2019.thepromiseofthisapproachiseliminationofanexpensivelayerof
26 hardware,bycollapsingtwoseparatenetworktransportsystems(opticalandrouting)into one. ( 3. Programmable(network(components( ESnet ssciencerequirementsreviewshavedocumentedtheneedforapplicationsand scienceworkflowstoreceivepredictablenetworkperformance.thismotivatesthegoalof buildingprotocolsandinterfacestosupportintelligentnetworkservices,asdescribedin section6.1above.esnetwillcontinuetoworkcloselywithleadingvendorstodevelopand prototypearchitecturesfornetworkprogrammability. Software MostofESnet slastinginnovationshavecomeintheformofnetworkarchitectureand networkmiddleware.asexplainedinsection6.2,theroleofsoftwaredevelopmentwill playanincreasingroleinthefacility sfuture,assciencerequirementsmotivatethe integrationofapplicationsandprogrammablenetworkcomponents. PartnershipsforInnovation ESnethasparticipatedinnumeroussuccessfultechnologypartnershipsandfirstCinCworld demonstrationswithnetworkingandtelecommunicationsfirms(includingtwoopenflow achievementsinthepastsixmonths).partnersamongvendorsincludeciena,infinera, Juniper,NECandBrocade;amongnetworks:Internet2,SURFnet,REANNZ;amongresearch institutions:ucberkeley,universityofvirginia,indianauniversity,universityofchicago, Stanford,andUCSanDiego.Inaddition,ESnetiscollaboratingwithNSFandwillshortly engagewithinternationalfundingagenciestopromotebestcpracticearchitecturesfordata mobilityinuniversitycampuses. 6.4&Power,&Space,&Cooling&& Theissuesofpower,spaceandcoolingarelessurgentfornetworkingthancomputing facilities inpartduetothedistributednatureofnetworks,andinpartduetolower absolutepowerconsumption. However,manynetworkcollocationfacilitiesweredesigneddecadesagoatmodestpower density.asnetworkscarrymorebitspersecond,withfixedenergycostsperbit transferred,networkprovidersarestartingtotakestepssuchaschargingformetered power,increasingcollocationrates,andlimitingrackdensity.it stooearlytoprojecthow thesedevelopmentswillaffectesnet soperatingcostsinthemediumandlongterm,butit s certainthatoperatingcostsassociatedwithpower,space,andcoolingwillincrease. ESnethasbeeninterestedinnetworkenergyconsumptionforsometime.Itwasthefirst nationalnetworktomonitorandvisualizeenergyconsumptioninrealtime,anditalso sponsoredanawardcwinningstudentresearchprojectonthetopicofpathcomputation and green powersources. Finally,ESnetistakingstepstoreduceitsownenergyfootprintbyinvestinginefficient bladeservers,implementingvirtualizationtechnology,andreplacingnumerousinefficient UPScomponentswithanenterpriseCgradesystem.Withintwoyears,ESnetwillmove
(alongwithnersc)tothecomputationalresearchandtheorybuilding,whichwillbethe world smostenergycefficienthpcdatacenter. & 6.5&Summary&Roadmap&and&Timeline 2013/2014 ESnet5backbonerequiresnew100Gbpswavelengthsasearlyaslate2013,with incrementalcapacityupgradesuntil2019orslightlybeyond. Testbedroadmapestablished.Programmabletestbeddeployed. Intelligentnetworkservicesintegrationwithapplications. 2014/2015 Outreachandtigerteamsfordatamobilityhaveasignificantimpactonproductivity ofsciencecollaborations.customerrelationshipmanagementtoolsdemonstrate utilityofsegmentingscientificcommunitiesandcarefullytrackingimpacts. Intelligentnetworkservicesprovidemorepredictableuseofthenetworkforsome importantmiddlewarecomponentsandapplications. 2016 Currentrouterplatformdoesnothavecapacityforexpansion;significantpurchase ofnewroutersthroughoutnetwork. 2017 1Tbpscapacityrequired(10x100Gbps)onallbackboneroutes. 2018 ExponentialdatagrowthonESnetcontinues(increasing10xevery48months), resultingindeploymentofadditionalwavesand>1.5tbpsbackbone.preparation toexpandontounused,nationalfiberopticfootprintcommences. 2020 ESnet5opticalfibercapacityisfullyutilized;ESnet6isdeployedonnationCwide darkfiberfootprint. 7.&Human&Resources& 7.1&Current&Workforce& ESnetstaffmembersareinspiredtoworkfortheorganization,andthereforeloyal,which hasresultedinlowturnover.evenintheheightofthedot.comboom,wheninternet engineerswereinshortsupply,theorganizationlostjusttwomembersofitscore engineeringteam.thatloyaltydidcontribute,however,toanagingworkforce.inthepast twoyears,sixkeystaffmembersretired withacombinedesnetworkhistoryofnearly 100years.TheywerereplacedbyearlyandmidCcareerstaffforthemostpart,allofwhom gainedvaluableexperiencebyparticipatinginesnet5upgrade. 27
Culturally,ESnethasastrongworkethic.Thisleadstoheroicperformanceintimesof crisis,butitalsoexacerbatesachronic(andlongcstanding)problemofunderstaffing comparedtopeerorganizations.theresultingoverloadwilleventuallyleadtofrustration andpoormorale,ifnotaddressed.recently,anddespitethedemandsoftheesnet5project, therehasbeengoodprogressinhiring,butthegapisnotentirelyclosed(especiallyinthe areaofoutreach,businessoperations,internetengineering,andsoftwareengineering). 80%ofESnetstaffmembersareengineers.ThecoreskillrequiredisInternet(orwideCarea network)engineering,forwhichthesupplyofoutstandingcandidatesislimited,especially attheopticallayer,duetoworldwidedemand.almostallofesnet sinternetengineers havesystemsandsecurityexperienceaswell,andmanyarealsoskilledsoftware developers.esnetstaffarealsoexpectedtohaveoutstandingcommunicationand collaborationskills. Forthemostpart,ESnethasbeenextremelysuccessfulinrecruitingworldCclasstalent. StaffhavepreviouslyworkedatGoogle,Oracle,Sun,WolframResearch,Level3Networks, Internet2,UCBerkeley,NCSA,NERSC,LBNL,JGI,BNL,AmesLab,LLNL,andmanyother prominentorganizations. 7.2&Future&Workforce& Likemanyoftheworld sresearchnetworks,esnetisdevotingmoreefforttodeveloping advancedservicesandsoftwarecapabilities,whichrequireshiringproportionallymore staff.driversforthechangeareexplainedinsection6.1,above.foranillustrationofthis largecscaletrendinthecontextofseveraldozeneuropeannetworkingorganizations,see Figure3. Figure3.Source:TERENACompendium2012,www.terena.org/compendium. 28
Inadditiontothisoverarchingtrend,staffincreaseswillbenecessarytosupport exponentialtrafficgrowth,transcatlanticnetworkextension,andpossibletranscpacific. Thefacilityisintheprocessofmodelingstaffrequirementsdrivenbynetworkscaleand complexity,aspartofthelargerprojectofprojecting5c7yearcosts(seesection3.3.1). 7.3&Obstacles&and&Strategies& MajorObstacle Persistentshortageofthemostskilledandqualifiedengineers,especially: experiencedinternetengineerswithabackgroundinscience,researchnetworking, oropticaltechnology engineerswithsuperbcommunicationandcollaborationskills softwaredevelopersandwebappdevelopersfamiliarwithnetworking engineerswhoarewomen Strategies sponsor5c6studentinternshipspersummer ensurethatfacilitydirectorispersonallyinvolvedinstrategicrecruitments,making callstopromisingcandidates ensurethatfacilitydirectoris alwaysrecruiting inpublicpresentations,by stressingthesatisfactionsofesnet smissionandtalentpool maximizeuseofsocialnetworkinginrecruitment buildfromwithin,givingpromisingearlyccareerstaffplentyofresponsibilityand interestingprojects fightbackwhensiliconvalleytriestorecruitkeystaff(bylisteningcarefully, identifyingthesourceoftemptation,andtryingtoconstructalocalremedy) activelyparticipateinhighcprofileconferencessuchasscandnanog,toidentify futureesnetstaff seekassistanceatthelableveltoincreaserepresentationofwomenonengineering teams o mostrecenthiresattheleadershiplayerhavebeenwomen,butthepipeline ofinternetengineerswhoarewomenisdisappointing,industrycwide o hiringstudentsandtargetedrecruitmentappeartobesuccessfulstrategies 8.&Conclusion& ThisdocumentisaroadmapforachievingESnet svisionofthefuture.inthatfuture, scientificprogresswillbecompletelyunconstrainedbythephysicallocationofinstruments, people,computationalresources,ordata.inaddition,collaborationsateveryscaleandin everyscientificdomainwillhavetheinformationandtoolstheyneedtoachievemaximum benefitfromglobalnetworksandemergingnetworkcapabilities. Inthenextdecade,ESnetisverywellCpositionedtoachievethisvision.Itskeydriverisdata intensityacrossthespectrumofscscience,anditskeyassetissuperbengineeringstaff.as 29
onereviewercommentedinthemostrecentoperationalassessmentreportforesnet, the entirestaffconscientiouslyandcontinuallyleadtheirfield. Becausesciencedataflowshaveexactingrequirementsandcontinuetogrowexponentially, ESnetwillincreaseactivitiesinappliedresearch,development,andinnovation.Inorderto avoidatechnologyroadmapthatdivergessubstantiallyfromtheindustrynorm,esnetwill engageandcollaboratewithvendors.inordertoassurethatitsactivitiesenableand acceleratediscovery,esnetwillbuildastrongoutreachprogram. MostnetworktrafficonESnetoriginatesorterminatesoutsidetheDOEcomplex.Forthis reason,esnetcannotsucceedunlessitfosterspartnershipsandpioneerstechnologies necessarytoensurethatarchitecturalchangesoccurnationallyandinternationally.esnet s missionrequiresittoinnovate,influence,andleadonaglobalscale. TheOfficeofScienceinvests$5Bannuallyinsupportingtransformationalresearchand operatingworldcclassfacilitiesthroughouttheunitedstates.esnetistheessential circulatorysystem forthisinvestment.whenesnet sdatactransportcapabilitiesare interruptedorevenslightlyimpaired(seesection5.1),theproductivityofevery collaborationfundedbyscsuffers.althoughtheinternetisoftentakenforgranted,the healthandsuccessofdoe snetworkingfacilityshouldnotbe.sccannotthrivewithouta strongandconstantlycinnovatingenergysciencesnetwork.& 30
Appendix&1:&Process&for&Developing&this&Strategic&Plan& & Thisstrategicplanhasbeenacollectiveeffort.Thefacilitysolicited broadinputfromitsstaffinseveralways:astaffcwideretreat; focusedmeetings;anonline virtualwhiteboard forideasand brainstorming;andcirculationofdraftsforcomment. Inaddition,thefacilityDirectorpresentedcoreideasfromtheplan toameetingofnationallaboratorynetworkingcoordinators;during aplenarytalkataninternationalnetworkingconference;andtoa subccommitteeofascac. AtwoCpagesummarywascirculatedtoASCRcomputingcenter directors.esnetalsoconsultedwiththenerscdirectorandstaff, theberkeleylabaldforcomputingscience,andtheesnetascr ProgramManager.Manysuggestionsforimprovementwere incorporatedintothefinaldraft. & 31
Appendix&2:&ESnet&Traffic&Growth&Projections&& ThefollowingchartshowsthehistoricalandprojectedfuturerelationshipbetweenESnet s monthlyacceptedtrafficvolume(ingbps),anditsbackbonebisectionalbandwidth(alsoin Gbps).Insimpleterms,theredlinemust stayabovetheblue inorderforthenetworkto beresilientagainstamajorfiberoutageandalsoremain(noncblocking.thegreenline showsprojectedgrowthrateinbackbonecapacity. ESnetanticipatesaddingnew100GwavestothenationalfootprintstaringasearlyasFall 2013.By2015,newopticalandroutingchassiswillberequired.In2016,thecomplexityof ESnet5willexceedthatofESnet4(intermsofdevicesundermanagement).In2017,ESnet willbeaterabitnetwork(10x100g).in2018and2019,esnetwillpreparetolightitsdark fiber,turningupthefirst200gwaveonesnet6in2020(atwhichtimethefacilitywillgo backandconvertesnet5from100gto200gtechnology). & 32
Appendix&3:&ESnet&Budget&Scenarios&and&Analysis& TheOfficeofSciencehasrequestedthatthisstrategicplanincludetwolongCterm budgetaryscenarios:oneinwhichprogrammaticneedsarefullyfunded,andanotherin whichfundingisconstrainedtogrowby1%annually. Overthenextdecade,ESnet scostswillbedominatedbyopticalandroutingequipment, plusthelaborrequiredtomaintainagrowingnumberofopticalwaves.boththe technologyroadmapanddownwardpriceccurveforhardwarecomponentsarewell understoodforthenext2c3years,butmorepoorlyunderstoodinthe5c10yearrange. Consequently,anylongCrangebudgetforecastwillhaveadegreeofuncertainty.ESnetis makingseveralassumptionsinpresentingthesebudgetscenarios. First,weassumethatnotechnologicalbreakthroughwillradicallychangethe facility scoststructureinthenext10years.incrementaltechnologyimprovements andsteadydeclinesincomponentcostsarefactoredin,butstepcfunction disruptionsarenot. Weassumethat200GbpstechnologywillnotachieveroughcostCparitywith 100Gbpstechnologyuntil2019,andthat400Gbpstechnologywillnotbeattractive inlongchauloperations,duetoitsshorterreach.againabreakthroughispossible, butwedonotfactoroneintothesebudgetarymodels. Finally,weassumethattrafficdemandwillcontinueathistoricalrates.With currentinformation,esnethasnoreasontobelieveotherwise. StartinginFY2016,thecosttolightadditionalwavesontheESnet5infrastructurewill increasesignificantly,asaresultofexponentialtrafficgrowth.asthe fullyfunded budget scenarioindicates,esnet sbudgetauthoritywillneedtoriseinordertomeetongoing demand.inapproximately2020,esnetwillhaveexhaustedspectralcapacityonthe currentfiber,andwillneedtodeployopticalgeartolightitsdarkcfiberfootprint(using 200Gwavetechnology);afterthat,ESnet5willbeconvertedfrom100Gto200G technology. Inthe1%budgetscenario,ESnetwillencountersevereproblemsbeginninginFY17.The facilitywouldcurtailcertainprogrammaticactivities(includingtranscatlanticextension), butmoreimportantlyitwouldstopaddingadditionalwaves,andbeginallocatingexisting capacitythroughpeerreview.assumingesnetstopsaddingwavesin2017,thenby2021, networkdemandwouldbe10xgreaterthanesnetcapacity.longbeforethat,thelackof adequatedatatransportwithinthedoewouldseverelyconstraintheproductivityofevery program,facility,andnationallab.smallerscientificprojectsmighttrytoadaptbyshipping disks,butmostlargerprojects(includingatlasandcms)wouldnothavethatoption,and theirscienceworkflowswouldbedevastated.thefollowingdiagram(which zoomsin on aportionofthediagramshowninappendix2)illustratesthetimingandeffectofthe1% fundingscenario. 33
ESnet&oversubscribed& wave&moratorium & Forthosereadersmorefamiliarwiththinkingaboutfundinginconnectionwith supercomputingfacilities,ananalogymaybehelpful.imagineanhpccenterthatis constrainedfromprocuringanynewsystems.overtime,centercapacitywouldbecome negligiblecomparedtothegrowingdemandforcomputation;eventually,thecenterwould beeffectivelyuseless.a wavemoratorium foresnetwouldhavethesameeffect. BecauseESnetfunctionsasavitalcirculatorysystemfortheentireDOEcomplex,capping itsgrowthwouldfundamentallycompromisethe$5binvestmentmadeeachyearbythe OfficeofScience. 34