Planning for the LHC T0-T1 Networking William E. Johnston Manager and Senior Scientist Lawrence Berkeley National Laboratory 1
Science Data Network (SDN) core Australia CA*net4 Taiwan (TANet2) Singaren JGI LBNL NERSC SLAC High-Speed Physical Connectivity to DOE Facilities and Collaborators (Summer 2005) LIGO PNNL SINet, AARnet TWC LLNL SNLL PAIX-PA Equinix, etc. YUCCA MT GA 42 end user sites Office Of Science Sponsored (22) NNSA Sponsored (12) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6) peering points IP core hubs high-speed peering points IP core LANL QWEST ATM CA*net4 France GLORIAD Kreonet2 MREN Netherlands StarTap TANet2 Taiwan (ASCC) SNLA Allied Signal SND core hubs AMES ARM KCP (LHCnet part DOE funded) ANL OSTI NOAA IP core: Packet over SONET Optical Ring and Hubs SINet (Japan) Japan Russia (BINP) ORNL INEEL-DC ORAU-DC LLNL/LANL-DC 4xLAB-DC GTN&NNSA ORAU SRS GEANT - Germany, France, Italy, UK, etc MAE-E JLAB PPPL Equinix International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings ( 10 G/s) OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less MIT BNL
The IP core is primarily a layer 3 infrastructure o However, supports layer 2 via MPLS o Directly connects sites o Provides global peering for sites The SDN core is primarily a layer 2 infrastructure o Targeted at providing virtual circuit services Site connectivity is evolving to be Metro Area Network rings Goal is two independent core rings that are interconnected at various locations with metro area rings 3
Target Architecture: IP Core+Science Data Network Core+Metro Area Rings Asia-Pacific Aus. Australia Sunnyvale LA San Diego IP core hubs SDN/NLR hubs Primary DOE Labs New hubs Metropolitan Area Rings Science Data Network Core (SDN) (NLR circuits) Albuquerque El Paso IP Core GEANT (Europe) New York Washington, DC 10-50 Gb/s circuits Production IP core Science Data Network core Metropolitan Area Networks Lab supplied International connections
Evolution of Step One: SF Bay Area MAN and West Coast SDN Asia-Pacific Aus. Australia Sunnyvale LA San Diego Science Data Network Core (SDN) (NLR circuits) IP Core (Qwest) Albuquerque GEANT (Europe) New York Washington, DC IP core hubs SDN/NLR hubs Primary DOE Labs New hubs Metropolitan Area Rings El Paso In service by Sept., 2005 planned Production IP core Science Data Network core Metropolitan Area Networks Lab supplied International connections
LHC Requirements 10G light paths from and BNL to o /LHCnet will provide10g circuit to Starlight and likely another to 32 AoA BNL must communicate with TRIUMF (Vancouver) o Data rates? Backup paths must be provided o 10G Starlight to MAN LAN o Others Access to T2 sites o What characteristics? o At least 1G guaranteed 6
Triumf Approach for LHC Requirements CANARIE LHC GEANT SL LHC SDN IP MAN LAN 32 AoA Starlight IP 60 Hudson (or 32 AoA) BNL Qwest SDN Sunnyvale SDN IP SDN core IP core All paths are 10 Gb/s 7
AsiaPac SEA Goal 2007/2008 10 Gbps enterprise IP traffic 40-60 Gbps circuit based transport Europe Aus. SNV Science Data Network (2nd Core 30-50 Gbps, National Lambda Rail) DEN Japan CHI Europe Japan NYC Europe Aus. Metropolitan Area Rings SDG hubs New hubs Metropolitan Area Rings ELP ALB IP Core ( 10 Gbps) Major DOE Office of Science Sites High-speed cross connects with Internet2/Abilene Production IP core Science Data Network core Lab supplied Major international ATL DC 10Gb/s 10Gb/s 30Gb/s 40Gb/s 8
Proposed Upgrade Schedule Fiscal Year 2005 o -ANL- Metropolitan Area Network Fiscal Year 2006 o BNL MAN Fiscal Year 2007 o Extend Science Data Network (SDN) from Seattle=>New York/Washington, DC at 10G. Jefferson Lab MAN. Fiscal Year 2008 o Science Data Network (SDN) loop around the country at10g. ORNL MAN. Fiscal Year 2009 o Double main Science Data Network (SDN) links to 20 Gigabits. Fiscal Year 2010 o Increase main Science Data Network (SDN) links to 30 Gigabits per sec. PPNL MAN. 9
Lambda Infrastructure Based on National Lambda Rail (incremental install 2005-2008) Seattle Boise Sunnyvale Denver KC Clev Pitts New York Wash DC LA San Diego Phoenix Albuq. Tulsa Dallas Atlanta Raleigh El Paso - Las Cruces San Ant. Houston Pensacola Baton Rouge Jacksonville NLR regeneration / OADM sites NLR wavegear sites
Near-Term Planning for ( mid-late 2005) ORNL OC192 NRL, UltraScienceNet, etc. Starlight sw Starlight SDN core IWire IP core Qwest hub (NBC Bld.) Qwest Notes Shared w/ IWire site equip. Switch/RTR site gateway router All circuits are 10Gb/s fiber (single lambda) / Qwest 11
Starlight Planning for BNL and Long Island MAN Ring (Qwest hub) GEANT Europe MAN LAN LI MAN Diverse dual core connection Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar Brookhaven National Lab, Upton, NY demarc Cisco 6509 10GE circuit 32 AoA, NYC 2.5G IP core IP core (Qwest) Washington (Qwest hub) Engineering Study for LI MAN (Syracuse Cleveland ) 60 Hudson, NYC SDN core (NLR circuits) LI MAN DWDM SDN core router NLR OADM Washington 1 GE or 10 GE 10GE circuit BNL IP gateway 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits 12
Starlight Planning for BNL and Long Island MAN Ring (Qwest hub) GEANT Europe MAN LAN LI MAN Diverse dual core connection Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar 2 1 Brookhaven National Lab, Upton, NY demarc Cisco 6509 10GE circuit 32 AoA, NYC 2.5G Washington (Qwest hub) IP core 7 IP core (Qwest) Engineering Study for LI MAN 4 3 6 5 (Syracuse Cleveland ) 60 Hudson, NYC SDN core (NLR circuits) LI MAN DWDM SDN core router NLR OADM Washington 1 GE or 10 GE 10GE circuit BNL IP gateway 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits 13
Starlight Planning for BNL and Long Island MAN Ring (Qwest hub) GEANT LI MAN Non-diverse dual core connection Europe MAN LAN Abilene NYSERNet SINet (Japan) CANARIE (Canada) HEANet (Ireland) Qatar Brookhaven National Lab, Upton, NY demarc Cisco 6509 10GE circuit LI MAN DWDM 32 AoA, NYC 2.5G Washington (Qwest hub) IP core IP core (Qwest) (Syracuse Cleveland ) Engineering Study for LI MAN SDN core router SDN core (NLR circuits) Washington 1 GE or 10 GE 10GE circuit BNL IP gateway 10 Gb/s circuits International LHCnet circuits (proposed) production IP core SDN/provisioned virtual circuits 14
Planning for BNL and Long Island MAN Ring LI MAN ring detail MAN LAN (?) NLR GEANT, etc. Other connections BNL internal Other connections Engineering Study for LI MAN **CONFIDENTIAL** 15