University of California, San Diego. CHERuB 100G campus gateway

Size: px

Start display at page:

Download "University of California, San Diego. CHERuB 100G campus gateway"

Janice Dean
8 years ago
Views:

1 4/30/2014 1

2 University of California, San Diego Science DMZ PI: P. Papadopulos, co-pi: L. Smarr 01/01/2013 to 12/31/2014 CHERuB 100G campus gateway PI: M. Norman, co-pi: T. Hutton, V. Polichar 01/01/2014 to 12/31/2015

3 UCSD and its environment Stem Cell Institute Salk Institute SDSC General Atomics CalIT2 In addition to the 3 main UCSD units: - General Campus - Medical School - Scripps I.O. NCMIR Physics Skaggs Medical School there are many other research organizations on and around campus. Scripps Institute of Oceanography Venter Institute 4/30/2014 3

4 Connecting YOU on UCSD Campus with the World By Creating a Big Data Freeway System NSF CC-NIE Has Awarded Prism@UCSD Optical Switch Phil Papadopoulos, SDSC, Calit2, PI CHERuB

5 A Researcher Defined 10 and 40Gbit/s Campus Scale Data Carrier high-bandwidth end-to-end optical connections routed by next generation Arista switches (7504) connects lab data producers with SDSC data-intensive computing & storage resources 10 Terabit/s of aggregate bandwidth, has full bisection similar to inmachine room clusters, but is deployed at a campus scale builds upon and upgrades the Quartzite "campus-scale network laboratory" NSF MRI (awarded 2006) adds IPv6 and OpenFlow Project in Brief existing optical fiber connection to the SDSC is being expanded to 120Gbps as a high-bandwidth bridge to cloud/parallel storage and NSF XSEDE resources

room clusters, but is deployed at a campus scale builds upon and upgrades the Quartzite "campus-scale network laboratory" NSF MRI (awarded 2006) adds IPv6 and

6 PRISM Puts SDSC s Big Data Gordon Supercomputer and Data Oasis Storage Into Your Lab 12

7 PRISM is Connecting CERN s CMS Experiment To Our Physics Department 80 Gbps PRISM Connection Has Been Made

8 UCSD is a Tier-2 LHC Data Center: CMS Flow into UCSD Physics Dept. Peaks at 2.4 Gbps Source: Frank Wuerthwein, Physics UCSD

9 Planning for climate change in California substantial shifts on top of already high climate variability SIO Campus Climate Researchers Need to Download Results from Remote Supercomputer Simulations to Make Regional Climate Change Forecasts Dan Cayan USGS Water Resources Discipline Scripps Institution of Oceanography, UC San Diego much support from Mary Tyree, Mike Dettinger, Guido Franco and other colleagues Sponsors: California Energy Commission NOAA RISA program California DWR, DOE, NSF

Cayan USGS Water Resources Discipline Scripps Institution of Oceanography, UC San Diego much support from Mary Tyree, Mike

10 average summer afternoon temperature GFDL A2 1km downscaled to 1km Hugo Hidalgo Tapash Das Mike Dettinger 10

11 Ultra High Resolution Microscopy Images Created at the National Center for Microscopy Imaging

12 NIH National Center for Microscopy & Imaging Research Integrated Infrastructure of Shared Resources Shared Infrastructure Scientific Instruments Local SOM Infrastructure End User FIONA Workstation Source: Steve Peltier, Mark Ellisman, NCMIR

Infrastructure Scientific Instruments Local SOM

13 PRISM Links Calit2 s VROOM to NCMIR to Explore Confocal Light Microscope Images of Rat Brains

14 Protein Data Bank (PDB) Needs Bandwidth to Connect Resources and Users Archive of experimentally determined 3D structures of proteins, nucleic acids, complex assemblies One of the largest scientific resources in life sciences Virus Hemoglobin Source: Phil Bourne and Andreas Prlić, PDB

acids, complex assemblies One of the largest scientific resources in

15 PDB Usage Is Growing Over Time More than 300,000 Unique Visitors per Month Up to 300 Concurrent Users ~10 Structures are Downloaded per Second 7/24/365 Increasingly Popular Web Services Traffic Source: Phil Bourne and Andreas Prlić, PDB

Structures are Downloaded per Second 7/24/365 Increasingly

16 Source: Phil Bourne and Andreas Prlić, PDB 2010 FTP Traffic RCSB PDB 159 million entry downloads PDBe 34 million entry downloads PDBj 16 million entry downloads

17 PDB Plans to Establish Global Load Balancing Why is it Important? Enables PDB to Better Serve Its Users by Providing Increased Reliability and Quicker Results How Will it be Done? By More Evenly Allocating PDB Resources at Rutgers and UCSD By Directing Users to the Closest Site Need High Bandwidth Between Rutgers & UCSD Facilities Source: Phil Bourne and Andreas Prlić, PDB

18 PRISM Will Link Computational Mass Spectrometry and Genome Sequencing Cores to the Big Data Freeway Source: proteomics.ucsd.edu ProteoSAFe: Compute-intensive discovery MS at the click of a button MassIVE: repository and identification platform for all MS data in the world

edu ProteoSAFe: Compute-intensive discovery MS at the click of a

19 SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

20 CHERuB*: SDSC-ACT partner to bring 100Gbps connectivity to UCSD Production late 2014 LBL - CMMAP NERSC - POLARBEAR, CAIDA UNL - OSG UWisc Madison - OSG FNAL - Tier-1 LHC UCSB NICS - CMMAP UCR UCSD/SDSC New 100G path Austin/TACC Pink line New CENIC 100G Blue lines Existing/planned ANI 100G Green lines Existing PacWave 100G Maroon lines XSEDE 10G network Thin lines Other existing 10G or lower SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO *Configurable, High-speed, Extensible Research Bandwidth

lines Existing/planned ANI 100G Green lines Existing PacWave 100G Maroon lines XSEDE 10G network Thin lines Other existing 10G or

fiber DWDM 100G transponders up to 3 add'l 100G transponders can be attached SDSC Juniper MX960 "Medusa" Nx10G 100G 100G UCSD/SDSC Gateway Juniper MX960 "MX0" New 2x100G/8x10G line card + optics New

21 The Plumbing (ask Tom Hutton) 818 W. 7th, Los Angeles, CA Hopkins Drive, La Jolla, CA Equinix/L3/CENIC POP SDSC NAP DWDM 100G transponders up to 3 add'l 100G transponders can be attached to CENIC/ PacWave switch L2 existing CENIC fiber DWDM 100G transponders up to 3 add'l 100G transponders can be attached SDSC Juniper MX960 "Medusa" Nx10G 100G 100G UCSD/SDSC Gateway Juniper MX960 "MX0" New 2x100G/8x10G line card + optics New 40G line card + optics 10G 10G Existing ESnet SD router UCSD/SDSC Cisco 6509 UCSD DYNES Key: PacWave, CENIC, Internet2, NLR, ESnet, StarLight, XSEDE & other R&E networks UCSD NEW Pink/black - existing UCSD 100G Dual Arista 7508 "Oasis" 256x10G DataOasis/ SDSC Cloud New 100G card/ optics 128x10G infrastructure SAN DIEGO SUPERCOMPUTER CENTER Green/dashed lines - new component/ SDSC equipment in proposal DYNES GORDON compute cluster Other SDSC resources mult. 40G connections 2x40G PRISM@UCSD Arista 7504 PRISM@UCSD - many UCSD big data users 4x10G add'l 10G card/optics UCSD Primary Node Cisco 6509 "Node B" mult. 40G+ connections at the UNIVERSITY OF CALIFORNIA; SAN DIEGO mult. 10G connections UCSD Production users

22 CENIC/ESnet 100G Connection enables Big Data science collaborations between NERSC and SDSC LBL - CMMAP NERSC - POLARBEAR, CAIDA UNL - OSG UWisc Madison - OSG FNAL - Tier-1 LHC UCSB NICS - CMMAP UCR UCSD/SDSC New 100G path Austin/TACC Pink line New CENIC 100G Blue lines Existing/planned ANI 100G Green lines Existing PacWave 100G Maroon lines XSEDE 10G network Thin lines Other existing 10G or lower SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

23 A Unique, Powerful, Data-Intensive Testbed for Scientific Discovery EDISON HPC SYSTEM 2 PF, 434 TB RAM GORDON HPD SYSTEM 0.3 PF, 364 TB RAM+SSD 150 GB/s 100 GB/s 6 PB DTN ESnet/CENIC DTN 4.5 PB 100 Gb/s SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

24 POLARBEAR Cosmology Telescope UC Berkeley/NERSC-UCSD/SDSC Goal: Measure B-mode polarization in the CMB from inflation era Data path: Chile (obs)- UCB/NERSC (analysis)- UCSD/SDSC (analysis) Data acquisition rates: 22 GB/mo. (current) 3 TB/mo. ( ) Map making data analysis NERSC & SDSC 100 MC realizations of 100 TB data = 10 PB SAN DIEGO SUPERCOMPUTER CENTER Atacama Desert, Chile at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

25 Next Generation Network Measurement CAIDA (SDSC)-NERSC CAIDA operates the UCSD Network Telescope, which collects Internet Background Radiation Data paths: global internet, ESnet Data rates: 3-4 TB/mo Using NERSC tape archive to replicate 100 TB historical data Other projects: network measurement tools, Future Internet Architecture 100 s TB archival data SDSC/NERSC unassigned IPv4 addresses SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

26 High Energy Physics LHC/US-CMS UCSD Tier-2 US-CMS collaboration Goals: Higgs boson, supersymmetry, BSM Data Paths: CERN- FNAL (Tier 1)-UCSD (Tier 2) via ESnet and CENIC/I2 Peak Bandwidths: Current: 10+5 Gbps 2015: 40 Gbps when LHC 14 Tev SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

27 Education & Training: UCSD Telemedicine Center SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

28 CHERuB Implementation Status January, 2014 Project funded, equipment on order February, 2014 Equipment received Production network switch upgraded March, 2014 Campus gateway upgraded, connected to regional 100G feed Successful border-to-regional Next steps (April/May): Connect Prism switch, Connect SDSC infrastructure, Connect production switch, Production Goal: September 2014 SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA; SAN DIEGO

29 Comet is a ~2000TeraFLOP System Architected for the Long Tail of Science NSF Track 2 award to SDSC $12M NSF award to acquire $3M/yr x 4 yrs to operate Production early 2015

Perspec'ves on SDN. Roadmap to SDN Workshop, LBL

Perspec'ves on SDN. Roadmap to SDN Workshop, LBL Perspec'ves on SDN Roadmap to SDN Workshop, LBL Philip Papadopoulos San Diego Supercomputer Center California Ins8tute for Telecommunica8ons and Informa8on Technology University of California, San Diego