Networking for Science + Software-Defined Networking (SDN): Hype vs. Hope

Networking for Science + Software-Defined Networking (SDN): Hype vs. Hope Inder Monga Chief Technologist and Area Lead HEAnet Conference 2013 Athlone, Ireland

Outline Energy Sciences Network Networking for Science Software-Defined Networking

ESnet was formed 26 years ago - 3 years after HEAnet

26 Years as a Mission Network Mission of DOE Office of Science: Deliver knowledge and tools for transforming our understanding of the universe. Mission of Energy Sciences Network: Accelerate research and discovery for DOE Office of Science. $5B/year for the US National Lab Complex, which includes: world's largest collection of scientific user facilities (32) supercomputers, accelerators, xray / neutron sources, electron microscopes, sequencers, fusion facilities, Energy Sciences Network >100 Nobel Prizes

Embedded in a US National Laboratory (Berkeley Lab) one of 3 CS Divisions at Berkeley Lab surrounded by scientific collaborations, largescale tools, Petabytes of data, 4000 researchers / staff advantages of proximity: cafeteria and hallway conversations 11/14/2013 6

Coupled with a Major Research University UC Berkeley is just downhill. 36,000 students, 1500 faculty hundreds with joint appointments at Berkeley Lab 11/14/2013 7

Our Advisory Board Cees De Laat David Clark Larry Smarr Jagdeep Singh David Foster Vint Cerf Kristin Raushenbach

Log scale From HEAnet Strategic Plan 2008-2013

Global Transfer Activities (LHC/ATLAS)

Clev. The 100G Energy Sciences Network (Fall 2013) Int l 100 Int l JGI 10 PNNL 10 10 Capability to scale to 13.2 Tbps 10 100 100 100 100 10 100 10 10 BNL 100 100 10 Int l 10 100 SUNN 100 100 10 SNLL LLNL 1 100 AMES 1 100 100 100 100 100 10 100 PPPL GFDL PU Physics 10 Int l 10 10 100 SNLA 100 100 1 U.S. commercial peerings U.S. R&E peerings (many) 100 100 ESnet routers Optical only site routers 100G 10-40G 1G 100 Metro area circuits Site provided circuits 100 Geographical representation is only approximate SF Bay Area Chicago New York Amsterdam SUNN STAR AofA AMST 100G testbed

276 miles 560 miles 618 miles ESnet Research Testbeds 100G Testbed High-speed protocol research Available since Jan 2012 Dedicated 100G wave from Oakland to Chicago to NYC Connects to 100G across Atlantic to Amsterdam (ANA-100G) SUNN NERSC LBNL Test% Hosts% ESnet 100G Testbed NERSC StarLight ANL Test% Hosts% 2/25/13 3 StarLight BNL MAN LAN NYC OpenFlow Testbed 10G Nationwide Footprint ESnet 10G OpenFlow Testbed Seattle HOUS LBNL Long Haul Dark Fiber Routes 12,924 miles Dark Fiber Testbed Continental-scale fiber footprint for disruptive research 284 miles Boise Albany Eugene 2/25/13 20 Cambridge 74 miles Buffalo Chepachet 204 miles Echo Springs 264 miles Stamford Pittsburgh 61 miles 336 miles South Bend Salt Lake City 152 miles Chicago New York Reno 95 miles Cleveland Sacramento Philadelphia Peoria Goodland Indianapolis 116 miles 198 miles Sunnyvale Denver 460 miles Washington DC 138 miles 422 miles 532 miles Los Angeles 551 miles 317 miles Phoenix 325 miles Silver City 119 mil es 500 miles Albuquerque 249 miles El Paso 863 miles Kansas City Tulsa 257 miles 278 miles 317 miles 205 miles St. Louis Dallas Jackson 130 miles 119 miles 228 miles 255 miles 212 miles 259 miles Louisville Nashville 150 miles Chattanooga 147 miles Seminary 22 8 miles 246 miles Selma 172 miles 27 5 mile s Atlanta 295 mil es 306 miles 212 miles Raleigh Charlotte 179 miles 13 7 miles 248 miles 275 miles Houston BayExpres Metro Fibers: 432 miles ChiExpress Metro Fibers: 167 miles NYExpress Metro Fibers: 6 miles

Network engineered for the Elephants 11/14/2013

Sensitive Elephants, Robust Mice 7.3 Gbps out / 9.8 Gbps in 1 ms RTT 1 Gbps out / 9.5 Gbps in 11 ms RTT 122 Mbps out / 7 Gbps in 51 ms RTT 60 Mbps out / 5 Gbps in 88 ms RTT How to build a lossless network service? > 80x reduction in data transfer rate at DOErelevant distances (ANL to NERSC) and speeds (10Gpbs). Infrastructure: ample network capacity Equipment: deep packet buffers ScienceDMZ: optimized end-site architecture perfsonar: automatic and continual verification of network health OSCARS: fast lanes http://www.es.net/assets/pubs_presos/sc13scidmz-final.pdf

Science DMZ Design Pattern for Data Transfer Dedicated Systems for Data Transfer Network Architecture Performance Testing & Measurement Data Transfer Node high performance tuned for data transfer proper tools Science DMZ dedicated and clean location for DTN easy to deploy - no need to redesign the whole network additional info: http://fasterdata.es.net/ perfsonar enables fault isolation verifies correct operation widely deployed in ESnet and other networks, as well as sites and facilities source: Eli Dart ESnet 11/14/2013 16

Prototype Science DMZ Border Router Enterprise Border Router/Firewall WAN 10G 10GE perfsonar Clean, High-bandwidth WAN path 10GE Site / Campus access to Science DMZ resources Science DMZ Switch/Router 10GE Site / Campus LAN 10GE source: Eli Dart ESnet High performance Data Transfer Node with high-speed storage Per-service security policy control points perfsonar 11/14/2013 17

Prototype Science DMZ Data Path Border Router Enterprise Border Router/Firewall WAN 10G 10GE perfsonar Clean, High-bandwidth WAN path 10GE Site / Campus access to Science DMZ resources Science DMZ Switch/Router 10GE Site / Campus LAN 10GE source: Eli Dart ESnet High performance Data Transfer Node with high-speed storage Per-service security policy control points perfsonar High Latency WAN Path Low Latency LAN Path 11/14/2013 18

Science DMZ is critical. Knowledgebase: http://fasterdata.es.net/ Science DMZ: http://fasterdata.es.net/science-dmz/ Security:http://www.internet2.edu/presentations/tip2013/20130115-dartscience-dmz.pdf CC-NIE: http://www.nsf.gov/pubs/2013/nsf13530/nsf13530.htm 11/14/2013 19

SDN is everywhere! SDN 2013 http://www.tomsitpro.com/articles/sdx-software-defined-kitchen-sink,1-1085.html

What is SDN? Loose definition: separation of data-plane from control plane In essence: enables programmability programmable Network Provisioning Network Monitoring Cloud/End-user Applications [Science] Applications Network Apps [NaaS] Network Virtualization Network Controller(OS) Statistics Topology Provisioning Protocols (SNMP, TL1) Protocol(s) (OpenFlow,?) Control Software Firmware Network ASICs Network Element control control control Control Firmware control Software control Firmware Firmware Network Firmware Network Firmware ASICs Network ASICs Network ASICs Network ASICs ASICs Network Element

Layer 3 Layer 2 Layer 1 What is the paradigm change? Internet today: - Built-in control in each layer - Multiple management domains SDN Approach: - Network-wide cross-layer view - OpenFlow enables programmatic access to network flows Layer 3 Control? Control (Network-wide view) Layer 3 Control OpenFlow OpenFlow Layer 2 Layer 1 Control Control Management Management 10/16/13 Inder Monga 23

Simple programming constructs OpenFlow 1.0 standard Rule Action Stats Packet + byte counters 1. Forward packet to zero or more ports 2. Encapsulate and forward to controller 3. Send to normal processing pipeline 4. Modify Fields 5. Any extensions you add! Switch Port VLAN ID VLAN pcp MAC src MAC dst Eth type IP Src IP Dst IP ToS IP Prot L4 sport L4 dport + mask what fields to match 11/14/2013 Inder Monga Slide courtesy Srini Seetharaman 24

OpenFlow usage Classic model, Simple example Controller OpenFlow Alice s Rule Switch Alice s App PC Decision? OpenFlow Protocol OpenFlow Alice s Rule Switch OpenFlow Alice s Rule Switch OpenFlow offloads control intelligence to remote software 11/14/2013 Inder Monga, WLCG GDB Alice 25

Why SDN? Bridging the application-network divide First workshop report for ESnet on intersite networking, 1986 Network community is still struggling to meet application requirements captured in 1986! Brute force approach (add more bandwidth) is not going to meet those requirements

Slide from Ian Foster

1-ID-1 1-ID-2 1-ID-3 1-ID-4 2-BM 2-ID-B 2-ID-E 2-ID-D 3-ID-B 3-ID-C 7 8-BM 8-ID-I 8-ID-E 9 11-ID-B 11-ID-C 11-ID-D 12-BM 12-ID-B 12-ID-C/D 15-ID 20 21-ID 21-ID-D 21-ID-E 21-ID-F 21-ID-G 22 23-ID-D 23-ID-B 30 32-ID-1 32-ID-2 34-ID Advanced Photon Source data rates 10000 1000 Data Rate (expected in the next 5-10 years) MB/s Data Rate (current) MB/s 100 10 1 Francesco de Carlo, APS 28 Slide from Ian Foster www.ci.anl.gov www.ci.uchicago.edu

Hope #1: Programmability will lead to greater predictability Burst movement of data using PhEDex Analysis triggered data movement (PD2P) Science applications increasingly need to deal with high performance, anyany bursts of data SDN enables multi-layer control packet and optical layer Control over individual flows ex. Route science flows around packet bottlenecks Routing non-tcp flows over WAN Many NRENs have access to fiber, optical and packet platforms.

Journey towards programmability Seamless multi-layer for handling elephant flows Layer123 SDN World Congress, Bad Homburg, October 2013 OTS Config Manager L0/L1 Topology Advanced Reservation System (OSCARS) Multi-Layer Topology App Multi-Layer Path Engine Multi-Layer Provisioning Multi-Layer SDN Control Layer Traffic Optimization Engine SDN Controller Floodlight OpenFlow & REST/JSON OpenFlow 1.0 Host A Host B OTS Virtualization WDM/ OTN/ Packet Infinera DTN-X Live Demo Nov 22: http://www.sdncentral.com/events/brocade-infinera-esnet-sdn-demo/

Abstractions are important (Scott Shenker, October 2011) http://opennetsummit.org/archives/oct11/shenker-tue.pdf The ability to master complexity is not the same as the ability to extract simplicity Abstractions key to extracting simplicity SDN is defined precisely by these three abstractions Distribution: centralized vs. distributed Forwarding: programming the fabric Specification: virtualization 11/14/2013 Inder Monga OFC/NFEC, 2013 31

What is the right abstraction for a (dynamic) collaboration? Set of (dynamic) point to point circuits Restricted & static routing policy Lots of meetings 11/14/2013 Inder Monga OFC/NFEC, 2013 32

Hope #2: Virtualization will simplify how applications program the network App 1 App n NB API Modeled as a Virtual WAN Network Element Simple, Multipoint, Network Virtualization Programmable simple complex Network slice Network Controller(OS)

Journey towards programmability Real network is too complex to program for applications SRS, Ciena, SuperComputing 2012, Salt Lake City Insights Virtualization is the killer-app for SDN (Scott Shenker) complexity pushed to the network hypervisor Architectural simplicity Flow programming only needed at edges of the network, core can be legacy

Thought experiment: Build an N-port virtual switch for a collaboration CERN T1 mile s kms France 350 565 Italy 570 920 UK 625 1000 Netherlands 625 1000 Germany 700 1185 Spain 850 1400 Nordic 1300 2100 USA New York 3900 6300 USA - Chicago 4400 7100 Canada BC 5200 8400 Taiwan 6100 9850 Source: Bill Johnston Universities/ physics groups Universities/ physics groups Universities/ physics groups The LHC Open Network Environment (LHCONE) Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups WAN Virtual Switch Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups LHC Tier 2 Analysis Centers 11/14/2013 Inder Monga OFC/NFEC, 2013 35

SDN is about system optimization Google s B4 SDN Network Utilization When the application and network work as a system, network resource optimization is possible Without knowledge of flows, networking can only do coarse characterization Fine discrimination of flows possible with SDN, meet application needs

Hope #3: SDN enables an opportunistic way to leverage all bandwidth without extra investment exploits the fact In general it s much cheaper to transport data than to store it, for example, vision of a diskless Tier3 for LHC

Is SDN ready for operations? 2013: SDN is here The innovator s dilemma, Clayton Christensen

Challenges = Opportunities? Cloud/End-user Applications 5) Who do you blame? Who do you call? Who debugs? [Science] Applications Network Apps [NaaS] Network Virtualization 4) Complexity of management from operations on virtual to physical reality? 2) Single point of failure or attack? Network Controller(OS) Statistics Topology Provisioning 3) Responsive to rapid topology changes? Flapping? 7) How does this interoperate with the current IP network? Protocol(s) (OpenFlow,?) control control control Firmware control control Firmware Firmware Network Firmware Network Firmware ASICs Network ASICs Network ASICs Network ASICs ASICs 1) Communication plane can be disrupted 6) Hardware will never be simple, manage capability differences

Journey towards programmability How to bridge the Internet with SDN networks? Treehouse, BGP over SDN infrastructure, ONS 2013 and ongoing Insights SDN networks can now peer with existing Internet New techniques need to be developed to scale controller-based networking Baby steps Project led by Josh Bailey, Google

The Bigger Picture: Organizational challenge to deal with SDN Layer 10 People People (network engineers**, sysadmins, operators**) (network engineers, + sysadmins, operators) (software engineers/devops) Layer 8-9 Network Operating System (control) Management, Tools, + Measurement New tools, service plane and management Layer 0-7 Network (control (API + and data data plane) plane) ** need to develop new skills

SDN Take-Away SDN is a journey R&E networks have been on for a while, but recently commercially formalized Innovator s dilemma gaps between established and the new ways, industry and researcher momentum will close those gaps Maturity will still take some time Focus on the problem being solved aka hope rather than the vendor hype Plan for the SDN future skillsets, training and hiring

Questions?

Problem: Mice and Elephant flow separation OSCARS, 2006-2013 SDN before it was called SDN Insights abstractions are key to success, regardless of the protocol can only learn by doing (lots of naysayers) Primary use will be different than the original usecase

Bringing it Together: A potential SDN R&E architecture (e2e resource broker) OneWan Switch OSCARS NSI NSI NRM (e2e resource broker) Service Plane Customer SDN Ctrl. OF SDN Ctrl. OF OSCARS Customer SDN Ctrl. OF Control Plane NERSC BNL FLA Router Transport ESnet SDN ORNL FLA Router FLA Router R&E Network SDN only at edges, efficient transport in core RON OF Univ. Data Plane

Three Inflection Points for Data-Intensive Science Abundant capacity (88 λ x 100Gbps) 2. Programmability Campus architectures newly optimized for data mobility (optimizing network architectures end-to-end) ESnet architecture (Science DMZ) + NSF grants.

What is common between modern networks and analog phone switches?

Labor-intensive, nearly static, error prone