Real-World Insights from an SDN Lab. Ron Milford Manager, InCNTRE SDN Lab Indiana University

Transcription

1 Real-World Insights from an SDN Lab Ron Milford Manager, InCNTRE SDN Lab Indiana University 1

2 A bit about IU, the GlobalNOC, and InCNTRE...

3 Indiana University s Network History 1998 University Corporation for Advanced Internet Development (aka Internet2) selects IU to build and operate the premier network supporting US research and education. The GlobalNOC is created National Science Foundation funds IU to provide highperformance networking between Asia Pacific and US National Lambda Rail (a facilities-based R&E network) selects IU to provide NOC services 2011 IU announces the formation of Indiana Center for Network Translational Research and Education (InCNTRE) 2013 InCNTRE under contract with ONF to assist with the development of OpenFlow conformance certification program

4 The IU GlobalNOC operates this 15,000+ mile network and...

5 ...these networks internationally...plus many more domestically

6 The IU GlobalNOC 27 Research and Education Networks o Architecture, Engineering, Monitoring, Installation o Exchanges, Regional, National, International o Optical, Layer2, Layer3 3 SDN/OpenFlow Networks o GENI -2010, Deployed 1st Nationwide R&E Testbed OpenFlow Network o NDDI Prototype Nationwide R&E Production Prototype OpenFlow Network IBM G8264 o NDDI AL2S I2 100Gb Advanced Layer 2 Services Production OpenFlow Network Juniper MX-80, Brocade MLXe-4

7 InCNTRE Mission Seeks to advance development, increase knowledge, and encourage adoption of OpenFlow and other standards-based Software-Defined Networking (SDN) technologies through: A Neutral Testing Facility Educational programs Training Internships Direct networking research Collaboration between faculty, students, and networking professionals

8 InCNTRE Initiatives Summer of Networking Internship Program Network Professional Training SDN Network Research o FlowScale, RouteFlow, XSP, GENI... SDN Lab o Interoperability Testing o OpenFlow Conformance Testing o Internships o Open Networking Foundation Research Associates

9 InCNTRE SDN Lab Research/Demo Networks o InteropNet OpenFlow Lab Interop Las Vegas 2011 Interop Las Vegas 2012 o SuperComputing 2011 SCinet Research Sandbox ONF Collaboration o Co - Chair Testing & Interoperability Working Group o Conformance Testing Program Development o Conformance Test Specifications OpenFlow Switch In Review Phase OpenFlow Switch Under development

10 ONF PlugFests

11

12

13 ONF PlugFests March 2012 Controllers -Big Switch, NEC, NTT Data, NOX Switches - Big Switch(Indigo), Broadcom, HP, IBM, Intel/WindRiver, Juniper, NEC October Companies, ~40 Switches, 4 testbeds(of 1.0, 1.2, Test Frameworks) Controllers Big Switch, HP, Huawei, Ixia, NEC, NTT Data, NTT MCL, NOX June 2013 Adding OpenFlow 1.3 & OF-Config Testing

14 What is OpenFlow and Software Defined Networking in 60 seconds...

15 OpenFlow is SDN, but SDN is not alway OpenFlow Software Defined Networking (SDN) is an architecture that separates and abstracts elements of the network. SDN centralizes the control plane from the high-speed data plane in network switching equipment. SDN allows programmatic control of a network instead of static configuration. OpenFlow is a protocol that allows the centralized control plane to control the behavior of distributed switching equipment and the API that exposes and abstracts the capabilities of the OpenFlow controlled equipment.

16 Features CLI, SNMP, TFTP Value Add Ethernet Switch Control Plane Embedded Operating System Data Plane Table-based (e.g., TCAM/CAM) high-speed forwarding engine

17 Control Plane Features OpenFlow Controller Value Add OpenFlow Protocol Embedded Operating System implements OpenFlow Data Plane Table-based (e.g., TCAM/CAM) high-speed forwarding engine

18 Features OpenFlow Controller OpenFlow Controller OpenFlow Controller Value Add OpenFlow Protocol Each switch connects directly with OF Controller

19 OpenFlow Abstraction Permits "Slicing" of the Network into virtual resources Allows "Slicing" network into virtual administrative domains Each Slice controlled by it's own controller and admin Network can be "Sliced" based on many criteria o o o o o o o Switch or Data Path Identifier (DPID) Port Src or Dst MAC Addresses Srcor DstIP Addresses Ethertype VLAN ID TOS Bits o Rest of 12-tuple (OpenFlow 1.0) Flowvisor Coming soon in controllers near you

20

21 InCNTRE's experience with some interesting demo networks...

22 InteropNet LV 2011 OpenFlow Lab 14 controller and switch vendors ~40 hardware and software Switches 5 Demos, each with their own controller and administrative slice of the network.

23 InteropNet LV 2011 OpenFlow Lab Worked with Stanford's Open Labs 3 Months planning, 3 weeks staging, 5 days in production OpenFlow 1.0 Switch Vendors o o Broadcom, Brocade, Dell, Extreme, Fulcrum, HP, IBM, Juniper, NEC, Netgear, Pronto Open vswitch on XenServer OpenFlow 1.0 Controllers o Big Switch, NEC, NOX Flowvisor Virtual Slicing o 6 Slices Switch & Port Source & Destination IP Subnet ~60,000 flowspace rules With right IP Subnet configured, could connect to almost any port and be in your slice

24 Supercomputing 2011 Distributed through show floor IBM, HP, NEC, Pronto Switches Beacon, NOX, NEC Controllers Inter-Connected to GENI and NDDI Prototype OpenFlow Networks Inter-Connected to ESnet, Internet 2, NLR, SCinetnon-OpenFlow Networks 9 Slices/Administrative Domains o o o o o Switch & Port Source & Destination MAC Address Source & Destination IP Subnet VLAN ID Ethertype 216 Flowspace Rules o Strict Port Assignments

25 Supercomputing 2011

26 IP Subnet: /24 MAC Address: 00:02:C9:10:F1:AC

27 Ethertype: 0x8902

28 IP Subnets: GENI Subnet, /24, / /24, /24, /24 IP Subnets: GENI Subnet, /24, / /24, /24, /24

29 VLAN IDs: 1327,

30 IP Subnet: GENI Subnet Ethertype: 0x27A0 VLAN: 233, 3715, 3716

31 Issues from SC11 Switch CPU limitations Too many messages to/from the controller Traffic flowing before controller can prepopulate tables No isolation of switch resources One heavily active or misbehaving slice can overrun the switch CPU, TCAM or memory Switches failing Open Floods traffic everywhere Can easily create loops & broadcast storms LLDP Behavior Different Across Vendors Controllers use LLDP for topology discovery Hybrid switches send LLDP FlowVisor Can t change controller IP without restarting FlowVisor Rule generation script unable to insert/remove individual flowspace rules Some switches not reconnecting correctly through FlowVisor. Required OpenFlow Process Restarts

32 Deep Dive into a our nationwide SDN/OpenFlow Network...

33 Network Development and Deployment Initiative (NDDI) Partnership among Internet2, Indiana University and the Clean Slate Program at Stanford University Building an OpenFlow-based network substrate to support Internet2 s new 100Gb/s services Services known as I2 s Advanced Layer 2 Service Software suite, developed by the IU, known as Open Exchange Software Suite (OE-SS)

34 Internet2 Network Layer 2 Services, 2013 Internet2

35 A bit about the software IU developed to provision services on AL2S...

36

37 OS3E Service Open Science, Scholarship, and Services Exchange First service to be built on top of the NDDI substrate. A nationwide distributed Layer-2 exchange persistent, on-demand & scheduled L2 VLANs Supports open inter-domain standards (IDCP) OS3E Use Cases High bandwidth paths for large file transfers Layer-2 connectivity between testbeds Distributed exchange for establishing IP peering

38 OESS Key Features Develop once, implement on many different types of switches Uses D-Bus to communicate with the OpenFlow controller allowing easy migration to other OpenFlow Controllers in the future. VLAN Provisioning time: <1 Second typically Automated fail-over to backup path Controller redundancy Automated measurement Easy management

39 Inter-domain Support Inter-domain support enables peering with other dynamic circuit networks. IntegradedOSCARS 0.6 with OESS for inter-domain component. Uses IDC (Interdomain Controller) protocol See a demo on the real network! km2m_4q&index=16 Initial peering with Internet2 at Los Angeles, Chicago, and Seattle Interconnecting with GEANT in New York Many more to come

40

41

42 You too can play with OESS Apache 2.0 license 3 deployment packages RPMs for RedHat/CentOS 6.tar.gz distribution VM image (virtualbox)

43 A bit about a production load-balancer built using OpenFlow...

44 Traffic load balancer as a service using OpenFlow Use existing switch hardware instead of a dedicated load balancer Built on Beacon Integration with other OpenFlow tools L2 Provisioning, VM Migration

45 Hash based on OpenFlow fields (IP src/dst) Divides IU Address Space into IP prefixes Uses Round Robin to Pre-Distribute Flows across sensors Monitors sensor utilization and dynamically adjusts Redirects specific types of traffic to specialized sensors (e.g. http or tcp/udp port) Application layer monitoring and failover Web UI for admin

46 Deployment IDS Cluster 2 Campuses 12 Sensor hosts per campus 10Gb/sec on all links 480Gb/sec total capacity

47

48

49 A bit about Internet2's "Innovation Platform" (i.e., 100G/s connection, SDN support, and Science DMZ)...

50 Innovation Platform 100 GigE Layer 2 Connection Internet2 innovation backbone delivered as 100G L1 Traditional regional and commodity providers Science DMZ Nx100G Nx10G High-Performance Layer 2/3 Switch/Router Nx10G Traditional Campus Border Router Nx10G Traditional L3 Campus Border Nx10G Security Campus Enterprise Network 50 4/11/2013, 2012 Internet2 SDN Control Server Performance Node Nx100G Dedicated science switches/servers in labs with high-speed storage and network access fasterdata.es.net Software Defined Networking R&E IP TR- CPS IP Network Layer 3 Static Layer 2 Traditional Services Traditional Switch Substrate Your Research Optical System Dark Fiber Dynami c Layer 2 GENI Experiment s GENI? Innovation Services Software Defined Networking Substrate innovation@internet2.edu

51 Long-term Architectural Aspiration 51 4/11/2013, 2012 Internet2

52 Software OSCARS UI Stack Today OESS UI IDC OSCARS API OSCARS IDC OESS Controller (Currently NOX) OpenFlow Flowvisor OpenFlow Vendor Software Vendor Switch 52 4/11/2013, 2012 Internet2

53 Software OSCARS UI Stack Future OESS UI IDC OSCARS API OSCARS NSI Extension IDC OESS Controller (Currently NOX) OpenFlow Flowvisor OpenFlow Vendor Software Vendor Switch 53 4/11/2013, 2012 Internet2

54 Software Stack Not fully Aggregate Manager OSCARS API IDC OESS UI OSCARS OSCARS UI NSI Extension IDC baked OESS Controller (Currently NOX) Experimenter OpenFlow Controller OpenFlow Flowvisor Aggregate Manager (FOAM) OpenFlow Vendor Software Vendor Switch 54 4/11/2013, 2012 Internet2

55