Converged Packet-Optical Software Defined Networking Tom Tofigh Marc De Leenheer AT&T ON.Lab
Outline SDN for Service Providers Background Use cases Packet/Optical Use Case Problem statement and conceptual solution Implementation Demonstration State of the Industry & Future Work
Explosive Growth Growth TRAFFIC OPERATOR COST VOICE ERA REVENUES NEW SERVICES DATA ERA * Graph Source - Accenture Analysis Service Provider Networks Time Unprecedented Traffic Growth Orders of magnitude increase in users, devices, apps Video, Mobile traffic exploding CAPEX continues to rise 2016 traffic = triple of 2011 More mobile devices than people Video: 79% of all traffic in 2018 AT&T spends $20 Billion per year on CAPEX
Turning Growth into Opportunity Open Open APIs Multi-vendor Multi-technology Open Source Scale Reduce CAPEX and OPEX Bring in cloud-style agility, flexibility, Scalability Monetize Deliver new and customized services rapidly DevOps model Lower operational complexity, increase visibility
Key Enabler: Software Defined Networking Control Apps Config Apps Mgmt Apps Closed SDN Network Operating System Agent OS Loader Merchant Silicon Whitebox Legacy
Service Provider Networks WAN core backbone Multi-Protocol Label Switching (MPLS) with Traffic Engineering (TE) 200-500 routers, 5-10K ports Metro Networks Metro cores for access networks 10-50K routers, 2-3M ports Cellular Access Networks LTE for a metro area 20-100K devices, 100K-100M ports Wired access / aggregation Access network for homes; DSL/Cable 10-50K devices, 100K-1M ports
Service Provider Network of the Future Core Packet-Optical POP Built like a Data Center Network Interface Network Interface Network Interface Network Interface Metro Packet-Optical Central Office Built like a Data Center Network Interface Network Interface Network Interface Network Interface Network Interface Network Interface Access Wired Access Wireles s Access Enterprise Access Wireles s Access Wired Access Wireles s Access Wired Access Wireles s Access Enterprise Access Wireles s Access
SDN Control Plane: Key Performance Requirements High Throughput: ~500K-1M paths setups / second ~3-6M network state ops / second High Volume: ~500GB-1TB of network state data ONOS Apps Global Network View / State Difficult challenge! high throughput low latency consistency high availability
ONOS: SDN Network OS for Service Providers Scalability, High Availability & Performance Northbound & Southbound Abstractions Modularity
ONOS: Distributed Network OS ONOS Instance 1 Apps NB Application Intent Framework ONOS Instance 2 Distributed Core ONOS Instance 3 (performance, scale-out, availability, state management, notifications) ONOS Instance N Southbound Core API Adapters Adapters Adapters Adapters Protocols Protocols Protocols Protocols
Application Intent Framework Flexible and intuitive northbound abstraction and interface for user or app to define what it needs without worrying about how. Application Intent Framework: APIs, Policy Enforcement, Conflict resolution Provision 10G path from Datacenter 1 to Datacenter2 optimized for cost Intents translated and compiled into specific instructions for network devices. Distributed Core Southbound Southbound Core API OpenFlow NETCONF Southbound Interface
SDN Use Cases for Service Providers Converged multi-layer packet/optical networks Central Office Re-architected as a Data center (CORD) Seamless SDN and peering with SDN- Segment routing with SDN control And many more Mobile backhaul ( RAN) Network Functions as a Service (NFaaS) multicast
20K-100K subscribers/co Control Central Office Re-architected as a Data center Key components Commodity hardware SDN Control Plane (ONOS) NFVI Orchestration (XOS, Openstack) Open Leaf Spine Fabric Simple on-prem CPE + vcpe Virtualized Access (PON OLT MAC + volt) Virtualized Functions Virtualized BNG Spine Switches Applications SDN Control Plane ONOS NFVI orchestration XOS Fabric Leaf Switches I O I O Simple Switch ONT Access Link I PON OLT MACs O volt vcpe vbng DHCP LDAP RADIUS Metro Core Link Subscriber Home Commodity hardware Central Office Re-architected as Datacenter Data
Mobile CORD Local Traffic. Mobile CORD DB HSS PGW-D Local video streaming service at mobile edge. Mobile CORD On-demand provisioning of vbbus at cell sites near big sport match On-demand provisioning of video caching application(vm) for local video caching service Functions like DNS and DPI also need to be deployed locally for traffic classification Other traffic of spectators is treated same as before; traverse from and to the Centralized Core SGW-D Centralized DC Local communications hosted by distributed EPC Virtualized EPC can also be deployed to host local and internal communications Communication between security staffs Remote monitoring of Security CAM
Analytic CORD GPON (Access) Apps Apps Apps ONOS + XOS LOG Customer Care Security Analytics Platform (XOS + Services) Diagnosis PON OLT MACs (Core) Key Building Blocks to Measure: Access, Fabric, s and VNFs
Seamless Peering: SDN- BGP speaker 1 BGP speaker 2 External Network ONOS Control plane SDN- 1 SDN- 2 ONOS 1 ONOS 2 SDN Network BGP routes ONOS intents OpenFlow entries External Network External Network External Network
Outline SDN for Service Providers Background Use cases Packet/Optical Use Case Problem statement and conceptual solution Implementation Demonstration State of the Industry & Future Work
Problem Statement Today packet and transport networks are separate. They are planned, designed and operated separately by different teams. This leads to significant inefficiencies. They are subject to under-utilized networks with significant pre-planning and highly over-provisioned for worst case. A lot of the path planning in these networks is off-line. Given these considerations, WAN links are typically provisioned to 30-40% average utilization. This allows the network service provider to mask virtually all link or router failures from clients. Such overprovisioning delivers admirable reliability at the very real costs of 2-3x bandwidth overprovisioning and high-end routing gear. S. Jain, et. al., B4: Experience with a Globally-Deployed Software Defined WAN, SIGCOMM 2013.
Multi-Layer Network without Converged Control Plane Logical Tunnels Full Mesh MPLS A B C D E Routers P1 P2 P3 P4 P5 s R1 R2 R4 R3 R5 R6 R7 100s of wavelengths per
Multi-Layer Network without Converged Control Plane A 50-100G C E 50-100G B D 100G P1 50-100G 50-100G P2 P3 P4 P5 R1 100G R2 100G R4 Peak rate provisioning R7 is necessary in optical transport 100G R3 R5 R6
Multi-Layer Network without Converged Control Plane A C E Multiple protection modes are applied (up to 4 times BW) Primary P1 Protected B P2 P3 P4 D P5 Light Paths R1 R2 Light Paths R4 R3 R5 R7 Static provisioning in transport networks R6
Conceptual Solution: Multi-Layer SDN Control BW Calendaring Control Apps Config Apps ONOS Mgmt Apps 1. Centralized Control of packet and optical 2. Multi-layer optimization based on availability, economics and policies Datacenter 1 Datacenter 2 Packet Network Optical circuit re-routed Optical Network
Benefits of Converged Control Plane Much faster bandwidth provisioning Drastically improve network utilization Perform dynamic restorations in response to packet and transport network failures Agile development and rapid deployment of new services
Implementation Code is king Less is more Vendor neutral Scalable Work focused on the three SDN layers Data plane Control plane Applications
Implementation Data plane Packet switches today Control of forwarding plane via OpenFlow Open and standardized Ideal scenario s have similar open and standard interface Reality Many s use legacy protocols, such as TL1 Vendor-specific, proprietary So we built an emulation platform (LINC-OE) Partnered with Infoblox
Emulation Basics Emulates optical layer topology from predefined table Includes characteristics of optical cross connect and Packet to Optical Link Interface (Add/Drop) Ports, links and switches are remotely reconfigurable by Mininet Supports OpenFlow 1.3+ Optical Add/Drop match actions Supports failure scenarios of links, ports, and Work in progress Emulates channel signal/power measurement Regenerator support Module Module O P M DROPs ADDs
Forwarding Model for s Match/action abstraction for s has three functions: add, drop, and forward Match is really about wavelength provisioning packets packets Match Add Packet port and traffic type Match Drop Optical port and lambda Action Transponder uses lambda and output to optical port Match Forwarding Optical port and lambda Action Transponder uses lambda and output to packet port Action Output to optical port (easy to extend when considering regenerators)
Forwarding Model for Packet and Layer Forwarding Constraints Match Action Match Action Egress Values Tuple on port3 Encaps, DEST xxx DEST xxx Capacity 10GbE LAG No Cost 10 Forwarding Constraints Match Action Match Action Egress Values, Dest xxx on port3 Pop Packet Tuple Forward to port12 Capacity 10GbE LAG No Cost 10 Lambda Forwarding Constraints Match Action Match Action Egress Values TPND port 5 PushOCH Lambda 66 Lambda 66 Forward to WDM port1 Capacity MUXPDR 100Gb Yes Cost 250 Loss 2.5 NF 4.75 PMD est 0.1 Disp est 40 Lambda Forwarding Constraints Match Action Egress Values l66 on port2 Forward WDM port6 Capacity 100Gb REGEN No Cost 250 Loss 2 NF 4.25 PMD est 0.05 Disp est 25 Lambda Forwarding Constraints Match Action Match Action Egress Values l66 on port3 Forward TPND port3 l66 on TPND port3 POP OCH and egress Capacity LAG 40Gb NA Cost 100
Transport Network Metering Model Metering Packets dropped Total packets Queue Port overflow status Queue Overflow count OAM Delay BFD Jitter Loss Metering Packets dropped Total packets Queue Port overflow status Queue Overflow count OAM Delay BFD Jitter Loss OTU- Frame/ OCH/TP ND Metering Errored seconds Severely errored seconds OTU- Frame/ OCH/TP ND OAM LOS LOF WDM LOL LBC Protection OTU- Frame/ OCH/TP ND Metering Errored seconds Severely errored seconds OTU- Frame/ OCH/TP ND OAM LOS LOF WDM LOL LBC Protection OTU- Frame/ OCH/TP ND Metering Errored seconds Severely errored seconds Red items implies access only if a regenerator is used OTU- Frame/ OCH/TP ND OAM LOS LOF WDM LOL LBC Protection
Implementation Control Plane Southbound protocol for s ONF Optical Transport Working Group OpenFlow 1.3+ experimenter messages Southbound abstractions simplify adding new protocols Converged topology Control both packet and optical layers Allows adding additional layers, e.g., OTN Discovery Automatic L3 topology discovery (LLDP) Static configuration of L0 topology L0 discovery work in progress Control Apps Config Apps SDN Network Operating System Mgmt Apps
Implementation Control Plane Path calculation takes place on the multi-layer graph Constraints and resource management Wavelength continuity, bandwidth, latency, Restoration Optical link failure causes disappearing packet links Control Apps Config Apps Mgmt Apps Packet layer restoration is tried first If unsuccessful, perform optical layer restoration Easily add multi-layer protection and restoration mechanisms SDN Network Operating System
Implementation Applications Multi-Layer GUI Bandwidth on Demand Bandwidth Calendaring Application Intent Framework: APIs, Policy Enforcement, Conflict resolution Distributed Core Southbound Southbound Core API OpenFlow NETCONF Southbound Interface
ONOS Multi-Layer Reference Platform LINC-OE First open source L0 emulator Based on OpenFlow 1.3+ Infoblox and ON.Lab ONOS HA, high performance, open source network OS ON.Lab and partners Open source platform Benefits Rapid prototyping, agile Adherence to common interfaces Scalability testing Common optical control plane for interoperability between vendors
Demo GUI https://wiki.onosproject.org/display/onos/packet+optical DO try this at home
Lessons Learned Feasibility Converged packet optical control plane is possible Offers scalability, HA, and performance Benefits Significant improvement in network utilization Drastic reduction in CAPEX and OPEX DevOps model for transport networks Deeper insights OpenFlow packet switches commercially available, resistance from L0 vendors Abstractions are critical: intent framework, multi-layer graph
Outline SDN for Service Providers Background Use cases Packet/Optical Use Case Problem statement and conceptual solution Implementation Demonstration State of the Industry & Future Work
Vertical Integration: Packet Switches Control Apps Config Apps Mgmt Apps Closed SDN Network Operating System Packet switches are undergoing this transformation right now! Agent OS Loader Merchant Silicon Whitebox Legacy
Vertical Integration: s fiber demux controller Control and config of WSS and transponders Signal Monitoring and Adjustment Metering and alarms mux Why is this de-aggregation not happening? Control Apps Config Apps Mgmt Apps SDN Network Operating System transponder WSS pass through add/drop Agent OS HAL Hardware Whitebox controller Legacy
What Makes Optical Devices Different? We need specialized mix of L0, L1, and L2 functions Physical impairments are too complex to monitor and manage externally Our analog transmission system is custom designed It s impossible to control all configuration and forwarding at scale You can t achieve sub-50ms failovers And so on None of this is fundamental! De-aggregation is inevitable
Open Optical Hardware Hardware Abstraction Layer Hides optical impairments, thermal instability, power balancing, etc. Can autonomously fix problems or perform maintenance OS Server-like environment for switches Manages various hardware sensors Boot loader, tools, switch management, etc. Agent Agent OS HAL Hardware Whitebox Open and standardized interface for forwarding, configuration, and observability Inviting all vendors to join us! controller Legacy
Disaggregated s API# API# API# W# MW# MW# Pluggable#Op?cs# G# Transponder# W# W# #(Wavelength#Switching,..)# 6" Martin Birk, Mehran Esfandiari, Kathy Tse, AT&T's direction towards a Whitebox, ONS 2015.
Architecture SDN Controller WOLU White box Optical Line Unit W White box Spine switch Spine switch Leaf & Spine Fabric L0 Device Controller L0 Device Controller L0 Device Controller Leaf switch Leaf switch Leaf switch W W WOLU WOLU WOLU WOLU WOLU WOLU WOLU W WOLU WOLU WOLU
Conceptual Solution: Disaggregated SDN Controlled Transponders and S Config Apps ONOS Control Apps Mgmt Apps Config Apps ONOS Control Apps Mgmt Apps 2 Leaf-Spine Fabric Leaf-Spine Fabric Channelized Traffic Transponder Transponder Channelized Traffic Metro λ Optical Network with disaggregated S λ Optical circuit re-routed
It's Happening Now! CALIENT s S-series Optical Circuit Switch (OCS) Up to 320 User Ports 640 Single Mode Fiber Termina ons 320x320, 160x160 op ons 10, 40, 100 Gbit/s per port and beyond 25ms typical setup me (<50ms Max) Less than 50ns latency Less than 3.5 db Inser on Loss Ultra low power (<45w), small size (7RU) TL1, SNMP, OpenFlow APIs Optimized for Datacenters and Software Defined Networks 5 CALIENT Technologies All Rights Reserved Company Confidential
Vendor-Specific Domains Second problem with Optical Transport Industry Transport networks suffer from vendor lock-in Domain consists of equipment from a single vendor Each domain requires vendor-specific NMS/EMS No data plane interoperability Profound impact on service providers Complex management & orchestration tools Problem identification & resolution Expensive Is this fundamental? NMS A Vendor A Service Provider Orchestration NMS B Vendor B
Why Vendor-Specific Domains? We monitor network state and performance in NMS We built intelligent alarm and event handling between boxes and NMS Our NMS is the only system that can control our transmission Failures are handled faster and more efficiently by our NMS And so on None of this is fundamental! Vendor-specific domains will disappear
Vendor-Neutral Domains Control Apps Config Apps Mgmt Apps SDN Network Operating System Common southbound abstractions X X X Vendor A Data plane interoperability is key X X Vendors can still innovate and diversify their hardware Vendor B
Proof of Concept Menlo Park, CA On-Demand Op cal Bandwidth Advanced Mul -Layer Restora on ONOS Mul -Layer Network Op miza on OF provider Ciena TL1 provider Fujitsu TL1 provider Huawei PCEP provider Layer Op cal layer O awa, Ontario Richardson, TX Plano, TX Domain A Domain B Domain C
Proof of Concept Open Networking Summit 2015 https://youtu.be/gsfywjyyfi4
Future Work Looking to work with vendors that offer OpenFlow/NETCONF support Something better than proprietary TL1 Experiments on data plane interoperability Drive adoption of DevOps model for transport networks Hardware deployments
Cap-Grow-Drain Strategy Cap-Grow-Drain = Bring SDN to backbone without fork lift upgrade Segment Routing Optical control Segment Routing Optical control Segment Routing (for MPLS network) (for MPLS network) (for MPLS network) ONOS ONOS ONOS Optical control Whitebox switches MPLS Network Cap Whitebox switches Whitebox switches MPLS Network Whitebox switches Whitebox switches MPLS Network Drain Whitebox switches New SDN Edge Optical Network Optical Network Grow Optical Network Route Big Flows to optical network Cap MPLS backbone don t grow the legacy MPLS backbone of proprietary routers Grow packet edge and optical core with SDN control plane and make the best use of packet-optical technologies Drain the MPLS backbone as most traffic transitions to new packet edge and optical core network
Summary Demonstrated converged packet/optical control plane for service providers Scalability, HA, performance Potential to dramatically decrease CAPEX & OPEX Innovative services using DevOps model Need the right abstractions Intent framework Multi-layer graph Datacenter 1 BW Calendaring Control Config Apps Apps ONOS Mgmt Apps Packet Network 1. Centralized Control of packet and optical 2. Multi-layer optimization based on availability, economics and policies Datacenter 2 Optical circuit re-routed Optical Network
Call to Action Open and standardize hardware interfaces Achieve control plane interoperability Eliminate vendor-specific domains Achieve L0 data plane interoperability Remove vendor-specific approaches (EMS & NMS) Control Apps Config Apps SDN Network Operating System Mgmt Apps Agent OS HAL controller X X X X X Hardware Whitebox Legacy If existing vendors don t take action, others will step in!
Software-defined Transformation of Service Provider Networks Join the journey @ onosproject.org