ENOS: a Network Opera/ng System for ESnet Testbed Eric Pouyoul (lomax@es.net) Technology Exchange Cleveland, Ohio, September 2015
Is ESnet really developing Yet Another Network Opera:ng System (YANOS)? Well, yes, sorry, but we had to! Focus on: Security Resource sharing (mul/- applica/ons) design panerns ODL, ONOS, other were not available at the /me. ESnet traffic is not typical. Not a controller (does not implement one) Will leverage any WAN, produc/on quality controller Co- design with hardware and network 2
ENOS Architecture REST API, Portal Layer 2 services MulEPoint L2 VPN Layer 3 SDX Netshell security, core services, API, SSHD Network Services Drivers Topology, NSI, OSCARS, Stats, Tests perfsonar, SNMP, Sampling Controller Drivers ODL, ONOS, 3 Database Immediately or eventually consistent VM Driver Linux containers (libvirt)
Netshell Design ENOS Services ENOS Modules Netshell Bundles, Python Karaf OSGi Java Virtual Machine 4
ENOS and SoKware Technologies Java Virtual Machine (JVM) for a produc/on- ready environment ENOS security leverages Java Security Manager Performance Python for rapid so^ware development Useful for experiments, quick prototyping The natural language of ENOS ENOS integrates Jython, a Python 2.7 interpreter running in the JVM. OSGi, Karaf for module, applica/on management. OpenDaylight for controlling SDN devices 5
>>> from net.es.netshell.api import TopologyProvider >>> graph = topo.getgraph(topologyprovider.weighttype.trafficengineering) >>> lbl = topo.getnode('lbl-mr2@es.net') >>> amst= topo.getnode('amst-cr5@es.net') >>> from org.jgrapht.alg import DijkstraShortestPath >>> path = DijkstraShortestPath.findPathBetween(graph, lbl, amst) >>> for link in path:... node = topo.getnodebylink(link.getid()... print "Node= " + node.getid() + "\tlinkid= " + link.getid() Node= urn:ogf:network:es.net:sunn-cr5 linkid= urn:ogf:network:es.net:sunn-cr5:to_lbl-mr2_ip-a:0 Node= urn:ogf:network:es.net:sacr-cr5 linkid= urn:ogf:network:es.net:sacr-cr5:to_sunn-cr5_ip-a:0 Node= urn:ogf:network:es.net:denv-cr5 linkid= urn:ogf:network:es.net:denv-cr5:to_sacr-cr5_ip-a:0 Node= urn:ogf:network:es.net:kans-cr5 linkid= urn:ogf:network:es.net:kans-cr5:to_denv-cr5_ip-a:0 Node= urn:ogf:network:es.net:chic-cr5 linkid= urn:ogf:network:es.net:chic-cr5:to_kans-cr5_ip-a:0 Node= urn:ogf:network:es.net:wash-cr5 linkid= urn:ogf:network:es.net:wash-cr5:to_chic-cr5_ip-a:0 Node= urn:ogf:network:es.net:aofa-cr5 linkid= urn:ogf:network:es.net:aofa-cr5:to_wash-cr5_ip-a:0 Node= urn:ogf:network:es.net:lond-cr5 linkid= urn:ogf:network:es.net:lond-cr5:to_aofa-cr5_ip-a:0 Node= urn:ogf:network:es.net:amst-cr5 linkid= urn:ogf:network:es.net:amst-cr5:to_lond-cr5_ip-a:0 6
Drivers Implements / uses Netshell Generic API: Topology, Monitoring, Provides an interface to services that are not implemented in ENOS, such as the OpenFlow controller Some core func/onali/es (DB, messaging) are provided by third part so^ware requiring a driver. Applica/on Generic API Third Party So^ware or Service Driver Netshell security, core services, API 7
ENOS Mul:point VPN Service Func:ons ENOS DHCP Virtual Machine Management Layer 2 Broadcast MAC Learning Virtualiza/on / Isola/on Layer 2 Provisioning Path Computa/on Open Daylight Driver OSCARS Driver ENOS Topology Service 8
ESnet PE Router (2+)x10GE ESnet SDN Testbed (n)x10ge Testbed Host Planned SDN Testbed node locations Planned SDN Testbed connectivity overlay (using OSCARS circuits) PNWG PNNL BOIS ANL FNAL STAR BOST AMST SACR DENV DENV KANS STAR CHIC AOFA BNL AMST LBL AOFA NEWY LOND LBL LLNL WASH CERN NERSC SUNN ALBQ NASH ORNL WASH CERN LANL SLAC ATLA SAND ELPA HOUS ATLA
ESnet SDN Testbed Hardware: Corsa Technology DP6440 4 x 100G CFP2 Ports 24x 10G SFP+ Ports DP6430 2 x 100G CFP2 Ports 24x 10G SFP+ Ports 100G 10G DP6420 48 x 10G SFP+ Ports 4 x 40G QSFP+ Ports Open vswitch + DP6410 24 x 10G SFP+ Ports OpenFlow 1.3 / 1.4 / 1.5+ FPGA- based SDN switch, with OpenFlow controller interface Highly scalable both in number of flows and speed Doing common ac/ons (defined by pipeline) very fast No broadcast, subset of OpenFlow matches and ac/ons.
ESnet SDN Testbed Node Physical View ESnet SDN Testbed Node Logical View ESnet PE Router ESnet PE Router (2+)x10GE (2+)x10GE (n)x10ge (n)x10ge Testbed Host Services VM
ESnet SDN Testbed Node Logical View Services VM So^ware switch and Services VM paired with every hardware switch. Most flows only pass through hardware switch (green) Flows requiring special handling go through so^ware switch (blue), possibly to Services VM (red). Hardware switch provides performance, stability So^ware switch and service VM provide flexibility, without compromising reliability
Design PaUerns at work: Traffic Mirroring. ANL OVS Flow Private OSCARS layer 2 circuit LBL Flow STAR SDN POP Private OSCARS layer 2 circuit CERN Private OSCARS layer2 circuit LBL SDN POP Shared OSCARS layer 2 circuits CERN SDN POP VM OVS OVS 13
ENOS Roadmap Year 1 (2013-14) Ini/ally only a proof of concept Focus Future ScienceDMZ Year 2 (2014-15) LBL funding (LDRD) Working prototype Mul/ple point VPN for high performance flows Year 3 (2015-2016) Deploy semi- produc/on mul/point VPN with traffic engineering. Support research projects: security, intent based networking, Open Source by Spring 2016 14
Lessons Learned and Summary Java + Python + SSHD is very powerful and yet simple combina/on. Aggrega/ng, grooming and normalizing data such as topology simplifies greatly network aware applica/ons. OSGI/Karaf allows us to painlessly change ODL version, ONOS, other OSGi based controller/applica/ons. Security is not trivial, especially when using third party so^ware: we needed to turn off security in order to use ODL. Will be open sourced very soon. A network operaeng system does really look like a computer operaeng system. Perhaps Linux itself could be the execueon environment for network policies. 15
Thank You / Ques:ons? Eric Pouyoul: lomax@es.net / epouyoul@lbl.gov 16