Open Network Linux A Path to an Open Source Network OS Rob Sherwood Big Switch Networks CTO
Outline: Open Network Linux (ONL) Context: What is ONL? Network OS Architecture View What does ONL provide? ONIE Compatible NOS Installer Supported Hardware Basic L3 Routing with Open Route Cache Platform Abstraction Layer ONLP Server-like Management Second stage more feature-full boot loader Plans for the Future How to Get Involved
ISN T NETWORKING ALREADY OPEN? Faster Throughput = More commercial value = More Proprietary All Hardware Specs Public X86 Servers ß Open Binary SDK for Packet Forwarding Closed OS, Open Control Plane APIs NPU, FPGA DC 1U ToRs, Spines Closed OS, Open APIs for Policy, Stats MulT- chassis Routers 10 Gb/s 100 Gb/s 1 Tb/s 10 Tb/s Aggregate Throughput à 2014 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM 3
ISN T NETWORKING ALREADY OPEN? Faster Throughput = More commercial value = More Proprietary All Hardware Specs Public X86 Servers ß Open Binary SDK for Packet Forwarding Closed OS, Open Control Plane APIs NPU, FPGA DC 1U ToRs, Spines DC 1U ToRs, Spines This talk Closed OS, Open APIs for Policy, Stats MulT- chassis Routers 10 Gb/s 100 Gb/s 1 Tb/s 10 Tb/s Aggregate Throughput à 2014 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM 4
LOTS OF OPEN SOURCE CONTROLLERS But what are they controlling? Open Source SDN Controllers??? This talk SDN OSS Software Switches SDN OSS Hardware Switches 2014 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM 5
ONL is a Linux Distribution for Bare Metal Switches A collection of software packages, utilities, drivers, and abstractions to run on OCP, bare metal, brite box hardware i.e., a NOS that ONIE would install Why not use an existing Linux distribution? Does build on existing distribution Debian Wheezy Need to create ONIE installers for many platforms Need to manage switch-specific hardware (e.g.,sfps) Switches are very similar to servers, but not quite
ONL Example Use-Cases 1. DIY packet forwarding platform, e.g., for students Platform driver code too complicated for most hobbyists 2. Reference hardware testing platform, e.g., OCP Certification Common core in open source for verification, reference 3. Building Block for Commercial Software Pull ONL in to a larger, commercially supported solution Not a Big Switch talk, but we use ONL in our products
Why Use ONL? Help ecosystem focus on innovation Many annoying software details to run an OCP switch Building platform drivers not high value asset; should be common Enables a reference NOS implementation Hardware without software is not useful Package up details and best practices into one place Help bootstrap the Open ecosystem and OCP adoption Allows commercial companies and DYI-folks to build OCP-based products faster
Outline: Open Network Linux (ONL) Context: What is ONL? Network OS Architecture View What does ONL provide? ONIE Compatible NOS Installer Supported Hardware Basic L3 Routing with Open Route Cache Platform Abstraction Layer ONLP Server-like Management Second stage more feature-full boot loader Plans for the Future How to Get Involved
Background: Generic NOS Architecture Applications Environment Agent(s) Forwarding Agent Platform Platform Abstraction Layer OS Distribution (Linux kernel, etc.) ASIC SDK Hardware CPU Misc Hardware (fans, LEDs, SFP, sensors) ASIC Per Switch Stack
Shipping now: ONL + Open Route Cache Applications ONLP APIs SensorD ONLP Dump S N M P Quagga/Xorp/Etc. Open Route Cache Orc-brcm Driver This API looks a lot like SAI ORC Driver API Platform ONL Platform APIs ONL Distribution (Linux + stuff) BRCM SDK Closed Source Hardware CPU (x86,ppc) Misc Hardware (fans, LEDs, SFP, sensors) BRCM Per Switch Stack
ONL Contributors/Supporters (So Far) Adds Forwarding Agents Open Network Linux Additional Drivers Provides Platform Drivers OTHERS
Kernel Linux Patches 3.9.6 Architecture Infrastructure Module (AIM) Linux 3.9.6 Linux 3.8.13 Specific component: `make C $ONL/components/ $arch/$component/ deb` Busybox Buildroot Etc. Open Route Cache FaultD Plaaorm- * (installers, etc.) ONLP (Plaaorm API Library) ONLP- $vendor ONL Loader Anatomy of `make onl- $arch` All components: `make C $ONL/builds/components` ONL- Specific.debs $ONL/debian/repo Generic.debs From Emdebian.org and Debian.org Via apt- cacher- ng SWI: `make C $ONL/builds/swi/$arch/all` Using MulTstrap and `onl- mkws` ONL Loader Switch Image (.swi) ONIE Installer SHAR wrapper Installer: `make C $ONL/builds/ installer/$arch/all` ONL ONIE Installer New from ONL 3 rd Party OSS Mixed
ONL Relative to ONIE An Installer Needs a NOS ~16MB ~3MB ~160 MB ONIE One Time Installer Normal Full- featured Boot Loader (w/busybox) Main Network OS Image (.swi) (w/real binaries) Github.com/ onie/onie Open Network Linux Common kernel and DTS files??
Open Network Linux Community Project has been public for ~1 year Code http://github.com/opennetworklinux/onl Website http://opennetlinux.org ~50K LOC (without Linux patches); 56K Makefile lines Community Traction Six platforms added by community; more coming Three companies have commercial projects w/onl ~14 code contributors from four companies
Outline: Open Network Linux (ONL) Context: What is ONL? Network OS Architecture View What does ONL provide? ONIE Compatible NOS Installer Supported Hardware Basic L3 Routing with Open Route Cache Platform Abstraction Layer ONLP Server-like Management Second stage more feature-full boot loader Plans for the Future How to Get Involved
ONL Provides an ONIE-Compatible Installer ONIE is great but needs an installer Installer packages entire NOS into a single image Installer is the glue between Platform and NOS Sets the $nos_bootcmd magic e.g., nos_bootcmd= diskboot 0x10000000 0:1 ; bootm 0x10000000 Formats local storage, e.g., flash or local disk Installs ONL loader onto local storage (see next)
Install Using ONIE then Boot ONL ~64MB uboot ENVs ONIE Free Space Boot Flash Boot Logic: 1. uboot POSTs 2. $nos_bootcmd is read from ENVs 3. run $nos_bootcmd If $nos_bootcmd returns, run ONIE On install, ONIE sets $nos_bootcmd to load ONL loader 4. Loader downloads specified SWI URL if not cached 5. Loader mounts rootfs as ramdisk with overlayfs 6. ONL loader kexec s SWI kernel ~2GB ONL Loader ONL config SWI s: ONL SWI #1 (cached) Mass Storage
ONL SUPPORTED HARDWARE X86 and PPC Support 2014 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM 19
Basic L3 Routing With Open Route Cache (ORC) 1. ORC creates a orcxx interface for each port 2. Quagga (or Xorp, etc.) installs routes into Linux kernel 3. Linux kernel sends RT Netlink Updates to ORC 4. ORC translates to SDK calls Hardware accelerated routing! Only unicast IPv4 now No IPv6, Multicast, ECMP support 5. Linux kernel is slow path to ASIC s fast path Open Route Cache Orc-brcm Driver BRCM SDK BRCM Quagga/Xorp/Etc.
ONL has a Platform Abstraction Layer BIOS does not provide inventory/drivers for all devices SFP has a life cycle, like USB; F2B vs. B2F power supplies Hot Pluggable devices: PSU, Fan trays, AC/DC power Not all platforms have BIOSs Need for per-platform memory maps of LEDs, Fans, Temp ONL Platform provides an Platform Abstraction Layer Not tied to Linux subsystem (but could be integrated) Goal is for trivial driver impls; leverage existing diag code OID based; single threaded, multi-application https://github.com/opennetworklinux/onlp/tree/master/modules/onlp/module/inc/onlp
ONL Provides Server-like Management Switches have flash, not hard drives Problem 1: Maximum flash cycle time limit disk writes Problem 2: Flash and ram more limited than typical servers Fix: Use overlayfs to overlay copy-on-write ram disk over flash ONL uses full-featured binaries For size, most switch OS s use stripped binaries, e.g., busybox Bigger binaries uses additional space, but ok with overlayfs Install/use proper Debian binaries using apt-get Useful for development or operations, e.g., gcc or Chef/Puppet
ONL Provides A Second-Stage Loader uboot (PPC) or Grub (x86) is the first stage loader ONIE is a 2 nd -stage loader; only runs at install-time ONL 2 nd Stage Loader Runs every time before NOS boots Built on Linux enables a full shell for rescue mode Load NOS via ssh/scp/http/tftp/nfs/ftp/local flash Net boot allows centralized NOS image management NFS book is very useful (future) Initialize ASIC/front panel ports inline boot
Outline: Open Network Linux (ONL) Context: What is ONL? Network OS Architecture View What does ONL provide? ONIE Compatible NOS Installer Supported Hardware Basic L3 Routing with Open Route Cache Platform Abstraction Layer ONLP Server-like Management Second stage more feature-full boot loader Plans for the Future How to Get Involved
Next Target: ONL + OpenFlow Agent OpenFlow Controller OpenFlow Protocol: v1.0-v1.3 Applications SensorD ONLP Dump S N M P Indigo OpenFlow Agent Indigo OF-DPA Driver Indigo API Platform ONL Platform APIs ONL Distribution (Linux + stuff) OF-DPA BRCM SDK Closed Source Hardware CPU (x86,ppc) Misc Hardware (fans, LEDs, SFP, sensors) BRCM Per Switch Stack
OpenFlow Datapath AbtracTon: OF- DPA Indigo driver OF- DPA SDK Chip open closed OF- DPA API 2014 BIG SWITCH NETWORKS, INC. WWW.BIGSWITCH.COM 26
How to Get Involved Join the mailing list and provide feedback Compile the code Try a new platform Feedback is fine, but patches are preferred J ONL is a Linux Distribution for bare metal switches Growing support for OCP and non-ocp switches Now supports ORC forwarding agent Indigo-based OpenFlow agent in progress Find out more at http://opennetlinux.org Documentation Videos Pre-compiled binaries