Here Now - an Open Source Project Near You

Here Now - an Open Source Project Near You The Linaro LNG Open Data Plane Initiative Mike Christofferson, Enea In conjunction with Ola Liljedahl, Arm

Enea - Powering Distributed, Connected Systems Increasing data traffic in communication devices require new and innovative software solutions to handle bandwidth, performance, and power requirements, as well as scalable systems management and availability solutions A robust product portfolio Enea operating systems software is heavily used in wireless Infrastructure (Macro, small cell), gateway, etc. Enea Solutions run in more than 50% of the world s 8.2M radio base stations. Enea provides a commercial Linux distribution, built by Yocto, with focus on realtime Proven, mature middleware solutions for over 10 years High Availability, Systems Management, and real-time database Global presence, global development, and headquartered in Stockholm, Sweden FOUNDED 1968 NO. OF EMPLOYEES 426 REVENUE ~70 M USD TEN OFFICES IN NORTH AMERICA, EUROPE AND ASIA

What Is Happening Super-linear growth in: Number of users Number of connected devices Number of over the top (OTT) applications and protocols Number of (standard) protocols (RFC s) Bandwidth usage Power consumption of network infrastructure 160 140 120 100 80 60 40 20 0 1995 2000 2005 2010 2015 2020 2025 2030 Logos and trademarks are used for illustration only and remain the property of their respective owners.

Increasing Diversity and Functionality Increasing and varying QoS requirements Realtime (e.g. VoIP, video conferencing, gaming) Streaming (e.g. music, video) Messaging (e.g. IM, Ajax, M2M/IoT) Bulk (web data, file sharing, OTA updates) More functionality and services implemented in the network Web caching Content delivery (CDN) Intrusion detection and prevention (IDS/IPS) User-specific service level agreements (SLA)

Consequences Need flexibility of software and programmable hardware Trend towards software-realised networking - function defined by software Need familiar programming environments with robust tools For TimeToMarket-driven development of new protocols and services Need portability Move functionality and applications between hardware platforms optimised for different power/performance/cost points Need high abstraction To enable innovation in efficient implementations E.g. OpenGL/OpenMAX Need efficient support for virtualization Decouple functionality (SW) from capacity (HW) Dynamic partitioning of common HW for different functions Simple, robust and incremental deployment of new services

Solution HW/OS Develop and deploy networking applications on general purpose processors/architectures Increasingly ARM and x86 Users and partners are drawn to the big ecosystems around these architectures Networking applications running in Linux user space Develop, debug and deploy using standard Linux tools Robust user space access to networking HW resources Linux enhanced to provide bare metal-like environment Bare Metal Linux Avoid TLB misses, interrupts, context switches, system calls, thread migration Direct HW access from user space Applications run isolated in user space on dedicated cores, unaffected by the Linux kernel and other applications Optional real-time support As needed by some wireless subsystems (<10μs interrupt response time)

Open Data Plane Overview What Is Open Data Plane? ODP is an open source cross-platform framework for data plane applications Common API for application portability Multiple implementations tuned to different platforms for performance Result: Easy app portability and performance Application Environment Applications run in Linux user space with essentially zero system overhead

Open Data Plane: The Time has Come Networking silicon vendors have evolved data plane SDKs for years No cross-industry group has sanctioned any common interface on diverse silicon The Linaro Networking Working Group - a consortium of 12 networking stakeholders surveyed the open source landscape Consensus: No ideal one-size fits all, implementation for diverse hardware/software approaches A truly open source & open contribution & cross-platform data plane interface, driven by a cross section of stakeholders, is needed Based on the OpenGL model: A software API at a higher level of abstraction, that could offer flexibility of implementations underneath that suit diverse needs. The Linaro non-profit open source software engineering organization is launching just such a collaboration So Linaro created OpenDataPlane(ODP).org with charter contributors

Open Data Plane API Standardized data plane API to enable Linux-based networking applications across any architecture Open support for ARM, Intel, MIPS & PowerPC! Structured to enable future innovation Lightweight abstraction preserves performance without prescribing lower level processing structure Access and management of HW accelerators Supports optional schedulers to provision easy management and traffic load balancing Proprietary SDKs sit underneath for OEM/operator software platform simplification (e.g. Supports DPDK on x86, USDPAA on QorIQ, etc) Application and services portability across a choice of hardware platforms Enabling an efficient, truly cross-platform standardized data plane processing model 9

EM SoCA BML ODP Foundational Principles Event Machine Work-driven many core data plane processing SoC Abstraction Portable API s for access to HW/SoC resources Bare Metal Linux (a.k.a. Bear Metal Linux) Minimal overhead and deterministic execution in Linux user space Application

ODP Foundational Principles (2) A data plane/networking API and runtime Loosely based on the NSN Event Machine Event/work-driven and polled programming models Portable API s for accelerators and offloads Runs in user space under Bare Metal Linux for best performance and determinism Common API, optimised implementations Separately owned and maintained API (e.g. OpenGL) Generic portable reference implementation from Linaro HW-optimised (possibly proprietary) implementations from networking SoC vendors Linaro maintains ODP for x86/dpdk

API and Concepts ODP loosely based on Event Machine, originally developed by NSN Generic framework for scalable multi-core programming, not limited to packet processing Event based abstraction and programming model for handling IO Supports packet flows, physical and virtual network interfaces, accelerators, SW endpoints, etc. Events represent different types of data: packets, timers, baseband data, HW notifications, SW messages... Supports scheduler based programming model (both HW & SW) Scheduling of IO events using different algorithms and knowledge of work in progress Implicit synchronisation and mutual exclusion between threads Supports different IO load balancing approaches Chose best configuration for traffic profile, latency/throughput requirements, and HW characteristics without changing the application Proven, already half a dozen HW-specific EM implementations

EM Basic Concepts Queue groups Cores/threads associated with queue groups Event handlers associated with queues Queues with events IP-fwd NAT Queues can be dynamically created and added to and removed from queue groups Scheduler GTP-U DPI RoHC Idle core Cores can be dynamically added to and removed from queue groups

Work Scheduling Logical flows/queues thousands to millions Pull work, on demand scheduling Clusters/Cores/Threads Processing packet from flow A Flows/ QoS classes WRR SP Work Scheduler Processing packet from flow B Scheduling algorithms WRR - Weighted Round Robin SP - Strict Priority Idle (power-gated) because of low load Actual scheduling algorithms implementation dependent Scheduler can enforce ordering/mutual exclusion Parallel, parallel/ordered and atomic queues Application doesn t need software mutex for protecting per-flow state Logical flows/queues mapped to hardware queues (if available)

Dynamic vs. Static Load Balancing Power Average latency/us Networking SoC s have hardware suitable for dynamic load balancing Queues associated with producer Queue and buffer mgmt in hardware Server NICs designed for termination Static load sharing (based on hashing) Queues associated with consumer Queue and buffer management in software/shared memory Static increases average and worst case latency and buffer space OK for ~8 cores but not many-core ready (some networking SoC s already have 30+ cores/hw-threads) Static makes core elasticity very difficult (per-core state with application level seamless/lossless handovers) Limits opportunity for power scaling 140 120 100 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0 80 60 40 20 0 Static loadsharing Dynamic loadbalancing 1 2 4 8 16 32 64 128 Number of cores Static Load Sharing with typical DVFS Dynamic Load Balancing Power gating + DVFS 0% 20% 40% 60% 80% 100% % Load

Issues Elasticity and Multi-core Load Balancing Traffic load and pattern varies over time Industry trend is to use more cores and more power-efficient cores To hit the sweet spot PPA (Power/Performance/Area) Enabled by inherent parallelism in networking Ideally use as many (or few) cores as traffic load and SLA s require and use them efficiently ODP Supports hardware scheduler and dynamic load balancing Cores can be added and removed No fixed allocation of cores for specific application Enables power or clock gating of idle cores Cores can share load dynamically Increased throughput, Decreased packet latencies, Increased core utilization

Scalable and Elastic Timer Support Issues Many protocols need timers, often several timers per flow/connection Millions of flows in core network means millions of timers Timers, like packets, are associated with flows/connections Need mutual exclusion of flow context when processing a timer event ODP ODP schedules timers together with packets Timers and packets can be synchronized and load balanced together

Power and Performance Management Issues Traffic load varies Daily variation and intermittent bursts Use as many or as few cores needed to meet bandwidth and QoS requirements Add and remove worker threads/cores Adjust clock frequency of active cores Power or clock gate inactive cores ODP Supports power/performance management Provides API for observing queue lengths Idle worker threads may yield to OS for background tasks or power down core Application can monitor traffic load and quickly react to increasing load

vswitch Integration Issues Efficient and robust integration with software or hardware-accelerated vswitch No loss of performance for virtualised networking applications using the dataplane API ODP ODP s queue-based I/O hides actual device implementation A queue may represent an actual network interface, a vswitch port, a pipeline of further processing stages (e.g. for encryption or encapsulation) etc. Allows for HW to copy packets between application and vswitch No shared memory between application and vswitch

Openness and Cross-platform ODP provides: Support for multiple architectures and platforms (e.g., ARM, x86, and MIPS) Open source and an open collaboration Not controlled by any single company Anyone may join in Reference implementations are open source Based on the Event Machine which currently is implemented on a number of different HW targets (using ARM/MIPS/PPC/x86 processors) Proven cross-platform support

Status Core API Definitions API Component Description Status BUFFER Shared memory, buffer pools, buffer types and access functions Preliminary done, but still work in progress CLASSIFICATION Ingress packet classification Preliminary work underway, CRYPTO Algorithmic and protocol offload for crypto, hashing, RNG Proposal being implemented IPC Inter-process communication control plane/data plane TBD PACKET I/O Network interface abstraction Done QUEUE Buffer queue management Done TIMER Protocol timers, periodic ticks Done SCHEDULER Ingress scheduling and distribution to threads/cores Done Version 0.2. of the API spec available now Version 1.0 by year end 2014

Status Implementations Platform Description Status linux-generic linux-dpdk Generic, portable reference implementation, uses Linux facilities (e.g. NetMap, crypto) Implementation for x86 using DPDK as the acceleration layer. Implements BUFFER/CRYPTO/PACKET- IO/QUEUE/TIMER/SCHEDULE R Just started linux-keystone2 HW-accelerated implementation for TI Keystone2 Tracking linux-generic linux-qoriq HW-accelerated implementation for FSL DPAA In progress Other implementations outside LNG also in progress...

Demos at Linaro Connect USA in Sep 14 Contributions and Interest from Major TEMs Cisco will demonstrate real app running on multiple HWimplementations of ODP Usage of API s Usage of HW acceleration through ODP API s (e.g. ordered and atomic scheduling, crypto) Portability NSN has had early influence on general architecture and APIs Huawei is promoting ODP in public presentations and expressing their support in meetings Ericsson s new influence in this project The ODP API Specification can be influenced by anyone in the open community

What s next? Adding more members to the ODP team Several companies in discussions, e.g. Aricent, Juniper & others downloading and commenting on ODP Developing NFV PoCs with ARM ecosystem partners, building on ODP Hardening and optimizing the performance of ODP implementations Developing liaisons with OpenDayLight, Open Networking Foundation, NFV Working Group Additionally, a new open source initiative to integrate open NFV building blocks with ODP Evangelizing ODP in the broad community

Thanks for Attending For more. Visit us in booth #201 Go to opendataplane.org linaro.org/projects/networking