Netvisor: Bare Metal Control Plane, Application level Analytics and Intrusion Detection Sunay Tripathi Pluribus Networks, Inc. 2455 Faber Rd, Palo Alto. CA. 94303. +1-650-714-0697 sunay@pluribusnetworks.com Roger Chickering Pluribus Networks, Inc. 2455 Faber Rd, Palo Alto. CA. 94303. +1-650-618-5490 rogerc@pluribusnetworks.com Jonathan Gainsley Pluribus Networks, Inc. 2455 Faber Rd, Palo Alto. CA. 94303 +1-650-618-5490 jon@pluribusnetworks.com ABSTRACT In this paper, we describe the architecture of Netvisor, the new network hypervisor that runs on ethernet switches. Netvisor controls all hardware tables, TCAMS, BST, and the learning and switching behavior of the switch chip. By capitalizing on the PCI-Express control plane of the latest generation of commercial off-the-shelf switch chips, Netvisor can memory map the entire register space into software for high speed/low latency multithreaded access. The Intel Ivy Bridge control processors in the newer switch designs from white box vendors have enough power and bandwidth to run complex multi-gigabit control plane applications. This opens the door for a new breed of applications running directly on Netvisor-enabled switches including the SDN control plane, rich on-switch analytics, and intrusion detection. Categories and Subject Descriptors D.4.4 [Operating Systems]: Network Communications; C.2.4 [Computer Communications Networks]: Network operating systems General Terms Algorithms, Management, Performance, Design, Security. Keywords Netvisor, Network Hypervisor, Control Plane, Network Analytics, vflow, merchant silicon based switches, Network programming APIs 1. INTRODUCTION For two decades top of the rack switches did not see major changes. The speeds increased from 10Mbps to 40Gbps Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 2014 Pluribus Networks Inc. Sigcomm HotSDN 14, August, 2014, Chicago, Illinois, USA Copyright 2014 ACM 1-58113-000-0/00/0004 $5.00. and many new protocols were added but operational behavior remained the same. Top of the rack switches were generally meant for switching packets at high speed with some human level static configuration via a command line interface. Each switch port was connected to a small number (typically one) of MAC addresses in a statically configured rack. As server virtualization and device mobility become ubiquitous, switches operate in a dynamic environment with hundreds of transient MAC addresses per physical port. MAC addresses migrate freely between different ports on the same switch as well as from switch to switch. Policy and isolation need to be applied dynamically on a MAC, IP address, or VLAN/VXLAN basis as virtual machines and mobile devices migrate through the network. Software intervention is required to manage these transient MAC addresses that roam throughout the network. The last decade also saw the growth of new web scale applications where a single client request results in a complex operation in the data center. Typically many servers coordinate complex searches, database queries, and advertisement placement to respond to a simple web request. This has caused huge growth in east-west traffic in addition to the traditional north-south traffic requiring change in deployment behavior and topologies. The above changes created the need for network programmability and the Software Defined Networking (SDN) movement. In this paper, we first take a look at issues with existing architecture and then discuss a Network Hypervisor that was implemented over the current generation of switches to address the issues with existing architecture. In section 4 we discuss some interesting new applications that are enabled by making the switch more programmable. The examples used throughout the paper are from a live production network running on switches controlled by Netvisor. Finally, we summarize the paper and describe our future direction.
2. ISSUES WITH EXISTING ARCHITECTURES In this section, we explore the issues with the existing top of the rack and edge switching architecture. 2.1 Control Plane Traffic in a Virtualized Environment In a modern data center containing thousands of virtual machines, ARP traffic alone can exceed 50Mbps. Network programmability requires control plane traffic, network address tables, and flow tables to be managed in software, which in turn requires a powerful platform with multiple CPU cores and enough memory to store the tables [5][4][6][3]. 2.2 Low Performance Control Processors Switch chips are becoming more and more powerful [1][2] and programmable but many switch vendors still use low powered processors with limited memory. Although the switch chips support multiple lanes of PCI-Express providing abundant bandwidth between the switch chip and the CPU, most current switch designs lack the memory and processing capability to implement a fully programmable software control plane on the switch itself. 2.3 Control Plane Decoupled from the Bare Metal Switch To add more programmability to switches, various controllers use Openflow [8] to decouple the control plane from the forwarding plane. To work around the issues described in sections 2.1 and 2.2, the various controllers are not implemented on the switch device but instead run on a connected server with adequate CPU and memory resources. Various efforts have been made to make the controllers distributed [7] and responsive. In principle, separating the control plane from the switch is a good way to side step the architectural issues of underpowered switch control planes. However, in data center virtualized environments the amount of control traffic (ARP, multicast setup, routing traffic, flow creation, etc.) between the switch and the controller is fairly high [11][12] and the delay in decision-making introduced by having the controller separated via the network from the switch introduces scaling issues even at a rack level and causes new challenges in terms of deployment models. 2.4 Growth of East-West Traffic and Cloud Computing Streaming, social media, and search are becoming the dominant applications driving the data center. A simple request from the client results in a complex operation in the data center where multiple servers coordinate with one another to generate a response. A leaf-and-spine architecture supports the increased east-west traffic better than the traditional fat tree architecture. The move towards the cloud and horizontally scaled applications is also forcing a change in the switching layer in the rack. There is a larger need to orchestrate the server, storage, and switch as one unit. This is forcing the switches to be more programmable and server-like so racks and pods of racks become the building blocks for next generation data centers. 2.5 Lack of Built-in Monitoring and Debugging Support Server operating systems and applications have built-in debugging capabilities. In contrast, many network switches have limited tools for understanding and debugging switching behavior. Network operators depend on physical probes and third party tools to debug their networks. Programmability and debugging go hand in hand and the lack of tools to debug and understand the network hampers the growth of programmability. 2.6 Intrusion From Within Intrusion has always been a big problem since the wide spread deployment of web infrastructure and Alan J. et. al. [9] have summarized the issue well. With virtualization, multi-tenancy, and bring your own device to work, the enterprise and data center is now becoming vulnerable to attack from within the network. Securing the programmable network requires forensics and intrusion detection inside the network in addition to at the edge. 3. NETVISOR ARCHITECTURE The goal of Netvisor is to make a switch programmable like a server. Netvisor leverages the new breed of switches by memory mapping the switch chip into the kernel over PCI-Express and taking advantage of powerful control processors, large amounts of memory, and storage built into the switch chassis. In the next sections we describe the Netvisor architecture and hardware platforms it can run on. 3.1 Bare Metal Network OS Netvisor integrates a switch chip into a server operating system over PCI-Express. The switch register space is memory mapped into the kernel where the kernel manages the MAC, IP and flow tables. There is no hardware MAC learning on the switch chip and when there is a MAC table miss in hardware the packet is forwarded to the kernel. The kernel keeps a much larger than is traditionally supported MAC table in main memory and uses the hardware table as a cache that is updated on a miss. The access into the switch for table updates is multi-threaded and protected by fine grain locks providing high bandwidth, low latency access for control plane operations and flow-related traffic. Figure 1 contrasts Netvisor on a Server-Switch using the current generation of switch chips with a traditional switch where the OS runs on a low powered control processor and low speed busses.
Figure 1: Server-Switch Compared to Traditional Switch 3.2 High Performance Flow Programming The 2-4 lanes of PCI-Express Gen2 connection between the CPU and the switch chip enables up to 8-16Gbps of bandwidth between the CPU and the switch. Coupling the switch chip to a modern Intel Ivy Bridge CPU over PCI- Express enables high performance network applications that run directly on the switch by creating flows to capture traffic relevant to the application and injecting packets into the switch from the CPU. Netvisor s vflow feature enables creation of flows that alter the behavior of the switch for packets that match a flow. Packets may be selected using any combination of L1-L4 attributes. Actions to apply to a matching flow include: Drop Redirect to CPU, port, or IP address Mirror to port or IP address Set VLAN or tunnel Copy to CPU Log statistics Log packets Set bandwidth minimum/maximum A flow may be created using Netvisor s CLI, C API, or Java API. Netvisor s control plane makes use of flows to redirect control plane traffic such as STP to the CPU. Netvisor processes control plane traffic, injecting packets into the switch as appropriate. Similarly, an application developer can create flows to redirect or copy selected traffic to an application for processing using Netvisor s C and Java APIs. The application can respond to redirected traffic by injecting packets into the switch, or the application can analyze or log copied traffic. Netvisor s OpenFlow [8] integration is implemented using the same APIs available to developers. The Netvisor C API may also be used to develop applications that run outside of the switch, using SSL to communicate with the Netvisor control plane. A REST API will be available in an upcoming version of Netvisor. 3.3 Hardware Platforms and Performance The current breed of switch designs based on Intel and Broadcom switch chips [1][2] have server-like characteristics. Some of these designs have been submitted to Open Compute Project [10] and many original design manufacturers are building switches based on OCP specifications that may be modified to include powerful multi-core control processors and 8-16GB or more of main memory. Such an enhanced design has the ability to run Netvisor. Figure 1 shows one such design with dual socket, server class control processors, 64GB RAM, and PCI- Express flash based storage where Netvisor can be a platform for large scale analytics, orchestration, and security applications. Table 1: Control Plane Data Transfer Per Second ipkts ibytes idrops opkts obytes odrops 8.91K 1.15M 0 2.85K 373K 0 The Table 1 shows the control plane I/O rates for a period of 1 second in a lightly virtualized rack. It shows the number of packets and bytes received and transmitted along with the number of dropped packets (none in this sample). The control traffic consists of software based MAC table to provide ARP suppression, congestion analytics, and application level analytics. The virtual machines were lightly loaded so application level traffic was at the level of a few thousand TCP sessions. ARP traffic was both sent and received by Netvisor to implement ARP suppression, while selected TCP traffic was copied to Netvisor to implement application analytics. Hence the lower transmit rate compared to receive rate. In a moderately loaded rack, the peak rates exceed 100K packets per second. 4. ANALYTICS The Netvisor architecture provides several advantages in the area of analytics. High bandwidth between the CPU and the switch allows deep visibility into the network data. Added memory and disk capacity enables a long history of network events. Powerful CPUs allow rich analytics to be applied to captured data. These factors are leveraged by Netvisor to provide the administrator with the data and tools needed to analyze network activity. 4.1 Congestion Analytics Netvisor tracks various port and system statistics over time, including congestion data that indicates when and where packets are dropped. Command 1 shows the congestion on port 41 between March 10, 10:06am and March 10, 10:12am, when network degradation was reported.
Command 1: Congestion on port 41 CLI> port-stats-show start-time 3-10,10:06 end-time 3-10,10:12 port 41 port timestamp obytes ocongdrops ---- -------------- ------ ---------- 41 03-10,10:06:40 40.2G 0 41 03-10,10:07:40 44.6G 0 41 03-10,10:08:40 64.3G 1.28M 41 03-10,10:09:40 70.8G 4.85M 41 03-10,10:10:40 51.1G 0 41 03-10,10:11:40 52.8G 0 Here we ve restricted the output to just the egress bytes and egress congestion drops for port 41, although many other counters may be displayed from the historical data. 4.2 Application Level Analytics Application level network activity is also tracked. Each TCP connection is logged, from client mac/ip/port to server mac/ip/port, along with the latency and bandwidth used by the connection. Command 2 shows the connections during the period congestion was seen. Command 2: Connection history CLI> connection-show start-time 3-18,10:06 end-time 3-18,10:12 sort-desc total-bytes client-ip server-ip port bytes ----------- --------------- ----- ----- 10.9.10.124 23.3.98.32 http 138M 10.9.10.124 23.3.98.32 http 96M 10.9.10.124 23.3.98.25 http 16M 10.9.10.124 23.3.98.40 http 10M 10.9.10.124 23.3.98.18 http 4.51M 10.9.10.79 54.230.147.179 nfs 3.39M 10.9.10.214 173.194.133.105 https 2.62M 10.9.10.214 173.194.133.105 https 2.16M 10.9.10.124 23.3.98.18 http 2.16M 10.9.10.79 23.194.103.3 http 1.28M 10.9.10.119 74.125.239.114 http 1.22M 10.9.10.119 74.125.239.114 http 1.18M 10.9.10.119 74.125.239.114 http 1.16M 10.9.10.119 74.125.239.114 http 1.12M 10.9.10.214 173.194.133.212 https 1.09M 10.9.10.214 173.194.133.105 https 1.08M 10.9.10.119 74.125.239.114 http 1.08M The data shows primarily HTTP connections during the timeframe. It is sometimes difficult to discover the issue from looking at single connections, so Netvisor provides tools to perform analysis and sorting on the data. The sum-by argument allows the user to specify which endpoints to aggregate data for. The sort-desc sorts the output in descending order based on the specified field. Command 3 shows the use of these options to determine which server is fielding the most traffic. Command 3: Data reduction on connection history CLI> connection-show start-time 3-18,10:06 end-time 3-18,10:12 sum-by server-ip,port sort-desc total-bytes count server-ip port bytes ----- --------------- ---- ------ 44032 23.3.98.32 http 18.1G 32944 23.3.98.25 http 16.5G 22948 23.3.98.40 http 10.4G 17975 23.3.98.18 http 6.71G 8091 173.194.133.105 https 1.50G 5290 74.125.239.114 http 555M 3959 54.230.147.179 http 339M 2834 216.137.37.33 http 192M 2083 74.125.239.113 https 162M 2164 23.194.103.3 https 152M 1982 173.194.133.212 https 133M 1705 74.125.239.111 https 130M 2091 199.175.56.184 http 120M 1882 174.35.40.38 http 119M The count column indicates the number of connections that were summed for the given server and port. The bytes transferred per connection were also summed to denote the total bytes transferred by the server during the specified timeframe. We could also query by client IP to find the client generating the most data or highest number of connections. The ability to specify which fields to sum by and which fields to sort on allow great flexibility in mining the data. Netvisor also compiles client-server relationships over time. In particular outstanding TCP SYN and completed TCP FIN counts are tracked to find misbehaving clients or network problems. Command 4 shows an example of the client-server relationships sorted by finished connections. Command 4: Client-server relationships CLI (network-admin@pn-dev02) > clientserver-stats-show sort-desc fin client-ip server-ip port syn fin ----------- ------------------ ----- ----- 10.9.19.79 10.9.18.27 8084 1.59K 21.9K 10.9.19.79 10.9.18.27 8443 1.61K 21.8K 10.9.9.186 10.9.20.10 ssh 1 13.7K 10.9.10.14 23.79.195.16 http 0 12.4K 10.9.10.65 173.252.13.7 https 0 6.49K 10.9.10.214 121.254.13.1 http 17 4.41K 10.9.9.186 10.20.9.137 ssh 8 4.25K 10.9.9.186 10.20.9.167 ssh 0 4.25K 10.9.9.186 10.20.9.157 ssh 1 4.25K 10.9.9.186 10.20.9.148 ssh 9 4.25K 10.98.1.18 10.9.10.110 http 10 3.50K 10.9.9.186 173.164.16.4 ssh 54 3.39K 10.9.9.73 10.9.9.9 nfs 0 3.02K
10.98.1.26 10.9.9.186 8080 0 2.65K 10.9.9.33 10.9.9.73 nfs 71 2.30K Real-time analysis is also possible, and merely involves changing the time specification. Command 5 shows application level activity in the last five minutes. Command 5: Connections in the last five minutes CLI> connection-show within-last 5m client-ip server-ip port bytes age ----------- -------------- ----- ----- --- 10.9.9.33 74.125.239.118 https 382 1s 10.9.10.4 184.51.150.56 https 0 2s 10.9.10.4 184.51.150.56 https 0 2s 10.9.10.192 74.125.239.100 https 1.36K 2s 10.9.10.4 184.51.150.56 https 0 2s 10.9.10.19 10.9.9.9 nfs 74 2s 10.9.10.192 74.125.239.100 https 5.41K 3s 10.9.10.79 10.9.10.110 http 10.4K 13s 10.9.10.192 74.125.239.98 https 7.88K 13s 10.9.10.192 74.125.239.98 https 9.59K 13s 10.9.9.186 10.9.9.73 nfs 74 16s 4.3 Packet Capture Netvisor s CLI includes a built-in application, vflow-snoop, which displays packets that match a flow or set of flows. Vflow-snoop is useful for troubleshooting. For example, to observe icmp traffic flowing through the switch the user runs the command: vflow-snoop scope local proto icmp action copy-to-cpu vflow-snoop prints metadata about each packet received including the ingress port and a timestamp, along with Ethernet and IP headers. Port 48 Size 82 Table 2. vflow-snoop output Time 22:02:15.82859320 Source Mac Destination Mac VLAN 1 Ethernet Type 00:00:24:d0:41:35 66:0e:94:21:f8:03 IP Source IP 10.9.9.1 Destination IP 10.9.9.210 Protocol ICMP Vflow may also be used to record packet capture to disk for later analysis. Packet captures are saved in a pcapcompatible format and may be accessed via nfs or sftp. 5. FUTURE WORK: INTRUSION DETECTION On-switch analytics and packet capture lay the groundwork for some future work in Netvisor: intrusion detection built into a switch. The persistent analytics connection history may be used to establish a baseline for what constitutes normal network activity[13]. To establish a baseline the connection-show and connection-stats-show commands are used to determine which hosts have communicated with each other and how much traffic has been handled by each host during a specified time interval. For a client host, the command: connection-show client-ip 10.9.9.86 starttime 03-16-2014 duration 1d displays all of the connections that have originated from the client with IP address 10.9.9.86 in the 24 hours starting at midnight on March 16, 2014. Similarly, the command: connection-show server-ip 10.9.9.86 starttime 03-16-2014 duration 1d displays all of the server connections to 10.9.9.86 in the same time period. The command: connection-show start-time 03-16-2014 duration 1d displays all TCP connections through the switch during the time interval and may be used to establish a comprehensive baseline for the entire network. Finer grained baselines based on traffic during shorter intervals throughout the day or week may be established by running a series of connection-show commands with starttime and duration parameters to pick out the activity during the intervals of interest. The connection-stat-show command displays the number of TCP connections and the amount of data handled by each host on the network. Connection-stat-show has start-time and duration arguments that may be used to gather data to establish baselines based on volume of traffic. Netvisor s C and Java APIs may be used by an application instead of the Netvisor CLI to gather baseline statistics. Once a baseline has been established, ongoing analytics may be monitored to detect deviations from the baseline by periodically running connection-show and connection-statshow or their C or Java API equivalents. As noted in Section 4.2 the connection-stats-show command can be used to detect imbalance between TCP SYN and FIN packets for detecting certain DDoS attacks. With sufficient memory and storage, the switch can run intrusion detection software such as Snort[14], Suricata [15] and Bro[16]. Vflows are established to capture packets
for inspection, which are fed into the intrusion detection software. When deviations from baseline activity or intrusions are detected, a variety of actions may be taken depending on the configuration established by the administrator. Actions include: Write a message to syslog Send an email or text message to an administrator Create a new vflow to block suspicious traffic Create a new vflow to reduce the bandwidth available to suspicious traffic 6. CONCLUSION The Netvisor architecture presented in this paper introduces a novel approach to make switches more programmable and realize the vision of Software Defined Network. By offering a fully multithreaded, low latency and high bandwidth OS on the bare metal switch, a new class of applications are enabled. The switch can be orchestrated just like a server while physical and virtual applications are deployed on the switch itself. The core of the Netvisor architecture has been implemented over four years and entered production recently in different types of environments. Some inbuilt applications like Cluster-Fabric to treat multiple switches as one logical switch, ARP suppression to scale virtualized environments, and application level debugging and analytics are making users realize the power of software defined networks. Collaborative work is ongoing with researchers, developers, and partners to migrate existing applications and develop new applications on Netvisor in booth proofof-concept and production networks. 7. REFERENCES [1] Ozdag Recep. Intel Ethernet Switch FM6000. DOI=http://www.intel.com/content/dam/www/public/u s/en/documents/white-papers/ethernet-switch-fm6000- sdn-paper.pdf. [2] Broadcom. Strata XGS Trident II Ethernet Switch Series. DOI=https://www.broadcom.com/products/Switching/ Data-Center/BCM56850-Series [3] G. Liao, D. Guo, L. Bhuyan, S. King. 2008. Software techniques to improve virtualized I/O performance on multi-core systems. 4th ACM/IEEE Symposium on Architectures for Networking and Communications Systems. ACM, 161-170. [4] S. Tripathi, N. Droux, T. Srinivasan, K. Belgaied. Crossbow: From H/W virtualized NICs to virtualized networks. Proceedings of the 1st ACM workshop on Virtualized infrastructure systems and architectures. VISA 2009, 53-62. [5] R. N. Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S. Radhakrishnan, V. Subramanya, and A. Vahdat. 2009. Portland: A scalable fault-tolerant layer 2 data center network fabric. ACM Sigcomm 2009 conference on Data Communications, 2009. [6] A. Greenberg, J. Hamilton, D.A.Maltz, and P.Patel. 2009. The cost of the cloud: research problems in datacenter networks. SIGCOMM Computer Communication Review. ACM, 68-73. [7] T.Koponen, M.Casado, N.Gude, J.Stribling, L.Poutievski, M.Zhu, R.Ramanathan, Y.Iwata, H.Inoue, T.Hama, and S.Shenker. 2010. Onix: A distributed control platform for large scale production networks. In USENIX OSDI, 2010. [8] N.McKeown, T.Anderson, H.Balakrishnan, G.Parulkar, L.Peterson, J.Rexford, S.Shenker, and J.Turner. 2008. Openflow: enabling innovation in campus networks. ACM Sigcomm CCR. 2008. [9] Allen J., Christie A., Fithen W., McHugh J., Pickel J., Stoner E. 1999. State of the Practice of Intrusion Detection Technologies. Technical Report CMU/SEI- 99-TR028. Carnegie Mellon University. [10] Open Computer Project. DOI=http://www.opencompute.org/wiki/Networking/S pecsanddesigns [11] A. Myers, E. Ng, and H. Zhang. 2004. Rethinking the service model: scaling Ethernet to a million nodes. HotNets, November 2004 [12] Kim, Changhoon, Matthew Caesar, and Jennifer Rexford. 2008. Floodless in seattle: a scalable ethernet architecture for large enterprises. ACM SIGCOMM Computer Communication Review. Vol. 38. No. 4. ACM, 2008. [13] Yu Gu, Andrew McCallum, Don Towsley. Detecting Anomalies in Network Traffic Using Maximum Entropy Estimation. In Proceedings of USENIX Internet Measurement Conference 2005, pages 345-350, Berkeley. USENIX Association. [14] Martin Roesch. Snort - Lightweight Intrusion Detection for Networks. In Proceedings of LISA '99: 13th Systems Administration Conference, pages 229-238, Seattle, 1999. USENIX Association. [15] Suricata Open Source IDS/IPS/NSM Engine. DOI=http://suricata-ids.org. [16] Vern Paxson. Bro: A System for Detecting Network Intruders in Real-Time. In Proceedings of the 7th USENIX Security Symposium, San Antonio, 1998.