JUNIPER NETWORKS FIREFLY HOST FIREWALL PERFORMANCE

White Paper JUNIPER NETWORKS FIREFLY HOST FIREWALL PERFORMANCE Copyright 2013, Juniper Networks, Inc. 1

Table of Contents Executive Summary...3 Performance Challenges in a Virtual Environment...3 Firefly Host Architecture (Fast, Distributed Connection Table, and Rule Base)... 4 High-Level Architecture... 4 Firefly Host Fast Implementation vs. Other Virtualization Options... 4 Firefly Host Connection Tables... 6 Firefly Host Rule Base Design...7 Basic Open Source Performance Validation...10 Testing Overview...10 Results...11 Conclusion...11 About Juniper Networks...12 List of Figures Figure 1: VMware hybrid model (i.e., firewall processing in Security VM)... 5 Figure 2: Firefly Host and Fast (i.e., firewall processing in hypervisor kernel)... 5 Figure 3: Shared connection table in traditional or products... 6 Figure 4: Firefly Host separate connection tables and firewall engines...7 Figure 5: Monolithic rule set... 8 Figure 6: Firefly Host separate rule bases for each protected VM... 9 Figure 7: Baseline, Firefly Host, and competition in TCP throughput test...11 2 Copyright 2013, Juniper Networks, Inc.

Executive Summary Virtualization adoption is on the rise for a variety of compelling reasons, number one of which is cost. Virtualization optimizes physical server hardware use, reduces power consumption, and lowers hardware maintenance expenditures for tremendous cost savings. For any CIO, it s a no-brainer to want to avoid a situation where hundreds or even thousands of servers are operating at a mere fraction of their computational capabilities. The second compelling reason is management elasticity. IT administrators who work on a daily basis with data center technologies (servers, switches, and so on) see the more subtle advantages of virtualization adoption. Namely, they understand the benefits of being able to easily manage and expand their systems infrastructure. For example, with virtualization technology, they can use live migration to move virtual machines (VMs) from one physical server to another in order to perform hardware maintenance without system downtime. They can also quickly spin up new VMs for departments via templates and clones rather than waiting weeks for new hardware to arrive. In essence, this flexibility means business can move at a more rapid pace. So if the cost, flexibility, and business drivers are clear, what might threaten the adoption of virtualization? Two elements come to mind performance and security. Nobody wants to virtualize servers if it means application performance will be drastically (or perhaps even mildly) diminished. To be in a situation where many VMs are sharing hardware is OK, provided all of the important applications get their necessary share of that hardware. Next, just because companies make a decision to virtualize servers doesn t mean that they want to eliminate or complicate the security of those servers. The imperative is to ensure that performance and security are maintained during virtualization adoption, and the ideal is to improve both through that process. Enter Juniper Networks Firefly Host*, a comprehensive virtualization security solution for virtualized data centers and clouds. Firefly Host was designed and built specifically for the virtual environment, and it enables organizations to implement security on their systems in more flexible and automated ways all while avoiding any costly reduction in performance. This paper will explain those fundamental principles and outline tests that validate the claims. Performance Challenges in a Virtual Environment Many of the security industry s first firewalls began by running as an application on Unix, Linux, or even Windows servers. This, however, required steps to harden server operating systems and then deploy the firewall application. Eventually, the industry shifted to an appliance model in which the firewall and hardened operating system were combined and pre-installed onto specialized hardware. Not only did this shift make deployment easier, it also allowed for the inclusion of ASICs and other performance-enhancing elements. Along with the shift in platform, the firewall itself started to morph into a combination of inspection and protection technologies, including things like antivirus, intrusion detection service (IDS), application identification, URL filtering, data leak protection, to name a few. Performance testing of these dedicated security devices became more difficult given the different types of features, but companies could still fairly easily examine basic firewall performance statistics (throughput, connection setup rate, etc.) by transmitting load on one interface of the device and reading the receive rate on the other end. These tests are easily repeatable and will vary little in setup and outcome. Performance testing of security in the virtual environment becomes much more complicated. Now we are back to a model in which the security application is running on generic servers instead of dedicated hardware appliances. This means that vast differences in potential performance numbers (one customer runs high-end Dell systems and another runs entry-level IBM with a totally different class of capabilities) can exist/occur. We also introduce the hypervisor itself, which will dynamically allocate resources in somewhat unpredictable ways. Also, companies can load very different amounts and types of VMs on these platforms (some may run 10 VMs per host with lightly used virtual network interfaces while others may push 50 or more, all with network-intensive applications). Combine all of these factors with the fact that there are drastically different approaches to securing the environment (e.g., implement an agent in the guest VMs, implement the security in a VM, implement the security partially in the kernel or completely in the kernel) and you have a very confusing set of parameters to sort through in order to understand performance implications in each environment. To help understand the issues, we will next examine the performance architecture implemented in the Firefly Host. *Formerly vgw Virtual Gateway Copyright 2013, Juniper Networks, Inc. 3

Firefly Host Architecture (Fast, Distributed Connection Table, and Rule Base) High-Level Architecture Early adopters of virtualization technology had two very basic choices to secure their new VMs: First, they could implement a guest agent firewall in each and every VM. Sometimes there will be an agent install available for the operating system, and sometimes reliance on iptables or similar technology is needed. The management and overhead of all these agents can be a real issue, even if an organization decides to live with present security concerns such as mixing the protection layer directly into the device that s in need of protection. From a security perspective, it is better to avoid full package agents to decrease the likelihood of compromise from the VM itself, as malware, rootkits, viruses, etc., often hide from or damage the security agent. Secondly, customers could use VLANs to funnel the traffic out of the virtual environment through traditional security devices and back into the virtual environment. However, this is inefficient and often lacks the granular control and dynamism needed to implement, for example, security on moving VMs, or VMs getting created on the fly via cloning. To improve upon these basic choices, security vendors have taken one of two approaches. In the first approach, they have opted to simply package their application as a virtual machine and do nothing custom. This model was easy for vendors and shifted the burden to the customer to figure out how to route guest VM traffic through this security application VM in order to protect the guests. This approach quickly fell by the wayside, as not only did it fail to reduce the management effort and security attachment to VMs, it actually created performance chokepoints in the architecture. The next approach was to build something specifically for this environment by utilizing hypervisor APIs to attach, retrieve, and analyze network packets. The Firefly Host is an example of this second approach. In fact, because it was built from scratch for the virtual environment, no trade-offs were necessary in its architecture (an existing product wasn t being force fit into the hypervisor). There are numerous benefits to be gained by creating a solution specifically for the virtual environment, but for the purposes of this paper, we will focus on the key Firefly Host design elements related to performance: VMware Fast APIs were used to create an entire enforcement and packet processing firewall in the kernel of the hypervisor. connection tables were allocated in isolation among virtual firewall instances. Patent-pending firewall rule base design allows distribution of workload to virtual firewall instances and reduces the number of rules processed for VMs. Firefly Host Fast Implementation vs. Other Virtualization Options Using hypervisor APIs to analyze packets is not as straightforward as it might appear. There are two fundamental methods use these APIs (known in VMware as VMsafe Networking APIs) to process all packets in the kernel, or use these APIs to shift packets to a VM in order to do all or even part of the processing. The initial VMware documentation of these approaches and APIs were literally called Fast and, and both of them are displayed in the following diagrams. 4 Copyright 2013, Juniper Networks, Inc.

Security VM Engine VM1 VM2 Rules vswitch Hypervisor Server 1 Figure 1: VMware hybrid model (i.e., firewall processing in Security VM) Security VM VM1 VM2 Policy Mgmt. and Logs Engine VM1 Rules VM2 Rules vswitch Hypervisor Server 1 Figure 2: Firefly Host and Fast (i.e., firewall processing in hypervisor kernel) Copyright 2013, Juniper Networks, Inc. 5

Using the Fast approach provides the best possible performance because packets don t have to be folded into a Security VM for processing. You don t incur a performance penalty in the transfer of the packets into the Security VM nor do you risk the VM becoming CPU bound and, thus, a bottleneck. However, this approach means that you must build a firewall engine completely in the hypervisor, which is a difficult engineering task. What s more, many companies have preexisting security solutions that they don t want (or can t) redesign to work in the hypervisor. Thus, most companies have avoided complete Fast integrations. The result, however, is a substantial performance penalty, as well as the introduction of failure points in your architecture for traffic processing, because all of the connections are funneling through a VM. It is easy to determine which implementation has been done with a deployed product by simply pausing the Security VM. If the Security VM is processing VM communication flows (VM-to-VM or VM-to-physical), then those flows will stop. If the Security VM is just a conduit between the management system and the firewall engine in the kernel (as is the case with Firefly Host) then the communication flows for VMs will not be impacted with the loss of the Security VM. Thus, there is a crucial distinction customers must make between Security VMs provided by various vendors is it processing firewall flows ( ) or just serving as a conduit to the kernel engine for policy deployment (Fast )? Firefly Host Connection Tables Not only was it important for Juniper to build the Firefly Host into the hypervisor using Fast mode to avoid forwarding packets up to a VM for processing, it was important so that the firewall connection table could be distributed. In a standard firewall product (either physical firewall appliance or traditional software firewall), there is a single connection table that holds information about all the communication between systems (sessions). This connection table is, in part, used as a mechanism to accelerate the processing of sessions. Instead of doing intensive analysis of every single packet, you can do it at the start of a session and then put an entry for that session into your table. Subsequent packets (belonging to that already validated session) are then bypassed by checking the connection table. The problem with this single table is it can get very large. In most cases, the larger it gets, the more expensive it becomes to access it for firewall processing. In addition, traditional products share the connection table, which means a handful of systems with high session rates can impact the firewall performance for all systems being protected. Security VM VM1 VM2 Connection Table vswitch Hypervisor Server 1 Server 2 Server 3 Figure 3: Shared connection table in traditional or products 6 Copyright 2013, Juniper Networks, Inc.

With Firefly Host, each time a VM is protected, there is a unique firewall engine and separate connection table instantiated in the hypervisor for that VM. Juniper loads different firewall engines on the VMs so that their processing is isolated from other VMs, and so that these packaged entities (security policy and connection table) can be tied to the VM and moved with VMware VMotion. In effect, although there is a single Firefly Host kernel module loaded on the host, it is logically used to load separate firewall instances for each protected VM. Each protected VM is currently allocated ~2 MB of memory via the Firefly Host kernel module protecting it. The allocated memory will grow if (or when) the protected VM session rate grows. The maximum memory allocation is currently 75 MB. With this model, each protected VM is getting a connection table big enough for 256,000 connections. This is an enormous amount of room for a single VM (many physical firewalls have smaller tables for all systems they are protecting). This large and separate connection table model means that instead of having one VM fill the connection table of a firewall and impact all the other VMs, it will impact only its own processing. Other protected VMs will be unaffected by that connection table growth. Security VM VM1 VM2 Policy Mgmt. and Logs Connection Table Connection Table vswitch Hypervisor Server 1 Server 2 Server 3 Figure 4: Firefly Host separate connection tables and firewall engines From a performance perspective, connection table size is clearly not a limiting factor for Firefly Host. The size is 256,000 multiplied by the number of protected VMs on a host and is isolated from misuse by rogue or session intensive VMs. The isolation of the connection table is key to high-performance firewall processing. Firefly Host Rule Base Design For performance reasons, Juniper built separate firewall engines and connection tables for each protected VM in the hypervisor. This was achieved via the Firefly Host kernel module. In addition, there was a goal to create an extremely flexible rule base structure (security rule bases are being layered onto complicated network communication flows in the middle of a data center) and a goal to separate the relevant rules for each VM. This was necessary because, similar to the connection table lookup process, there is a computational price to traverse the rule base to find rules that should be applied to a given VM. Rules are typically processed in a top down fashion so, if you have a large number of rules in a single firewall, it will take progressively longer for the firewall engine to find and apply the correct rule to a connection. However, if you create different firewall engines, you can apply unique rule bases to each engine and avoid having to expend energy trying to optimize a single monolithic rule base (e.g., pushing frequently used rules to the top and/or limiting the number of rules that are added). With separate firewall rule sets, one or more VMs with extensively long or complicated rules won t impact the processing of a different VM, which may have a very simple rule set. Copyright 2013, Juniper Networks, Inc. 7

Separate Connection Tables and Engines + Rule Base Security VM VM1 VM2 Monolithic Rule Base Rule 1 VM1 Related Connection Table Rule 2 VM1 Related Rule 3 VM1 Related Rule 4 VM1 Related Rule... VMX Related Rule 120 VM2 Related Rule 123 VM1 Related Hypervisor vswitch Server 1 Server 2 Server 3 Figure 5: Monolithic rule set The monolithic rule base is in contrast to Firefly Host with which different rule bases (global, group, individual VM) can be created, combined, and applied to specific VMs from the management interface. 8 Copyright 2013, Juniper Networks, Inc.

Separate Connection Tables and Engines + Rule Base Security VM VM1 VM2 Rule Base for VM1 Rule 1 VM1 Related Policy Mgmt. and Logs Rule 2 VM1 Related Rule 3 VM1 Related Rule 4 VM1 Related Rule 5 VM1 Related Engine + Rule Base Engine + Rule Base Rule Base for VM2 Rule 1 VM2 Related vswitch Hypervisor Server 1 Server 2 Server 3 Figure 6: Firefly Host separate rule bases for each protected VM In the Firefly Host rule base design, the logical rule set for each VM (and the connection table) can easily be moved when a VM VMotions. There is no limitation imposed on VMotion because no synchronization issues exist. This is different than other solutions that rely on traditional firewall state synchronization mechanisms, where all new or changed states are synchronized to all firewall cluster members. These work fine when the cluster is relatively small (e.g., less than 10 members). However, this mechanism is far too slow when there are dozens or even hundreds of cluster members. This cluster size is exactly what starts to happen in large VMware deployments, where customers have extremely large ESX/ESXi host deployments and each will have firewalls that need to be synchronized. Firefly Host compartmentalizes the entire rule base and connection table for each VM, and can simply move that information with the VM when the VM moves. In addition, the Firefly Host per-vm rule base and connection table design provides much greater flexibility when a new policy is installed. Unlike a regular firewall and installation action, it only impacts the VM(s) you are changing. Aside from the performance and VMotion-related benefits, the separate rule base is also important from a security perspective. The separate rule base can be applied to specific VMs. The process for determining a specific VM is done by using the unique Virtual Machine Identifer (VMID). Once Firefly Host learns the VMID via an integration with VMware vcenter, it can apply the exact policy intended for that VM and only that VM. This helps avoid situations where a generic monolithic rule base has rules catching on VMs that it shouldn t (e.g., because of cloning or IP spoofing). Combining VMID with per-vm rule base allows Firefly Host to easily support environments with overlapping IPs. The next portion of this white paper describes performance tests and related results from the architecture described above. The intent of the testing was to stress the architecture in an easily repeatable manner. In addition, it was important to shape the tests to address concerns that companies typically have in production deployments. Copyright 2013, Juniper Networks, Inc. 9

Basic Open Source Performance Validation The VMware vsphere 4.1 platform introduces many new performance advantages (asynchronous transmits, large receive offloads, etc.). VMware has conducted performance testing to demonstrate that its virtualization platform is capable of scaling to meet the needs of network intensive application VMs and highly compressed ratios of network active VMs. 1 To demonstrate the platform s capabilities, the VMware report used netperf, which is an open source client/server tool. 2 Similar to netperf, there is also an open source client/server tool called nuttcp. 3 Generally, nuttcp is more up-todate and active than netperf. Either one of these tools can be effective in baselining a setup for basic network operations. Juniper conducted three simple tests with nuttcp. These were in line with VMware s report with the exception of adding the security (VMware didn t test vshield in its report). Using an open source tool allows companies to reproduce the basic performance tests in their labs without requiring high-end testing tools. Testing Overview The following technologies were used to conduct the testing: ESXi Server Load Generation System VMs Under Load Make HP ProLiance DL180 G6 HP ProLiant DL180 G6 VMware vsphere 4.1 VM CPU 2x Intel Xeon E5540 2.53 Ghz, 4 cores/socket (16 cores total with hyper threading) 2x Intel Xeon E5540 2.53 Ghz, 4 cores/socket (16 cores total with hyperthreading) One vcpu RAM 48 GB 48 GB 3 GB NICs 2X Intel 520 2 Port 10 Gbps 2x Intel 520 2-Port 10 Gbps VMXNET3 OS vsphere ESXi 4.1 348481 64 Bit Ubuntu 11.04 64 Bit Ubuntu 10.04 Load Generation Not Applicable Nuttcp 7.1.3 Server Nuttcp 7.1.3 Server Firefly Host Series Used Firefly Host Series 4.5 FCS Important notes about the testing configuration: The ESXi host was configured with four virtual switches one was a management network, while the other three were mapped to three subnets (10.10.10, 11, 12/24) and cross-connected directly to three interfaces on the load generation system. A separate physical load generation system (or systems) is recommended to make sure that this component doesn t become a bottleneck, and also to ensure a proper time source for network transfer calculations (as opposed to VMbased calculations, which can drift and skew results). It is important that the VMs under load are using VMXNET3. Otherwise, performance numbers can suffer greatly. 4 No special performance modifications to vsphere or the Firefly Host were performed (the default installation was used in both cases). A set of scripts was developed to start nuttcp on varied numbers of VMs, then to parse and calculate the results. Those scripts are available upon request from your Juniper representative. The scripts simply start nuttcp on a TCP port and push standard 1,500 byte packets at maximum rate over this connection using basic nuttcp commands. 1 VMware vsphere4.1 Networking Performance VMware Inc. 2011. http://www.vmware.com/resources/techresources/10181. 2 www.netperf.org/netperf/ 3 www.wcisd.hpc.mil/nuttcp/ 4 Performance Evaluation of vmxnet3 Virtual Network Device VMware Inc. 2009. www.vmware.com/resourcecs/techresources/10065. 10 Copyright 2013, Juniper Networks, Inc.

Results Three different types of tests were performed to show the impact of securing a host with Fast or architectures: 1. The baseline was without security. 2. The Firefly Host solution test was performed after loading the Firefly Host kernel module and protecting the relevant VMs with an open security policy. This means that the packet traversed the entire firewall engine and was then accepted. logging was turned on during the testing, and relevant log entries were analyzed as correct during each protected run. 3. The competitor 1 solution implemented a VMware architecture and the competitor 2 solution implemented a hybrid slow/fast path VMware architecture, and, as with Firefly Host testing, both an open policy and logging were enabled. Virtualization Performance Comparison 40.00 35.00 Unsercured 30.00 Firefly Host Competitor 1 25.00 Competitor 2 Gbps 20.00 15.00 10.00 5.00 0.00 8 VMs 12 VMs 16 VMs 20 VMs 24 VMs 28 VMs 32 VMs Number of VMs Figure 7: Baseline, Firefly Host, and competition in TCP throughput test Further analysis of this data shows that the Firefly Host, on average, had only a 6% impact on the baseline during extreme levels of system stress (lower percentages of approximately 2-3% are seen at more normal data rates). Though all security processing comes at some performance cost, these tests show that the Firefly Host has minimal negative impact. This is especially evident when compared to competitive solutions that haven t completely implemented their firewall enforcement engine inside the hypervisor kernel and, thus, severely impact the network traffic flow of VMs. Conclusion Virtualization is here to stay because it offers such compelling benefits in terms of cost savings, management flexibility, and business agility to name just a few. And today as organizations drive toward the adoption of further virtualization, there is no longer any need for them to let performance, or security concerns, hinder that advance. The Juniper Networks Firefly Host, which was purpose-built for the virtual environment, provides organizations with the necessary protections without performance trade-offs. This is thanks to its use of VMware Fast APIs that create an entire enforcement and packet processing firewall in the hypervisor kernel, separate connection tables for enhanced firewall processing, and patent-pending firewall rule base design that allows distribution of workload to virtual firewall instances and reduces the number of rules processed for VMs. With Firefly Host, organizations can augment their investment in virtualization and boost their potential for achieving maximum cost savings and optimizing management elasticity. Copyright 2013, Juniper Networks, Inc. 11

About Juniper Networks Juniper Networks is in the business of network innovation. From devices to data centers, from consumers to cloud providers, Juniper Networks delivers the software, silicon and systems that transform the experience and economics of networking. The company serves customers and partners worldwide. Additional information can be found at www.juniper.net. Corporate and Sales Headquarters Juniper Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA Phone: 888.JUNIPER (888.586.4737) or +1.408.745.2000 Fax: +1.408.745.2100 www.juniper.net APAC and EMEA Headquarters Juniper Networks International B.V. Boeing Avenue 240 1119 PZ Schiphol-Rijk Amsterdam, The Netherlands Phone: +31.0.207.125.700 Fax: +31.0.207.125.701 To purchase Juniper Networks solutions, please contact your Juniper Networks representative at +1-866-298-6428 or authorized reseller. Copyright 2013 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos and QFabric are registered trademarks of Juniper Networks, Inc. in the United States and other countries. All other trademarks, service marks, registered marks, or registered service marks are the property of their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice. 2000437-003-EN Nov 2013 Printed on recycled paper 12 Copyright 2013, Juniper Networks, Inc.