NETWORK FIREWALL PRODUCT ANALYSIS

Transcription

1 NETWORK FIREWALL PRODUCT ANALYSIS Cyberoam CR2500iNG CyberoamOS v Ryan Liles, Joseph Pearce, Bhaarath Venkateswaran Overview NSS Labs performed an independent test of the Cyberoam CR2500iNG CyberoamOS v10.04 firewall. Although Cyberoam CR2500iNG is currently marketed and sold as a unified threat management (UTM) system, this product was configured as a firewall and subjected to thorough testing at the NSS Labs facility in Austin, Texas, based on the network firewall methodology v4.0 available on This test was conducted free of charge and NSS Labs did not receive any compensation in return for Cyberoam s participation. While the companion Network Firewall Comparative Analysis Reports (CAR) on security, performance, management, and total cost of ownership (TCO) will provide comparative information about all tested products, this individual Product Analysis Report (PAR) provides detailed information that is not available elsewhere. Product Cyberoam CR2500iNG CyberoamOS v10.04 Throughput 8,733 Mbps Stability & Reliability Firewall Enforcement Security Effectiveness 100% 100% 100% Following testing, the Cyberoam CR2500iNG was rated by NSS at 8.7 Gbps, which is significantly less than the 28Gbps (TCP) performance claimed by Cyberoam. NSS Labs rated throughput is based on the average of the following test results: Real World Protocol Mix (Perimeter), Real World Protocol Mix (Core), and 21 KB HTTP response based capacity tests.

2 Table of Contents Overview... 1 Security Effectiveness... 3 Firewall Policy Enforcement... 3 Performance... 5 UDP Throughput... 5 Latency UDP... 6 Maximum Capacity... 6 HTTP Connections per Second and Capacity... 7 Application Average Response Time HTTP... 8 HTTP Connections per Second and Capacity (With Delays)... 8 Real- World Traffic Mixes... 9 Stability & Reliability Management & Configuration Total Cost of Ownership (TCO) Installation (in Hours) Purchase Price and Total Cost of Ownership Value: Total Cost of Ownership Per Protected Mbps Detailed Product Scorecard Test Methodology Contact Information Table of Figures Figure 1: Raw Packet Processing Performance (UDP Traffic)... 5 Figure 2: Concurrency and Connection Rates... 7 Figure 3: HTTP Connections per Second and Capacity... 8 Figure 4: HTTP Connections per Second and Capacity (With Delays)... 9 Figure 5: Real- World Traffic Mixes NSS Labs, Inc. All rights reserved. 2

3 Security Effectiveness This section verifies that the device under test (DUT) is capable of enforcing a specified security policy effectively. NSS Labs firewall testing is conducted by incrementally building upon a baseline configuration (simple routing with no policy restrictions and no content inspection) to a complex real world multiple zone configuration supporting many addressing modes, policies, applications, and inspection engines. With each new security policy, test traffic is passed across the DUT to ensure that only specified traffic is allowed and the rest is denied, and that appropriate log entries are recorded. The DUT must support stateful firewalling either by managing state tables to prevent traffic leakage or as a stateful proxy. The ability to manage firewall policy across multiple interfaces/zones is a required function. At a minimum, the DUT must provide a trusted internal interface, an untrusted external/internet interface, and one or more DMZ interfaces. In addition, a dedicated management interface is preferred. Firewall Policy Enforcement Policies are rules that are configured on a firewall to permit or deny access from one network resource to another based on identifying criteria such as: source, destination, and service. A term typically used to define the demarcation point of a network where policy is applied is a demilitarized zone (DMZ). Policies are typically written to permit or deny network traffic from one or more of the following zones: Untrusted This is typically an external network and is considered to be an unknown and non- secure. An example of an untrusted network would be the Internet. DMZ This is a network that is being isolated by the firewall restricting network traffic to and from hosts contained within the isolated network. Trusted This is typically an internal network; a network that is considered secure and protected. The NSS Labs firewall tests verify performance and the ability to enforce policy between the following: Trusted to Untrusted Untrusted to DMZ Trusted to DMZ Note: Firewalls must provide at a minimum one DMZ interface in order to provide a DMZ or transition point between untrusted and trusted networks NSS Labs, Inc. All rights reserved. 3

4 Test Procedure Baseline Policy Result Simple Policy Complex Policy Static NAT Dynamic / Hide NAT SYN Flood Protection Address Spoofing Protection TCP Split Handshake Spoof Testing determined that Cyberoam CR2500iNG running CyberoamOS v10.04 correctly enforced complex outbound and inbound policies consisting of many rules, objects and applications. NSS engineers verified that the device successfully inspected traffic and took the appropriate action based upon the policy, raising the appropriate alerts and creating associated log entries NSS Labs, Inc. All rights reserved. 4

5 Performance There is frequently a trade- off between security effectiveness and performance. Because of this trade- off, it is important to judge a product s security effectiveness within the context of its performance (and vice versa). This ensures that new security protections do not adversely impact performance and security shortcuts are not taken to maintain or improve performance. UDP Throughput This test uses UDP packets of varying sizes generated by BreakingPoint Systems traffic generation tool. A constant stream of the appropriate packet size with variable source and destination IP addresses transmitting from a fixed source port to a fixed destination port is transmitted bi- directionally through each port pair of the DUT. Each packet contains dummy data, and is targeted at a valid port on a valid IP address on the target subnet. The percentage load and frames per second (fps) figures across each in- line port pair are verified by network monitoring tools before each test begins. Multiple tests are run and averages taken where necessary. This traffic does not attempt to simulate any form of real- world network condition. No TCP sessions are created during this test, and there is very little for the state engine to do. The aim of this test is purely to determine the raw packet processing capability of each in- line port pair of the DUT, and its effectiveness at forwarding packets quickly in order to provide the highest level of network performance and lowest latency. Megabits per Second Byte Packets 845 5, Byte Packets 9, Byte Packets 11, Byte Packets 16, Byte Packets 19, Byte Packets Mbps ,650 9,300 11,230 16,200 19,300 Latency (µs) Latency Figure 1: Raw Packet Processing Performance (UDP Traffic) 2013 NSS Labs, Inc. All rights reserved. 5

6 Cyberoam CR2500iNG s raw packet processing performance with UDP traffic was significantly impacted with smaller packet sizes, specifically 256, 128 and 64 byte packets. A 70% average drop in maximum UDP throughput when handling smaller sized packets should be considered atypical in a device aimed at enterprise deployments. Latency UDP Firewalls that introduce high levels of latency lead to unacceptable response times for users, especially where multiple security devices are placed in the data path. These results show the latency (in microseconds) as recorded during the UDP throughput tests at 90% of maximum load. Note that in the table below, the latency across the board is very high for the Cyberoam CR2500iNG, and is significantly higher for 128 and 64 byte sized packets. These latency numbers are considered an atypical result on a true legacy firewall, and is undesirable on firewalls that are deployed in enterprise environments. This device would be unsuitable for deployment in the network core in any size network. Latency - UDP Microseconds 64 Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets 270 Maximum Capacity The use of advanced, hardware- based testing tools and traffic generators allows NSS engineers to create true real world traffic at multi- Gigabit speeds as a background load for the tests. The aim of these tests is to stress the inspection engine and determine how it handles high volumes of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data, and these tests provide an excellent representation of a live network at various connection/transaction rates. Note that in all tests, the following critical breaking points - where the final measurements are taken - are used: Excessive concurrent TCP connections - unacceptable increase in open connections on the server- side 2013 NSS Labs, Inc. All rights reserved. 6

7 Excessive response time for HTTP transactions - excessive delays and increased response time to client Unsuccessful HTTP transactions - normally there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency causing connections to time out 3,500,000 3,000,000 3,100, ,000 2,999, , ,000 Concurrent Connections 2,500,000 2,000,000 1,500,000 1,000, , , , , , , ,000 Connections per Second 500,000 50,000 0 without data TCP Connections/Sec 215,000 HTTP Connections/Sec 179,000 HTTP Transactions/Sec 360,000 with data 0 Concurrent TCP Conns 3,100,000 2,999,000 Figure 2: Concurrency and Connection Rates As highlighted in Figure 2, Cyberoam CR2500iNG s on- board memory management and connection tracking mechanisms are highly efficient to the point where they successfully track and maintain 2.9 and 3.1 million concurrent stateful connections with and without data. HTTP Connections per Second and Capacity The aim of these tests is to stress the HTTP detection engine and determine how the DUT copes with network loads of varying average packet size and varying connections per second. By creating genuine session- based traffic with varying session lengths, the DUT is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packet- based background traffic. This provides a test environment that is as close to real world as it is possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability. Each transaction consists of a single HTTP GET request and there are no transaction delays (i.e. the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data. This test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads. Note that in Figure 3, the throughput for the CR2500iNG decreases substantially with response object sizes of less than 4.5KB NSS Labs, Inc. All rights reserved. 7

8 12, ,000 Megabits per Second 10,000 8,000 6,000 4,000 2, KB Response 21 KB Response 10 KB Response 4.5 KB Response 1.7 KB Response 140, , ,000 80,000 60,000 40,000 20,000 0 Connections per Second CPS 25,000 50,000 84, , ,000 Mbps 10,000 10,000 8,491 5,650 3,675 Figure 3: HTTP Connections per Second and Capacity Application Average Response Time HTTP Application Average Response Time - HTTP (at 90% Max Load) Milliseconds 2,500 Connections Per Second 44Kbyte Response ,000 Connections Per Second 21Kbyte Response ,000 Connections Per Second 10Kbyte Response ,000 Connections Per Second 4.5Kbyte Response ,000 Connections Per Second 1.7Kbyte Response 0.24 HTTP Connections per Second and Capacity (With Delays) Typical user behavior introduces delays between requests and reponses, e.g. think time, as users read web pages and decide which links to click next. This group of tests is identical to the previous group except that these include a 5 second delay in the server response for each transaction. This has the effect of maintaining a high number of 2013 NSS Labs, Inc. All rights reserved. 8

9 open connections throughout the test, thus forcing the firewall to utilize additional resources to track those connections. The Cyberoam CR2500iNG performance dropped by 50% on both 21KB and 10KB response testing when delay was introduced, which would indicate a lower performance level in real- world deployments. 12,000 90,000 Megabits per Second 10,000 8,000 6,000 4,000 2, KB Response 21 KB Response w/ Delay 10 KB Response 10 KB Response w/ Delay 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 Connections per Second CPS 50,000 28,000 84,908 37,000 Mbps 10,000 5,600 8,491 3,700 Figure 4: HTTP Connections per Second and Capacity (With Delays) Real- World Traffic Mixes The aim of this test is to measure the performance of the device under test in a real world environment by introducing additional protocols and real content, while still maintaining a precisely repeatable and consistent background traffic load. Different protocol mixes are utilized based on the intended location of the device under test (network core or perimeter) to reflect real use cases. For details about real world traffic protocol types and percentages, see the NSS Labs Network Firewall Test Methodology, available at Although Cyberoam CR2500iNG performed adequately on the Real World Perimeter Protocol mix test, NSS Labs would caution against deploying this firewall in any latency- sensitive environment because of the latency issues highlighted in section NSS Labs, Inc. All rights reserved. 9

10 12,000 Megabits per Second 10,000 8,000 6,000 4,000 2,000 10,000 6,200 0 Real World Protocol Mix (Perimeter) Real World Protocol Mix (Core) Mbps 10,000 6,200 Figure 5: Real- World Traffic Mixes 2013 NSS Labs, Inc. All rights reserved. 10

11 Stability & Reliability Long- term stability is particularly important for an in- line device, where failure can produce network outages. These tests verify the stability of the DUT along with its ability to maintain security effectiveness while under normal load and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or crash) while under hostile attack will not pass. The DUT is required to remain operational and stable throughout these tests, and to block 100 per cent of previously blocked traffic, raising an alert for each. If any non- allowed traffic passes successfully - caused by either the volume of traffic or the DUT failing open for any reason - this will result in a FAIL. Test Procedure Blocking Under Extended Attack Result Passing Legitimate Traffic Under Extended Attack Protocol Fuzzing & Mutation Power Fail Redundancy YES Persistence of Data YES Failover Legitimate Traffic Failover Time To Failover 5.4 Seconds Stateful Operation YES Active- Active Configuration YES Cyberoam CR2500iNG v10.04 was able to withstand the stability test and remained functional throughout. The CR2500iNG has a built in hardware bypass module which will fail open if enabled in bridge or transparent mode deployments. It will fail closed when disabled. When the resources on the device are exhausted and it is no longer possible to analyze traffic for any reason, the device will begin to drop traffic and the device will fail closed NSS Labs, Inc. All rights reserved. 11

12 Management & Configuration Security devices are complicated to deploy; essential systems such as centralized management console options, log aggregation, and event correlation/management systems further complicate the purchasing decision. Understanding key comparison points will allow customers to model the overall impact on network service level agreements (SLAs), estimate operational resource requirements to maintain and manage the systems, and better evaluate required skill / competencies of staff. As part of this test, NSS performed in- depth technical evaluations of all the main features and capabilities of the enterprise management systems offered by each vendor, covering the following key areas: General Management and Configuration how easy is it to install and configure devices, and deploy multiple devices throughout a large enterprise network? Policy Handling how easy is it to create, edit and deploy complicated security policies across an enterprise? Alert Handling how accurate and timely is the alerting, and how easy is it to drill down to locate critical information needed to remediate a security problem? Reporting how effective and customizable is the reporting capability? For a complete analysis of enterprise management capabilities and total cost of ownership, refer to the TCO and Management Comparative Analysis Reports (CAR) NSS Labs, Inc. All rights reserved. 12

13 Total Cost of Ownership (TCO) Implementation of security solutions can be complex, with several factors affecting the overall cost of deployment, maintenance and upkeep. All of these should be considered over the course of the useful life of the solution. Product Purchase the cost of acquisition. Product Maintenance the fees paid to the vendor (including software and hardware support, maintenance and other updates). Installation the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, and set up desired logging and reporting. Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and other updates. Management day- to- day management tasks including device configuration, policy updates, policy deployment, alert handling, and so on. For the purposes of this report, capital expenditure (CAPEX) items are included for a single device only (the cost of acquisition and installation). Operational expenditure (OPEX) items (ongoing management and labor costs) for multiple devices plus centralized management systems are modeled in the separate Management and TCO Comparative Analysis Reports (CAR). Installation (in Hours) This table details the number of hours of labor required to install each device using local device management options only. This will reflect accurately the amount of time taken for NSS engineers, with the help of vendor engineers, to install and configure the DUT to the point where it operates successfully in the test harness, passing legitimate traffic and blocking/detecting prohibited/malicious traffic. This closely mimics a typical enterprise deployment scenario for a single device. Centralized management options are covered in the Management CAR. Costings are based upon the time required by an experienced security engineer ($75 per hour fully loaded), allowing us to hold constant the talent cost and measure only the difference in time required for installation. Readers should substitute their own costs to obtain accurate TCO figures. Product Cyberoam CR2500iNG CyberoamOS v10.04 Installation (Hrs) 8 Purchase Price and Total Cost of Ownership Calculations are based on vendor- provided pricing information. Where possible, the 24/7 maintenance and support option with 24- hour replacement is utilized, since this is the option typically selected by enterprise 2013 NSS Labs, Inc. All rights reserved. 13

14 customers. Prices are for single device management and maintenance only; costs for central device management (CDM) solutions may be extra. For additional TCO analysis, including CDM, refer to the Management CAR. Product Cyberoam CR2500iNG CyberoamOS v10.04 Purchase Maintenance / year Year 1 Cost Year 2 Cost Year 3 Cost 3 Year TCO $19,999 $2,752 $23,351 $2,752 $2,752 $28,855 Year 1 Cost is calculated by adding installation costs ($75 per hour fully loaded labor x installation) + purchase price + first- year maintenance/support fees Year 2 Cost consists only of maintenance/support fees Year 3 Cost consists only of maintenance/support fees This provides a TCO figure consisting of hardware, installation and maintenance costs only. Additional management and labor costs are excluded, since they are modeled extensively in the Management and TCO CARs. Value: Total Cost of Ownership Per Protected Mbps There is a clear difference between price and value. The least expensive product does not necessarily offer the greatest value if it offers significantly lower performance than only slightly more expensive competitors. The best value is a product with a low TCO and high level of secure throughput (security effectiveness x performance). The following table illustrates the relative cost per unit of work performed: Mbps- Protected Product Throughput 3 Year TCO Price / Mbps- Protected Cyberoam CR2500iNG CyberoamOS v ,733 $28,855 $3 Price- per- Protected- Mbps was calculated by taking the three- year TCO and dividing it by the rated throughput. Three- Year TCO/Throughput = Price/Mbps- Protected. Prices are for single device management and maintenance only; costs for central device management (CDM) solutions may be extra. For additional TCO analysis, including CDM, refer to the TCO and Management CARs NSS Labs, Inc. All rights reserved. 14

15 Detailed Product Scorecard The following chart depicts the status of each test with quantitative results where applicable. More detailed scorecards that include enterprise management capabilities and TCO calculations are available in the appropriate Comparative Analysis Reports (CAR.) Description Result Security Effectiveness Firewall Policy Enforcement Baseline Policy Simple Policy Complex Policy Static NAT Dynamic / Hide NAT SYN Flood Protection Address Spoofing Protection TCP Split Handshake Performance UDP Throughput Mbps 64 Byte Packets 3, Byte Packets 9, Byte Packets 9, Byte Packets 11, Byte Packets 16, Byte Packets 19,300 Latency - UDP 64 Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets NSS Labs, Inc. All rights reserved. 15

16 Connection Dynamics - Concurrency & Connection Rates Theoretical Max. Concurrent TCP Connections 3,100,000 Theoretical Max. Concurrent TCP Connections w/data 2,999,000 Maximum TCP Connections Per Second 215,000 Maximum HTTP Connections Per Second 179,000 Maximum HTTP Transactions Per Second 360,000 HTTP Connections per Second & Capacity CPS Connections Per Second 44Kbyte Response 25,000 5,000 Connections Per Second 21Kbyte Response 50,000 10,000 Connections Per Second 10Kbyte Response 84,908 20,000 Connections Per Second 4.5Kbyte Response 113,000 40,000 Connections Per Second 1.7Kbyte Response 147,000 HTTP Average Application Response Time Connections Per Second 44Kbyte Response ,000 Connections Per Second 21Kbyte Response ,000 Connections Per Second 10Kbyte Response ,000 Connections Per Second 4.5Kbyte Response ,000 Connections Per Second 1.7Kbyte Response 0.24 HTTP Connections per Second & Capacity with Delays CPS 5,000 Connections Per Second 21Kbyte Response with Delays 28,000 10,000 Connections Per Second 10Kbyte Response with Delays 37,000 Real World Traffic Real World Protocol Mix (Perimeter) 10,000 Real World Protocol Mix (Core) 6, NSS Labs, Inc. All rights reserved. 16

17 Stability & Reliability Blocking Under Extended Attack Passing Legitimate Traffic Under Extended Attack Protocol Fuzzing & Mutation Power Fail Redundancy Persistence of Data Failover - Legitimate Traffic Failover - Time to Failover Stateful Operation Active- Active Configuration YES 5.4 Seconds Management & Configuration See Management Comparative Analysis Report (CAR) Total Cost of Ownership Ease of Use Initial Setup (Hours) 8 Time Required for Upkeep (Hours per Year) See CAR Expected Costs Initial Purchase (hardware as tested) $19,999 Installation Labor Cost (@$75/hr) $600 Annual Cost of Maintenance & Support (hardware/software) $2,752 Initial Purchase (enterprise management system) Annual Cost of Maintenance & Support (enterprise management system) Management/Upkeep Labor Cost (per See CAR See CAR See CAR Total Cost of Ownership Year 1 $23,351 Year 2 $2,752 Year 3 $2,752 3 Year Total Cost of Ownership $28, NSS Labs, Inc. All rights reserved. 17

18 Test Methodology Methodology Version: Network Firewall v4.0 All Test IDs in this report refer to the methodology document, not necessarily to sections in this report. A copy of the test methodology is available on the NSS Labs website at Contact Information NSS Labs, Inc. 206 Wild Basin Rd Building A, Suite 200 Austin, TX (512) info@nsslabs.com v This and other related documents available at: To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) or sales@nsslabs.com NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this report is conditioned on the following: 1. The information in this report is subject to change by NSS Labs without notice. 2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners NSS Labs, Inc. All rights reserved. 18