What to Look for When Evaluating Next-Generation Firewalls Using independent tests to compare performance, cost and functionality
Table of Contents Why Use Independent Tests in Evaluations?... 3 What to Look for in a Next-Generation Firewall... 3 1. The NSS Labs Next-Generation Firewall Security Value Map... 4 2. The Network World Next-Generation Firewall Clear Choice Test... 7 3. The ICSA Labs Next-Generation Firewall Evaluation Report and Certifications... 9 Recap... 12 Page 2
Why Use Independent Tests in Evaluations? It is never easy to evaluate and compare complex technology products. Vendors provide feature lists and documentation, but feature lists never tell the whole story. Testing products in your own environment is costly and time consuming. That is why reports from independent test labs can be so valuable. These organizations: n Have the time and resources to perform thorough, detailed tests. n Offer in-depth expertise in the technologies, often from years of experience with the leading products in the field. n Provide unbiased results. In this paper, we will summarize the results of three independent tests that can help you select a Next-Generation Firewall for your organization. We also provide links to the portions of the research that have been made public, so you can examine the details yourself. What to Look for in a Next-Generation Firewall Traditional firewalls fail to provide adequate security against today s threats. Often they: n Provide unbiased results. n Provide little or no ability to protect against malware and advanced attacks. n Cannot decrypt and inspect Secure Sockets Layer (SSL) traffic. n Have no application awareness or ability to control application usage. n Are limited by hardware architectures that can t handle peak web traffic. Page 3
These liabilities create security vulnerabilities and force enterprises into expensive workarounds like deploying separate gateway antivirus products and intrusion prevention systems (IPS). Next-Generation Firewalls remedy these problems. But because they are more complex, they are also harder to evaluate. The criteria you should be considering when evaluating Next-Generation Firewalls include: n Are limited by hardware architectures that can t handle peak web traffic. n Cost effectiveness, as measured by the cost to scan a given volume of traffic. n Performance, particularly the ability to handle high volumes of traffic at wire speeds when all security functions are active. n Features such as: Gateway malware blocking and intrusion prevention. Decryption and inspection of SSL traffic Application intelligence and control User authentication and management Fortunately, independent test lab results can help you evaluate these factors across appliances from the leading vendors. Page 4
1. The NSS Labs Next-Generation Firewall Security Value Map Let s start by looking at a test report that summarizes security effectiveness and cost effectiveness on one chart. Austin, Texas-based NSS Labs is an independent research and analysis organization with in-house testing capabilities. In 2012, it conducted extensive tests designed to measure the cost effectiveness and performance of seven leading Next-Generation Firewall products. The results are summarized on what NSS Labs calls a Security Value Map, shown in Figure 1. Figure 1: The NSS Labs 2012 Next-Generation Firewall Security Value Map Page 5
The Y (vertical) axis shows the block rate, a summary of security-effectiveness tests. The products that are highest on the chart provide the best security against threats. The X (horizontal) axis shows the price per protected Mbps, which represents the three-year total cost of ownership divided by the performance (measured in Mbps scanned). Products on the right side of the chart have the lowest price per protected Mbps and the greatest cost effectiveness. Note that the scale on the axis showing the price per protected Mbps is logarithmic, so each grid line to the right represents a doubling of bang for the buck. For those products represented by two points on the graph, the point down and to the left represents security and performance under real-world conditions, with results adjusted for tests of evasions, stability and leakage of malicious traffic. Products with a single point on the graph tested 100% on all evasion, stability and blockage tests. Results The Dell SonicWALL SuperMassive E10800 running SonicOS 6.0 was positioned in the Recommend quadrant, indicating high security effectiveness and high cost effectiveness. Of the seven Next-Generation Firewalls evaluated in the assessment, only three vendors earned NSS Labs highest rating of Recommend. Of these three, the Dell SonicWALL SuperMassive E10800 achieved the Highest Overall Protection. Only one other appliance had a (slightly) higher block rate, but at roughly triple the price per protected Mbps. For More Details A copy of the NSS Labs Next-Generation Firewall Security Value Map is available at: http://o-www.sonicwall.com/us/en/14233.html. Page 6
2. The Network World Next-Generation Firewall Clear Choice The Clear Choice Tests Network World is a leading provider of information, intelligence and insight for network and IT executives. In April 2012, it performed an in-depth analysis of Next-Generation Firewalls, testing real-world performance metrics and SSL decryption capabilities. Summaries of the first set of its Clear Choice tests are shown in Figure 2. These Mixed-HTTP Content Handling tests involved simulating enterprise network traffic with objects ranging from 1KB to 1.5MB in size and a variety of content types, including JPEG images, PDF documents and binary files. These tests were designed to most closely approximate the loads handled by firewalls in enterprise networks. Figure 2: The Network World Clear Choice Tests: Mixed-HTTP Content Page 7
The testers varied the conditions of the tests by running them: 1. With only the firewall turned on. 2. With the firewall and IPS features turned on. 3. With the firewall, antivirus, antispyware and IPS features all turned on. The tests were further varied by sending the traffic in cleartext and again encrypted using SSL. Another set of tests was run for Static HTTP Content Handling, a slightly more artificial form of test where all of the objects in the traffic were either 100KB or 512KB. Again, the tests were varied for clear text and SSL traffic. Figure 3: The Network World Clear Choice Tests: Static HTTP Page 8
Results The Dell SonicWALL SuperMassive E10800 came out on top in Network World s performance tests for Next-Generation Firewalls. In the Mixed-HTTP Content Handling tests, the Dell SonicWALL SuperMassive appliance had the best performance on five of the six tests, and was dramatically faster with SSL traffic than the other devices. In fact, in the most demanding test in this series scanning SSL traffic with firewall, antivirus, antispyware and IPS features turned on the Dell SonicWALL appliance outperformed the second-fastest device by 18% and the other two devices by more than 100%. In the Static HTTP Content Handling tests, the Dell SonicWALL appliance had the best performance on 14 of the 16 tests. The article that accompanied the publication of the test results noted: [Dell] SonicWALL s SuperMassive can decrypt SSL traffic very fast in fact, these one-off tests show it to be the fastest device by far. For More Details A copy of the article detailing the Network World Next-Generation Firewalls Clear Choice test results is available at: http://www.sonicwall.com/us/en/15796.html. 3. The ICSA Labs Next-Generation Firewall Evaluation Report and Certifications ICSA Labs, an independent division of Verizon Business based in Mechanicsburg, Pa., provides vendor-neutral testing and certification of security products and solutions. Page 9
The Next-Generation Firewall Evaluation Report In July 2012, ICSA Labs published a detailed report evaluating the Dell SonicWALL E-Class Network Security Appliance (NSA) Series of Next-Generation Firewalls. During the course of testing, ICSA Labs evaluated application intelligence and control, user-based authentication, malware protection, user-side protection, server-side protection and false positives. The results are summarized in Figure 4, and some of the key findings are reviewed below. Area of Evaluation Effectiveness User-Based Authentication 100.00% Application Identification and Control 100.00% User Protection 98.34% Server Protection 94.60% Figure 4: Key results from the ICSA Labs evaluation of the E-Class NSA Series User-Based Authentication The results are summarized in Figure 4, and some of the key findings are reviewed below. Effectiveness: 100% The testers evaluated the ability of the Dell SonicWALL E-Class NSA Series to authenticate users and apply security policies based on user characteristics. The testers set up Active Directory domain controllers and created three unique user groups. The tests verified that: n Users from a variety of computers and operating systems were able to authenticate correctly. n The appliance could make access control policy decisions based on the user s identity. Page 10
Application Identification and Control Effectiveness: 100% The testers evaluated the ability of the Dell SonicWALL E-Class NSA Series to provide access control for applications needed for business, to limit or prevent access to applications that pose a risk to security or productivity, and to manage bandwidth to give priority to high-value applications. The testers set up three departments (Finance, Marketing and IT) and simulated the effort of users in those groups to access social media, online gaming, streaming media, instant messaging and web email sites. Acceptable-use policies were created for example, to allow marketing to post status updates on Facebook but not to play online games like Mafia Wars and Farmville there. The appliance was set up to decrypt and read SSL traffic as well as HTTP traffic. In the final set of tests, the Dell SonicWALL E-Class NSA Series was able to: n Identify all of the tested applications. n Enforce the acceptable-use policies correctly for each user group. n Control bandwidth utilization by application. User Protection Effectiveness: 98.3% The testers evaluated the Dell SonicWALL E-Class NSA Series on anti-malware, intrusion prevention and content filtering features that is, capabilities to detect and block malware, prevent exploits targeting application vulnerabilities, and restrict access to undesirable web sites. They measured the system s ability to protect users against attacks on Adobe, Microsoft, Mozilla and Oracle applications, polymorphic and non-polymorphic malware samples, and attempts to surf to undesirable web sites frequently compromised by hackers. Page 11
Server Protection Effectiveness: 94.6% The testers evaluated the ability of the Dell SonicWALL E-Class NSA Series system to block attacks on servers. They launched a series of attacks against exploitable, high-sensitivity vulnerabilities in enterprise applications from Microsoft, HP, Oracle, Symantec, IBM and others. The testers found that the Dell SonicWALL appliance was able to provide high effectiveness against these attacks, without negatively impacting normal/legitimate traffic [or] causing false positives. ICSA Labs Certifications ICSA Labs also certifies firewalls based on a detailed battery of tests. The labs have certified network firewalls from over 20 vendors at the Corporate and Small/Medium Business levels. Dell SonicWALL was the first, and is currently one of only two, Next-Generation Firewall providers to achieve the more exacting ICSA Labs Firewall-Enterprise certification. For More Details A copy of the ICSA Labs Next-Generation Firewall Evaluation Report for the E-Class NSA Series is available at: http://www.sonicwall.com/us/en/15804.html. Recap Independent testing organizations are widely trusted because they have the resources, expertise and perspective to provide detailed, unbiased information on technology products. The three sets of tests reviewed here provide useful information to people evaluating Next-Generation Firewalls. Page 12
In the NSS Labs Next-Generation Firewall Security Value Map, the Dell SonicWALL SuperMassive E108000 was one of three outstanding performers in terms of block rate and by far the leader in cost effectiveness (the combination of the block rate and price per protected Mbps). In the Network World Next-Generation Firewall Clear Choice tests, the Dell SonicWALL SuperMassive had the best performance in five of the six Mixed-HTTP Content Handling tests and in 14 of the 16 Static HTTP Content Handling tests. In the most demanding of these tests scanning encrypted SSL traffic with firewall, antivirus, antispyware and intrusion prevention features turned on the Dell Sonic- WALL Next-Generation Firewall outperformed its rivals by between 18% and 194%. In the ICSA Labs Next-Generation Firewall Evaluation Report, the Dell SonicWALL E-Class NSA Series scored between 95% and 100% on batteries of tests evaluating Next-Generation Firewall capabilities. These included features related to application intelligence and control, user authentication and management, and the ability to block malware and intrusions and protect against application vulnerabilities. In addition, Dell SonicWALL is one of only two vendors to have passed ICSA Labs most demanding firewall tests for Firewall-Enterprise certification. For more information on evaluating Next-Generation Firewalls, download Why Protection and Performance Matter at: http://www.sonicwall.com/app/projects/file_downloader/document_lib.php?t=wp&id=114 Page 13