McAfee Web Gateway Security Appliance Test A test commissioned by McAfee, Inc. and performed by AV-TEST GmbH Date of the report: January 24 th, 2013 (last update: February 20 th, 2013) Executive Summary In December 2012 and January 2013, AV-TEST performed a review of McAfee s web gateway security solution for the enterprise to determine their malware detection and blocking capabilities. McAfee commissioned AV-TEST to run an independent test of McAfee Web Gateway. AV-TEST was not able to successfully contact any competitor prior to the test, to include competing appliances in the test. In order to ensure a fair review, the sponsor has not supplied any samples or had any influence or any prior knowledge regarding the samples being tested. The following test scenarios are standard tests that AV-TEST does on a regular basis for gateway antimalware solutions. There are two ways to stop malware at the gateway level today. These are signature-based detection options that protect against known vulnerabilities and malware, and heuristic/generic detection options which work on both known and unknown vulnerabilities and malware, including Zero-Day. These tests cover both. 1. Zero-Day Testing: Testing of the effectiveness of dynamic URL filtering capabilities and protection against zero-day malware by accessing real URLs that host malicious downloads, 2. PE Malware Test: Detection of relevant current malicious Win32 portable executable (PE) files, also referred as Zoo viruses, which are not older than 3 months at the start of the review, 3. Non-PE Malware Test: Detection of current malicious non-pe files, such as PDF exploits, as well as files including malicious scripts and macros for Microsoft Office and other applications, which are also not older than 3 months, Breaking out the data by test shows that McAfee performs very good in each test: Zero-Day PE Malware Non-PE Detected Samples 189 120,213 9,234 Total Samples 200 120,448 9,362 Detect Rate 94.50% 99.80% 98.63% Figure 1: Summary of the appliance test results 1
Overview With the increasing volume of malware, targeted attacks and advanced persistent threats spreading through the Internet these days, the danger of getting infected is higher than ever before. In the year 2000, AV-TEST received more than 170,000 new unique samples, and in 2011, the number of new samples grew to over 18,000,000 and reached 34,000,000 in 2012. The growth of these numbers is displayed in Figure 2. New unique samples added to AV-TEST's malware repository (2000-2012) 35.000.000 30.000.000 25.000.000 20.000.000 15.000.000 10.000.000 5.000.000 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 Dec Nov Oct Sep Aug Jul Jun May Apr Mar Figure 2: New malware samples per year To protect the enterprise network against the enormous number of threats a multilayered security setup is recommended. The layers at a minimum should include the enterprise firewall, a web- and content-filter for every kind of traffic, which is the topic of this document, and an endpoint protection product as the last barrier for the malware. A clever combination of those layers makes it hard for the attacking site to infiltrate the enterprise network. Product Tested The following product was tested, using the latest signature updates available: Product Software Version AV-Engine McAfee Web Gateway 7 7.3.0.2.0 McAfee Gateway Anti-Malware Engine (1) (1) McAfee Gateway Anti-Malware Engine includes both signature based and behavioral components Figure 3: Version details of the tested product 2
Methodology and Scoring Platform The test environment was set up according to figure 4. The client and webserver were virtual machines on a VMware ESXi host. The client was configured to use the McAfee security gateway as HTTP-proxy. Over a second line the client collected verification information like file checksums for each sample. The webserver hosted the Zoo samples for the PE malware detection test, the false positive samples and the Non-PE malware samples. For the Zero-Day test, the webserver downloaded files from the sample URLs and notified the client after each URL. The client then fetched the URL over its HTTP-proxy and verified the checksum, which was received from the webserver. Figure 4: Test platform overview Testing methodology AV-TEST received a preconfigured appliance from McAfee and did not need to make any changes regarding the security and network settings. During the analysis of the results, the testers noted that PDF files were mainly blocked by policy and not by the malware engine. A web gateway can use policies for all kinds of contents as well as black and white-listing. Because the purpose of the test was to assess malware detection, the testers decided to retest the PDF samples with the relevant policy being disabled. 1. Internet Access. The appliance had access to the Internet at all times in order to use any inthe-cloud queries. 2. Product Configuration. The product was run with the configuration supplied by McAfee. A media blocking policy has been disabled. The appliance was able to perform automatic signature updates all the time. 3. Testing. All files, except for the malicious URLs, were downloaded via http from the virtual webserver to the client system using a Java client with http proxy set to the appliance. For 3
the URL testing, an additional client has been used without proxy configuration to download the reference samples from the Internet. 4. Analysis. The downloaded files were compared with the original files (reference files at URL testing) by MD5 hash. For verifying the results, the appliance report files were analyzed. The static set of files consisted of 120,448 malicious PE files (Zoo malware) and 9,362 non-pe files. The dynamic tests were performed using 200 working malicious URLs. Test Results Test #1: Zero-Day protection rate Zero-day threats are typically identified through the gateway s ability to open up content for inspection coupled with whatever proactive scanning abilities and cloud intelligence a vendor may provide. A block can be based on URL filtering or Web Reputation services, by signatures or heuristic scanning of the provided content, and other inspection and filtering technologies. In the case of the blocking of malicious URLs McAfee has a protection rate of 94.5%. In a previous Web gateway comparative test conducted by AV-TEST in 2010/2011, McAfee Web Gateway achieved a protection rate in this category of 90.5%. Test #2: PE malware detection The total number of malicious samples tested was 120,448. This includes the following number of samples: 7,481 Backdoors, 2,875 Bots, 4,882 Viruses, 6,459 Worms, 89,071 Trojan Horses, 6,237 potentially unwanted applications (PUA) as well as 3,443 rogue applications (e.g. Fake AV). This test focuses on the generic malware detection and blocking capabilities, especially on the signature-based detection as well as generic and heuristic technologies. McAfee detected almost all Zoo samples, as they did in the previous 2010/2011 test (see figure 5): Type of Malware Number of Samples Detected Samples % Total 120,448 120,213 99.80% Backdoors 7,481 7,473 99.89% Bots 2,875 2,874 99.97% Potentially Unwanted Applications 6,237 6,222 99.76% Rogue Software 3,443 3,438 99.85% Trojan Horses 89,071 88,878 99.78% Viruses 4,882 4,876 99.88% Worms 6,459 6,452 99.89% Figure 5: Zoo detection results Test #3: Non-PE malware detection While many companies incorporate complete blocking of PE files for security reasons, so a product would filter out 100% of the test cases shown in the Test #2 (when this option is enabled), many other file formats like DOC or PDF exist which can also be very dangerous due to exploits, scripts or macros, but blocking the file type would not be an option. The following test set includes 9,362 samples with 109 batch scripts, 98 Java class files, 3,172 HTML files with embedded JavaScript or malicious IFRAMEs, 5,978 PDF exploits and 5 Shell-based exploits. 4
The detection of McAfee was very high against the tested malicious non-pe files (see figure 6). Type of Malware Number of Samples Detected Samples % Total 9,362 9,234 98.63% Batch Scripts 109 104 95.41% Java Classes 98 98 100.00% HTML/JS 3,172 3,054 96.28% PDF Documents 5,978 5,973 99.92% Shell Scripts 5 5 100.00% Figure 6: Non-PE files detection results McAfee Web Gateway achieved a similar protection rate in the 2010/2011 comparative test. Conclusion With the given configuration, McAfee scored very well in all reviewed categories. Compared to the previous test, McAfee showed a repeated very good performance. However, the results can vary over time and may vary with other configurations. The rising number of malware and the huge number of new malwares per day definitely require a multi-layered protection as McAfee provides. Copyright 2013 by AV-TEST GmbH, Klewitzstr. 7, 39112 Magdeburg, Germany Phone +49 (0) 391 60754-60, Fax +49 (0) 391 60754-69, Web http://www.av-test.org 5