TEST METHODOLOGY. Endpoint Protection Evasion and Exploit. v4.0

TEST METHODOLOGY Endpoint Protection Evasion and Exploit v4.0

Table of Contents 1 Introduction... 3 1.1 Inclusion Criteria... 3 2 Product Guidance... 5 2.1 Recommended... 5 2.2 Neutral... 5 2.3 Caution... 5 3 Security Effectiveness... 6 3.1 False Positive Testing... 7 3.2 Exploits... 7 3.3 Evasions... 7 3.3.1 HTTP Evasion... 8 3.3.2 HTML Obfuscation... 8 3.3.3 Payload Encoding... 8 3.3.4 File Compressors... 9 3.3.5 Packers (Executable Compressors)... 9 3.3.6 Layered Evasions... 9 4 Performance... 10 4.1 Time to Start an Application (Warm Start)... 10 4.1.1 Outlook (current version)... 10 4.1.2 Microsoft Word (current version)... 10 4.1.3 Excel (current version)... 10 4.1.4 PowerPoint (current version)... 10 4.1.5 Adobe Reader (current version)... 10 4.1.6 Internet Explorer (current version)... 10 4.1.7 Firefox (current version)... 10 4.1.8 Chrome (current version)... 10 4.2 File Copy Times/Speeds from a USB Drive to a Local Folder... 11 4.2.1 Net increase in time to copy clean file 500K... 11 4.2.2 Net increase in time to copy clean file 1MB... 11 4.2.3 Net increase in time to copy clean file 3MB... 11 4.2.4 Net increase in time to copy clean file 10MB... 11 5 Total Cost of Ownership and Value... 12 Appendix A: Change Log... 13 Contact Information... 14 2

1 Introduction NSS Labs defines the endpoint as a client workstation where the most common usage is by a user or employee performing business tasks. The endpoint is contrasted to servers, which host, for example, databases, websites, ERP applications, and file and print services. Servers are not used to surf the Internet, check email, edit documents, or perform the wide array of tasks performed by the endpoint. Endpoint protection (EPP) has 3 main functional components. The foundation is a positive security model of a firewall, which limits communications based on application type and permitted source and destination address. The other two components are malware protection and intrusion prevention, which inspect traffic permitted through the firewall and via removable media. Traditionally, these are based on a negative security model (exception- based) that utilizes combinations of signatures and heuristics to determine bad content. Some products also employ threat isolation technology in place of, or to augment these. Threat isolation is intended to isolate unknown content in such a way as to keep malware from affecting the endpoint without attempting to determine if the content is good or bad. The term antivirus has largely been replaced by anti- malware or malware protection to incorporate protection against a more encompassing array of threats that typically includes viruses, worms, rootkits, Trojans, spyware, adware, and other rogue applications. NSS intentionally refers to this capability as malware protection in order to shift the focus from the technology to the end goal. (For example, some whitelisting approaches can achieve the desired end goal of protecting against malware, even though they are not considered to be anti- malware products). Endpoint Protection Product Malware Protection Intrusion Prevention Firewall Intrusion prevention refers to the technology that protects a system from exploits against vulnerable applications. The attacks can be initiated by an external attacker (attacker initiated) or an unsuspecting target user (target initiated). Attacker- initiated attacks include vulnerabilities within applications that can be remotely executed without user intervention. Examples include attacks against the TCP/IP stack, running services on a system, such as file and print sharing, management daemons, etc. Target- initiated attacks include actions initiated by the user that result in compromise to any of the following: Web browsers (Internet Explorer/Firefox) and plug- ins such as ActiveX, Adobe Flash, JavaScript Web- enabled technologies such as audio and video (Apple QuickTime, Windows Media, and RealPlayer) Desktop publishing (Adobe Acrobat and Microsoft Word, Excel, and PowerPoint) Applications that piggyback on HTTP, such as instant messaging, P2P (Skype, torrent applications) Other client applications that can be exploited without user knowledge 1.1 Inclusion Criteria In order to encourage the greatest participation, NSS invites all security vendors claiming Endpoint Protection Evasion and Exploit capabilities to submit their products at no cost. Vendors with major market share, as well as challengers with new technology, will be included. 3

Endpoint Protection Evasion and Exploit products should be supplied as a software executable, where possible, with the appropriate packaging and documentation. For all public tests, generally available (GA) software is required. Software will be installed on a system that meets the minimum requirements of the endpoint protection product. Endpoint protection products may include a separate management station. Vendors are encouraged to provide information and support to configure and review these systems to the best of their abilities. 4

2 Product Guidance NSS issues summary product guidance based on evaluation criteria that is important to information security professionals. The evaluation criteria include: Security effectiveness Resistance to evasion Stability Performance Value Each product is given a guidance rating. 2.1 Recommended A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommended rating from NSS regardless of market share, company size, or brand recognition. 2.2 Neutral A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration during the purchasing process. 2.3 Caution A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS should not be short- listed or renewed. 5

3 Security Effectiveness The ultimate goal of any attack on a computer system is to gain access to a target host and attempt to perform an unauthorized action. The unauthorized action could be reading of a system file, accessing a memory location, execution of malicious code, or any number of other actions. Unauthorized access of this nature is considered an intrusion. Computer systems are designed with many levels of protection to prevent unauthorized access and grant authorized access. However, intruders may circumvent these levels of protection by targeting vulnerable services, invoking back door privilege escalation, or replacing key operating system files. Endpoint protection products are designed to protect against remote attacks through continuous monitoring or isolation of the network traffic and protected operating system/applications using a software agent installed on the host operating system. Given that most endpoint protection products are designed to protect laptop and desktop clients and client applications rather than servers/server applications, NSS tests endpoint protection products by attempting to compromise client applications using target initiated exploits, including those against web browsers such as Microsoft Internet Explorer and Mozilla Firefox; email clients such as Microsoft Outlook, Mozilla Thunderbird, and Lotus Notes; desktop publishing and office productivity tools such as Adobe Acrobat, Microsoft Word, PowerPoint, and Excel; and media players such as Windows Media Player, Apple QuickTime, and Real Audio/Video. This section verifies that the product under test (PUT) is capable of accurately detecting and blocking, or otherwise isolating, a wide range of common exploits, while remaining resistant to false positives. For enterprise products, the latest signature pack is acquired from the vendor, and the PUT is deployed with the policy recommended by the vendor. For consumer products, NSS considers it unacceptable for a product of this nature to be sold without a default policy and/or recommended settings. No custom signatures are permitted in the testing all signatures used must be available to the general public at the time of testing. Procedure: 1. Prior to installing the endpoint protection product, NSS validates the baseline vulnerabilities and successful attacks for each host configuration. It is important to note that NSS only utilizes live exploits that have been validated in the NSS lab in order to ensure the most accurate test possible. 2. The target host systems are restored to a clean, uncompromised state, as installed and configured by the vendor. The EPP software is updated to ensure the latest protection. 3. NSS validates that the EPP does not interfere with legitimate access to the target host and its protected applications. Policies must allow legitimate communication. 4. The protected system is subjected to a battery of attacks. Between each attack, the system is restored to a clean state. The results are recorded for inclusion within NSS Endpoint Protection report. The security effectiveness of the PUT will be tested with live exploits and threats targeting real operating systems and various client applications. It is important to note that the vendor has no advanced knowledge of the attacks selected for the test. The test results therefore reflect a real- world scenario in which there is no ability to perform custom tuning for a lab environment. This approach differs considerably from any other public testing methodology currently in existence. 6

3.1 False Positive Testing The ability of the PUT to identify and allow legitimate traffic while blocking threats and exploits is of equal importance to providing protection against malicious content. This test will include a varied sample of legitimate application traffic, which should properly be identified and allowed. After completion of the false positive test and prior to the exploit testing being performed, those signatures or rules that were deemed to cause the false positive alerts will be disabled within the security policy. 3.2 Exploits NSS tests EPP products using target initiated exploits against client applications (i.e., Apple QuickTime, Adobe Acrobat, MS Word, Internet Explorer, etc.) In addition, since desktop clients rarely run server applications (i.e., HTTP, SMTP, DNS, and DB servers), NSS does not test server/attacker Initiated exploits against server applications as part of its endpoint protection methodology. Type Missed Tested Caught % Target Initiated X (#) Y (#) Z (%) NSS verifies that the PUT is capable of correctly blocking malicious attacks comprising of exploits. NSS security effectiveness testing leverages the deep expertise of NSS engineers who utilize multiple commercial, open source, and proprietary tools as appropriate. All of the live exploits and payloads in the NSS live exploit test have been validated in the NSS lab such that: a reverse shell is returned a bind shell is opened on the target allowing the attacker to execute arbitrary commands a malicious payload is installed a system is rendered unresponsive This test goes far beyond pressing the button on a test tool. In short, NSS engineers trigger vulnerabilities for the purpose of validating that an exploit was able to successfully target the victim and gain privilege escalation or perform some unauthorized task on the protected system. For threat isolation testing, NSS defines success based upon the product successfully isolating the malicious binary delivered from the exploit and executed/installed on the system. No traces of the malicious sample should remain in the system once the isolated task is closed. NSS defines failure based upon the exploit successfully downloading installing/executing malware, and where traces of the malicious code (or the effects of that code having executed, such as changes to the underlying OS or its configuration) remain on the host system once the task is closed. 3.3 Evasions Attackers can modify basic attacks to evade detection in a number of ways. If a PUT fails to detect a single form of evasion, any exploit can bypass protection, rendering it ineffective. NSS verifies that the PUT is capable of detecting and blocking basic exploits when it is subjected to varying common evasion techniques. 7

Wherever possible, the PUT is expected to successfully decode the obfuscated traffic to provide an accurate alert relating to the original exploit, rather than alerting purely on anomalous traffic detected as a result of the evasion technique itself. A number of common exploits are executed across the PUT to ensure that they are detected in their unmodified state. These will be chosen from a suite of common basic exploits for which NSS is certain that all vendors will have protection. 3.3.1 HTTP Evasion Per RFC 2616, the HTTP protocol allows the client to request and the server to use several compression methods. These compression methods not only improve performance in many circumstances, they completely change the characteristic size and appearance of HTML documents. Furthermore, small changes in the original document can greatly change the final appearance of the compressed document. This property of these algorithms could be used to obfuscate hostile content for the purpose of evading detection. The deflate compression method is a Lempel- Ziv coding (LZ77), specified in RFC 1951. The gzip compression method is specified in RFC 1952. Compression (Deflate) Compression (Gzip) Chunked encoding 3.3.2 HTML Obfuscation Malicious HTML documents exploit flaws in common web browsers, browser plug- ins, and add- ons in order to gain control of the client system and silently install malware such as Trojans, rootkits, and key loggers. Therefore, it is important that security products charged with protecting end systems must correctly interpret HTML documents. Many security products use simple pattern matching systems with very little semantic or syntactic understanding of the data they are analyzing. This leaves them vulnerable to evasion through the use of redundant, but equivalent, alternative representations of malicious documents. This test suite uses a number of malicious HTML documents that are transferred from server to client through the DUT. Each malicious HTML document is served with a different form of obfuscation, as follows: Base- 64 encoding Base- 64 encoding (shifting 1 bit) Base- 64 encoding (shifting 2 bits) Base- 64 encoding (random space injection) UTF- 16 character set encoding (big- endian) UTF- 16 character set encoding (little- endian) UTF- 32 character set encoding (big- endian) UTF- 32 character set encoding (little- endian) JavaScript escape encoding 3.3.3 Payload Encoding This test attempts to confuse the IPS into allowing an otherwise blocked exploit to pass using various encoding options that are standard within the Metasploit framework: x86/call4_dword_xor This encoder implements a Call+4 Dword XOR Encoder. 8

x86/countdown This encoder uses the length of the payload as a position- dependent encoder key to produce a small decoder stub. x86/fnstenv_mov This encoder uses a variable- length mov equivalent instruction with fnstenv for getip. x86/jmp_call_additive This encoder implements a Jump/Call XOR Additive Feedback Encoder. x86/shikata_ga_nai This encoder implements a polymorphic XOR additive feedback encoder. The decoder stub is generated based on dynamic instruction substitution and dynamic block ordering. Registers are also selected dynamically. 3.3.4 File Compressors The file compressors used include but are not limited to the following: WinZip 7- Zip WinRAR BZip GZip 3.3.5 Packers (Executable Compressors) The packers used include but are not limited to the following: UPX ASPack Expressor RLPack Mew 3.3.6 Layered Evasions This test attempts to bypass the PUT by combining evasion techniques. For example, UTF encoding + Gzip compression + chunked encoding. 9

4 Performance Host- based software can have a considerable impact on the usability of a workstation. This section outlines the specific use cases to be executed and measured. They are designed to represent the most common tasks performed by corporate employees. Each test is first performed without the PUT to establish a baseline. The PUT is then installed and the test is run again to determine the impact on performance. Each test is executed at least 385 times, providing a margin of error of 5%. In addition, the results that are more than 2 standard deviations from the mean (statistical outliers) are then discarded. 4.1 Time to Start an Application (Warm Start) 4.1.1 Outlook (current version) 4.1.2 Microsoft Word (current version) 4.1.2.1 The net increase in time to open a Word document 500K 4.1.2.2 The net increase in time to open a Word document 1MB 4.1.2.3 The net increase in time to open a Word document 3MB 4.1.2.4 The net increase in time to open a Word document 10MB 4.1.3 Excel (current version) 4.1.3.1 The net increase in time to open an Excel file 500K 4.1.3.2 The net increase in time to open a Excel file 1MB 4.1.3.3 The net increase in time to open a Excel file 3MB 4.1.3.4 The net increase in time to open a Excel file 10MB 4.1.4 PowerPoint (current version) 4.1.5 Adobe Reader (current version) 4.1.6 Internet Explorer (current version) Open to the default web page on the local system 4.1.7 Firefox (current version) Open to the default web page on the local system 4.1.8 Chrome (current version) Open to the default web page on the local system 10

4.2 File Copy Times/Speeds from a USB Drive to a Local Folder Copy Microsoft Word, Excel, and PDF files to the destination location. 4.2.1 Net increase in time to copy clean file 500K 4.2.2 Net increase in time to copy clean file 1MB 4.2.3 Net increase in time to copy clean file 3MB 4.2.4 Net increase in time to copy clean file 10MB 11

5 Total Cost of Ownership and Value Organizations should be concerned with the ongoing, amortized cost of operating security products. This section evaluates the costs associated with the purchase, installation, and ongoing management of the PUT, including: Product Purchase the cost of acquisition Product Maintenance the fees paid to the vendor (including software support, maintenance, and updates) Installation the time required to install the PUT on the endpoint, apply updates and patches, and configure it Upkeep the time required to apply periodic updates and patches 12

Appendix A: Change Log Version 0.9- Draft 02 May 2014 Original Document 13

Contact Information NSS Labs, Inc. 206 Wild Basin Rd, Building A, Suite 200 Austin, TX 78746 USA +1 (512) 961-5300 info@nsslabs.com www.nsslabs.com This and other related documents available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) 961-5300 or sales@nsslabs.com. 2014 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this document is conditional on the following: 1. NSS Labs reserves the right to modify any part of the methodology before, or during, a test, or to amend the configuration of a device under test (DUT) where specific characteristics of the DUT or its configuration interfere with the normal operation of any of the tests, or where the results obtained from those tests would, in the good faith opinion of NSS Labs engineers, misrepresent the true capabilities of the DUT. Every effort will be made to ensure the optimal combination of security effectiveness and performance, as would be the aim of a typical customer deploying the DUT in a live network environment. 2. The information in this document is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this document are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this document. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This document does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This document does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this document are the trademarks, service marks, and trade names of their respective owners. 14