Test Report November 2007 Firebox X family from Unified Threat Management Technology Report
Vendor Details Name: Watchguard Address: 505 Fifth Avenue South, Suite 500, Seattle, WA 98104, United States Telephone: +1.206.613.6600 Website: www.watchguard.com Product: Firebox X e-series, Edge, Core, and Peak Models Test Laboratory Details Name: West Coast Labs, Unit 9 Oak Tree Court, Mulberry Drive Cardiff Gate Business Park, Cardiff, CF23 8RS, UK Telephone: +44 (0) 29 2054 8400 Date: November 2007 Issue: 1.0 Author: Rob Tanner Contact Point Contact name: Rob Tanner Contact telephone number: +44 (0) 29 2054 8400 2
Contents Test Objectives 4 Checkmark Certification 5 The Product 6 Test Report 7 Test Results 11 West Coast Labs Conclusion 13 Products Features Buyers Guide 14 Appendix A, B & C 17 Appendix D - Test Methodology & Specifications 23 3
Test Objectives Through a series of rigorous tests, West Coast Labs is providing a thorough examination of all the main technology components in the devices under test in a real world test environment to establish the level of functionality and performance of both. For the purposes of this Product Test Report, West Coast Labs tests the following technologies: Firewall VPN Anti-Spam URL Filtering IPS West Coast Labs engineers test and evaluate each device in a controlled environment. Throughout the test period, each product has internet access and is configured as recommended to update online. The testing environment mirrors that of a small to medium sized business and the internal interface of the firewall is connected to a 100Mbs network, and traffic loads are set accordingly. Products are tested in accordance with the functionality and performance criteria, which form the Checkmark certification programs for Firewall, VPN, IPS, URL Filtering and Anti-Spam. (See Appendix D). 4
Checkmark Certification Upon completion of all testing within each UTM technology subset, individual product results are manually analyzed and confirmed, and Checkmark accreditation is awarded on achieving the following standards: Checkmark Anti-Spam Certification Premium 97% and over Catch Rate Standard 90% and over Catch Rate Checkmark Anti-Virus Certification 100% Detection Checkmark Anti-Spyware Certification 100% Detection Checkmark IPS Certification 100% Detection and Prevention Rate Checkmark URL Filtering Certification Premium 100% adherence to policy Standard 95% and over adherence to policy Checkmark Enterprise Firewall Certification 100% adherence to policy Checkmark VPN Certification 100% adherence to policy 5
The Product Introduction The Firebox X family of UTM solutions consist of Peak, Core, and Edge e-series models that address an entire gamut of security issues relevant to small, medium, and large enterprises. The solutions ship in the form of gateway appliances that are specifically designed to automatically counter zero day threats using advanced proxy technologies and policy controls, and include antivirus, anti-spyware, IPS, VPN, URL content filtering, anti-spam, and firewall modules. The higher end appliances boast multiple Gigabit Ethernet ports, while all e-series devices are designed for optimum throughput rates and performance. The Core and Peak models have an integral LCD display panel. 6
Installation and Configuration The primary method of installation is via an easy to operate set-up wizard. All configuration activities are ordinarily carried out using either a standard web browser to access an SSL-encrypted web page for the Edge device, or via the dedicated installation of the WatchGuard System Manager (WSM) Windows application for control of the Core and Peak models. West Coast Labs opted to manage all Firebox X devices using a single instance of the WSM in order to provide continuity between the test set ups. Network interfaces were configured in internal (LAN), external (WAN), and optional modes, to match test requirements and available IP address space. Using the graphical point and click functionality of the WSM, West Coast Labs quickly set-up and configured the appropriate proxies to scan HTTP, POP3, and SMTP traffic for malware, in line with device capabilities. The devices ship with a complete set of useful and accurate documentation that proved effective during the installation process, and throughout the subsequent test period. 7
Operations and Features The devices each contain an anti-malware component that works seamlessly with other UTM modules to provide a powerful protection barrier against a wide range of threats, including viruses, spyware, trojans, DoS atttacks, SQL injections, worms, web exploits, and buffer overflows*. The integral spamblocker engine proved effective at preventing unsolicited email from reaching end-users. Once it is enabled and an appropriate policy defined, the spamblocker module intercepts and checks all email routed through the devices. Supported protocols include SMTP and POP3. The appliances ship with a URL content filtering module known as WebBlocker. This module operates as an HTTP proxy server, providing a full-featured and accurate web filtering mechanism. Using the policy engine, Administrators can define categories of URL that end-users are unable to access from within the corporate network. Specific URL exceptions are easily configured, providing organisations with the ability to fine tune access restrictions to their individual requirements. The Firebox X family uses a powerful, flexible firewall engine that, when coupled with the inbuilt fine-grained control of the policy module, provides a comprehensive and proven defence against common forms of automated and manual attack. The firewall, policy management, and ruleset configurations proved simple to set up and adjust via the management console. The firewall is deployed in a secure posture by default. Each device provides the inherent ability for a VPN to be configured, allowing secure connections to be established between any IPSec 8
and PPTP compliant products on distinct internal networks, in a simple and intuitive fashion. Remote Mobile User VPN (MUVPN) and manual modes are also optionally available, providing added flexibility while catering for a wide range of deployment scenarios. All VPN set-up and management is carried out using the standard management console. A tightly coupled, easily configured IPS engine is also included on each appliance. This component is designed to prevent threats from traversing the device and reaching the protected network. The engine actively and,if required, automatically blocks attack sources either permanently or on a temporary basis, with black and white lists available for more granular control. The WSM contains a Policy Manager component that displays each packet filter and proxy as a graphical icon, to represent a policy. It is straightforward to configure the source and destination for network traffic, and to specify whether this traffic is allowed or denied. Associated rules can also be set for logging and notification functionality. The devices include a number of pre-configured packet filters and proxies out of the box. For example, should an end-user require a packet filter for all telnet traffic, they can simply select the pre-defined telnet policy and modify the parameters to suit their individual needs. In addition, end-users also have the option to create custom packet filters or proxies and set the ports, protocols, and other parameters to match their requirements. * West Coast Labs were not required to formally test anti-virus and antispyware functionality to any Checkmark standard. 9
Reporting The WSM incorporates a real-time connection monitoring module, known as HostWatch, to display an operational picture of live network communications. This feature is potentially useful to administrators as it provides the ability to view current activity, allowing inappropriate connections to be terminated, thus further protecting network integrity. In addition to HostWatch, the WSM includes a powerful reporting system, allowing administrators to perform comprehensive analyses of network throughput, identification of security threats, and to assess end-user behaviour. The reporting engine supports data export in a number of formats, such as HTML and XML, thus ensuring flexibility and compatibility with a variety of third-party reporting systems. There is also a specific NetIQ compatible export format. 10
Results Anti-spam functionality was tested by pointing a live internet domain feed at the appliances, this feed was comprised of spam, ham (genuine), and gray email. Depending on the particular appliance, both SMTP (Core and Peak) and POP3 (Edge) based emails were processed from the external feed and relayed to an internal mail server. The Core and Peak devices use the same firmware and produced the following results. Type of Mail Detected as Genuine Detected as Spam GENUINE 100% 0% SPAM 6% 94% The Edge anti-spam test results are shown in the table below Type of Mail Detected as Genuine Detected as Spam GENUINE 100% 0% SPAM 4% 96% Throughout firewall testing, the appliances successfully blocked all spoofing, DoS, and malformed packet attacks as well as correctly logging the associated events. The appliances simultaneously allowed all legitimate network traffic to traverse each device under test and further blocked network probes that attempted to discover which services were active. IPS testing proved that the devices were equally as competent in preventing repeated attack attempts using different threats from the same source IP address, as they were in protecting against different threats from multiple source IP addresses. It is also worth noting that an extensive range of threats are set to an automatic block in 11
the default policy as an extra layer of protection. Testing involved attempting to exploit vulnerable computers within the protected network emulating a real-world attack from an internet-based source. West Coast Labs tested the effectiveness of the WatchGuard WebBlocker URL filtering module, by configuring the filtering policy to block certain undesirable content, while simultaneously allowing all other content. To aid testing in this area, a proprietary in house URL loader application based upon a well known web browser was used in conjunction with a manually harvested and verified set of several thousand URLs. Each device correctly blocked all content types specified by the policy, while allowing legitimate traffic through to the requesting end-user computer. For VPN testing purposes, West Coast Labs configured an IPSec VPN between the devices under test, emulating a real-world connection between a branch office and a remote office. West Coast Labs then applied a policy to only allow specific access to resources or services; web, email, and ftp servers on both the branch and remote networks. Additional services were active on both networks, however, all access to these services was denied. West Coast Labs subsequently tested that the access policy was operating correctly and that only the specified services were allowed, while all additional, non-specified services were denied. Port probes, network traffic analyzers, and other commonly used security tools and techniques were used in attempts to circumvent policy and to ensure that all data passing through the VPN connection were encrypted. West Coast Labs is pleased to award the WatchGuard Firebox X e- Series the Anti-Spam, Enterprise Firewall, IPS, URL Filtering, and VPN Checkmarks. 12
Conclusion The devices share a number of underlying software components and are similar in functionality. In addition to being systematically easy to use, configure, and manage, the Firebox X e-series provide effective protection against multiple real-world security threats. All devices benefit from thorough, well-written, and accurate documentation, as well as a secure internet support portal that provides simple, intuitive account management. In addition, an easy to use product upgrade area is also provided online that allows for the effortless upgrade of features and functionality. This is achieved by simply downloading and applying a license key to the product, as and when required and is an especially beneficial service to corporate customers and system administrators who have other calls on their time. The Firebox X e-series successfully blends versatility and performance with simple and effective management features, to produce a range of powerful and practical solutions, potentially benefiting businesses of any size. 13
The Products Firebox X Core, Edge and Peak * The Firebox X family of unified threat management solutions delivers an award-winning combination of network security, performance, ease of use, and value. They integrate multiple security functions into a single, easy-to-deploy platform that can easily extend and scale as businesses requirements change. Security capabilities include stateful packet firewall, VPN, authentication, proactive zero day attack prevention, gateway antivirus, intrusion prevention, anti-spyware, spam blocking, and URL filtering. This level of unified threat management protects the network from the constant threat of malicious attacks, while our intuitive user interface gives customers both granular control and genuine ease of use. WatchGuard product lines include: Firebox X Core for corporate and branch offices Firebox X Edge for small businesses, remote offices, and telecommuters Firebox X Peak for demanding network environments** *Product information provided by Watchguard. **Not tested by West Coast Labs 16
The Products - Key Selling Points* Stronger security to block zero day attacks Built-in security features including protocol anomaly detection and pattern matching proactively defend against sophisticated network attacks Powerful security services boost protection in critical attack areas WebBlocker Manage your users Web surfing to increase productivity, prevent legal liabilities, and decrease security risks by blocking access to malicious or inappropriate Web content SpamBlocker Get the best anti-spam service in the industry, blocking up to 97% of unwanted email in real time regardless of content, format, or language Gateway AntiVirus/Intrusion Prevention Service Rely on robust signature-based protection at the gateway to stop known viruses, trojans, spyware, SQL injections, and policy violations Unmatched ease of use Easy to set up and manage, the Firebox X has an intuitive user interface to streamline administration. Includes smart defaults, wizards, and drag-and-drop VPN. Scalable and upgradeable Get more performance and security capabilities by applying a simple license key no hardware to buy. *Product information provided by WatchGuard 17
Best support package in the industry Includes hardware warranty with advance hardware replacement, concise threat alerts, expert advice, technical support, software updates, and innovative education resources *Product information provided by WatchGuard 18
Appendix A - Firebox X Core features Security Features IP Address Assignment Network Address Translation Stateful Packet Firewall Port Independence Static NAT (Port Translation) Deep Application Inspection Firewall Static Dynamic NAT Spyware Blocking PPPoE Client One-to-One NAT Application Proxies - HTTP, SMTP, FTP, DHCP Server IPSec NAT Traversal DNS, TCP DoS and DDoS Prevention DHCP Client Policy-based NAT Progressive DDoS Prevention DHCP Relay Logging/Reporting Protocol Anomaly Detection Dynamic DNS Client Multi-appliance Log Aggregation Behavioral Analysis High Availability* WebTrends Compatible Reports (WELF) Pattern Matching HA Active/Passive HTML Reports Fragmented Packet Reassembly Configuration Synchronization XML Log Format Protection Malformed Packet Protection Session Synchronization Encrypted Log Channel Static Blocked Sources List VPN Tunnel Synchronization Syslog Dynamic Blocked Sources List WAN Failover SNMP Time-based Rules VPN Failover Alarms/Notifications Instant Messaging and P2P Allow/Deny WAN Modes SNMP Virtualization Spill-over* Email VLAN* Round Robin Management System Alert - Bridging Failover Management Software - Tagging ECMP WatchGuard System Manager (WSM) - Routed Mode Weight Round Robin* Certifications VPN Traffic Shaping EAL4+ - Encryption Quality of Service* (DES, 3DES, AES 128-, 192-, 256-bit) - 8 Priority Queues - Diffserve West Coast Labs Checkmark: - Modified Strict Queuing Firewall Level 1, VPN, URL Filtering, Intrusion Prevention, Anti-Spam IPSec Routing Support & Maintenance - SHA-1, MD5 - IKE Pre-Shared Key, Firebox 3rd Party Certificate* PPTP Server Static Routes 1-Year Hardware Warranty PPTP Passthrough RIPv1, v2 90-Day LiveSecurity Service Subscription Dead Peer Detection (RFC 3706) Dynamic Routing:* BGP4 OSPF Hardware-based Encryption Policy-based Routing* Drag-and-Drop Tunnels with Fireware Rules Modes of Operation User Authentication Transparent/Drop-in Mode (Layer 2) XAUTH Routed Mode (Layer 3) RADIUS LDAP Windows Active Directory RSA SecurID Web-based Local Authentication *Available with Fireware Pro advanced appliance software upgrade 19
Appendix A - Firebox X Core Specifications Specifications Firebox X550e Firebox X750e Firebox X1250e Firewall Throughput 300+ Mbps 300+ Mbps 300+ Mbps VPN Throughput 35 Mbps 50 Mbps 100 Mbps Gateway AV/IPS Optional Optional Optional URL Filtering Optional Optional Optional Spam Blocking Optional Optional Optional Interfaces 10/100 4 8 0 Interfaces 10/100/1000 0 0 8 Security Zones (incl.) 4 8 8 Concurrent Sessions 25,000 75,000 200,000 Nodes Supported (LAN IPs) Unlimited Unlimited Unlimited Serial Port 1 1 1 VLAN* 25 25 25 Branch Office VPN Tunnels (inc/max) 35/45 100/100 400/400 Mobile User VPN Tunnels (inc/max) 5/75 50/100 400/400 Local User Authentication DB Limit 250 1,000 5,000 Model Upgradeable No Yes No Fireware Pro Advanced Appliance Software Optional Optional Optional Advanced Networking Features Multi-WAN Load Balancing** Yes Yes Yes Multi-WAN Failover/VPN Failover Yes Yes Yes Traffic Shaping/QoS** Yes Yes Yes Port Independence Yes Yes Yes High Availability (Active/Passive)** Yes Yes Yes Dynamic Routing** Yes Yes Yes Policy-based Routing** Yes Yes Yes Hardware Warranty 1 Year 1 Year 1 Year LiveSecurity Service Initial Subscription 90 Day 90 Day 90 Day RoHS/WEEE compliant Yes Yes Yes Power Consumption U.S.: 60 Watts Rest of World: 860 Cal/min or 250 BTU/hr Throughput rates will vary depending on environment and configuration *Available with Fireware Pro advanced appliance software upgrade 20
Appendix B: Firebox X Edge Features Security Features IP Address Assignment Management Software Stateful Packet Firewall Port Independence Web GUI Outbound Deep Application Inspection Static WatchGuard System Manager (WSM) v9.0 or higher - HTTP PPPoE Client Logging/Reporting - FTP DHCP Server Syslog - POP3 DHCP Client WebTrends Compatible Reports (available to WSM users) Protocol Anomaly Detection DHCP Relay HTML Reports (available to WSM users) Pattern Matching Redundancy Features Encrypted Log Channel Fragmented Packet Reassembly WAN Failover Appliance Software Protection Malformed Packet Protection Traffic Management and v8.x or higher Prioritization Static Blocked Sources List Policy-based Traffic Prioritization Wireless Security Capabilities VPN VPN Traffic Prioritization Wireless Guest Services Encryption (DES, 3DES) Quality of Service 802.11b/g (4 prioritization queues) IPSec - Interactive WPA - SHA-1, MD5 - High WEP - IKE Pre-Shared Key, Firebox Certificate - Medium Certifications IPSec Passthrough - Low ICSA IPSec PPTP Passthrough Advanced Networking West Coast Labs Checkmark: Firewall Level 1, VPN, URL Filtering, Intrusion Prevention, Anti-Spam Dead Peer Detection (RFC 3706) Static NAT Support & Maintenance Hardware-based Encryption Dynamic NAT 1-Year Hardware Warranty User Authentication 1:1 NAT 90-Day LiveSecurity Service Subscription XAUTH IPSec NAT Traversal - LDAP Policy-based Port Address Translation - Windows Active Directory Up to 8 External IP Addresses Local Authentication Static Routes Windows NT Dynamic Routes Windows 2000 Modes of Operation Windows 2003 Integrated 3-Port Switch (Layer 2) Routed Mode (Layer 3) *Available with Fireware Pro advanced appliance software upgrade 21
Appendix B: Firebox X Edge Specifications Specifications Firebox Firebox Firebox X10e/X1e-W X20e/X20e-W X55e/X55e-W Firewall Throughput 100 Mbps 100 Mbps 100 Mbps VPN Throughput 35 Mbps 35 Mbps 35 Mbps Gateway AV/IPS Optional Optional Optional URL Filtering Optional Optional Optional Spam Blocking Optional Optional Optional Interfaces 10/100 6 6 6 Security Zones (incl.) 2 2 2 Concurrent Sessions 6,000 8,000 10,,000 Nodes Supported (LAN IPs) 15 30 Unlimited (upgradeable to 20) Serial Port 1 1 1 Branch Office VPN Tunnels (inc/max) 5 15 25 Mobile User VPN Tunnels (inc/max) 1/11 5/25 5/55 Local User Authentication DB Limit 200 200 200 Model Upgradeable No Yes No WAN Failover Optional Optional Included RoHS/WEEE compliant Yes Yes Yes Hardware Warranty 1 Year 1 Year 1 Year Model Upgradeable Yes Yes N/A LiveSecurity Service Initial Subscription 90 Day 90 Day 90 Day Power Consumption U.S.: 12 Watts Rest of World: 172 Cal/min or 41 BTU/hr *Throughput rates will vary depending on environment and configuration 22
Appendix C - Firebox X Peak Features Security Features X8500e-F Fiber Interface Modes of Operation Stateful Packet Firewall Multi-mode Fiber (MMF) Transparent/Drop-in Mode (Layer 2) Deep Application Inspection Firewall 1000 Base SX Routed Mode (Layer 3) Spyware Blocking 850 nm Network Address Translation Application Proxies - LC Connectors Static NAT (Port Translation) HTTP, SMTP, FTP, DNS, TCP DoS and DDoS Prevention IP Address Assignment Dynamic NAT Progressive DDoS Prevention Port Independence One-to-One NAT Protocol Anomaly Detection Static IPSec NAT Traversal Behavioral Analysis PPPoE Client Policy-based NAT Pattern Matching DHCP Server Logging/Reporting Fragmented Packet Reassembly DHCP Client Multi-appliance Log Aggregation Protection Malformed Packet Protection DHCP Relay WebTrends Compatible Reports (WELF) Static Blocked Sources List Dynamic DNS Client HTML Reports Dynamic Blocked Sources List High Availability XML Log Format Time-based Rules HA Active/Passive Encrypted Log Channel Instant Messaging and P2P Allow/Deny Configuration Synchronization Syslog Virtualization Session Synchronization SNMP VLAN VPN Tunnel Synchronization Alarms/Notifications - Bridging WAN Failover SNMP - Tagging VPN Failover Email - Routed Mode WAN Modes Management System Alert VPN - Spill-over Management Software - Encryption - Round Robin WatchGuard System Manager (WSM) (DES, 3DES, AES 128-, 192, 256-bit) - IPSec: - Failover Certifications * SHA-1, MD5 * IKE Pre-Shared Key, Firebox 3rd Party Certificates PPTP Server - ECMP EAL4+ PPTP Passthrough - Weight Round Robin Dead Peer Detection (RFC 3706) Traffic Shaping Support & Maintenance Hardware-based Encryption Quality of Service 1-Year Hardware Warranty Drag-and-Drop Tunnels with Fireware Rules - 8 Priority Queues 90-Day LiveSecurity Service Subscription User Authentication - Diffserve XAUTH - Modified Strict Queuing - RADIUS Routing - LDAP Static Routes - Windows Active Directory RIPv1, v2 RSA SecurID Dynamic Routing: BGP4, OSPF Web-based Policy-based Routing Local Authentication 23
Appendix C - Firebox X Peak Specifications Specifications Firebox Firebox Firebox Firebox X5505e X6500e X8500e X8500e-F Firewall Throughput* 2.0 Gbps 2.0 Gbps 2.0 Gbps 2.0 Gbps VPN Throughput* 400 Mbps 600 Mbps 600 Mbps 600 Mbps Gateway AV/IPS Optional Optional Optional Optional URL Filtering Optional Optional Optional Optional Spam Blocking Optional Optional Optional Optional Interfaces 10/100/1000 8 8 8 8 (4 copper /4 fiber) Serial Port 1 1 1 1 Security Zones (incl.) 8 8 8 4 RJ45, 4 SFP GBIC Concurrent Sessions 500,000 750,000 1,000,000 1,000,000 Nodes Supported (LAN IPs) Unlimited Unlimited Unlimited Serial Port 1 1 1 1 VLAN 75 75 75 75 Branch Office VPN Tunnels (incl./max.) 400/400 400/400 400/400 400/400 Mobile User VPN Tunnels (incl./max.) 400/400 400/400 400/400 400/400 Local User Authentication DB Limit 5,000 6,000 8,000 8,000 Model Upgradeable Yes Yes No No Advanced Networking Features Multi-WAN Load Balancing Yes Yes Yes Yes Traffic Shaping/QoS Yes Yes Yes Yes Port Independence Yes Yes Yes Yes High Availability Yes Yes Yes Yes (Active/Passive) Dynamic Routing/Policy-based Routing Yes Yes Yes Yes LiveSecurity Initial Subscription 90 Day 90 Day 90 Day 90 Day RoHS/WEEE Compliant Yes Yes Yes Yes Power Consumption U.S.: 80 Watts Rest of World: 1146 Cal/min or 273 BTU/hr *Throughput rates will vary depending on environment and configuration 24
Appendix D Test Methodology and Specifications Firewall Test Environment The test environment will consist of three distinct networks: the external (Internet), DMZ and internal (protected). The external network may include a telnet host, Web server, FTP server, DNS server and a "hacker" client to simulate the internet. The DMZ network may include a Web server and FTP server. The internal network may include a DNS server, SMTP server, file/print server, Web server and a "hacker" client. Machines on the internal and DMZ networks are not configured in a secure manner: they rely totally on the protection of the firewall. The firewall is the only link between the DMZ, internal & external networks. The link between the firewall and the external network is via a simple router. No packet filtering will be configured on this router: all protection must be provided by the firewall. Network monitors, protocol analysers and security monitors are employed on the external, DMZ and internal networks. Firewall Configuration The firewall is to be configured to provide the various services and enforce the various restrictions specified in this document. All firewalls are to be provided initially with an "out of the box" configuration, although vendors will be invited to remotely access their products if they wish to provide a best fit configuration. Network ranges will be provided to vendors as appropriate. No patches or configuration options will be allowed which are not available to the general public either in a current release or via a recognised and generally available support source. The configuration of all machines on the three networks will remain constant between tests. 25
Appendix D Test Methodology and Specifications Firewall Service Configuration The firewall is to be configured to allow the following outbound services: Internal to External: DNS, FTP (active and passive), HTTP, SSL/HTTPS, SSH, Telnet, SMTP Internal to DMZ: FTP, HTTP, SSL/HTTPS, SSH External to Internal: DNS, SSH, and SMTP External to DMZ: DNS, FTP, HTTP, SSL/HTTPS, SSH, SMTP DMZ to Internal: syslog, SNMP Firewall Test Specifications The testing is designed to ensure that the firewall technologies under test achieve a basic level of protection against a number of common hostile attacks, from both inside and outside the organization. A range of tests will be carried out using a variety of firewall scanning tools: these will be configured with full knowledge of both the firewall and network configuration: Test that all specified outbound services (and no others) are available from internal clients. Test that all specified inbound services (and no others) are available to external clients. Test that the firewall management console is not available to any users unless authenticated. Test that the firewall is resistant to a range of known Denial Of Service (DOS) tests. Test that the firewall does not allow uncontrolled access to either the internal or DMZ networks. Test that the underlying OS is hardened and not vulnerable to known OS-specific attacks. 26
Appendix D Test Methodology and Specifications Tests will be repeated in the following manner: Probe the internal network from the Internet Probe the DMZ from the Internet Probe the firewall from the Internet Probe the external network from the internal network (test security policy) Probe the DMZ from the internal network Probe the firewall from the internal network Management of the firewall will be evaluated using the following criteria: Local console must be secure. Management console should not be open to the external network. The firewall configuration should be fully protected and tamper proof (except from an authorised management station). Authentication should be required for the administrator for local administration. Authentication and an encrypted link should be available for remote administration. All attacks should be logged with date and time. 27
Appendix D Test Methodology and Specifications VPN Test Environment The VPN Test Environment will be based on the specification for Firewall as shown above, although another network will be specified as a Remote Office (RO). This may contain a telnet host, DNS server, SMTP server, FTP server, file/print server, Web server and client machines. Network monitors, protocol analysers and security monitors will also be deployed on the RO network. VPN Configuration and Service Configuration Initial configuration of the RO firewall should allow no inbound traffic to services hosted on the RO network. Clients on the RO should have access to the DNS, HTTP and SSL/HTTPS servers on the External network. The initial configuration of the VPN should allow unrestricted traffic flow between the RO and the main Internal network. This should include as a minimum ICMP, DNS, FTP (active and passive), HTTP, SSL/HTTPS, SMTP. 28
Appendix D Test Methodology and Specifications VPN Test Specifications The testing is designed to ensure that VPN technology achieves a basic level of security performance in that it: Allows a secure point-to-point link between two networks and between a roaming client and a network (optional) Provides authentication and access control mechanisms to restrict resource access on a per-user or per-group level Provides packet filtering or proxy services within the tunnel to restrict tunnel traffic to specific protocols or source/destination points Enforces a reasonable level of encryption and data integrity. Penetration Tests A range of penetration tests will be carried out using commonlyavailable scanning tools All tools will be configured with full knowledge of both the VPN and network configuration: Check that VPN management console is not available to any users unless authenticated and that the remote management link (if available) is encrypted or can be disabled Check that the VPN configuration is fully protected and tamper proof and that the VPN is resistant to a range of known Denial Of Service (DOS) attacks Check that the VPN has no known vulnerabilities and that it does not allow uncontrolled access to the networks behind it if traffic is restricted (see Services) Check that the VPN does not pass mis-configured packets to the networks behind it if traffic is restricted (see Services) and that the VPN correctly enforces access control policy on a per user and/or per group basis 29
Appendix D Test Methodology and Specifications Additional Tests Stage 1: Probe the VPN from the protected network with no tunnel established Stage 2: Probe the VPN from the external network with no tunnel established Stage 3: Attempt to establish tunnels using incorrect credentials Stage 4: Establish a valid tunnel (gateway-gateway and optionally client-gateway) and ensure that data is being encrypted Stage 5: Probe the remote network from the local network with valid gateway-gateway tunnel established attempt to violate tunnel traffic policy (eg. pass prohibited protocols, etc.) Stage 6: Probe the remote network from the local network with valid client-gateway tunnel established attempt to violate tunnel traffic policy (eg pass prohibited protocols, etc). Stage 7: Probe the remote network from the local network with valid gateway-gateway tunnel established attempt to violate access control policy (eg. user to access restricted resources). Stage 8: Probe the remote network from the local network with valid client-gateway tunnel established attempt to violate access control policy (eg. user to access restricted resources). 30
Appendix D Test Methodology and Specifications Management Management of the VPN will be evaluated using the following criteria: Local console must be secure and the Management console should not be open to the external network The VPN configuration should be fully protected and tamper proof (except from an authorised management station) Full authentication is required for the administrator for local administration Full authentication and an encrypted link is required for remote administration. If the remote link cannot be encrypted, there should be the ability to disable it. IPS Test Environment The network structure will be the same as for the VPN testing (if the DUT supports it, otherwise it should be the same as for firewall testing) with deployments of Network monitors, protocol analysers and security monitors on each network. Configuration The configuration of the DUT should be the same as the VPN testing (if the DUT supports it, otherwise it should be the same as for firewall testing). 31
Appendix D Test Methodology and Specifications Testing The IPS testing module is designed to ensure that the Intrusion Detection and/or Prevention technology contributes to a basic level of protection for an organization against hostile attacks. All IPS testing will be conducted with full knowledge of the configuration of the DUT. The testing will include a variety of different testing methodologies using both proprietary and established tools and code. Further exploration and attempted exploitations will take place dependant upon the DUT and results received from scans made. The IPS will be expected to monitor all traffic between the external and internal networks. Machines on the internal network are not configured in a secure manner. Network monitors, protocol analysers and security monitors are employed on the external and internal networks. The configuration of all machines remains constant between tests. A full range of tests will be carried out using tools, which will be configured with full knowledge of the network configuration. Tools used will include port scanners and vulnerability testers. Attacks will be launched including denial of service attacks and targeted buffer overflows. The internal network will also be subject to attack using spyware, worms and Trojans drawn from the West Coast Labs AV, Spyware and Trojan test suites. The IPS will be tested for reactions to: multiple, varied attacks (flood and swarm). obfuscated URLs and obfuscated exploit payloads speed adjustments in packet sending fragmented packets The testing will also review IPS logs and alerts, matching them to vulnerability scans. They will also be matched to password cracking activity. 32
Appendix D Test Methodology and Specifications Anti-Spam test Environment WCL has a number of domains available which act as honeypots for spam, receiving genuine, not canned spam. These domains receive varying levels of spam and are intended to mirror different email environments. Within each domain are designated user accounts with a variety of email practices and needs. Test Methodology During the course of testing, test engineers use several different internal and external accounts to send emails that simulate real life email transactions common in a business environment. These include requesting meetings, distributing notifications to groups and sending non-business related social emails. Emails are also sent from web-based accounts to simulate external users sending non-business related emails and home workers. Individual user accounts are subscribed to several mailing lists and daily newsletters for grey mail purposes. Each solution is configured initially to fit in with the test network using the vendor s recommendations and is placed into the stream of live mail to ascertain how it copes in an out-of-the-box situation. The only alteration made to standard working practices is that all emails should be forwarded on (although with altered headers or some sort of flag marking the offending mail as spam) to allow for later classification. For ascertaining the level of performance, each solution will receive a set number of emails. These are then classified by hand into genuine, spam and grey mail by test engineers with full knowledge of the mailing lists that have been previously signed up for. These figures are then compared with the figures given by the solution to give an overall detection rate. 33
Appendix D Test Methodology and Specifications Each solution will be assessed in three specific areas Management/ Administration, Functionality, and Performance. 1. Management/Administration. Ease of Setup/Use; Logging and reporting function; Rule creation. Customization; Content Categories; Product Documentation 2. Functionality Email Processing; Allow/Blocking of Email; Quarantine Area; Blacklist/ Whitelist 3. Performance Volume or % of spam detected; False positive rate Spam incorrectly passed thru; Legitimate mail blocked 34
Appendix D Test Methodology and Specifications URL Filtering Test Environment The tests replicate a number of hits on sites or emails received that fall outside of a prescribed Acceptable Usage Policy, along with providing genuine sites as a control group. URL Filtering Test Methodology TEST I - A proprietary piece of software loads in a list of URLs from a file. This switches through the list changing web page every 6 (six) seconds until it either runs out of URLs or receives an END command. The HTML code from each web page is appended to a log. The designated test engineer will then look through these logs to ascertain if any pages have been passed through the solution. TEST II - The list from TEST I is re-run through the software. This is accompanied by two human operators manually following a prespecified list of URLs in a pre-specified order, and also by a background load provided by specialist hardware. The logs are then appended again to a log file and will be checked further. Test Specifications Basic assessment of the solutions under test will consist of attempts to access material via the web in contravention of the security policy. A standardised user session will be employed for this purpose with reproducible http requests being generated. It is expected that attempts to access web sites outwith the terms of the security policy will be blocked, and that all such attempts will be logged and recorded. 35
West Coast Labs Disclaimer While West Coast Labs is dedicated to ensuring the highest standard of security product testing in the industry, it is not always possible within the scope of any given test to completely and exhaustively validate every variation of the security capabilities and / or functionality of any particular product tested and / or guarantee that any particular product tested is fit for any given purpose. Therefore, the test results published within any given report should not be taken and accepted in isolation. Potential customers interested in deploying any particular product tested by West Coast Labs are recommended to seek further confirmation that the said product will meet their individual requirements, technical infrastructure and specific security considerations. All test results represent a snapshot of security capability at one point in time and are not a guarantee of future product effectiveness and security capability. West Coast Labs provide test results for any particular product tested, most relevant at the time of testing and within the specified scope of testing and relative to the specific test hardware, software, equipment, infrastructure, configurations and tools used during the specific test process. West Coast Labs is unable to directly endorse or certify the overall worthiness and reliability of any particular product tested for any given situation or deployment. Revision History Issue Description of Changes Date Issued 1.0 WatchGuard Firebox X Core 21st May 2007 and Firebox X Edge UTM Testing 34
US SALES T +1 (717) 243 5575 EUROPE SALES T +44 2920 548 400 GLOBAL HEADQUARTERS West Coast Labs Unit 9 Oak Tree Court Mulberry Drive Cardiff Gate Business Park Cardiff CF23 8RS, UK