TEST METHODOLOGY. Hypervisors For x86 Virtualization. v1.0

Transcription

1 TEST METHODOLOGY Hypervisors For x86 Virtualization v1.0

2 Table of Contents 1 Introduction The Need For Virtualization About This Test Methodology And Report Inclusion Criteria Product Guidance Recommended Neutral Caution Hypervisor Functionality Testing Capabilities Reviewed Outside The Testing Lab Virtual Machine Deployment Virtual Switch Configuration Support IEEE 802.3ad Link Aggregation Support IEEE 802.1q VLAN Tags Support Of The Combination Of IEEE 802.3ad Link Aggregation And IEEE 802.1q VLAN Tags Scalability and Performance Maximum Capacity Maximum Concurrent TCP Connections Maximum TCP Connections Per Second Maximum HTTP Connections Per Second Maximum HTTP Transactions Per Second HTTP Capacity With No Transaction Delays KB HTTP Post And Response Size 2,500 Connections Per Second KB HTTP Post And Response Size 5,000 Connections Per Second KB HTTP Post And Response Size 10,000 Connections Per Second KB HTTP Post And Response Size 20,000 Connections Per Second KB HTTP Post And Response Size 40,000 Connections Per Second HTTP Connections Per Second And Capacity (With Delays) KB HTTP Post And Response With Delay KB HTTP Post And Response With Delay Application Average Response Time: HTTP NSS Labs, Inc. All rights reserved. 2

3 5 Stability Protocol Fuzzing And Mutation Of Management Interface Protocol Fuzzing And Mutation of Virtual Network Interface Redundancy Of Hard Drive Disks Redundancy Of Virtual Switch Physical Interfaces Persistence Of Data Power Failure Recovery Management And Configuration Total Cost of Ownership And Value Appendix A: Test Environment Appendix B: Change Log Contact Information NSS Labs, Inc. All rights reserved. 3

4 1 Introduction 1.1 The Need For Virtualization Hypervisors are an integral part of virtualization and cloud computing. The primary function of a hypervisor is to abstract and distribute computing resources so that multiple operating system environments can share a single physical server while each appearing to run on a dedicated server. These are referred to as virtual machines. It is the hypervisor s job to control the physical server s processor, memory, network, disk, and other resources and to allocate those resources as required by each virtual machine. Most corporate data centers are populated with servers operating at a fraction of their capacity. Virtualization is one of the most prevalent ways for enterprises to consolidate and reduce unused resources. The demand to reduce operational costs by utilizing multitenant cloud computing technology has pushed enterprises to virtualize at a rapid pace. 1.2 About This Test Methodology And Report NSS Labs test reports are designed to address the challenges faced by information technology (IT) professionals in selecting and managing products designed for virtual and cloud based infrastructure. The scope of this particular report includes: Hypervisor Functionality Scalability and Performance Stability Management and Configuration Total Cost of Ownership (TCO) In order to evaluate accurately the performance and scalability a hypervisor provides, all underlying components, including virtual switches (vswitches), must be assessed. As hypervisor deployment increases in enterprise data centers and public cloud offerings, the performance, stability, and scalability of the hypervisor become critical. The hypervisor should not degrade network performance to the virtual machines it hosts, nor should it impact response time to mission critical applications. Hypervisors must support the use of networking technologies such as VLAN tagging (802.1Q) and link aggregation (802.3ad) to allow for multitenant environments, along with high bandwidth scenarios. This new virtual environment must offer equal or better stability, reliability, and responsiveness, and at the same time must be as flexible, as the physical environment it is replacing. The following capabilities are considered essential as part of a hypervisor: Provide virtualized hardware Host multiple operating system guests Provide network access through virtual switch architecture Remote administration Operate at layer 3 (IPv4 and IPv6) Note that the ability to handle IPv6 traffic is becoming increasingly important. This methodology will be used to test IPv4 and IPv6 deployments NSS Labs, Inc. All rights reserved. 4

5 If a vendor does not claim IPv6 support, there will be no penalty; the product will appear with the IPv4 results only. 1.3 Inclusion Criteria In order to encourage the greatest participation, and to allay any potential concerns of bias, NSS invites all leading hypervisor vendors to submit their Type 1 hypervisor products at no cost. Vendors with major market share, as well as challengers with new technology, will be included. To be considered a data center device, any virtual solution submitted must be capable of at least 10Gbps of throughput (as claimed by the vendor). Each physical server will have two management ports running at 1 GigE to a dedicated management network for command and control functionality. Additionally, two data ports will be connected to dedicated switches, which support VLAN tagging as well as link aggregation, and negotiated at 10 GigE for virtual network access to an untrusted network. Hypervisors submitted must support x bit hardware, IPv6 and SMP with at least 4 vcpu cores per hypervisor. Once installed in the test lab, the hypervisor will be configured for the use cases defined in this methodology. Each use case will have its own metrics to be measured and compared against a bare metal deployment to determine the impact of the virtualized environment. These use cases are defined in Section 3 of this methodology NSS Labs, Inc. All rights reserved. 5

6 2 Product Guidance NSS issues summary product guidance based on evaluation criteria that is important to information technology professionals. The evaluation criteria are weighted as follows: Functionality The hypervisor should provide base functions including virtual machine deployment as well as basic virtual switching functionality for fault tolerance and traffic isolation. Scalability The hypervisor should allow for adding more virtual machines without impacting the performance of current running virtual machines. Performance The hypervisor should allow a virtualized guest (i.e., virtual machine) to operate with the same performance it would have in a physical server deployment. Stability - Long- term stability is particularly important for a hypervisor, where the failure of a virtual machine, physical disc drive, or single network link should not cause system outages. This includes maintaining uptime and access to all hosted virtual machines while receiving malformed packets at its management and virtual switch interfaces. Management - In particular, how difficult is it to configure and deploy multiple virtual machine operating system types as well as what core features are available through an API (i.e., create, delete, clone, revert to snapshot ). Value Customers should seek low TCO and high effectiveness and performance rankings. Products are listed in rank order according to their guidance rating. 2.1 Recommended A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommended rating from NSS, regardless of market share, company size, or brand recognition. 2.2 Neutral A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration during the purchasing process. 2.3 Caution A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS should not be short- listed or renewed NSS Labs, Inc. All rights reserved. 6

7 3 Hypervisor Functionality Testing This section verifies the functionality of the hypervisor during deployment of virtual machines as well as various hypervisor configuration options. Multiple operating systems will be selected and used as baselines as well as test targets throughout the remainder of the tests. Some information will be gathered outside the testing lab (see Section 3.1); however, areas related to networking support and virtual machine deployment are evaluated and measured by NSS in the testing lab. Client connections to virtual machines will originate from outside the hypervisor, thus confirming that all measurements of the hypervisor will be related only to the load generated by the hosted virtual guests. 3.1 Capabilities Reviewed Outside The Testing Lab The following areas will be investigated prior to testing: Feature set comparison Security hardening standards available Network protocol support: 802.1Q, 802.3ad, QoS, Jumbo Frames Storage protocol support: iscsi, NFS, FCoE, image versus block storage Disk encryption External logging capabilities Ability to acquire system clock time from an external source Life cycle for security patches, defect patches, and updates Partner ecosystem Maturity of product Breadth of supported hardware configurations Dynamic IP Address Grouping Contributions or participation in the open source community Training materials and classes availability 3.2 Virtual Machine Deployment The hypervisor should be able to support a multiple operating system environment. The hypervisor will be configured to confirm that it is capable of hosting multiple virtual machines, each of which is running a different operating system (i.e., Linux and Windows). During this test, the ease of deployment will be measured along with any exceptions that need to take place for optimal performance of the hosted multiple operating system environment. Note this test is not designed to measure the density capabilities of a specific operating system. 3.3 Virtual Switch Configuration The hypervisor should support the use of various virtual switch configurations. There are many reasons for different types of configurations, including: Performance Network Isolation Fault Tolerance 2013 NSS Labs, Inc. All rights reserved. 7

8 Split management functionality (i.e. network storage versus management access) The following tests highlight the two most prominent network standards used to accomplish these goals Support IEEE 802.3ad Link Aggregation The hypervisor should support the configuration of link aggregation as defined by IEEE 802.3ad such that two virtual network interfaces can be combined into one link to provide fault tolerance (i.e., Active/Passive mode) capabilities Support IEEE 802.1q VLAN Tags The hypervisor should support the configuration of VLAN tags as defined by IEEE 802.1q such that virtual machines on different VLAN tags cannot communicate with each other without the support of a router or external switch to interconnect them Support Of The Combination Of IEEE 802.3ad Link Aggregation And IEEE 802.1q VLAN Tags The hypervisor should support the combined configuration of link aggregation (IEEE 802.3ad) and VLAN tags (IEEE 802.1q) without any adverse effects to either network standard NSS Labs, Inc. All rights reserved. 8

9 4 Scalability and Performance This section measures the scalabilty and performance of the hypervisor using multiple traffic conditions that emulate real- world deployment scenarios. Individual implementations will vary based on usage; however, these quantitative metrics provide a gauge as to whether a particular hypervisor is appropriate for a given environment. 4.1 Maximum Capacity The use of network testing tools allows NSS engineers to create real- world traffic at multi- Gigabit speeds as a background load for the tests. The aim of these tests is to stress the hypervisor and to determine how it copes with high volumes of TCP connections per second, application layer transactions per second, and concurrent open connections. This is achieved by generating increasing load targeting one or more instances of virtual machines running an HTTP server. All packets contain valid payload and address data, and these tests provide an excellent representation of a live network at various connection/transaction rates. Note that in all tests, the following critical breaking points (where the final measurements are taken) are used: Excessive concurrent TCP connections unacceptable increase in open connections on either the client- side or the server- side Excessive response time for HTTP transactions excessive delays and increased response time to client Unsuccessful HTTP transactions normally, there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency is causing connections to time out. The tests will begin with a baseline of a single physical (bare metal) machine and a single VM as follows: Single Bare Metal Machine Test Linux Single Virtual Machine Test Linux Single Bare Metal Machine Test Windows Single Virtual Machine Test Windows These tests will determine if a single virtual machine maintains the same maximum performance capabilities as the physical (bare metal) machine it is designed to replace. Virtual machines are then added in pairs (i.e., 2, 4, 6, 8,...), until the maximum number of virtual machines hosted can be determined without impacting the performance of any hosted virtual machine. Both oversubscription models (i.e., oversubscription of RAM and oversubscription of CPU) will be tested to confirm where performance degradation occurs. The following test will be used to determine these breaking points Maximum Concurrent TCP Connections This test is designed to determine the maximum concurrent TCP connections of the SUT with no data passing across the connections. This type of traffic would not typically be found on a normal network, but it provides the means to determine the maximum possible concurrent connections figure. An increasing number of Layer 4 TCP sessions are opened through the device. Each session is opened normally and then held open for the duration of the test as additional sessions are added up to the maximum possible. Load is increased until no more connections can be established, and this number is recorded NSS Labs, Inc. All rights reserved. 9

10 4.1.2 Maximum TCP Connections Per Second This test is designed to determine the maximum TCP connection rate of the SUT with one byte of data passing across the connections. This type of traffic would not typically be found on a normal network, but it provides the means to determine the maximum possible TCP connection rate. An increasing number of new sessions are established through the SUT, ramped slowly to determine the exact point of failure. Each session is opened normally, one byte of data passed to the host, and then the session is closed immediately. Load is increased until one or more of the breaking points defined earlier is reached Maximum HTTP Connections Per Second This test is designed to determine the maximum TCP connection rate of the hypervisor with a 1 byte HTTP response size. The response size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum HTTP connections per second rate. Client and server are using HTTP 1.0 without keep- alive, and the client will open a TCP connection, send one HTTP request, and close the connection. This ensures that all TCP connections are closed immediately upon the request being satisfied, thus any concurrent TCP connections will be caused purely as a result of latency of the host on which the hypervisor is installed. Load is increased until one or more of the breaking points defined earlier is reached Maximum HTTP Transactions Per Second This test is designed to determine the maximum HTTP transaction rate of the hypervisor with a 1 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum connections per second rate. Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send ten HTTP requests, and close the connection. This ensures that TCP connections remain open until all ten HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined earlier is reached. 4.2 HTTP Capacity With No Transaction Delays The aim of these tests is to stress the resources of the hypervisor and determine how the hypervisor copes with network loads of varying average packet size and varying connections per second. By creating genuine session- based traffic with varying session lengths, the hypervisor is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packet- based background traffic. This provides a test environment that is as close to real world as it is possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability. Each transaction consists of a single HTTP GET or POST request and there are no transaction delays (i.e., the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads NSS Labs, Inc. All rights reserved. 10

11 Connections per Second 44Kbyte Response 21Kbyte Response 10Kbyte Response Figure 1 HTTP Capacity 4.5Kbyte Response 1.7Kbyte Response CPS 2,500 5,000 10,000 20,000 40,000 Mbps 1,000 1,000 1,000 1,000 1,000 Mbps KB HTTP Post And Response Size 2,500 Connections Per Second Maximum 2,500 new connections per second per Gigabit of traffic with a 44KB HTTP post or response size average packet size 900 bytes maximum 140,000 packets per second per Gigabit of traffic. With relatively low connection rates and large packet sizes, all hosts should be capable of performing well throughout this test KB HTTP Post And Response Size 5,000 Connections Per Second Maximum 5,000 new connections per second per Gigabit of traffic with a 21KB HTTP post or response size average packet size 670 bytes maximum 185,000 packets per second per Gigabit of traffic. With average connection rates and average packet sizes, this is a good approximation of a real- world production network, and all hosts should be capable of performing well throughout this test KB HTTP Post And Response Size 10,000 Connections Per Second Maximum 10,000 new connections per second per Gigabit of traffic with a 10KB HTTP post or response size average packet size 550 bytes maximum 225,000 packets per second per Gigabit of traffic. With smaller packet sizes coupled with high connection rates this represents a very heavily used production network KB HTTP Post And Response Size 20,000 Connections Per Second Maximum 20,000 new connections per second per Gigabit of traffic with a 4.5KB HTTP post or response size average packet size 420 bytes maximum 300,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any host KB HTTP Post And Response Size 40,000 Connections Per Second Maximum 40,000 new connections per second per Gigabit of traffic with a 1.7KB HTTP post or response size average packet size 270 bytes maximum 445,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any host. 4.3 HTTP Connections Per Second And Capacity (With Delays) Typical user behavior introduces delays between requests and reponses, e.g., think time, as users read web pages and decide which links to click next or as users complete web forms for online submission. This group of tests is identical to the previous group except that these include a 5 second delay in the client GET or POST for each HTTP transaction. This has the effect of maintaining a high number of open connections throughout the test NSS Labs, Inc. All rights reserved. 11

12 KB HTTP Post And Response With Delay Max 5,000 new connections per second per Gigabit of traffic with a 21KB HTTP response size - average packet size 670 bytes - maximum 185,000 packets per second per Gigabit of traffic. 5 second transaction delay resulting in an additional 50,000 open connections per Gigabit over the test described in Section With average connection rates and average packet sizes, this is a good approximation of a real- world production network, and all sensors should be capable of performing well throughout this test KB HTTP Post And Response With Delay Max 10,000 new connections per second per Gigabit of traffic with a 10KB HTTP response size - average packet size 550 bytes - maximum 225,000 packets per second per Gigabit of traffic. Repeated with background traffic loads of 25%, 50%, 75%, and 100% of maximum throughput of IPS. 5 second transaction delay resulting in an additional 100,000 open connections over the test described in Section With large average packet sizes coupled with very high connection rates, this represents a very heavily used production network, and is a strenuous test for any sensor. 4.4 Application Average Response Time: HTTP Test traffic is passed across the infrastructure switches and into the virtual switch of the hypervisor simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The results recorded at each post or response size (44 KB, 21 KB, 10 KB, 4.5 KB, and 1.7 KB HTTP responses) load level of 90% of the maximum throughput with zero packet loss as determined previously NSS Labs, Inc. All rights reserved. 12

13 5 Stability Long- term stability is particularly important for a hypervisor, where failure can produce system unavailability. The following tests verify the stability of the hypervisor while under normal load and allowing access to all hosted virtual machines. Hypervisors that are not able to sustain uptime and access to all hosted virtual machines while under the conditions defined in this section will not pass. The hypervisor is required to remain operational and stable throughout these tests while allowing access to all host virtual machines. If any traffic does not pass successfully to any hosted virtual machines, caused by either the volume of traffic or the hypervisor resource exhaustion (i.e., CPU or memory), this will result in a FAIL. 5.1 Protocol Fuzzing And Mutation Of Management Interface This test stresses the management stack of the hypervisor by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and other well known test tools/suites. Traffic load is a maximum of 350Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a simple PASS/FAIL; the hypervisor is expected to remain operational and capable of maintaining throughout the test. 5.2 Protocol Fuzzing And Mutation of Virtual Network Interface This test stresses the virtual network stack of the hypervisor by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and other well known test tools/suites. Traffic load is a maximum of 350Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a simple PASS/FAIL; the hypervisor is expected to remain operational and capable of maintaining throughout the test. 5.3 Redundancy Of Hard Drive Disks The hypervisor should support redundant hard drive configurations (such as RAID 6) such that the hypervisor can maintain operation of all hosted virtual machines during a hard disk drive failure. The hypervisor should also maintain operation of all host virtual machines during the redundant hard drive disk rebuilding period following a hard drive disk failure. 5.4 Redundancy Of Virtual Switch Physical Interfaces The hypervisor should support redundant network interfaces associated to one virtual switch such that the failure of one physical interface will not impact the operation of the hypervisor nor impact access to the hosted virtual machines NSS Labs, Inc. All rights reserved. 13

14 5.5 Persistence Of Data The hypervisor should retain all configuration data, locally stored virtual machines, and locally logged data once restored to operation following power failure. 5.6 Power Failure Recovery The hypervisor should, following power recovery, reapply power to previously running virtual machines or it should possess the ability to designate certain virtual machines to power on upon completion of its boot process NSS Labs, Inc. All rights reserved. 14

15 6 Management And Configuration Hypervisors can be complicated to deploy; essential systems such as centralized management console options and event logging further complicate the purchasing decision. Understanding key comparison points will allow customers to model the overall impact on network service level agreements (SLAs), estimate operational resource requirements to maintain and manage the systems, and better evaluate required skill/competencies of staff. As part of this test, NSS will perform in- depth technical evaluations of the main features and capabilities of the centralized enterprise management systems offered by each vendor, covering the following key areas: General Management and Configuration how easy is it to install and configure the hypervisor, and deploy multiple hypervisors throughout a large enterprise or cloud- computing environment? Virtual Machine Deployment how easy is it to create, edit, and deploy virtual machines across an enterprise or cloud- computing environment? Alert Handling how accurate and timely is the alerting, and how easy is it to drill down to locate critical information needed to remediate a problem? Reporting how effective and customizable is the reporting capability? For additional information concerning enterprise management testing, refer to the separate Management Questionnaire document NSS Labs, Inc. All rights reserved. 15

16 7 Total Cost of Ownership And Value Organizations should be concerned with the ongoing, amortized cost of operating virtualization products. This section evaluates the costs associated with the purchase, installation, and ongoing management of the hypervisor. Product Purchase the cost of acquisition. Product Maintenance the fees paid to the vendor (including software and hardware support, maintenance and updates.) Installation the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, initial tuning, and set up desired logging and reporting. Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and firmware updates NSS Labs, Inc. All rights reserved. 16

17 Appendix A: Test Environment The aim of this procedure is to provide a thorough test of all of the main components of the hypervisor in a controlled and repeatable manner and in the most real- world environment that can be simulated in a test lab. The Test Environment The NSS test network is a multi- Gigabit infrastructure that can accommodate both Gigabit copper and 10 Gigabit fiber interfaces. The hypervisor is configured for the use case according to the test methodology. Traffic generation equipment, such as the hosts generating HTTP posts and requests, is connected to the external network. During this test, the receiving equipment is defined as the hosted virtual machines running on the hypervisor under test. For this test, there are no receive ports defined for the load generation equipment. The hypervisor s virtual switches are connected via 10 GigE interfaces to a dedicated physical switch allowing uplink to the test lab environment as well as to the test equipment emulating the HTTP clients. There is no firewall connected between the hypervisor under test and the test equipment. Cluster of 40 Host Physical Server Hypervisors Physical Test Switches Physical Core Switches Test Equipment and Management Servers Guest Virtual Machines Virtual Appliances Virtual Switches Untagged No 801.1q VLAN External Virtual Switch nic4 40 x 10G 10 GigE Physical Switch 802.1q VLAN ID X 802.1q VLAN ID Y 802.1q VLAN ID Z Untagged No 801.1q VLAN Internal Virtual Switch nic3 nic2 40 x 1G 10 GigE Physical Switch 1 GigE Physical Switch 4 x 40G 40 GigE Physical Switch 40 GigE Physical Switch 40 x 10G MGMT Virtual Switch nic1 1 GigE Physical Switch 40 Gig 10 Gig - Data 1 Gig - MGMT Figure 2 Example Stack Reference Architecture Overview The management interfaces of the hypervisor are connected to a dedicated 1 GigE physical switch, which is a different physical switch than the 10 GigE switch mentioned previously. This switch also contains uplink to the test lab environment to allow for testing of the security of the hypervisor s management interface NSS Labs, Inc. All rights reserved. 17

18 Appendix B: Change Log Version August 2013 Original Document 2013 NSS Labs, Inc. All rights reserved. 18

19 Contact Information NSS Labs, Inc. 206 Wild Basin Rd, Building A, Suite 200 Austin, TX USA +1 (512) v131101a This and other related documents available at: To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) or sales@nsslabs.com NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this document is conditional on the following: 1. NSS Labs reserves the right to modify any part of the methodology before, or during, a test, or to amend the configuration of a device under test (DUT) where specific characteristics of the DUT or its configuration interfere with the normal operation of any of the tests, or where the results obtained from those tests would, in the good faith opinion of NSS Labs engineers, misrepresent the true capabilities of the DUT. Every effort will be made to ensure the optimal combination of security effectiveness and performance, as would be the aim of a typical customer deploying the DUT in a live network environment. 2. The information in this document is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this document are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this document. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This document does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This document does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this document are the trademarks, service marks, and trade names of their respective owners NSS Labs, Inc. All rights reserved. 19