Make Optimizing Security Protection in Virtualized Environments a Priority

G00229651 Make Optimizing Security Protection in Virtualized Environments a Priority Published: 15 February 2012 Analyst(s): Neil MacDonald As the virtualization of servers and desktops becomes more common, endpoint protection platform (EPP) solutions are starting to adapt to the needs of these environments. Although most solutions will run in a virtual guest environment without modification, the performance impact, especially of scheduled scans, will affect the density of virtual machines (VMs) per server. Techniques such as randomized scanning, scanning of offline guests, gold image whitelisting, scan result caching and randomizing signature updates provide improved performance in virtual environments. Several vendors have added or improved these techniques; others are notably behind. Explicit support and optimization for virtualized environments should become a mandatory part of any endpoint security tool evaluation. Key Findings Not all anti-malware vendors have delivered scanning solutions that are optimized for virtual environments. Multiple simultaneous scans aren't the only cause of resource contention issues. Signature and engine distribution scenarios also create significant overhead. Resource contention issues can occur on virtualized server or desktop workloads, although the issue is more pronounced with hosted virtual desktops (HVDs). Resource contention issues caused by security protection will alter the economics of full-scale HVD implementations if anti-malware solutions are not optimized. Pilot programs, which typically have low server utilization rates, may not surface this problem. Agentless antivirus (AV) has received a significant amount of market attention, but there are limitations associated with this approach that must be considered. Recommendations Look for solutions that natively support and optimize anti-malware scanning in a virtualized environment in your EPP evaluation for servers and desktops. Favor vendors that offer solutions spanning physical and virtual servers and desktops with a consistent management and reporting interface and a consistent way to set policies across all environments

To reduce complexity and consolidate licensing costs, use the same anti-malware-scanning solution across desktops and servers virtual or physical where possible. What ou Need to Know Endpoint security protection capabilities must evolve to explicitly support virtualized environments ideally, providing virtualization exploitative solutions and protecting virtualized desktop and server workloads. Different vendors are at different levels of maturity in their virtualized protection capabilities, so this must become a key requirement in server and desktop security protection solutions moving forward. Don't assume that your vendor provides the best solution for virtual environments. Use the framework provided here to evaluate EPP solutions against each other, and to better understand the different approaches available. Analysis Virtualization of the data center server and desktop workloads is happening rapidly; 1, 2 however, most host-based protection software was designed with the now-outdated notion that the security stack was running on dedicated hardware. Agent-based security protection controls for endpoints, such as EPP (see "Magic Quadrant for Endpoint Protection Platforms"), create significant resource contention issues when multiple, separate, independent and uncoordinated anti-malware scans are initiated from within virtualized sessions on the same physical hardware. The issue can occur with real-time, on-access scanning; however, this problem is more pronounced with scheduled on-demand scans within HVD environments, where all workloads are typically protected by EPP agents set to perform anti-malware scanning or to update their signature files at the same time creating "AV storms." In many cases, server and desktop VM images are "cookie cutter" images of each other, built from common enterprise high-assurance "gold" templates. It makes no sense to scan the same set of files against the same signature set over and over. Furthermore, this inefficiency affects the alreadytenuous value proposition of HVD 3 by reducing the number of images that can be stored on a single physical host because of input/output or other hardware constraints. There are multiple ways that anti-malware vendors should address the need to secure virtualized environments from simply supporting a virtualized environment to exploiting native integration with the virtualization platform to optimize anti-malware protection. We provide a multilevel framework to evaluate the anti-malware protection in a virtualized endpoint environment (see Figures 1 and 2 for examples of vendors and solutions). Page 2 of 13 Gartner, Inc. G00229651

Figure 1. Example of Virtualization Capabilities by Vendor in Shipping Solutions (Part 1) Capability Vendor and Version 1a. Support of the product running in virtualized environments: Support of the product running in a full VM 1b. Support of the product running in virtualized environments: Support of the product running in a terminal service environment 1c. Support of the product running in virtualized environments: Supporting the product's ability to look inside virtualized application containers Symantec McAfee Trend Micro Kaspersky Sophos Microsoft SEP 11 SEP 12.1 VirusScan Move 1.0 Move 2.0 Office Scan DeepSecurity v6 MP4 v8 v10 Forefront Endpoint Protection 1d. Support of the product running in virtualized environments: Support of the product running in a virtualized application container 2. Randomized or staggered scanning (via virtualization scan controller) 3. Build a whitelist of files from VM image templates (if gold image built with agent for initial cache) (if gold image built with agent for initial cache) N (future) Source: Gartner (February 2012) Gartner, Inc. G00229651 Page 3 of 13

Figure 2. Example of Virtualization Capabilities by Vendor in Shipping Solutions (Part 2) Vendor and Version Symantec McAfee Trend Micro Kaspersky Sophos Microsoft Capability SEP 11 SEP 12.1 VirusScan Move 1.0 Move 2.0 Office Scan DeepSecurity v6 MP4 v8 v10 Forefront Endpoint Protection 4. Intelligent signature file update (random based on VM boot) 5. Intelligent engine update (with VDI plug-in) 6. Extend scanning to offline VMs Partial (off by default; VHD only) 7. Update signature files and engines of offline VMs 8. Offload on-demand scanning to a security VM running on the same machine N (future) N (future) 9. Offload on-demand scanning to a security VM running on another machine 10. Intelligent caching of anti-malware scans across VMs Partial (via v10 "Allow List") 11. Hybrid security VM/lightweight agent architecture 12. Full agentless onaccess scanning 13. Agentless File Integrity Monitoring N (future) N (future) N (future) N (future) Source: Gartner (February 2012) Page 4 of 13 Gartner, Inc. G00229651

In Figures 1 and 2, we provide information on shipping solutions. However, nearly every vendor has now publicly stated its intention to deliver virtualization-optimized solutions most of them with a specific option to integrate in the VMware vsphere hypervisor for agentless anti-malware scanning (see Figure 3). The information in Figure 3 is based on information provided by the vendors. Some vendors declined to include information on future versions, and this information is subject to final changes before release. Figure 3. Example of Virtualization Capabilities by Vendor Expected in Future Solutions Capability Vendor and Future Version 1a. Support of the product running in virtualized environments: Support of the product running in a full VM 1b. Support of the product running in virtualized environments: Support of the product running in a terminal service environment 1c. Support of the product running in virtualized environments: Supporting the product's ability to look inside virtualized application containers 1d. Support of the product running in virtualized environments: Support of the product running in a virtualized application container McAfee Move 2.5 (April 2012) Kaspersky Security for Virtualization 1.1 (February 2012) 2. Randomized or staggered scanning 3. Build a whitelist of files from VM image templates 4. Intelligent signature file update 5. Intelligent engine update (non-vmware) for VMware (non-vmware) for VMware 6. Extend scanning to offline VMs 7. Update signature files and engines of offline VMs 8. Offload on-demand scanning to a security VM running on the same machine 9. Offload on-demand scanning to a security VM running on another machine 10. Intelligent caching of anti-malware scans across VMs 11. Hybrid security VM/lightweight agent architecture (non-vmware) for VMware (non-vmware) N (future) 12. Full agentless on-access scanning (for VMware only) 13. Agentless file integrity monitoring Source: Gartner (February 2012) Gartner, Inc. G00229651 Page 5 of 13

We provide an explanation of the importance of each capability: 1. Support of the product running in virtualized environments: Because virtualization can occur at different layers, there are multiple ways a vendor should support virtualized environments: Support of the product running in a full VM: One of the first things an organization should do is ensure that its EPP vendor's solutions run correctly in a virtualized environment and support the major virtualization platforms. This should include virtual server workloads, HVDs and VMs that may run locally on a machine for example, in a Windows VM on Macintosh or on Windows using hosted virtualization, or in a Windows VM on a local desktop hypervisor. Ideally, the ability to be run and fully supported inside a VM would extend to the vendor's EPP management console as well. Support of the product running in a terminal service environment: The organization should ensure that support extends to remote desktop services (presentation virtualization) from Citrix and Microsoft. Nearly all vendors have done this at this point, because x86 terminal service technology has been around for more than two decades. Supporting the product's ability to look inside virtualized application containers: Enterprises are increasingly using application virtualization solutions from vendors such as Microsoft, VMware, Symantec and others as an easier way to package and distribute applications and as a way to manage composite desktop images. Security solutions need to be able to see into these containers to perform real-time malware scanning and other EPP functions, such as application control. Support of the product running in a virtualized application container: It is also possible that the security or management software itself could be containerized using application virtualization technologies to enable the security protection to be more easily composited into a workload image. However, this has not yet been delivered because of the low-level OS integration requirements of security agents required to be able to scan outside of the virtualized application container. 2. Randomized or staggered scanning: Once the agent is running in a VM, most organizations encounter resource contention issues with EPP solutions that are not optimized for virtualized environments. In most cases, this is because all anti-malware scheduled scans are set to start at the same time, which creates memory, CPU and disk contention. To reduce contention, enterprises should look for EPP providers to enable randomization of the scans or grouping of scans of machines to scan at staggered starting times by policy. 3. Build a whitelist of files from VM image templates: Many VMs (especially with HVD) are based on clones or templated images, or they are thinly provisioned from a common base of high assurance images. One way is to include the EPP agent in the base image and initiate a scan to build the initial cache. A better approach is to ingest a gold image to construct a whitelist of files that do not have to be subsequently scanned. Because there is a slight risk that the gold image has been compromised, a periodic scan should be performed of these templates (see No. 6 below). The whitelist should be based on a hash of the file (not file name, path or directory) to reduce the risk that a whitelisted file is compromised and not detected. If Page 6 of 13 Gartner, Inc. G00229651

the solution doesn't support this, the entire system must be scanned periodically; however, this can also be optimized (see No. 10 below). 4. Intelligent signature file update: Like the problem of scheduled on-demand scans initiating at the same time, resource contention is created if all the VMs attempt to update their signature file at the same time (for example, during the boot process). At a minimum, the EPP solution must allow signature file updates to be staggered or randomized. An improved solution would be to have one machine update its signature file and then act as a relay to update the others. The best solution would be to have an architecture optimized to use a single, shared signature file across all engines saving the redundancy of maintaining hundreds of identical files. EPP solutions that have a live cloud lookup mechanism (such as Trend Micro's solutions to its Smart Protection Network) should also have techniques to avoid network resource contention (for example, a relay mechanism such that if one VM asks the cloud about a particular file, it can share that update with other VMs). 5. Intelligent engine update: Most EPP providers have the ability to update their security engines as a way to counter new threats that can't be addressed using signature file updates alone. In the same way that signature file updates need to be updated intelligently (as in the previous section), EPP engine updates need to able to be intelligently distributed as well in similar ways. 6. Extend scanning to offline VMs: Most organizations maintain libraries of offline VM images (for example, templates, clones, snapshots and users with persistent HVD sessions). The antimalware scanning process must be extended to offline VMs. This becomes important to ensure that dormant VMs do not contain embedded malware. Even though the dormant VM may have an agent of its own, it is not actively being updated or scanned. Also, files that were previously scanned and not determined to be malicious may be malicious when the VMs are rescanned with an updated signature set. One solution would be to programmatically mount the VMs in a quarantined state to scan the VMs in a live state. However, a better solution would be for the EPP vendor to be able to understand and crawl the specific VM image format to scan for malware. 7. Update signature files and engines of offline VMs: Offline VMs with EPP solutions installed will be immediately out of date when they boot. This can create a delay in the boot process or responsiveness if the EPP agent is required to update its signature file and/or engine at boot time. One solution would be for the EPP provider to provide a way to update the EPP signature file and/or engine while the VM is offline. For signature files, this could be simply a file replacement so that when the VM is brought up, not only has it been scanned, but it also has the latest engine and set of signature files and rules. Note that this is not an issue with agentless anti-malware scanning (see No. 12 below). 8. Offload on-demand scanning to a security VM running on the same machine: Most EPP offerings provide a combination of on-demand (scheduled) and on-access (real time) scanning for malware. To optimize on-demand scanning, one VM should be responsible for the scheduled scanning of all the VM images. This works for file-based malware (on-demand) but not for on-access scanning. Also, because live VM images are held open, they are not easily accessible from other VMs for scanning. One solution here is to use snapshots of live machines to create a copy that is then scanned for malware. One advantage to placing protection Gartner, Inc. G00229651 Page 7 of 13

capabilities in a separate VM is that the VM can be throttled to limit the overall impact on the server. 9. Offload on-demand scanning to a security VM running on another machine: There is no technical reason that consolidated on-demand scanning has to be performed from a VM running in the same physical server. Any physical server that can access the same shared storage could perform this function. To do this effectively will require integration with tools such as snapshotting and virtual server management tools. 10. Intelligent caching of anti-malware scans across VMs: In addition to the ability to create a whitelist or common cache (see No. 3 above), there are occasions when a set of VMs will need to be completely scanned. Ideally, the EPP solution would support caching the results of VM scans and enable the sharing of this cache across VMs. For example, this approach is used by Symantec Endpoint Protection (SEP) 12's Shared Insight Cache. In this way, the scanning engine can intelligently avoid rescanning the same files against the same signature set. This capability should support online and offline VMs: If the scanning engine knows that a given file has been scanned (typically based on its hash) and the signature set hasn't changed, then don't scan it again. An improved approach would support differential intelligence. If a signature file receives an update, then the engine would be smart enough to only scan differentially with the net new signatures, as opposed to rescanning with the entire set. This approach requires the use of a shared cache, which is coordinated using a security VM, or one of the agents takes on this role. 11. Hybrid security VM/lightweight agent architecture: Some solutions (for example, McAfee's MOVE) have developed a hybrid architecture that combines in-vm agents with a security VM that oversees the scanning across VMs. In this way, the vendor is able to provide optimization of on-demand scanning by using the security VM to coordinate scanning and using the in-vm agent for on-access scanning. Another advantage to this approach is that it can be designed in such a way that it is virtualization-platform-neutral. 12. Full agentless on-access scanning: All the approaches to this point still require a local agent for real-time, on-access scanning as files are accessed. Introspection techniques offer the ability to remove the agent entirely This capability was first commercialized in VMware with vsphere 4.1 and VMware's vshield Endpoint set of APIs. The first vendor to support this was Trend Micro. Since then, multiple providers have announced road maps and intentions to support this, but none have yet shipped. Agentless anti-malware scanning is an extremely useful way to extend protection to VMs that have no protection at all (for example, virtual appliances), and to provide immediate and up-to-date protection from previously dormant VMs. There is a significant amount of marketing hype around this approach and significant interest from clients; 4 however, there are trade-offs to being completely outside the VM that you are trying to protect (see Figures 4 and 5). Page 8 of 13 Gartner, Inc. G00229651

Figure 4. Pros of Agentless Anti-Malware Scanning Anti-malware scanning is orchestrated by a separate security virtual machine (VM), optimizing scanning across VMs, and improving performance and resource virtualization The security VM provides separation of the security controls from the containers it protects, providing tamper resistance. Because there is no vendor-supplied agent installed in each VM, keeping engines and signature files up to date is simplified. When new or offline VMs are booted, they are protected immediately, and there is no local signature file that requires updating. Anti-malware scanning extends security protection to third-party VMs and virtual appliances where anti-malware agents may not have been installed. The security VM can be throttled to restrict its impact on system resources. Source: Gartner (February 2012) Figure 5. Cons of Agentless Anti-Malware Scanning The solution ties you to a specific virtualization platform (currently only VMware offers this capability). It requires VMware's vsphere 4.1 or higher and licensing of vshield Endpoint. It requires code (vshield Endpoint) to be installed on each physical server at the hypervisor layer, which will support this functionality. This is a significant issue when considering the protection of cloud-based workloads in infrastructure-as-a-service offerings where the provider won't provide direct hypervisor access. The solution is not truly agentless. Stub code is supplied by VMware that instruments each VM using VMware Tools to install the code. VMware provides stub code for Windows-based VMs only. There is no option to protect Linux-based workloads. The solution is limited to file-based anti-malware scanning. There's no ability to protect memory. There's no behavioral monitoring, application control or device control. Because malware is detected outside the VM container it is protecting, there is a limited ability to interact with users to let them know why a particular piece of executable code was not allowed to run. Although malware can be quarantined and simple malware can be removed, it is difficult to remove deeply rooted malware that requires multiple simultaneous interactions with the registry, boot record, Windows system files and so on in order to be completely removed. Source: Gartner (February 2012) Gartner, Inc. G00229651 Page 9 of 13

13. Agentless file integrity monitoring (FIM): Introspection provides access to all file, disk, network and CPU activity, and could be used for a large number of information security protection mechanisms beyond just agentless anti-malware scanning. Introduced in August 2011, vsphere 5.0 provided the next generation of VMware's vshield set of APIs, including support for agentless FIM. Trend Micro is the first to support agentless FIM with its Deep Security offering. 14. Agentless data loss prevention (DLP): Agentless DLP scanning is offered only from VMware providers using RSA's (the parent company of VMware and security division of EMC) DLP engine. This capability was introduced with vsphere 5.0 in 2011. Because only VMware provides this, and there is no option to replace the RSA engine that provides this capability, it is not shown in Figures 1, 2 and 3. 15. Agentless application control: This capability is not yet available, so it is not included in Figures 1, 2 and 3. The ability to control what applications are allowed to launch (also referred to as whitelisting) is a powerful, foundational security capability, and we expect VMware to enable this capability by year-end 2012. Tactical Guidelines Migrating server and desktop workloads from physical to virtual does not eliminate the need for endpoint protection. Ensure that pilot programs for HVD include specific testing to surface EPP resource contention issues. As you evaluate EPP solutions, use this framework to ask providers specific questions about their support and optimization for virtualized environments. Explicit support and optimization for virtualized environments must become a mandatory part of any endpoint security tool evaluation. Look for independent, third-party comparison testing of the performance of anti-malware scanning in virtualized environments. Be wary of vendor-sponsored studies. 5 Configuring EPP products for randomization is a first step to reduce the impact on server, network and storage infrastructures. Don't overlook the need to scan offline VM images. Keep these images up to date from a signature file, engine and patch perspective, or consider an agentless approach that helps address these issues. Don't assume that agentless AV is always the best approach. Alternatives using hybrid architectures and intelligent caching also optimize performance without creating hypervisor lock-in. Innovation in this space is rapid, so demand specific road map commitments from your incumbent EPP providers. If your incumbent EPP provider can't deliver, switch vendors even if this means using a different solution to protect physical versus virtualized environments. Page 10 of 13 Gartner, Inc. G00229651

Recommended Reading Some documents may not be available as part of your current Gartner subscription. "Magic Quadrant for Endpoint Protection Platforms" "Radically Transforming Security and Management in a Virtualized World: Concepts" "Radically Transforming Security and Management in a Virtualized World: Considerations" "Tactical Guidelines for Evaluating Virtualization Security Solutions" "VMware Pushes Further Into the Security Market With Its vshield Offerings" "Tactical Guidelines for Evaluating Virtualization Security Solutions" "McAfee Leverages Intel for Deeper Security" "How to Devise a Server Protection Strategy" "What Can Desktop Virtualization Do for our Organization?" "Forecast: Hosted Virtual Desktops, Worldwide, 2010-2014 (2010 Update)" Evidence 1 In "Forecast: Hosted Virtual Desktops, Worldwide, 2010-2014 (2010 Update)," Gartner research projects that the total number of HVDs will grow sharply to 70 million users by 2014. 2 In "Virtual Machines Will Slow in the Enterprise, Grow in the Cloud," we estimated that, at yearend 2011, 49% of the x86 in data centers that could be virtualized had been virtualized. 3 In "Forecast: Hosted Virtual Desktops, Worldwide, 2010-2014 (2010 Update)," Gartner survey data indicates that a reduction in total cost of ownership is the highest priority cited for the adoption of HVD. However, nonoptimized anti-malware scanning will affect the ability of servers to scale to support HVD sessions and will ultimately impact the overall TCO of HVD by increasing hardware requirements beyond the approximately 1.4 to 1.6 times the cost of using physical hardware. 4 Gartner's information security team has received an increasing number of calls looking for solutions to reduce resource contention created by anti-malware scanning in virtualized environments. In a majority of these calls, Trend Micro's early embrace of the agentless approach using introspection was brought up proactively by Gartner clients as a possible solution. However, as discussed in this research, there are alternative approaches, and each approach has its pros and cons. 5 Third-party test reports of anti-malware scanning in virtualized environments are appearing, providing alternative sources of resource utilization information and comparisons, such as www.dennistechnologylabs.com/reports/security/anti-malware/symantec/dtl_sm_vdi.pdf and Gartner, Inc. G00229651 Page 11 of 13

www.slideshare.net/unit4itsolutions/tolly-report. However, some of these tests are vendorsponsored, so confirmation in your own test lab setting with typical user workloads and applications is required. Page 12 of 13 Gartner, Inc. G00229651

Regional Headquarters Corporate Headquarters 56 Top Gallant Road Stamford, CT 06902-7700 USA +1 203 964 0096 European Headquarters Tamesis The Glanty Egham Surrey, TW20 9AW UNITED KINGDOM +44 1784 431611 Japan Headquarters Gartner Japan Ltd. Aobadai Hills, 6F 7-7, Aobadai, 4-chome Meguro-ku, Tokyo 153-0042 JAPAN +81 3 3481 3670 Latin America Headquarters Gartner do Brazil Av. das Nações Unidas, 12551 9 andar World Trade Center 04578-903 São Paulo SP BRAZIL +55 11 3443 1509 Asia/Pacific Headquarters Gartner Australasia Pty. Ltd. Level 9, 141 Walker Street North Sydney New South Wales 2060 AUSTRALIA +61 2 9459 4600 2012 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This publication may not be reproduced or distributed in any form without Gartner s prior written permission. The information contained in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. This publication consists of the opinions of Gartner s research organization and should not be construed as statements of fact. The opinions expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company, and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartner s Board of Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartner research, see Guiding Principles on Independence and Objectivity on its website, http://www.gartner.com/technology/about/ ombudsman/omb_guide2.jsp. Gartner, Inc. G00229651 Page 13 of 13