Virtual Perimeter Extending IPS and Firewall Protection Internal to the Network

White Paper February 2005 McAfee Network Protection Solutions Virtual Perimeter Extending IPS and Firewall Protection Internal to the Network

Extending IPS and Firewall Protection Internal to the Network 2 Introduction Perimeter Firewall Necessary But Not Sufficient 3 3 Internal Firewall Extending Protection Inside the Perimeter 4 McAfee IntruShield Pioneering Internal Firewall + IPS Integration 4 Virtualization The IntruShield Approach 6 Enabling the Virtual Perimeter 6 Typical Applications of the Virtual Perimeter 7 Benefits of Virtual IPS 8 Conclusion 9

Extending IPS and Firewall Protection Internal to the Network 3 Introduction Whilst most organisations would not consider connecting their corporate networks to the Internet without a properly-configured firewall in place, more advanced forms of intrusion detection and prevention - such as Intrusion Prevention Systems (IPS) - are relatively new. To date, traditional IDS and IPS appliances have been considered as completely separate from firewalls, but essentially - at a low level - they perform very similar tasks. For example, they each have multiple interfaces, and are designed to be installed in-line between trusted and untrusted networks or subnets. But there the similarity ends. Firewalls are essentially policy enforcement devices, designed to match traffic against a set of access control rules, and accept or deny traffic based on those rules. For example, a firewall could allow all FTP traffic to one particular server on the DMZ, but deny FTP to any other machine. On the other hand, IPS devices are designed to detect potential exploit traffic rather than enforce policy. For example, if the firewall has already allowed through the FTP traffic destined for the FTP server, the IPS device now watches that FTP traffic for suspicious patterns. Whenever an IPS device detects suspicious traffic it is capable of dropping the packet immediately and blocking the rest of the suspicious flow, thus preventing suspicious traffic from entering the protected network. Given that at a hardware level, a firewall and an IPS appliance look very similar, it seems only logical that the functionality should converge. It is a natural evolutionary step for the IPS device to offer firewall features, since this allows it to be used not only at the perimeter, but also within the network core, providing both IPS and firewall capabilities without the need to install, configure and manage two separate appliances at multiple points throughout a corporate network. Perimeter Firewall Necessary But Not Sufficient It is apparent that firewalls are not always effective against many intrusion attempts. The average firewall is designed to deny clearly suspicious traffic such as an attempt to telnet to a device when corporate security policy forbids telnet access while allowing some network traffic such as Web traffic to an internal Web server to pass through. The problem, however, is that many exploits attempt to take advantage of weaknesses in the very protocols that are allowed through our perimeter firewalls. Once the Web server has been compromised, this can often be used as a springboard to launch additional attacks on other internal servers. For example, the shallow inspection performed by most firewalls today would be incapable of detecting an attempted buffer overflow contained within what would appear to be perfectly normal Web traffic passing through port 80. This makes the perimeter defences of the typical corporate network very porous - we might think we have installed an impermeable hard shell, but there are many small holes which might allow through more than we bargained for. Standard Firewall Configuration Remote DMZ Public Firewall Switch Private Executive Servers The inadequacies inherent in current defenses have Perimeter firewall provides a hardened shell against external attacks. Internal driven the development of a new breed of security users have unrestricted access to corporate assets. products known as Intrusion Prevention Systems (IPS). These systems are proactive defenses mechanisms designed to detect malicious packets within normal network traffic (something that the current breed of firewalls do not actually do, for example) and stop intrusions dead, blocking the offending

Virtual Perimeter Extending IPS and Firewall Protection Internal to the Network 4 traffic automatically before it does any damage rather than simply raising an alert as, or after, the malicious payload has been delivered. As with a typical firewall, the network IPS (NIPS) has at least two network interfaces, one designated as internal and one as external. As packets appear at either interface they are passed to the detection engine. At this point, the IPS device functions much like any IDS would in determining whether or not the packet being examined poses a threat. However, if it should detect a malicious packet, it will raise an alert while discarding the packet and marking that flow as bad. As the remaining packets that make up that particular TCP session arrive at the IPS device, they are discarded immediately. Legitimate packets are passed through to the second interface and on to their intended destination. The traditional perimeter firewall is still necessary, but is no longer enough. The next step is to integrate the firewall capability with the IPS in order to extend the firewall protection inside the perimeter and harden the core of the network. Internal Firewall Extending Protection Inside the Perimeter Firewalls are typically employed only at the network perimeter. However, many attacks intentional or unintentional are launched from within an organisation. Virtual private networks, laptops, and wireless networks all provide access to the internal network that can often bypass the firewall. As a result, the typical corporate network could be thought of as having a hard shell but a soft center. Internal Firewall Configuration Remote DMZ Public Switch Firewall Private Executive Due to the high cost of deploying expensive firewall devices, firewalls are not commonly deployed inside the network perimeter. This is especially true for points in the Servers network that are already candidates for IDS and/or IPS appliances. In addition, the increase in the complexity IntruShield Manager v2.1 provides an internal firewall solution to block and burden on the administrator is an equal disincentive malicious traffic and enforce security policies internally with its virtual firewall for internal firewall deployment. This is exemplified in the technology. fact that most firewalls are required to have different subnet addresses on each interface, thereby necessitating costly and complex internal network renumbering when deployed inside the perimeter. What is needed in the enterprise is a low-cost alternative to the traditional firewall that can provide all the power of Access Control Lists (ACLs) to filter and permit/deny packets based on IP address, port number and protocol while eliminating the more costly firewall elements (both in terms of cost, and deployment effort) such as Network Address Translation (NAT) and Virtual Private Networks (VPN). McAfee IntruShield Pioneering Internal Firewall + IPS Integration Given that, at a hardware level, a firewall and an IPS appliance look very similar, it seems only logical that the functionality should converge. It is a natural evolutionary step for the IPS device to offer firewall features. This will allow it to be used not only at the perimeter, but also within the network core to provide both IPS and firewall capabilities without the need to install, configure and manage two separate appliances at multiple points throughout a corporate network.

Extending IPS and Firewall Protection Internal to the Network 5 Moving the firewall inside the network has considerable advantages when the firewall is also part of the IDS/IPS appliance. As such, McAfee IntruShield s next-generation intrusion prevention technology pioneers the integrations of IPS and internal firewall functions. Since IPS appliances tend to be deployed far more extensively within the network perimeter than firewalls, they are better placed to provide enhanced protection between internal networks, subnets and VLANs. A huge advantage of having IPS devices providing firewall capabilities is that such devices are normally installed as a bump in the wire. This means that it is neither necessary nor desirable to allocate IP addresses to the various interfaces on the device. As a result, no internal IP addresses need to be altered, and routing issues are not a concern. This is in stark contrast to traditional firewall devices that usually necessitate the major reworking of IP address ranges when being installed inside the perimeter. In addition, the provision of multiple ports on a single appliance provides the means to firewall traffic between a number of different networks using a single device. This offers a more scalable solution that is ultimately easier to manage, and incurs lower costs than, multiple firewalls. With firewall capabilities integrated into an IPS appliance, the opportunity arises to impose far more granular security policies within the corporate network. For example, consider a typical perimeter firewall device that can be configured to allow inbound SMTP and outbound HTTP traffic. Although it is possible to create rules that provide more restrictive access, this is often not done because of potential problems with enforcing an overly strict rule-set across a large corporate network. A single misplaced rule could eliminate crucial access to an important service for an entire group of users. An IPS, on the other hand, is often deployed within the core of the network and is placed in-line between multiple subnets. As such, it is possible to enforce different security policies for each pair of ports and thus a different security policy for each subnet. Outbound HTTP traffic can therefore be permitted from all client machines, while being denied from HTTP servers, thus preventing the propagation of harmful Trojans should an HTTP server be infected. Likewise, the inbound SMTP traffic can be denied to all machines other than the mail server. Placed at key points within the core network (and even in front of the perimeter firewall, if required), the IPS appliance with firewall capabilities is far more capable of handling Denial Of Service (DOS) attacks, since it can filter out specific types of malicious packets at a low level and simply discard them without subjecting them to further analysis or processing. Likewise, it is far more capable of handling nuisance traffic such as Peer-to-Peer file sharing within the network perimeter. Nor is it necessary to rely on the IPS device for the prevention of purely malicious traffic. Instead, by simply applying ACL rules that restrict access to certain resources from specific parts of the corporate network, it can also be used to enforce generic security policy. For example, a typical requirement might be to ensure that only the HR subnet has access to the HR databases. Rules can be defined to ensure that no access is allowed to the HR server from any other host on the network other than those within a specific IP address range. As well, access to the development servers can be restricted purely to those machines on the development network, and so on. Therefore, even if attackers were able to compromise an internal server from which they were able to gain access to the rest of the network, it would not be possible for them to access files within the HR or development subnets that are protected by the internal IPS/firewall combination. The provision of firewall capabilities on the IPS device has one other important advantage, especially when deployed in the core of the network where high speed data transfer often up to Gigabit speeds and beyond is much more likely than at the perimeter. When it is possible to identify a subset of the traffic that is passing through a particular device for example, if it is known that a particular appliance should never see any HTTP traffic then it is possible to filter out that traffic at a lower level in the appliance, thereby preventing it from being analyzed by the IDS/IPS detection engine. This can have a dramatic effect on the performance of an IPS device. Given that the analysis of packets and streams for malicious traffic is the most difficult task it has to perform, the elimination of a significant portion of that traffic based on simple firewall rules can help to ensure that the detection engine is never stressed. Once again, this can have the effect of improving scalability of the overall security solution and allowing much higher total levels of traffic to be handled by all the IPS devices on the network.

Extending IPS and Firewall Protection Internal to the Network 6 Virtualization The IntruShield Approach Virtualization is at the core of the IntruShield IPS product. It has been designed this way from the ground up, and the virtualization features are designed and built into the hardware architecture. This ensures maximum performance in high-bandwidth (multi- Gigabit) deployments, since there are no complex features which have been bolted on in the software as an afterthought. This is the only way that an IPS device can support such large numbers of Virtual IPS (VIPS) in a single appliance without compromising either security or performance. Traditionally, there is a trade-off in the security world between security strength and performance increasing one always decreases the other. IntruShield s advanced hardware design utilizing custom-designed ASICs eliminates this problem. Multi-Gigabit traffic levels can be handled easily, even when all signatures and features are enabled, and when supporting 1000 VIPS per device. With IntruShield, Virtual IPS domains can be set up for specific departments, geographic locations or functions within an organization. Security policies can then be set for each Virtual IPS. The VIPS functionality can be implemented in three ways: 1. By attributing Virtual Local Area Network (VLAN) tag(s) to a set of network resources 2. By protecting a block of IP addresses utilizing Classless Inter-domain Routing (CIDR) blocks 3. By dedicating IntruShield system interfaces to protect the network resources in particular department, geography or organizational function. It is possible to create sub-interfaces beneath an interface node, or VIPS nodes within child domains both of these are different manifestations of the Virtual IPS and allow the allocation of multiple policies to the same physical interface. A VIPS within a child domain can be allocated to an administrator who only has rights to that child domain and nothing else. Thus, when the administrator logs in, he or she will be able to configure and allocate policies to the VIPS under his control without affecting any other interfaces or VIPS in the system. CIDR-based VIPS implementation allows granularity down to an individual host level. For example, DoS attacks can be identified and responded to with unique policies for individual hosts. Virtualisation permits the separation of a single physical IPS device into multiple logical devices based on port, IP address range, VLAN tag, right down to an individual host if required. Where there are multiple segments to monitor or a need to monitor aggregated traffic like on Gigabit uplinks, for example a multi-port box and more granularity in the inspection process can make for a much more cost-effective and efficient solution. The IntruShield Virtual IPS (VIPS) feature achieves this by enabling an administrator to configure multiple policies for multiple unique environments, all monitored with a single IntruShield sensor. For example, suppose one port of a sensor is connected to the SPAN port on a switch and is configured with a specific environment detection policy. The rest of the ports on the sensor could then have policies completely different from the policy on the SPAN port. Or, if required, or they can use the same policy. In addition, each port could be segmented by multiple VLAN tags or CIDR addresses, each with its own customized security policy. Enabling the Virtual Perimeter By combining the Virtual IPS and Virtual Firewall capabilities, it is now possible for enterprises to create powerful and innovative Virtual Perimeters within the core of the network. Virtual Perimeter technology and brand new concept unique to McAfee IntruShield delivers highly granular and customised protection for a single network segment, collection of hosts, or even a IntruShield Virtual Perimeters Remote DMZ Public Firewall Switch Private Executive CIDR based Virtual Perimeter IP Address Based Virtual Perimeter Servers IntruShield Virtual Perimeters provide individual security zones for systems or groups of systems. Unique Firewall and IPS policies can be created and enforced for each perimeter providing the ultimate in protection

Extending IPS and Firewall Protection Internal to the Network 7 single host. By defining multiple Virtual Perimeters inside the physical network perimeter, protection is offered against both internal and external threats. In addition, perimeter-grade protection is extended to the core of the network, thereby enhancing security by allowing creation and deployment of much more granular security policies. Because the IPS is designed to be deployed throughout a corporate network rather than simply at the perimeter, it is often desirable to be able to deploy multiple security policies on a single box. Further yet, it may even be desirable to deploy multiple security policies on a single port, where a group of separate clients or subnets are sharing a single port (or port pair, when operating in-line). McAfee IntruShield provides this capability, allowing the administrator to apply unique security policies for a range of IP addresses on a port, right down to an individual host if necessary. Thus multiple Virtual Perimeters can be deployed within a physical network perimeter, allowing a far more scalable and fine-grained security policy to be deployed than is possible with traditional perimeter-based firewall solutions. This is essential since, more often than not, a single security policy is simply too restrictive to apply across an entire corporate network. This means that many organisations are required to purchase and install a number of point security products both firewall and IPS/IDS in order to achieve the granularity of control that they require. This not only significantly increases the initial acquisition and deployment costs, but also increases the costs of post-install administration and configuration. The Virtual Perimeter concept eliminates restrictive global one size fits all security policies and helps to reduce deployment and management costs by logically segregating the traffic passing through an IPS appliance and applying multiple discrete security policies. Policies can be applied for each network, subnet, or even to each separate host being protected by the device. This is known as virtualisation, and provides increased control and flexibility in defining security policies, while actually reducing the number of devices that need to be deployed. This is an extremely effective strategy in light of the gradual erosion and possible disappearance of the traditional network perimeter, and to date McAfee is the only company to have implemented this feature. Typical Applications of the Virtual Perimeter Combining the VIPS with the integrated firewall capability enables the establishment of multiple Virtual Perimeters that are secured by the same appliance. For example, different policies can be applied to different interfaces allowing one pair of interfaces to monitor the DMZ with a predominantly Web-based policy in in-line mode, while another interface monitors the internal network in SPAN mode using the Default policy. Typical applications of this technology include: In front of the perimeter firewall to filter DOS attacks The stealthier firewall that is part of the IPS appliance is invisible to attackers, and thus cannot be compromised. Behind the perimeter firewall A Virtual Perimeter can be deployed throughout the internal network to restrict access to valuable resources. For example, access to HR servers can be restricted to hosts on the HR subnet Network core policy enforcement This can enforce a more granular security policy. For example, it becomes easier to restrict certain forms of traffic from specific hosts or subnets, thus preventing propagation of Trojans or the use of nuisance applications such as Peer-to-Peer file sharing which can consume valuable network bandwidth. Internal environments Ideal for internal environments where NAT is typically not required or desired. For example, because no IP address is assigned to the ports of the IPS appliance, there is no affect on the IP addresses of networks on each side of the device, and no routing problems to account for Dropping malicious traffic Drop valid, non-malicious traffic that violates security policy. For example, if Telnet is not allowed throughout the network, it can be disabled at each IPS device and all Telnet traffic will be dropped without even

Extending IPS and Firewall Protection Internal to the Network 8 having to pass it to the IPS inspection engine. The addition of firewall functionality make the IPS appliance a broader policy enforcement device. Maximizing performance Maximize the performance of the IPS appliance by eliminating both nuisance traffic and traffic that does not conform to security policy at a lower level without passing to the inspection engine. The inspection engine is therefore able to spend more processing cycles on inspecting normal traffic flows. Multiple Virtual Perimeters The administrator is able to create multiple Virtual Perimeters within a physical network perimeter, thus allowing a far more fine-grained security policy to be deployed than is possible with traditional perimeterbased firewall solutions. For example, it would be possible to allow outbound Web access for all desktops but block SMTP to avoid dangers of a worm which may have its own SMTP engine. This also offers a much more scalable solution that is not only easier to manage than multiple firewalls, but helps to significantly lower costs. Port blocking Blocking certain ports and/or protocols throughout internal networks in the event of a worm outbreak. This could offer very powerful protection against internal worm propagation until patches are deployed Bandwidth management Identify critical delay-sensitive voice and video data and use the firewall to pass this directly through to the protected environment, bypassing the IPS detection engine completely and thus minimising latency Benefits of Virtual IPS Virtualisation offers a number of clear benefits, including: Granular security policies This provides enhanced security and control through the ability to tailor a particular policy to the requirements of the traffic passing between two networks, two subnets, or even two hosts. Unlike other solutions on the market, IntruShield is capable of providing this granularity without having to deploy multiple devices. This level of granularity can be applied at device level, VLAN level, port level, subnet level, or even to individual hosts if required. Granular administrative control With the ability to define administrative domains and apply them in line with the Virtual IPS structure, it is possible for each administrator to see only that portion of the device which is directly under his control. Thus, it is impossible for the administrator responsible for the HR systems to accidentally (or intentionally) reconfigure or destroy security policies for the finance department. It also ensures that reports and real-time views are all directly applicable to each administrator, and do not contain data from other VIPS. Reduced costs This comes mainly from two sources: 1) the reduction of the number of appliances which are deployed due to the VIPS capability that allows an administrator to support multiple security policies and multiple networks on a single device, 2) the reduction in management costs attendant with the smaller number of devices and the ability to configure and deploy all security policies and devices from a single, central point if required. As well, the granular admin control features can provide a far simpler and more directly-relevant view of the security infrastructure for each administrator, thus making the job simpler of each of them. Managed services The VIPS capability provides the means to logically segregate traffic within an IntruShield appliance to ensure that a complete managed IPS service can be offered to multiple clients using a single device without compromising the integrity of neither the customers network or overall security. Each client would be contained within a virtual perimeter secured by a Virtual Firewall and protected by one or more unique, customized security policies. One or more ports can be allocated to a client, or it is even possible to support multiple clients on a single in-line port pair by defining a VIPS for each one based on IP address ranges or VLAN tags. Enhanced security The Virtual Firewall capability brings firewall functionality to the core of the network, allowing the administrator to define multiple virtual perimeters inside the physical network perimeter and inside the protection of the traditional firewall device. Since each firewall can be configured to handle precisely the traffic it is expected to see in that part of the network, control can be much greater. This improves overall security. In addition, the security policies can also

Extending IPS and Firewall Protection Internal to the Network 9 be applied at a much more granular level than would normally be available with a traditional IPS appliance. Each policy can be defined to include signatures and settings which directly pertain to the traffic that will be seen by the specific VIPS, thereby allowing for irrelevant signatures to be omitted. Enhance performance Thanks to the Virtual Firewall capability, the detection engine of each IntruShield appliance has less traffic to deal with. This ensures that processing power is reserved for relevant traffic only, and thus helps to increase the overall bandwidth of genuine traffic that can be transmitted through each device on the network. Where traffic can be directly identified as harmless by the firewall module, it can also be passed through the device with no analysis, thus decreasing latency on delay-sensitive traffic (such as voice or video). Increased accuracy The granular security policies ensure that irrelevant or noisy signatures can be omitted from a VIPS. If it is determined that a particular network has only Solaris Web servers, for example, then all IIS signatures can be omitted. This results in a huge reduction in false positives, increasing the efficiency of the administrator and reducing the amount of time that would need to be spent on forensic analysis. Mixture of IDS and IPS on a single device The ability to support both IDS and IPS functionality on different ports on the same device greatly increases the flexibility of the solution, further reducing costs by eliminating the needs for separate appliances for the two capabilities. Scalability Since virtualization has the effect of: 1) reducing the number of physical devices which need to be installed, 2) increasing the number of policies that can be deployed on each device (up to 1000 VIPS are supported on each appliance), 3) enhancing the overall security via the use of granular security policies and virtual firewall perimeters, 4) reducing the number of false positives (thus reducing the need for forensic analysis) and 5) reducing the management burden on the administrator, the ability for a small group of administrators or even a single administrator to handle a large deployment is greatly enhanced. Conclusion The ability to create multiple Virtual IPS in a single appliance represents a major milestone in the Network Intrusion Prevention field. McAfee IntruShield is the first system to offer this capability to provide enhanced security and a more scalable and costeffective IPS/firewall solution. Via the application of multiple discrete security policies in a single device, this delivers highly granular and customised protection for a single network segment, collection of hosts, or even a single host. By defining multiple Virtual Perimeters inside the physical network perimeter, the IntruShield system provides the following advantages: Increased flexibility IPS/IDS and firewall policies can all be deployed on a single appliance, and multiple policies can even be supported on a single port More granular control Security policies can be tailored exactly to suit the traffic which is crossing each Virtual Perimeter Enhanced security coverage Protection is offered against both internal and external threats, perimeter-grade protection is extended to the core of the network, and false positives are reduced by allowing creation and deployment of much more granular security policies. Reduced capital outlay Fewer devices are required because each appliance can support up to 1000 Virtual IPS. Subsequent management burden is also reduced as a result. Enhanced performance The integrated firewall is also capable of enhancing the performance of the IPS appliance by removing irrelevant, nuisance, or clearly malicious traffic at a much lower level, thus eliminating the need for that traffic to be passed to the detection engine.

Extending IPS and Firewall Protection Internal to the Network 10 Minimizing latency Delay-sensitive traffic, such as voice and video data, can benefit from this approach since the integrated firewall can detect such traffic and pass it straight through to the protected zone, bypassing the detection engine completely and thus minimizing latency. McAfee is the only company to offer Virtual IPS, Virtual IDS and Virtual Firewall capability in a single device, and the only company to support the concept of the Virtual Perimeter. McAfee, Inc. 3965 Freedom Circle, Santa Clara, CA 95054, 888.847.8766 McAfee and/or additional marks herein are registered trademarks or trademarks of McAfee, Inc. and/or its affiliates in the US and/or other countries. The color red in connection with security is distinctive of McAfee brand products. All other registered and unregistered trademarks herein are the sole property of their respective owners. 2005 McAfee, Inc. All Rights Reserved. 6-NPS-VPR-002-0205