A Guide to Domain Isolation for Security Architects

Transcription

1 A Guide to Domain Isolation for Security Architects Microsoft Corporation Published: March 28, 2005 Author: Jim Holtzman Editor: Scott Somohano Abstract Designed for network architects of enterprise organizations that are investigating using IPsec in Microsoft Windows to deploy domain isolation, this white paper explains how to assess the enterprise environment, plan domain isolation, and assess the implications of deploying domain isolation in an enterprise environment. Read this guide after you have developed a working knowledge of domain isolation.

2 This is a preliminary document and may be changed substantially. The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication. This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. Unless otherwise noted, the example companies, organizations, products, domain names, addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, address, logo, person, place, or event is intended or should be inferred Microsoft Corporation. All rights reserved. Microsoft, Active Directory, Windows, Windows NT, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.

3 Contents A Guide to Domain Isolation for Security Architects... 5 Knowledge Prerequisites... 5 Determining the Current State of Your IT Infrastructure... 6 Identifying Current State... 6 Network Segmentation... 6 Network Infrastructure Devices... 7 Network Communication... 8 Mapping Active Directory Infrastructure... 8 Host Configuration... 9 For More Information Planning Domain Isolation Trust State Trustworthy Untrustworthy Determining Isolation Groups Communication Paths and IPsec Policies Next Steps Considering Operational Issues for IPsec-Secured Traffic The Impact of IPsec on Performance Network Performance Processor Performance Crossing Security Boundaries With IPsec Traffic NAT Traversal Creating Firewall Filters to Permit ISAKMP, AH, and ESP Traffic Different Trust Environments Operational Considerations Non-IPsec-compatible Support NLB Clusters Network Inspection Technologies Summary See Also... 22

4

5 A Guide to Domain Isolation for Security Architects 5 A Guide to Domain Isolation for Security Architects The goal of domain isolation is to isolate trustworthy computers from untrustworthy computers by providing an additional level of protection for the traffic that is sent on a network. By using IPsec you add an additional defense-in-depth layer to the mechanisms that currently secure your network. IPsec-secured communication changes the format of the TCP/IP packets transmitted over your network, which requires careful study and planning to mitigate risks and ensure success. This paper presents an overview of the three steps a security architect should follow to prepare for IPsec deployment: 1. Determine the current state of your IT infrastructure 2. Plan isolation groups 3. Consider operational issues for IPsec-secured traffic Note: This is preliminary documentation and subject to change. Knowledge Prerequisites Prior to reading this paper you should read Domain Isolation with Microsoft Windows Explained at In addition, make sure that you have a thorough understanding of the following topics before you begin IPsec domain isolation: TCP/IP and core networking principles. Microsoft Windows Authentication, including use of the Kerberos V5 protocol and public key infrastructure (PKI). Active Directory concepts, including Active Directory structure and tools; manipulating users, groups, and other Active Directory objects; and use of Group Policy. Windows system security, including security concepts such as users, groups, auditing, and access control lists (ACLs). A thorough understanding of Windows Server 2003 IPsec policies. For foundational material, see the Windows Server 2003 Technical Reference at

6 A Guide to Domain Isolation for Security Architects 6 A good understanding of interoperability issues. See Interoperability Considerations for IPsec Server and Domain Isolation at Determining the Current State of Your IT Infrastructure Before starting the planning process for a domain isolation project, you need to collect and analyze up-to-date information about the current state of your IT infrastructure. A thorough and accurate assessment of your infrastructure provides the basis for planning what to isolate and how isolation should be enacted. Your assessment should consist of three main components: network configuration, Active Directory, and host configuration. Identifying Current State Determining the current state of your network involves gathering information about network segmentation and infrastructure devices and performing an analysis of network communication. Network Segmentation The question to answer is, What is the physical layout of my network? You need to determine what networks are in place and what hosts exist on those networks. This information can be presented in a number of ways but is often most effectively illustrated through schematic diagrams. Figure 1 illustrates a very simple network diagram.

7 Figure 1 An Example Network Diagram A Guide to Domain Isolation for Security Architects 7 Network Infrastructure Devices Next in your assessment, review the devices that make up the network infrastructure - routers, switches, load balancers, and firewalls. These devices are required to process IPsec-secured traffic after the isolation solution is implemented. For this reason, it is important to determine whether these devices can handle the technical and physical requirements of the isolation design. Some of the characteristics to examine include: Response to datagram change. IPsec without encryption changes the offset for the destination ports and protocols within the TCP/IP packet. These changes might limit the functionality of traffic analysis applications like Cisco NetFlow. Normal and peak utilization. Peak use information can help determine whether the device is capable of using IPsec or if a memory or firmware upgrade is needed. If problems arise after IPsec is implemented, normal utilization data can assist in determining if the cause is related to high utilization of the device. Whether the device is performing network address translation (NAT). To allow IPsec-secured traffic to pass through a NAT device, you must ensure that IPsec NAT-T is supported on your IPsec peer computers. Internal vs. external subnets and firewall configuration. When a firewall or filtering router exists between IPsec peers, it must be configured to forward IPsec traffic on UDP source and destination port 500 for Internet Security Association and Key Management Protocol (ISAKMP), UDP source and destination port 4500 (IPsec NAT-T), IP protocol 50 for Encapsulating Security Payload (ESP), or IP protocol 51 for Authentication Header (AH).

8 A Guide to Domain Isolation for Security Architects 8 For specific considerations for network infrastructure devices, see Considering Operational Issues for IPsec-Secured Traffic" in this paper. Network Communication A thorough analysis of the communication flow on your network will provide two vital pieces of information: What communication is currently taking place between trusted and untrusted hosts What communications are at risk of failing when secured with IPsec When you examine traffic flow, look closely at how all network devices interact. Keep in mind the following questions: How do servers and clients communicate with each other? Are there any security devices or projects that are implemented or planned that could impact the isolation project? Are certificates used to provide authentication? Do you have subnets that are dedicated to specific types of traffic (for example, Macintosh workstations and UNIX workstations, or mainframe-to-terminal connections)? Mapping Active Directory Infrastructure After the network, Active Directory is the second most important source from which to gather information. You should understand the forest structure, including domain layout, organizational unit (OU) architecture, Group Policy, and site topology. This information provides the logical placement of computers, their configuration, and the impact of making changes to Active Directory as a result of implementing IPsec. For Active Directory, determine: Number of forests. Because the forest is the security boundary in Active Directory, it is necessary to understand the current Active Directory architecture to determine which hosts should be isolated and how to accomplish that isolation. Cross-forest IPsec communication is possible if external trusts are configured. Names and number of domains. IPsec using Kerberos V5 authentication assumes that computers are joined to any domain that is accessible via a two-way trust between the hosts that are attempting to secure their communications.

9 A Guide to Domain Isolation for Security Architects 9 Number and types of trusts. Trusts affect the logical boundaries of isolation and can define how IKE authentication occurs. Active Directory trusts can affect Kerberos-based IKE authentication. Names and number of sites. Site architecture should be aligned with network architecture. OU structure and Group Policy. Organizational units can be used to apply IPsec policy. Review the OU structure to determine how Group Policy is used and how OUs are organized. Existing use of IPsec policy. Multiple IPsec policies that are assigned in dfferent Group Policy objects (GPOs) are not merged. The IPsec policy applied is the one assigned to the GPO that is closest to the computer in the Active Directory domain structure. Host Configuration The final piece of your current network environment assessment is a review of client and server host computers. IPsec-based communication depends on host systems meeting specific operating system and application requirements. Useful host information to gather includes: Operating system, service pack, and hotfix versions. The operating system version is a key factor in determining the ability of a host to communicate with IPsec. It is also important to track the current state of service packs and hotfixes that can be installed because these are often used to determine that the minimum security standards have been met. Domain membership. This information is used to determine whether a computer can obtain IPsec policy from Active Directory or whether it will need to use a local IPsec policy. Non-domain-joined hosts can use a local IPsec policy applied via scripts. Physical location. This information is simply the location of the device in your organization. It can be used to determine whether a device should participate in a particular isolation group based on its location or the location of the hosts that it communicates with regularly. Hardware type and role. This information could be used to determine the eligibility of a particular computer to participate in isolation and in which isolation group to place the computer.

10 For More Information A Guide to Domain Isolation for Security Architects 10 Various methods can be used to gather data from the devices on your network. These methods range from high-end, fully automated systems to completely manual data collection. Generally, the use of automated methods to gather data, such as Microsoft Systems Management Server, is preferred over manual methods for reasons of speed and accuracy. For more information about how SMS 2003 can help perform asset management, see the demonstration and the asset management datasheet on the SMS 2003 Asset Management page at For more information regarding determining your current network configuration, see "Chapter 3: Determining the Current State of Your IT Infrastructure" available at Planning Domain Isolation With a good understanding of your company s network and host configuration, you are ready for the next step, planning isolation. Allowing or blocking host communication on the basis of a trust relationship requires the determination of trust state. Trust for your enterprise is likely delineated by a well-defined corporate security policy that may involve virus, patch, and password considerations. Trust State When describing IPsec hosts in this paper, "trust" does not refer to Active Directory trusts. This paper defines the trust state of a host computer as follows. Trustworthy A host computer can be considered trustworthy when it meets the following criteria: Operating system: A trusted host must run an operating system that is IPseccapable such as Windows 2003, Windows XP, or Windows 2000 SP4. Domain membership: A trusted host must belong to an IPsec domain which means it is capable of participating in Active Directory and processing GPOs. Untrustworthy Any host that cannot meet the criteria for a trusted state is untrustworthy. Examples of host computers that are untrustworthy include: Computers running Windows Millennium Edition

11 Computers running Windows 98 Computers running Windows 95 Computers running Windows NT A Guide to Domain Isolation for Security Architects 11 Non-Windows-based computers and devices, such as Macintosh, UNIX, and Pocket PCs. Computers not joined to a trusted domain Other non-microsoft remote access or VPN clients It is important to note the classification of untrustworthy does not reflect on the use of the operating system in general. Rather, untrustworthy is used strictly in relation to the implementation of domain isolation in this simplified presentation. After you have determined the trust state of the hosts in your domain, the next step is to combine hosts with the same level of trust requirements into logical groups for the purpose of planning the network traffic allowed between groups. Determining Isolation Groups The primary goal of a domain isolation plan is to restrict the communication between trustworthy and untrustworthy hosts. For planning purposes, it is useful to combine hosts into logical groups called isolation groups. An isolation group is a conceptual device for grouping host computers that share the same inbound and outbound network traffic requirements. In Windows, isolation groups are implemented by using security groups in Group Policy. An isolation group s inbound and outbound access controls can be implemented by IPsec policy using permit/block actions, or by using IPsec policy in combination with Group Policy network access rights. Isolation groups are ultimately used to design the secured communication paths of your network. Grouping is based on host IPsec capability and security/trust access requirements. Figure 2 illustrates a simple isolation group design.

12 Figure 2 An Example Isolation Group Design A Guide to Domain Isolation for Security Architects 12 In this example, four groups are created for domain isolation: Isolation domain. This is the default group for all trusted computers and will likely be the group with the largest number of hosts. Membership in this group is the same as the membership in the Windows domain. If the domain includes bidirectional trusted domains, members of those domains would be part of the isolation domain. All hostto-host communication within this group is secured by using IPsec. Untrusted systems. This group contains all untrusted computers not capable of IPsec communication, such as Macintosh and UNIX computers. Boundary isolation. This group contains trusted hosts that will be allowed to communicate with untrusted and trusted hosts. These computers will be exposed to a higher level of risk because they can receive incoming communications directly from untrusted computers. For example: Remote Installation Services (RIS) servers which support downloading operating system images to computers not yet IPsec capable. Exemption lists. This group contains computers that cannot be secured with IPsec or should not be secured with IPsec but must be available to all systems on the network. For example: Domain controllers, and DNS, DHCP, and WINS servers. All computers on your network that share the same communications security policy are assigned to one of the isolation groups. After the groups are designed and host membership decided, you re ready to plan the network traffic between the isolation groups.

13 A Guide to Domain Isolation for Security Architects 13 Communication Paths and IPsec Policies At this point in the planning process, the communications traffic requirements that will be allowed to pass between the groups need to be documented, along with the form the communications will take. Figure 3 illustrates communication paths for the isolation groups. Figure 3 Example Communication Paths After IPsec policies are in place for the isolation groups, communication between the groups will proceed as shown in the following table. Path Groups Communication 1 From untrusted hosts to hosts in the boundary group 2 From boundary group hosts to untrusted hosts Unsecured, clear text TCP/IP connections. Hosts attempt to communicate with IPsec-secured traffic but can fall back to non-ipsecsecured communication (the Fall back to clear option).

14 A Guide to Domain Isolation for Security Architects 14 3 Between hosts in the isolation domain and boundary groups 4 From isolation group hosts to untrusted hosts 5 From untrusted hosts to trusted hosts Hosts attempt to use IPsec to protect the traffic. This traffic might require encryption, depending on the security requirements. If the IKE negotiation fails, the communications will fail because there is no Fall back to clear option when IKE negotiation fails. Hosts attempt to communicate with IPsec-secured traffic but will fall back to non-ipsec communication. Blocked from initiating communication. Though not included in the diagram or table, all hosts are allowed to communicate with hosts in the exemption lists group using unsecured, plaintext connections. Next Steps With a complete understanding of your network and a plan for controlling traffic with isolation groups, you are now ready for the next steps in your utilization of IPsec to secure your network. These steps include: Create IPsec policies. Creating IPsec policies involves creating filter lists, filter actions, and IPsec policies consisting of rules that match filter lists with filter actions. For a thorough discussion of these tasks, see "Chapter 5: Creating IPsec Policies for Isolation Groups" at Create a test lab. To ensure that IPsec policy functions as expected and provides the appropriate level of security, test specific IPsec policy configurations on clients and servers in a lab environment, and then conduct pilot or beta tests in a limited operational environment before conducting a full-scale deployment. For more information about creating a test lab, see Setting Up IPsec Server and Domain Isolation in a Test Lab at

15 A Guide to Domain Isolation for Security Architects 15 Deploy IPsec Across Your Network. For a through discussion of IPsec deployment, see Deploying IPsec in the Windows Server 2003 Deployment Kit at Considering Operational Issues for IPsec- Secured Traffic IPsec provides greater security at the expense of network and processing performance and compatibility with other services, tools, and features. The following section describes these and other important operational considerations for managing IPsec policies. The Impact of IPsec on Performance IPsec has some costs in network and processor performance. The exact cost cannot be predicted because it depends on many factors, such as the level of application traffic and the number and configuration of IPsec filters deployed. However, the following factors should be understood and reviewed for your deployment. Network Performance IKE SA establishment results in slower connection establishment. When a security association (SA) is initially established between two computers, a delay of up to three seconds occurs while security is negotiated. This time might vary, depending on policy design, the authentication method that is selected, network roundtrip time, and the load placed on the servers required to establish the connection. Accordingly, end users might notice brief initial delays when IPsec-secured connections are established. Typically, after SAs are established, no further delays occur. Per-packet volume overhead results in reduced throughput and increased network utilization. For each IPsec-secured packet, the AH and ESP headers add an additional 24 to 36 bytes of data, depending on which method of encapsulation is used and whether IPsec tunneling is used. This decreases the effective IP payload size for upper-layer protocols such as TCP and UDP. As a result, traffic that is secured by IPsec uses more packets than traffic that is not secured by IPsec. The Microsoft IT department observed a mean bandwidth usage increase of 3 percent when they deployed IPsec (ESP-Null) across their network. Different enterprise environments might experience a different amount of increased usage, depending on the mix of applications present. The increase in bandwidth usage is

16 A Guide to Domain Isolation for Security Architects 16 likely to have a more significant effect in a WAN scenario where the traffic is already near capacity, than in a LAN over Fast Ethernet connections. Processor Performance SA maintenance affects scalability and results in increased CPU and memory overhead. Performance degradation and lower throughput might be experienced by servers that must establish IPsec-secured communications with many clients. In particular, full ESP encrypted traffic places the highest burden on server CPU and can gain the most benefit from the use of offload network adapters. Typically, for each active IPsec client, 5 to 20 kilobytes (KB) of memory is used on a server. The IKE authentication method has the greatest impact on memory usage for each active IPsec client. For example, Kerberos V5 authentication typically requires less memory per SA than certificate authentication. Per-packet processing overhead results in increased CPU and memory overhead. Each IPsec-secured packet creates additional performance overhead for encryption, authentication, and other packet processing functions. Each packet that is not secured by IPsec must still pass through all IPsec filters. Therefore, when you configure an IPsec policy, make sure that you create the minimum number of filters required for your environment. The impact of IPsec on throughput varies in proportion to the rate at which traffic is sent or received. Typically, active IPsec policies with 500 or fewer filters result in only a minor impact on throughput. In IPsec for Windows XP and Windows Server 2003, the per-packet processing performance is significantly improved, compared to the perpacket processing performance in IPsec for Windows The effects of IPsec on CPU performance are typically negligible on clients because clients often have ample, spare CPU processing power and memory to accommodate IPsec. However, servers that have highly loaded CPUs might incur a significant increase in load, depending on the amount of traffic that must be secured by IPsec, the number of SAs that must be processed, and the cryptographic algorithms that are selected. If your servers already have high loads on their CPUs, consider using IPsec offload network adapters before deploying IPsec. IPsec offload network adapters accelerate cryptographic operations used to secure IPsec packets. However, these adapters do not accelerate filter processing time. IPsec-enabled network adapters can be used with Windows 2000, Windows XP, and Windows Server Such network adapters typically complete IPsec operations at such a high rate that there is minimal change in transmission speeds, resulting in minimal performance degradation. However, such adapters generally have a maximum number of SAs that they can support, after which additional SAs are processed by Windows IPsec.

17 A Guide to Domain Isolation for Security Architects 17 Crossing Security Boundaries With IPsec Traffic When you design an IPsec policy, it is important to consider security boundaries, such as NATs, firewalls, and different domains forest trust environments, that IPsec-secured traffic must cross. Within a TCP/IP packet, IPsec changes the offsets for the destination ports and protocols. These changes might adversely affect the transmission of data over network devices and should be researched prior to IPsec deployment. NAT Traversal If traffic between the client and a server must pass through a network address translator (NAT), then IPsec cannot secure the traffic (the IKE negotiation will fail when translated by a network address translator). IPsec NAT traversal (NAT-T) allows IPsec peers that are behind NATs to detect the presence of NATs, negotiate IPsec SAs, and send ESPprotected data, despite the fact that the addresses in the IPsec-protected IPv4 packets change. IPsec NAT-T does not support the use of AH across network address translation. For detailed information about how IPsec NAT-T works, see IPsec NAT Traversal Overview, the August 2002 Cable Guy article at IPsec NAT-T is supported by Microsoft Windows Server 2003, Windows XP Service Pack 2 (SP2), and by Windows XP Service Pack 1 and Windows 2000 with a free Web download. However, due to the behavior of IPsec and NATs, Windows XP SP2 by default no longer supports establishing IPsec NAT-T SAs to servers that are located behind NATs to avoid a perceived security risk. If there is a requirement for IPsec communication across a NAT device, it is recommended that public IP addresses be used for all servers that can be connected to directly from the Internet. If this configuration is not possible, then the default behavior of Windows XP SP2 can be changed to enable IPsec NAT-T security associations to servers that are located behind a network address translator. Windows Server 2003 based servers also require a hotfix in order to be reached by hosts on the other side of a NAT device. The hotfix will be included in the Windows Server 2003 SP1 release. (This hotfix can currently be obtained from Product Support Services.) To ensure that a server is reachable behind NAT for IPsec traffic, you must configure NAT with static translation entries that map IKE (using UDP port 500) and IPsec NAT-T (using UDP port 4500) traffic to the server. For additional information, see the following articles:

18 A Guide to Domain Isolation for Security Architects 18 Problems with Using Network Address Translators The Cable Guy, October 2004 at IPsec NAT-T is not recommended for Windows Server 2003 computers that are behind network address translators on the Microsoft Support Site at L2TP/IPsec NAT-T update for Windows XP and Windows 2000 on the Microsoft Support Site at Creating Firewall Filters to Permit ISAKMP, AH, and ESP Traffic In some cases, IPsec-secured traffic might need to pass through a router, firewall, or other filtering device. In the case of a router, unless the router filters TCP and UDP traffic or other upper-level protocol headers, no special configuration is required to permit the IPsec-secured traffic to be forwarded. In the case of a filtering router or a firewall, you must configure these devices to allow IPsec-secured traffic to be forwarded. In order for IPsec-secured communications to pass through a firewall or other filtering device, you must configure the firewall to permit IPsec traffic on UDP source and destination port 500 (ISAKMP), UDP source and destination port 4500 (IPsec NAT-T) and IP Protocol 50 (ESP). You might also need to configure the firewall to permit IPsec traffic on IP protocol 51 (AH) to permit troubleshooting by IPsec administrators and to allow the traffic to be inspected while it is still IPsec-protected. For more information see How to Enable IPsec Traffic Through a Firewall on the Microsoft Support Site at Different Trust Environments When you design an IPsec policy, consider any logical boundaries that might affect IPsec-secured communications. For example, your domain and trust environment is critical in determining an appropriate IKE authentication method. Kerberos V5 authentication is recommended for use in a two-way (mutual) domain and forest trust environment. You can use Kerberos V5 for IKE authentication across trusted domains, if the domains are in the same forest or different forests. If the two domains are in different forests, you must configure two external trusts, one for each direction, between the domains. The external trusts must use the fully qualified domain name (FQDN) of the domains, and you must configure the IPsec policy to allow an IKE initiator to communicate to any domain controller in the forest domain hierarchy, so the initiator can obtain a Kerberos ticket from a domain controller in the responder s domain. If the domains are on opposite sides of a firewall, then you must configure the firewall to allow

19 A Guide to Domain Isolation for Security Architects 19 Kerberos traffic over UDP destination port 88, TCP destination port 88, as well as UDP destination port 389. For more information, see Active Directory in Networks Segmented by Firewalls, available at If the use of Kerberos is not possible because two-way trusts across forests cannot be established as in some large enterprise environments, you can use a PKI and digital certificates to establish IPsec-trusted communication. For an example of how Microsoft deployed their PKI, see Deploying PKI Inside Microsoft at Operational Considerations In addition to the impact of IPsec on security boundaries, other operational areas of your network will likely be affected by an IPsec deployment. These areas include: non-ipseccompatible support, NLB clusters, and network inspection technologies. Non-IPsec-compatible Support In most organizations, there will be a number of workstations, or servers, that cannot communicate using IPsec. Such hosts include: Non-Windows-based computers and devices, such as Macintosh computers, UNIX computers, and personal digital assistants (PDAs). Older versions of the Windows operating system. Computers running Windows NT, Windows 95, Windows 98, and Windows Millennium Edition do not support Group Policy or IPsec policies. Windows-based computers not joined to a trusted domain. Stand-alone computers cannot perform IKE authentication using the Kerberos V5 protocol. A computer that is joined to a domain that is not trusted by the forest being used as the trust boundary for IKE authentication will not be able to participate in a domain isolation solution. Other non-windows-based remote access or VPN clients. If a non- Windows-based IPsec virtual private network (VPN) client is being used for an organization's remote access solution, it typically is not compatible with the built-in IPsec for Windows. If the built-in IPsec for Windows cannot be used, the host will not be able to participate in an IPsec network isolation solution. In an IPsec-secured network, these hosts would likely be considered untrustworthy and communication with them would be blocked. However, business requirements often exist for these computers to communicate with trusted hosts in the isolation domain.

20 A Guide to Domain Isolation for Security Architects 20 To deal with this situation, you can create a boundary group that contains hosts that will be allowed to communicate with untrusted systems. These hosts will be exposed to a higher level of risk because they can receive incoming unsecured communications directly from untrusted computers. Computers in the boundary group are trusted hosts that can communicate both with other trusted hosts and with untrusted hosts. Boundary hosts will attempt to communicate using IPsec by initiating an IKE negotiation to the originating computer. If no IKE response is received within three seconds, the host will use plaintext without IPsec. Because plaintext communication is allowed, these hosts must be secured in other ways or monitored for acceptable risk. For more information about planning and deploying Boundary groups, see "Chapter 4: Designing and Planning Isolation Groups" available at NLB Clusters There are challenges implementing IPsec on Network Load Balancing (NLB) clusters and server clusters. NLB allows multiple servers to be clustered together to provide high availability for a service by providing automatic failover to other nodes in the cluster if an individual server fails. IPsec directly opposes NLB in that it prevents different computers from handling the same client connection. Because IPsec is negotiated between individual hosts before any data is transmitted, the same node in the cluster must respond to established IPsec connections or the traffic will be dropped. If the particular node in the cluster fails, existing IPsec connections to that server cannot detect the failover and attempts to rebuild the security association until the preset timeout period. In Windows 2000 and Windows XP, IPsec must be idle for 5 minutes before the client attempts to re-authenticate. This results in a two-to-six minute interruption in service for that client. Windows XP SP1 (with the NAT-T update), Windows XP SP2, and Windows Server 2003 reduce the preset time-out period to one minute. There will always be a delay if a client happens to be connected to a cluster node that fails. For more information about IPsec and configuring NLB clusters, see the following articles on the Microsoft Support Site: How to Configure Network Load Balancing to Work with IPsec at IPsec is not designed for failover at How to Configure IPsec on an Exchange Server 2003 Back-End Server That Is Running on a Windows Server 2003 Server Cluster at

21 Network Inspection Technologies A Guide to Domain Isolation for Security Architects 21 Within a TCP/IP packet, IPsec without encryption changes the offsets for the destination ports and protocols. These changes adversely affect many applications running on network devices like routers that monitor and manage traffic on the network. Some network applications are updated to support IPsec, but many are not yet compatible: Router access control lists (ACLs) cannot examine protocol and port fields in ESPencrypted packets, and therefore the packets are dropped. ACLs based only on IP address are forwarded as usual. Cisco NetFlow on routers cannot analyze packets between IPsec members based on protocol or port. Analysis based only on IP address is unaffected. Router-based Quality of Service (QoS) cannot use ports to classify packets. Hostbased classification of packets supported by Windows is unaffected, as is routerbased QoS using only IP addresses. Weighted Fair Queuing (WFQ) and other flow-based router traffic prioritization methods might not queue packets as intended resulting in suboptimal network utilization. Protocol analyzers, such as Microsoft Network Monitor, are unable to parse ESPencrypted traffic. In general, IPsec defeats network-based prioritization and port- or protocol-based traffic management. For encrypted packets, there is no workaround the host itself must handle any traffic management functions. For unencrypted, authenticated-only packets, the devices and applications must be aware of how IPsec changes packets to be able to do anything with them other than route them to the correct host. If it is not possible to upgrade monitoring or management devices to support IPsec, it is vital that this information be recorded and figured into your isolation design. Network Monitor added an ESP parser in version 2.1 to aid troubleshooting of unencrypted IPsec packets. Network Monitor is available in a light version in Windows Server 2003, and a full version for SMS. The SMS version adds formatting of data, advanced diagnostic capabilities, and other features for capture management. The version of Network Monitor that is provided with Windows Server 2003 includes parsers for the ISAKMP (IKE), AH, and ESP protocols. Network Monitor parsers for ESP can parse inside the ESP packet only if ESP null-encryption is being used. Network Monitor cannot parse the encrypted portions of IPsec-secured ESP traffic when encryption is performed in software. However, if encryption is performed by an IPsec hardware offload network adapter, the ESP packets are decrypted when Network Monitor

22 A Guide to Domain Isolation for Security Architects 22 captures them on either the source or the destination and, as a result, they can be parsed and interpreted into the upper-layer protocols. If you need to diagnose ESP software-encrypted communication, you must disable ESP encryption and use ESP-null encryption by changing the IPsec policy on both computers. Summary The use of IPsec to isolate computers that belong to an Active Directory domain requires careful planning. Three steps presented in this paper will prepare the security architect for the necessary research and cross-team communication. The three steps presented in this paper are: Determine the current state of your IT infrastructure, plan isolation groups, and consider operational issues for IPsec-secured traffic See Also Server and Domain Isolation Using IPsec and Group Policy Windows Server IPsec page Deploying IPsec Improving Security with Domain Isolation: Microsoft IT implements IP Security (IPsec) Using Microsoft Windows IPsec to Help Secure an Internal Corporate Network Server Problems with Using Network Address Translators Domain Isolation with Microsoft Windows Explained