Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems

Transcription

1 IBM Proventia Network Multi-Function Security (MFS) Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems December 19, 2007 Overview Introduction This document describes how to configure an L2TP/IPsec VPN tunnel from a Proventia Network MFS running firmware 3.11 or later to Windows operating systems. Scope This document provides examples of settings. For specific instructions on how to configure actual settings, refer to the documentation listed in Related documentation. Related documentation Refer to the Proventia Manager Help and the IBM Proventia Network Multi-Function Security (MFS) Policy Configuration Guide for more information about settings and policies. For related procedures for configuring the Windows XP or Windows Vista systems, refer to the documentation provided with your system. In this document This document contains the following topics: Topic Page Before You Begin 3 Required Tasks 7 Configuring Access Policies 8 Creating NAT Policies 11 Configuring VPN Users and IP Address Pools 12 Configuring the Security Gateway 14 IBM Internet Security Systems 1

2 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Topic Configuring the Windows XP Client for L2TP/IPsec VPN Connection Using CHAP Authentication Configuring the Windows Vista Client for L2TP/IPsec VPN Connection Using CHAP Authentication Page Configuring the RADIUS Client 20 VPN Certificate Authentication 21 Configuring Certificates on the Windows Client 24 Troubleshooting 26 Contents of document subject to change. 2

3 Before You Begin Before You Begin Introduction This topic includes information about the types of VPN connections, compatible Windows systems, a topography graphic, and checklists to help you gather the information you need to configure an L2TP/IPsec VPN for your Proventia Network MFS appliance and Windows XP/Vista systems. Types of VPN connections You can use two types of authentication for L2TP/IPsec VPN connections from a Proventia Network MFS appliance to a Windows client, as shown in the following table. To use this authentication method... See this topic... Certificate (recommended) VPN Certificate Authentication on page 21 Pre-shared Key Configuring the Windows XP Client for L2TP/IPsec VPN Connection Using CHAP Authentication on page 16 Configuring the Windows Vista Client for L2TP/IPsec VPN Connection Using CHAP Authentication on page 18 Table 1: VPN tunnel types Intended use This document explains how to configure VPN from a Proventia Network MFS appliance to any of the following systems: Windows XP Windows XP with Service Pack 1 installed Note: Patch required. See NAT-T support patch from Microsoft on page 16. Windows XP with Service Pack 2 installed Note: See NAT-T behavior in Windows XP SP 2 on page 16. Windows Vista The procedures are not designed for operational use without modification. A knowledgeable IPsec network administrator or advanced user should design new, custom polices for operational use. NAT devices, routable IP addresses and advanced parameters This information applies to Proventia Network MFS appliance firmware 3.11 and later. If you have clients connecting from routable IP addresses, as well as from behind a NAT device, add the following advanced parameter to your Proventia Network MFS appliance in Configuration Firewall Advanced Parameters. Name Type Default Value l2tp.ipsec.allowanyip Boolean True Table 2: Advanced parameter for firmware 3.11 and later IBM Internet Security Systems 3

4 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Name Type Default Value True: the appliance accepts any IP address as a remote ID from clients who have a routable IP address even when the appliance is configured to use an FQDN. Computers connecting from behind a NAT device must use the correct FQDN in the Remote ID field. False: the appliance only accepts a matching FQDN remote ID. Table 2: Advanced parameter for firmware 3.11 and later (Continued) Topography The following graphic illustrates the network topography of a Proventia Network MFS appliance configured for VPN with a Windows XP/Vista system. The example used in this document is based on the topography depicted. Internal Network External Network Eth0 internal /24 L2TP IP address pool /254 ` ` ` L2TP IP address pool endpoint Proventia Network MFS Eth Eth Internet Windows XP/Vista Client Remote Client IP ` Figure 1: Topography for VPN tunnel from Proventia Network MFS appliance to Windows XP/Vista system Contents of document subject to change. 4

5 Before You Begin Checklist for mandatory information Use the following checklist to gather information you must have before you configure your VPN tunnel. Mandatory Information Proventia Network MFS Unit A External IP address Note: This is the IP address that you use where appears in the examples in this document. Proventia Network MFS Unit A Internal IP Address Subnet A IP address/mask L2TP IP address pool range Note: This is the IP address that you use where /254 appears in the examples in this document. L2TP IP address pool endpoint Note: This is the IP address that you use where appears in the examples in this document. Preshared key (minimum of 8 characters) Note: Windows XP stores the pre-shared key in cleartext in the registry, accessible by administrators. Active Directory stores IPsec configuration policies and pre-shared keys in cleartext. Consider using signed certificates identifying the Proventia Network MFS and Windows XP client for better security. Access Policies NAT Policies Table 3: Mandatory information checklist IBM Internet Security Systems 5

6 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Checklist for optional information Use the following checklist to gather optional information for configuring your VPN tunnel. Optional Information IKE Phase 1 (Main Mode) Authentication MD5 SHA1 IKE Phase 1 Encryption 3DES DES AES Note: If you select AES, select an AES key length: IKE Phase 1 Key Lifetime Seconds IKE Phase 1 Key Lifetime Kbytes IKE Phase 1 Diffie-Hellman Group Group1 Group2 Group5 IKE Phase 2 (Quick Mode) Authentication MD5 SHA1 IKE Phase 2 Encryption 3DES DES AES Note: If you select AES, select an AES key length: IKE Phase 2 Key Lifetime Seconds IKE Phase 2 Key Lifetime Kbytes IKE Phase 2 Diffie-Hellman Group None Group1 Group2 Group5 Table 4: Optional information checklist Contents of document subject to change. 6

7 Required Tasks Required Tasks Introduction This topic describes the tasks required to establish an L2TP/IPsec connection between the Proventia Network MFS appliance and Windows clients using certificate authentication. Guideline You are creating a VPN tunnel in which the original IP addresses are preserved in the ESP, so you do not need NAT for the subnets. Required tasks for certificate authentication To establish the L2TP/IPsec connection using certificate authentication, you must complete the following tasks. Task Description 1 Configure access policies. Reference: See Configuring Access Policies on page 8. 2 Configure NAT policies. Reference: See Creating NAT Policies on page Add IP addresses for remote Windows clients. Reference: See Configuring VPN Users and IP Address Pools on page Configure the security gateway. Reference: See Configuring the Security Gateway on page Configure the Windows XP or Vista client for certificate authentication. References: See Configuring the Windows XP Client for L2TP/IPsec VPN Connection Using CHAP Authentication on page 16. See Configuring the Windows Vista Client for L2TP/IPsec VPN Connection Using CHAP Authentication on page 18. Table 5: Required tasks for L2TP/IPsec using certificate authentication IBM Internet Security Systems 7

8 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Configuring Access Policies Introduction This topic describes how to configure access policies on your Proventia Network MFS appliance. You must enable three default access policies and create one access policy on the Proventia Network MFS appliance to allow all traffic from subnet A to subnet B. Enable the following policies to allow inbound traffic: ISAKMP_UDP IPsec_NAT-T L2TP Create the following policy to allow inbound traffic: L2TP_Pool_access ISAKMP_UDP general settings Define the ISAKMP_UDP inbound access policy general settings as follows: Item Setting Enabled Action Log Enabled Comment Selected Allow Not selected (optional) Access policy to allow traffic from remote Windows XP client Table 6: ISAKMP_UDP general settings ISAKMP_UDP remaining settings Define the remaining ISAKMP_UDP inbound access policy settings as follows: On this subtab... Select this item... With this setting... Protocol Any N/A Source Address Single IP Address Any Source Port Any Any Destination Address Network Address/#Network Bits (CIDR) Self Destination Port Any 500 Table 7: ISAKMP_UDP remaining settings IPsec_NAT-T general settings Define the IPsec_NAT-T inbound access policy general settings as follows: Item Setting Enabled Action Log Enabled Comment Selected Allow Not selected (optional) Access policy to allow traffic from remote Windows XP client Table 8: IPsec_NAT-T general settings Contents of document subject to change. 8

9 Configuring Access Policies IPsec_NAT-T remaining settings Define the remaining IPsec_NAT-T inbound access policy settings as follows: On this subtab... Select this item... With this setting... Protocol Any N/A Source Address Single IP Address Any Source Port Any Any Destination Address Network Address/#Network Bits (CIDR) Self Destination Port Any 4500 Table 9: IPsec_NAT-T remaining settings L2TP general settings Define the L2TP inbound access policy general settings as follows: Item Setting Enabled Action Log Enabled Comment Selected Allow Not selected (optional) Access policy to allow traffic from remote Windows XP client Table 10: L2TP general settings L2TP remaining settings Define the remaining inbound access policy settings as follows: On this subtab... Select this item... With this setting... Protocol Any N/A Source Address Single IP Address Any Source Port Any Any Destination Address Network Address/#Network Bits (CIDR) Self Destination Port Any 1701 Table 11: L2TP remaining settings IBM Internet Security Systems 9

10 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems L2TP_Pool_access general settings Define the L2TP_Pool_access internal access policy general settings as follows: Item Setting Enabled Action Log Enabled Comment Selected Allow Not selected (optional) Access policy to allow traffic out to remote Windows XP client Table 12: L2TP_Pool_access general settings L2TP_Pool_access remaining settings Define the remaining L2TP_Pool_access internal access policy settings as follows: On this subtab... Select this item... With this setting... Protocol Any N/A Source Address Network Address/#Network Bits (CIDR) L2TP IP pool range Example: /254 Source Port Any Any Destination Address Single IP Address Any Note: The Any setting allows VPN clients full access to internal resources and to the Internet. If Eth0 is used, clients can access internal resources but not the Internet. Destination Port Any Any Table 13: L2TP_Pool_access remaining settings Contents of document subject to change. 10

11 Creating NAT Policies Creating NAT Policies Introduction This topic describes how to create NAT policies for your Proventia Network MFS appliance. You must add a source NAT (Network Address Translation) rule on the Proventia Network MFS appliance to bypass NAT and to ensure that the appliance does not translate packets that travel between subnets. Source NAT Rule general settings Create a source NAT rule with general settings as follows: Item Setting Name Enabled Comment WinXP_BypassNAT_Src Selected Source NAT Rule to bypass NAT Table 14: Source NAT Rule general settings Source NAT Rule remaining settings Define the remaining source NAT rule settings as follows: On this subtab... Select this item... With this setting... Protocol Any N/A Source Address Any L2TP IP address pool Example: /254 Destination Address IP range SysEth0Net Note: Additional rules may be needed. Destination Port Any Any Translated Address Do Not Translate N/A Table 15: Source NAT Rule remaining settings Note: Make sure that the source NAT rule is in the top position in the source NAT rules table because NAT rules are applied in order from top to bottom. IBM Internet Security Systems 11

12 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Configuring VPN Users and IP Address Pools Introduction This topic describes how to configure VPN users and IP address pools for your Proventia Network MFS appliance. You must add the following items for remote Windows clients: IP addresses that the appliance assigns to Windows clients when they connect Username/password pairs Consideration You can use the Proventia Network MFS appliance to enter the IP addresses in the IP pool and add username/password pairs, or you can use a RADIUS authentication server. For more information about RADIUS authentication, go to the task: Configuring the RADIUS Client on page 20. Adding IP addresses To add the IP addresses on the appliance, do the following: 1. In the navigation pane, expand the Firewall/VPN node. 2. Select Settings. 3. Click the VPN Advanced tab. 4. Click the L2TP IP Pool tab. 5. Click Add. 6. Type the IP address range for the L2TP endpoint in the IP Range field. Note: These are the IP addresses that you want to assign to the remote Windows clients. L2TP IP address range options are shown in the following table: If you want to... Use a static IP address range Use an Address Name network object Use a Dynamic Address Name network object Then do this... Select Static Address Range, and then type the starting and ending IP addresses in the IP Address Range field in dotted decimal format. Select Address Name, and then select an address entry from the list. Note: Click Configure to add or edit an address name. Select Dynamic Address Name, and then select a name from the list. Note: Click Configure to add or edit a dynamic address name. 7. Click OK. 8. Click Save Changes. Creating a VPN Users List entry with username/ password pairs To create a VPN Users list entry with username/password pairs, do the following: 1. In the navigation pane, expand the Firewall/VPN node. 2. Select Settings. 3. Click the VPN Advanced tab. Contents of document subject to change. 12

13 Configuring VPN Users and IP Address Pools 4. Click the VPN Users tab. 5. Click Add. 6. Type the VPN user's name in the User Name field. 7. To set the user's password, click Set Password, and type the user's password in the Password field. 8. Type the user's password in the Confirm Password field, and then click OK. 9. Do one of the following: In the Proventia Manager interface, click Save Changes. In the SiteProtector interface, click OK. IBM Internet Security Systems 13

14 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Configuring the Security Gateway Introduction This topic describes how to configure the Proventia Network MFS appliance security gateway. Accessing the security gateway To access the security gateway, do the following: 1. In the navigation pane, click Configuration Firewall. 2. Click the Security Gateways tab. 3. Click the L2TP/IPsec Remote Client Security Gateway tab. Configuring the security gateway Add or edit the L2TP/IPsec remote client security gateway with the settings shown in this topic. Configuring the general settings To configure the general settings, do the following: 1. Type a name. 2. Select the Enabled check box. 3. Type a comment. 4. Configure the General tab, the IKE Configuration tab, and the IPsec Configuration tab, and then click OK. Configuring the General tab Configure the General tab with the settings shown in the following table. Item Setting Disable L2TP Tunnel Authentication L2TP Host Name L2TP Endpoint IP Address Clear this check box. Enter the device name of the firewall as shown on the top right corner of your appliance s LMI. Select Static Address, and then type the address in the IP Address field in dotted decimal format. Important: The L2TP endpoint IP address for the appliance must be a fixed, globally unique IP address, and should not be in the L2TP IP address pool and not used for any other interface on the appliance, such as the internal network. Examples: L2TP IP address pool endpoint: L2TP IP address pool: /254 Table 16: General tab settings Contents of document subject to change. 14

15 Configuring the Security Gateway Configuring the IKE Configuration tab Configure the IKE Configuration tab with the settings shown in the following table. Item Setting IKE Exchange Type Encryption Algorithm Authentication Algorithm Authentication Mode Pre-Shared Key Select Main Mode. Select 3DES. Select SHA1. Select Pre Shared Key. Type the pre-shared key used on the Window XP/Vista side. Note: This must be at least 8 characters. Life Time Secs Type DH Group Descriptor Select Group 2. Local ID Remote ID Select Dynamic Address and then select SysEth1IP. Select FQDN and then type the FQDN (Fully Qualified Domain Name) that clients send. Example: test.com Table 17: IKE Configuration tab settings Configuring the IPsec Configuration tab Configure the IPsec Configuration tab with the settings shown in the following table: Item Setting Encapsulation Mode Security Protocol Authentication Algorithm Encryption Algorithm Select Transport. Select ESP With Auth. Select SHA1. Select 3DES. Life Time Secs Type Life Time KBytes Type Table 18: IPsec Configuration tab settings IBM Internet Security Systems 15

16 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Configuring the Windows XP Client for L2TP/IPsec VPN Connection Using CHAP Authentication Introduction This topic describes how to configure the Windows XP client for an L2TP/IPsec connection using CHAP authentication (pre-shared keys). NAT-T support patch from Microsoft These instructions assume that the Windows client is behind a NAT appliance. Microsoft released a patch for Windows XP that added support for NAT-T within the IKE negotiations of L2TP/IPsec. You must install this patch if you are running Service Pack 1 and if the Windows client is behind a NAT device. The patch is located at: NAT-T behavior in Windows XP SP 2 The default behavior for NAT-T within Windows XP Service Pack 2 has changed. For more information, visit the Microsoft Web site at: Procedure To configure the Windows XP client, do the following: 1. On the taskbar, click Start Control Panel Network Connection. 2. Click File New Connection. 3. Click Next. 4. Select Connect to the Network at My Place, and then click Next. 5. Select Virtual Private Network Connection, and then click Next. 6. Type the connection name, and then click Next. 7. Type the VPN server host IP address, and then click Next. Note: The VPN server host IP address is the external Eth1 interface IP address. 8. Click Finish. 9. Type your username and password. Note: This is the same username and password used on the appliance. Both the username and password are case sensitive. 10. Click Properties. 11. Click the Security tab and then Click IPsec Settings. 12. Type the pre-shared secret key, and then click OK. Note: This is the same key you used for the appliance. 13. Click Settings. 14. In the Data encryption field, select Optional encryption (connect even if no encryption). 15. Select Allow these protocols. 16. Clear all the default check boxes, and then enable the Challenge Handshake Authentication Protocol (CHAP) check box. 17. Click OK. Contents of document subject to change. 16

17 Configuring the Windows XP Client for L2TP/IPsec VPN Connection Using CHAP Authentication 18. Click Yes. 19. Click Connect. Note: A Windows host connecting from behind a NAT device often reports its DNS suffix as the remote ID when connecting to an L2TP VPN gateway. You may have to change the DNS suffix on the Windows host to match the Remote ID value specified in the L2TP remote client security gateway on the appliance. IBM Internet Security Systems 17

18 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Configuring the Windows Vista Client for L2TP/IPsec VPN Connection Using CHAP Authentication Introduction This topic describes how to configure the Windows Vista client for an L2TP/IPsec connection using CHAP authentication (pre-shared keys). Procedure To configure the Windows Vista client, do the following: 1. In the Windows desktop, click Start Control Panel. 2. Double-click Network and Sharing Center. 3. Click Set up a connection or network. 4. Click Connect to a workplace and then click Next. 5. Select Use my Internet connection (VPN). 6. Select I ll set up an Internet connection later. 7. Type the Internet address, the IP address of the VPN server, and then click Next. 8. Type the username and password, and then click Create. 9. Click Close. Configure Windows Vista to use CHAP authentication To configure the VPN connection in Windows Vista, do the following: 1. On the taskbar, click Start Control Panel. 2. Double-click Network and Sharing Center. 3. Click Manage your network connection. 4. Double-click the virtual connection you created. 5. Click the Properties tab. 6. Click the Options tab, and then clear the Include Windows logon domain check box. 7. Click the Security tab, and then select Advanced (custom settings). 8. Click the Settings tab, and then select Optional encryption from the Data Encryption window. 9. Clear the MS-CHAP2 check box, and then enable the CHAP check box. 10. Click OK, and then click Yes. 11. Click Networking, and then select L2TP IPsec VPN from the menu. 12. Click IPsec Settings, and then type the pre-shared key. Note: This pre-shared key is the same as the appliance key. 13. Click OK. 14. Double-click the new VPN connection, type the username and password, and then click Connect. Note: The username and password are the same as the appliance username and password. Contents of document subject to change. 18

19 Configuring the Windows Vista Client for L2TP/IPsec VPN Connection Using CHAP Authentication Connecting behind a NAT device To connect the VPN connection in Windows Vista behind a NAT device, do the following: 1. On the taskbar, click Start Connect to. 2. Right-click Network, and then click Properties. 3. Click Properties. 4. Right-click the VPN connection you want to configure, and then click Properties. 5. Click Continue if prompted to do so. 6. Click the Networking tab, and then enable the Internet Protocol Version 4 (TCP/ IPv4) check box. 7. Click Properties. 8. Click Advanced, and then click the DNS tab. 9. Type the DNS suffix for the connection, and then click OK three times. Note: The DNS suffix must be the same as the appliance remote ID. IBM Internet Security Systems 19

20 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Configuring the RADIUS Client Introduction This topic describes how to configure the Proventia Network MFS appliance to allow RADIUS authentication. Configuring the appliance to allow RADIUS authentication To configure the appliance, do the following: 1. Go to the Firewall/VPNSettingsVPN Advanced tab. 2. Define the RADIUS settings as shown in the following table: Item Enabled Primary Server IP Address Primary Server Subnet Mask Primary Server Auth Port Primary Server Acct Port Secret with Primary Server Primary Server NAS ID Setting Select this check box. Type the IP address of the primary RADIUS server. Type the subnet mask of the primary RADIUS server. Select the authentication port for the primary RADIUS server. Note: You can use the default port numbers, but the standard port number from earlier versions is available in the list. Select the user account port for the primary RADIUS server. Note: You can use the default port numbers, but the standard port number from earlier versions is available in the list. Type the shared secret for the appliance to use as a client when communicating with the primary RADIUS server. Note: A shared secret is proof of identity, which can be a certificate or a pre-shared secret key. Type the NAS ID of the primary RADIUS server or type 0 or 1. Note: NAS operates as a client of RADIUS. The client is responsible for passing user information to designated RADIUS servers. If you do not plan to use a NAS server ID, then you must type a 0 or If you use a backup RADIUS server, select Use Backup Server, and then configure the RADIUS settings. Contents of document subject to change. 20

21 VPN Certificate Authentication VPN Certificate Authentication Introduction This topic explains how to generate a public/private key set, generate a certificate request, and install the certificate issued by a Trusted Certificate Authority (CA). You must download a RootCA certificate in DER format before you perform the procedures in this topic. Contact IBM Internet Security Systems Customer Support or your Trusted Certificate Authority for details. Note: This topic assumes that you are using the same Trusted CA you used for Gateway A and Gateway B. If you are not, then you must install a Trusted CA certificate and Certificate Revocation List from each CA you used. Uploading the Trusted Certificate Authority s certificate To upload the certificate, do the following: 1. Log on to Proventia Manager. 2. In the navigation pane, expand the Firewall/VPN node. 3. Select Certificate Management. 4. Select Trusted Certificate Authorities. 5. Click Browse. 6. Go to the file that contains the DER encoded certificate that you received from your Trusted CA. 7. Select the file, and then click Open. 8. Click Upload. The certificate displays in the Trusted Certificates section on the Trusted CA Certificates page. Creating the public/private key pair and certificate request To create a key and certificate request, do the following: 1. In the navigation pane, expand the Firewall/VPN node. 2. Select Certificate Management. 3. Select Create Self Certificate. 4. Click Generate Certificate Request. 5. Enter information as shown in the following table: Item Setting Key-ID Type 1 Subject Type the hostname of gateway A. Department Organization City State Postal code Type your department (optional). Type your organization (optional). Type your city (optional). Type your state (optional). Type your zip code (optional). IBM Internet Security Systems 21

22 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Item Country Code Setting Type your two character country code (optional). Type the address of the primary administrator (optional). Domain Name Type the FQDN of gateway A. IP Address Type Algorithm Type RSA_MD5. Key Length Type Click Submit Request. The system creates the public/private key pair automatically, and then places a certificate request into the certificate store. 7. In the Certificate Request section, locate the private key with the Key-ID you set in Step Select the key, and then click Display. 9. Copy the text from the text area of the dialog box and give it to your CA as a PKCS#12 formatted certificate request. Your CA issues a certificate. 10. Download the certificate from the CA in DER format as instructed by the CA. 11. In the Upload Public Key section, enter the Key-ID value. 12. Go to the certificate, and then highlight it. 13. Click Upload. The certificate appears on the Self Certificates page. Note: You must install the Trusted CA certificate before you can successfully install the issued certificate. Installing a Certificate Revocation List Before you can install a Certificate Revocation List, you must acquire the latest Certificate Revocation List (CRL) in DER format from your Trusted Certificate Authority (CA). Please contact your CA if you need assistance with downloading the CRL. To install a certificate revocation list, do the following: 1. In the navigation pane, expand the Firewall/VPN node. 2. Select Certificate Management. 3. Select Certificate Revocation List. 4. Click Browse. 5. Go to the file that contains the CRL that you received from your Trusted CA. 6. Select the file, and then click Open. 7. Click Upload. The CRL appears in the Certificate Revocation List. Contents of document subject to change. 22

23 VPN Certificate Authentication Setting the security gateway object to use certificates To set the security gateway object to use certificates as the authentication method, do the following: 1. In the navigation pane, expand the Firewall/VPN node. 2. Select Settings. 3. Click the Security Gateways tab. 4. Click the Auto Key IPsec Security Gateways tab. 5. Locate and highlight Gateway B, and then click Edit. 6. In the Authentication Mode field, select RSA Signed. 7. Click OK. 8. Click Save Changes. Note: The Proventia Network MFS appliance uses the Local ID Type and Local ID Data fields to determine which certificate to send. Make sure these values correspond to your certificates. Example: USER FQDN = address from certificate request Reference: See the IBM Proventia Network Multi-Function Security (MFS) Policy Configuration Guide for more information about certificates. IBM Internet Security Systems 23

24 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Configuring Certificates on the Windows Client Introduction This topic describes how to install and verify certificates on the Windows client. Installing the certificate Follow the instructions from your Certificate Authority to configure the client certificate. The following instructions are for the Microsoft Certificate Authority included with the Windows 2000 Server. To install the certificate, do the following: 1. On the client computer, login with an account with administrative privileges. 2. Open the Web site for the Microsoft Certificate Authority. Example 3. Select Request a certificate, and then click Next. 4. Select Advanced Request, and then click Next. 5. Select Submit a certificate request to this CA using a form, and then click Next. 6. Complete the form with information for your organization. Important: Do not use the field. Due to the design differences between Windows 2000 and Windows XP, you must leave this field blank. 7. Select IPsec Certificate from the Intended Purpose list. 8. Select Microsoft Base Crypto Provider v Select one of the following key sizes: Select one of the following for the Hash Algo field: SHA-1 MD5 11. Select Store certificate in the local computer certificate store. 12. Click Submit. 13. Click Install Certificate. The certificate installs automatically. Verifying the certificate To verify that the certificate was installed correctly on the client, do the following: 1. On the client computer, login with an account with administrative privileges. 2. Run MMC.EXE. 3. Select FileAdd/Remove Snap In. 4. Click Add. 5. Select Certificates, and then click Add. 6. Select Computer Account, and then click Next. 7. Select Local Computer, and then click Finish. Contents of document subject to change. 24

25 Configuring Certificates on the Windows Client 8. Click Close. 9. Click OK. 10. Expand the Certificates tree. 11. Right-click the Personal folder and the Certificates folder. The certificates should be listed. 12. Double click on the certificate. Note: The certificate states This certificate cannot be verified up to a trusted certificate authority. This is because the Microsoft Certificate Authority Root Certificate is not installed on this computer. 13. Click the Certificate Path tab. 14. Highlight the root CA certificate in the tree, and then click View Certificate. The following message displays: This CA Root certificate is not trusted. To enable trust, install this certificate in the Trusted Root Certification Authorities store. 15. Click the Details tab, and then click Copy to File. 16. Complete the wizard and export the CA Root Certificate. Note: Consider using the DER format. 17. Browse to the certificate file you exported, and right click it. 18. Click Install Certificate. 19. Click Next. 20. Select Place all certificates in the following store. 21. Click Browse, and then select Trusted Root Certification Authorities. 22. Click OK. 23. Click Next, and then click Finish on the Certificate Import Wizard. 24. Click Yes on the Root Certificate Store dialog to add the certificate. 25. Return to the MMC application, and view the local certificate again in the Personal Certificates folder. 26. Exit the MMC application. IBM Internet Security Systems 25

26 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Troubleshooting Introduction This topic describes Windows client error messages and provides steps for troubleshooting them. Error 789 Error 789 displays the following text: Error 789: The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer. Possible solutions include the following: If the Windows client is configured for certificate authentication, make sure the certificate is installed on the Windows client. Perform the procedure Configuring Certificates on the Windows Client on page 24. The IPsec service may not be running on the Windows client. To verify, start the IPsec service, type the following command: net start policyagent If the command fails, uninstall any third party programs that replace the IPsec stack on Windows, such as SAFEnet Softremote. Error 781 Error 781 displays the following text: Error 781: no valid certificate If you see this error, then a problem exists with the certificate that the L2TP/IPsec client is attempting to use. To identify the problem, do the following: To verify that the trusted root CA that issued the certificate is installed on the Windows client, perform the procedure Configuring Certificates on the Windows Client on page 24. Verify that the certificate was imported correctly so that it exists for the local computer certificate. Do the following: Run the MMC.EXE file. Add the certificate snap-in for Local Certificate management. Double-click the certificate, and verify that the following text appears on the General tab: You have a private key that corresponds to this certificate If this text does not appear, then you may not have imported the certificate from an PKCS#12 container. All certificates imported for use in L2TP/IPsec must be in PKCS#12 format. Contents of document subject to change. 26

27 Troubleshooting Copyright IBM Corporation 2003, All Rights Reserved. IBM and the IBM logo are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. ADDME, Ahead of the threat, BlackICE, Internet Scanner, Proventia, RealSecure, SecurePartner, SecurityFusion, SiteProtector, System Scanner, Virtual Patch, X-Force and X-Press Update are trademarks or registered trademarks of Internet Security Systems, Inc. in the United States, other countries, or both. Internet Security Systems, Inc. is a wholly-owned subsidiary of International Business Machines Corporation. Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product and service names may be trademarks or service marks of others. References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates. IBM Internet Security Systems 27

28 Configuring L2TP/IPsec VPN Connections from Proventia Network MFS to Windows Systems Contents of document subject to change. 28