Research Report about IPsec VPN

Research Report about IPsec VPN Twan talitwan@os3.nl Jonel jspellen@os3.nl Fangbin, fliu@os3.nl 2005-12-13 Abstract IPsec VPN supplies a secure transport medium for the private network in a public environment. In this case research, different aspects of IPsec VPN has been investigated, such as the implementation of IPsec VPN, scalability and security. Although, IPsec supplies a secure transfer method over the internet, it is still vulnerable for some certain sort of attack such as sniffing and so forth. Also, the scalability of IPsec VPN is a big problem for its success, although it achieves a low cost through applying the pubic network medium. 1

Contents 1 Introduction 3 1.1 What is VPN?.......................... 3 1.2 What is IPsec........................... 3 1.3 Why IPsec............................. 3 2 Components of IPsec VPN 4 2.1 IPsec Protocols.......................... 4 2.2 IKE Management......................... 5 3 Building an IPsec connection 5 3.1 Transport or tunnel-mode.................... 6 3.2 NAT traversal........................... 6 3.3 IPsec between two hosts or networks.............. 7 3.3.1 Linux to Linux...................... 7 3.3.2 Windows to Windows.................. 7 3.3.3 Linux to Windows.................... 8 4 Scalability of IPsec VPN 9 5 Security of IPsec VPN 10 5.1 ISAKMP Vulnerability Id:20051114-01013........... 10 5.1.1 Recommendations.................... 10 5.2 Vulnerability Advisory IPSEC id: 20050509-00386...... 11 5.2.1 Solution.......................... 11 5.3 Weak Encryption......................... 12 5.3.1 The algorithm...................... 12 5.4 Authentication cracking tool................... 12 5.4.1 How IKECrack works.................. 13 5.5 Microsoft PPTP protocol used with VPN........... 13 6 Conclusion 13 2

1 Introduction 1.1 What is VPN? VPN is an abbreviation for Virtual Private Network. VPN is built up on the basis of the existing network. Through transferring the private data over public domain, the cost for it is reduced significantly. Since data is transferred over a public medium, the confidentiality must be protected. Various kinds of encryption can be used by VPN. There are two main ways of encryptions, Symmetric Cryptography and Asymmetric Cryptography. With symmetric cryptography, the same key is used for both of encrypting and decrypting the messages. On the other hand, with asymmetric cryptography, two keys are used for encryption and decryption. In most of cases, the asymmetric encryption is used to authenticating each other, while the symmetric encryption is applied to supply the confidentiality of the data. Some popular symmetric encryption algorithms include DES, AES, 3DES and so forth. Some famous asymmetric algorithm consists of such as RSA, DSA, and so forth. 1.2 What is IPsec IPsec is a collection of some special internet protocols to supply a secure data transfer over the network layer. The standard IP (Internet Protocol) supports no security mechanism when it was designated firstly. With the increasing demands of the internet security, some new protocol have been developed for the network layer, such as AH, ESP, and so forth. IPsec becomes the name of the collection of all these protocols. Therefore, IPsec can be utilized in multiple application for security purpose. For example, the confidentiality and integrity of data transferred over the network can be guaranteed, the authentication of data sender can be realized, the relay and the analysis of the data transferred over the public internet can be prevented, and so forth. 1.3 Why IPsec IPsec has became a much more popular VPN security technology than many other concurrent such as PPTP, L2TP, and so forth because of some utility 3

advantages supplied by it. Firstly, many VPN product vendors support these protocol, since these protocol is compatible with many other protocols. In this way, the vendor s products will be quite compatible with other vendor s product. Further, the automatic key exchange mechanism make the integrating of new equipment into the VPN network much easier. Another significant advantage IPsec supplies is that, it works in the network layer, which means that the user application will not need to apply this protocol, they can just enjoy a secure network transfer transparently. All the way, the IPsec has been a standard method for the VPN technologies. 2 Components of IPsec VPN In this section, the main working theory of IPsec by VPN will be introduced. IPsec uses various protocols to process different kinds of security tasks, such as origin authentication, data confidentiality keeping, access control and so forth. Further, two main transfer modes are used within IPsec which are transport mode and tunnel mode. These two modes are used for various links over the transfer parties. Also, to establish a secure data transfer, a confidential key exchange mechanism is used by IPsec. Finally, various implementation modes of IPsec are supplied for various purposes. 2.1 IPsec Protocols There are two main protocols to support the security on IPsec. The first one is AH standing for Authentication Header, and the other is ESP for Encapsulated Security Payload. AH is used for the transport origin authentication, further the access control and anti-replay protection and so forth. The ESP can be used to supply a secure link to transfer confidentiality data, and to achieve a limited traffic flow confidentiality. These two protocols can be used individually as well as be used together. When used individually, a secure communication link can be built up for the transfer parties. When used with each other, a combination of security service can be supplied. 4

For the authentication, a key exchange algorithm must be implemented so that the confidentiality and integrity can be kept. The method for key exchange will be discussed in the coming chapters. 2.2 IKE Management As mentioned in the previous section, both of the transfer parties need to authenticate each other before a confidentiality transfer can be initiated. For this purpose, a special protocol called Internet Key Exchange is used for communication parties to negotiate, create and process the security associations used for their transfer. With Security Association, a number of security policies is defined, for the connections. There are two phases needed to build up a secure communication link. In the first phase, the goal is to establish a secure transfer channel so that in the second phase, the IPsec security associations can be exchanged safely. In the second phase, the real IPsec security associations are negotiated and built up. In each of these two phases, various mode is used such as main mode or passive mode. Also various protection algorithm is used for each mode, such as AES, DES, Deffie-Hellman for various goals. 3 Building an IPsec connection Before encrypted data can travel from one side to the other, a number of key exchanges has to be done. These exchanges, necessary to negotiate a session key, are called phases. The first, main phase is mainly responsible for the encryption negotiation. The second, quick phase initializes the SA s, Security Association, with the pre-shared keys or certificates. The SA s are the IPsec end-points and encrypt data with a session key. When a IPsec connection expires, only the quick phase is needed to negotiate a new session key and rebuild the connection. This expiring is critical, because excessive use of the same session key will weaken the encryption. The key exchange in the first, main phase can be done in main-mode or aggressive-mode. The aggressive-mode skips the encryption mechanism negotiation, thus it is recommended to use main-mode. Main-mode also makes eaves-dropping more difficult (see chapter 5.1.1). This alone makes its use preferred. 5

There are several ways to define keys. Keys can be pre-shared and used as a shared secret or another way is the use of the public/private key mechanism. The last mechanism is also known as certificates and is the most recommended method while it reveals the least about the cipher. That way it s more difficult to crack the connection. Thus to establish a VPN connection, the following procedures are executed: IKE: Phase 1: main-mode or aggressive-mode (encryption negotiation) IKE: Phase 2: quick-mode (setup of SA s) IPsec: starting tunnel (network data traffic) 3.1 Transport or tunnel-mode IPsec can be utilized for multiple sorts of transfer endpoints. This can be either encryption in transport-mode or tunnel-mode. Tunnel-mode is useful when encryption is only needed between two firewalls (site to site). An example is a network connection between two remote sites. The traffic between these to sites has to be secure. In this mode, the protection of data is not fully provided, on the other hand, a host does not need to know about IPsec, which makes security easy and invisible. Transport-mode encrypts the data stream completely and adds an new IP header in front of the ESP packet. Transport-mode is recommended when users work off-site on a foreign network. This method is also referred as end to end encryption. The data stream that leaves a host is yet encrypted and can not be sniffed. This can lead to some error prone situations, especially when the network s firewall or router is blocking certain ports or uses NAT. To work around the problem with NAT, NAT traversal, also known as NAT-T was invented. 3.2 NAT traversal Many experts believe that NAT is an bad solution to the IP shortage due to IPv4. IETF designed IPsec that way that it shouldn t work over a NAT router. That way they thought that people would move to IPv6 - designed by the IETF as well - sooner. Unfortunately IPv4 was propagated that much 6

that people circumvented this problem instead of moving to IPv6. To solve issues with NAT routers, ESP has to travel through like other TCP or UDP packets. So instead of being used as a protocol equally like IP, ESP is encapsulated in an UDP packet. This way it is possible to connect VPN s over a NAT setup. The use of NAT-T is not recommended. It makes things more complicated than they already are. When a situation arises where a VPN must be build over a NAT router, a better solution is to let the NAT router forward all the incoming traffic to a default host. This host then can handle the ESP traffic and do firewalling. There are NAT routers available on the market that are IPsec aware. These routers can handle ESP traffic in a more delicate way without bludging it into UDP streams. They can even build the transport-mode tunnel with the other side. [6] 3.3 IPsec between two hosts or networks A SA, Security Authority, that connects a VPN to a LAN is also known as a VPN gateway. This is because of its gateway-ing nature. The gateway can connect one LAN to another. Multiple hosts can by this gateway connect to the other side. 3.3.1 Linux to Linux.or unix to unix. To build a VPN connection between to Linux hosts is simple. Only IPsec is needed with a pre-shared key as minimum. When using Kernel 2.6 the setup of IPsec is even more convenient because of the build-in encryption capabilities. Kernel 2.4 needs to be patched against the userland sources to fully support IPsec. Independent of kernel version, Openswan or FreeSwan is necessary to build the meant connection. Although both kernels work, version 2.6 is recommended. [18] 3.3.2 Windows to Windows VPN technology was made available to Windows 2000 and XP out-of-thebox. It s fairly easy to interconnect modern Windows machines. With some extra software it is also possible to connect Windows 98 and ME as well. There s a lot of third-party software on the market that does the same thing as the already available implementation. This software on the other hand 7

can come in handy when creating certificates (discussed later). The authentication of the underlaying IPsec connection is done primarily by PKCS12-certificates, although XP (not 2000) supports pre-shared keys. The creation of certificates is the hardest part of setting up a VPN connection. Openssl on a Linux box can be useful when creating certificates. The Microsoft implementation of VPN differs somewhat from the standard implementation. Microsoft uses an additional tunnel over IPsec to establish a connection. This extra tunnel is created by PPTP or L2TP. The reason why Microsoft had choosen to do things this way, is that they believe that certificates can only be used to authenticate hosts, not humans. PAP or CHAP are used to check user credentials and therefor an extra layer, PPTP or L2TP is needed. [16] Thus to establish a VPN connection, the following procedures are executed: IKE: Phase 1: main-mode or aggressive-mode (negotiation) IKE: Phase 2: quick-mode (SA setup, host authentication) IPsec: starting tunnel (underlaying tunnel) L2TP: starting additional tunnel (network data transfer. user authentication) 3.3.3 Linux to Windows As said before, Windows uses an extra tunnel, authenticated with PAP or CHAP, over the IPsec connection. PPTP is known to be insecure (see chapter 5.5), which makes L2TP the preferred standard. Thus, when it comes to connecting Windows machines to Linux, extra software is needed. This software establishes the PPP connection over this tunnel, so that Linux can talk to the Windows host. The PAP or CHAP authentication can be done by the very available PPPd, bundled with every linux or unix distribution. Further L2TPd is needed, which is available as a RPM package. When PAP authentication is used, one can use a single daemon, L2TPNSd, to do both creating a L2TP tunnel and do the PPP authentication. One drawback of PAP is that it isn t encrypted. Although this isn t a big issue, because of the underlaying 8

IPsec tunnel, it is not recommended. Instead, CHAP or MS-CHAP is the preferred standard. Microsoft tends to use own products or standards and therefor MS-CHAP has somewhat better support.[16] It is also recommended to setup the IPsec connection with certificates. The only difference is that on Linux, PEM certificates can be used. Even DER format is supported on Linux. To make the IPsec connection work, one has to be sure that on both ends, the same root certificate is used to sign the individual ones. To eliminate problems, it is possible to use the same certificate on both Linux and Windows. The Windows variant still has to be in PKCS12 format.[17] The IPsec tunnel across Internet must be setup with routable Internet addresses, but the PPTP or L2TP tunnel can be established with private or non-routable addresses. When the latter is the case, it has the advantage that the connection is assured. It s certain that the non-routable traffic will not get to the other side without the L2TP tunnel. When routable addresses are used, traffic can still reach another host without the use of the tunnel. So it is recommended that PPTP or L2TP tunnels are made with private, non-routable Internet addresses. 4 Scalability of IPsec VPN Although the IPsec supplies a good protection for the confidentiality data transferred over the public network, it has also raised some drawback for its application. One of them is the limitation on the scalability of IPsec over VPN. Since IPsec VPN is implemented by the tunnel transfer mode, the tunnel server for the package processing will be extremely loaded when multiple packages need to be sent to local endpoints. Another drawback is that IPsec VPN does not support broadcast. IPsec VPN is designated for the point-to-point communication which is secured with tunnel mode. With this mode, the message sent over the internet are all the unicast. This characteristic also make the relay operation for the message impossible since the whole body of the original data is encrypted and packaged with the IPsec package as explained in the section 3.1. Also the bandwidth will be affected by the multiple unicast package. 9

5 Security of IPsec VPN 5.1 ISAKMP Vulnerability Id:20051114-01013 A group at the University of Oulu (Finland)[7] developed a test suite called OUSPG PROTOS ISAKMP to generate abnormal ISAKMP traffic. As they used this test suite against various IPSEC implementations, they found them to be vulnerable[8]. The severity of these vulnerabilities varies by vendor. These flaws may expose denial-of-service conditions, formatproven very susceptible to attack string vulnerabilities, and buffer overflows. All these could shut down devices and slow transmission of data across the Internet. In some cases, they could also allow hackers to execute code and hijack a device. Cisco and Juniper, two of the largest networking technology vendors, acknowledged that some of their products are at risk. OpenSWAN an open source software, may be used in many Linux and BSD based appliances is also vulnerable. The OUSPG PROTOS ISAKMP Test Suite does not test Internet Key Exchange version 2 (IKEv2), it is based on IKEv1. ISAKMP consists of two phases. In phase 1, the two parties negotiate a SA to agree on how to protect the traffic in the next phase. In phase 2 keying material is derived and the policy to share it is negotiated. In this way, security associations for other security protocols are established. Multiple ISAKMP implementations behave in anomalous way when they receive and handle ISAKMP Phase 1 packets with invalid and/or abnormal contents. By applying the OUSPG PROTOS ISAKMP Test Suite to a variety of products, several vulnerabilities can be revealed that can have varying effects. 5.1.1 Recommendations These are the suggestions recommended by the NISCC[9] when mitigate to IPSec against the issues discussed in this advisory: If possible, use packet filters and accepts ISAKMP negotiations only from trusted IP-addresses 10

Avoid using aggressive mode* in phase 1 [*In aggressive mode, fewer exchanges are made and with fewer packets during the negotiation stage. The weakness of using this mode is that both sides have exchanged information before there is a secure channel.] The solution to this situation is by looking at the Vendor Information. They came out with patches for platform specific remediation. 5.2 Vulnerability Advisory IPSEC id: 20050509-00386 In May 2005 a vulnerable[10] was identified about three attacks that apply to certain configurations of IPsec. These configurations use Encapsulating Security Payload (ESP) in tunnel mode with confidentiality only, or with integrity protection being provided by a higher layer protocol. Some configurations using AH to provide integrity protection are also vulnerable. In these configurations, an attacker can modify sections of the IPsec packet, causing either the clear text inner packet to be redirected or a network host to generate an error message. In the latter case, these errors are relayed via the Internet Control Message Protocol (ICMP); because of the design of ICMP, these messages directly reveal segments of the header and payload of the inner datagram in clear text. An attacker who can intercept the ICMP messages can then retrieve plaintext data. The attacks have been implemented and demonstrated to work under realistic conditions. 5.2.1 Solution Any of the following methods[11] can be used to rectify this issue: 1. Configure ESP to use both confidentiality and integrity protection. This is the recommended solution. 2. Use the AH protocol alongside ESP to provide integrity protection. However, this must be done carefully: for example, the configuration where AH in transport mode is applied end-to-end and tunneled inside ESP is still vulnerable. 3. Remove the error reporting by restricting the generation of ICMP messages or by filtering these messages at a firewall or security gateway. 11

5.3 Weak Encryption In some cases an IPSec connection is made with the use of client software. The user and group password are mostly store in the local user profile file. If a weak encryption is used it can be revealed with the knowledge of a good cryptographer. In this article some describe that he has found a way how to reveal the password of a Cisco VPN Client. He describes[12]: The main problem of the method used to encrypt the passwords is, that the whole procedure is deterministically and no user input is used. This effectively means that the encryption keys the Cisco Client calculates can also be calculated by any other program whensoever this program knows the algorithm. This algorithm was now reversed. 5.3.1 The algorithm The algorithm which is used to encrypt a given user/group password is shown below The current date as a string is retrieved (e.g. Mon Sep 19 20:00:00 2005) Then a SHA-1 Hash h1 is computed (20 Bytes) h1 is modified and a new Hash h2 is calculated h1 is again modified and h3 is calculated The 3DES key is made of h2 and the first 4 bytes of h3 The password is encrypted using 3DES in CBC Mode. The IV consists of the first 8 bytes from h1. The algorithm computes a last hash h4 from the encrypted password The key enc UserPassword in our profile file now looks like this: h1 h4 encrypted password 5.4 Authentication cracking tool A tool who can crack the IKE/IPSec authentication is IKECrack. It is an open source tools which is design to brute force or dictionary attack the password key/password used with Pre-Shared Key [PKE] IKE authentication. This tool was built to demonstrate proof-of-concept ad will work with RFC 2409 based aggressive mode PSK authentication. 12

5.4.1 How IKECrack works IKE Aggressive Mode BruteForce Summary Aggressive Mode IKE authentication is composed of the following steps[13]: 1. Initiating client sends encryption options proposal, DH public key, random number [nonce i], and an ID in an un-encrypted packet to the gateway/responder. 2. Responder creates a DH public value, another random number [nonce r], and calculates a HASH that is sent back to the initiator in an unencrypted packet. This hash is used to authenticate the parties to each other, and is based on the exchange nonces, DH public values, the initiator ID, other values from the initiator packet, and the Pre- Shared-Key [PSK]. 3. The Initiating client sends a reply packet also containing a HASH, but this response is normally sent in an encrypted packet. 5.5 Microsoft PPTP protocol used with VPN PPTP (Point-to-Point Tunneling Protocol) is a Microsoft VPN protocol published as an RFC in 1999 for secure remote access. This protocol has en is been used in many Microsoft based networks, firewall appliances, and even pure Linux and Open Source environments[14]. In 2003 Joshua Wright created ASLEAP[15] tools to prove that a password based authentication system like Cisco LEAP is not a secure because of one glaring weakness, it relies on humans to memorize strong passwords. ASLEAP just happens to make that point abundantly clear since it had the ability to scan through a 4 GB pre-computed password hash table at a rate of 45 million passwords a second using a common desktop computer for cracking passwords. Better Solution is to use L2TP (Layer 2 Tunneling Protocol) protocol with VPN. 6 Conclusion IPsec is very useful, if used the right way. Use main-mode, not aggressivemode. When connecting Windows to Windows or Windows to Linux, use 13

L2TP instead of PPTP. Last but not least, use CHAP or MS-CHAP instead of PAP. IPsec is most proper to be applied in the communication of point-topoint type. As a result, the scalability of IPsec has been limited with more number of nodes added to the network. Also, since IPsec supply the confidentiality and integrity of the original data through encrypt them and adding its new header, many process options in the original header can not be accessed when the package is transferred in the network. IPsec provide a nice way to secure the data when it is transferred through the public network through building up a secure link between sender and receiver. But, on the other hand, it is still possible to be attacked by some kind of sniffing attack, or man-in-the-middle attack for example on the local network of the each end point before data is sent by the gateway over the network. Using the well known encryption algorithms is better than making your own encryption algorithm. Because those well known algorithms have been and are tried to be cracked by thousand of people everyday. This is why it s better using known encryption algorithms. Second in many cases people make mistakes in the implementation of those encryptions algorithms in their product. Now a day MD5 hash algorithm is considered cracked. For the implementation for IPSec (HMAC) it would be better to use SHA-1 or other strong hash algorithms. The same for DES encryption, it is also consider cracked. AES and RSA are stronger encryption. Despite of the complexity, IPsec has been able to work together with many other services supplied by multiple network infrastructures, such as. Therefore, IPsec has became almost a standard secure communication services. 14

References [1] Wipul Jayawickrama: Demystifying IPSec, Information Security Management System, 2003 [2] Sheila Frankel, Karen Kent, Ryan Lewkowski, Angela D. Orebaugh, Ronald W. Ritchey, Steven R. Sharma: Guide to IPsec VPNs, Computer Security Division, Information Technology Laboratory, National Institute of Standards and Technology, Gaithersburg, MD 20899-8930, January 2005 [3] The Illusion of Security: Using IPsec VPNs to Secure the Air, Trapeze Networks, [4] George Hadjichristofi: IPSec Interoperability and Scalability, Computer Engineering, Virginia Tech, 2003 [5] Prakash Iyer, Victor Lortz, Ylian Saint-Hilaire: Scalable Deployment of IPsec in Corporate Intranets, Intel Architecture Labs Internet Building Blocks Initiative, 2000 [6] Charlie Kaufman, Radia Perlman, Mike Speciner: Network Security, Private Communication in a PUBLIC World, second edition 2002 [7] PROTOS Test-Suite: c09-isakmp, University of Oulu, November 2005, [8] IPSEC / ISAKMP Vulnerability wrapup, SANS, November 2005, [9] Vulnerability Advisory 273756/NISCC/ISAKMP, NISCC, November 2005, [10] Bill Brenner, News Writer: High-severity vulnerability in IPsec, Search- Security.com, May 2005, [11] NISCC Vulnerability Advisory IPSEC - 004033, NISCC, May 2005 [12] Geschrieben von HAL: Cisco Password Encryption reversed, EvilScientists, October 2005 [13] Anton T. Rager: IKECrack, http://ikecrack.sourceforge.net/, 2001-2002 [14] George Ou: PPTP VPN authentication protocol proven very susceptible to attack, ZDnet.com, December 2004 15

[15] Joshua Wright: Asleap behind the wheel, http://asleap.sourceforge.net/, sourceforge.net, 2004, [16] Jacco de Leeuw: Using a Linux L2TP/IPsec VPN server, http://www.jacco2.dds.nl/networking/freeswan-l2tp.html [17] Nate Carlson: Configuring an ipsec tunnel between openswan and windows 2000 / xp, http://www.natecarlson.com/linux/ipsecx509.php#casetup [18] Xelerance Corporation: Openswan, http://www.openswan.org 16