Highly Available VPNs on Linux

Transcription

1 Highly Available VPNs on Linux October 2002 Version 1.1 Ken Bantoft

2 Table of Contents 1. Introduction Solution Overview Heartbeat Introduction Installation Configuration Program Control Troubleshooting External References and Documentation FreeS/WAN Introduction Zebra Introduction Installation Program Control Basic Configuration GRE Tunneling Advanced Configuration ii

3 1.Introduction VPNs on Linux aren't new, by any means. Even back into 1995, NIST had an IPSec implementation, as well as the various homebrew solutions using SSH + pppd, or even telnet and pppd to create point to point links across the Internet. FreeS/WAN is currently the most popular IPSec implementation on Linux, and the most mature. It has been around for several years, and is nearing the 2.0 release. The 1.x series have been stable enough that nearly every Linux distribution vendor (with the notable exception of RedHat) is shipping with FreeS/WAN included. FreeS/WAN works great for nearly every possible VPN situation - road warriors, inter-op with other vendors' products, and recently, Opportunistic Encryption. Some major organizations are now running FreeS/WAN with hundreds, possibly thousands of IPSec tunnels concurrently. Which leads to a serious problem if your primary FreeS/WAN server goes down, you lose you entire network. Fortunatly, there's a fairly simple and elegant solution that will buy you complete failover, and it's all based on Open Source products. 1.1.Solution Overview When an IPSec gateway fails, all the remote sites aren't notified of the failure. This works in our favour, since we seamlessly want to replace the now defunct gateway with a new system. Since all IPSec tunnels are bound to either DNS hostnames or IP addresses, if we keep the old IP address (now known as the Service Address) then we can just takeover the tunnels. The solution uses two systems, a primary and a backup. During regular operation, all traffic passes through the primary server. In the event of any network card/system failure, the backup server takes over the IP address(es) and starts up FreeS/WAN to keep the tunnels alive. In addition to FreeS/WAN, Heartbeat (from the Linux-HA Project), and zebra (from IP Infusion) are required. Heartbeat takes care of taking over the IP address(es), and zebra takes care of any dynamic routing required. You can run without zebra if you have a simple network, but most enterprises will need zebra running either OSPF or BGP.

4 2.Heartbeat 2.1.Introduction Heartbeat is the basic heartbeat subsystem from Linux-HA. It will run scripts ( service <name> start ) during its' initization, and when a system changes state (ie: backup -> primary). It will also perform IP address takeover using gratuitous ARP. It works correctly for 2 node configurations, and probably larger configurations. Heartbeat is based on resources an IP address is a resource, a service (ie: ipsec, named, xinetd) is also a resource. When the decision is made to change state (ie: backup -> primary) each of the resources is aquired, and notification is send out to other nodes. Heartbeat uses UDP to send keep-alives and notifications to other nodes in the cluster. These keepalives may also be MD5 or SHA1 hased for security purposes, so a rogue node can't takeover resources. The master node, which has all the shared services for the cluster, will be monitoring the services also. If at anytime there is a problem with the service it will shut that service down on the master and transfer the service to a slave node that is reporting that everything is working. If a backup node detects that the master has gone down it will attempt to aquire all resources, by taking over the IP address(es), and starting all the services listed in the config file. 2.2.Installation Getting Heartbeat installed is simple. Get the latest package from uncompress, build, and install. Sample installation: tar -xzvf heartbeat tar.gz cd heartbeat make make install

5 2.3.Configuration To configure the Linux-HA software you must configure three files: authkeys, ha.cf, and haresources. These files are kept in the /etc/ha.d directory and are configured by using your favourite text editor. Authkeys The authkeys file holds the authentication information: Authentication file. Must be mode 600 Must have exactly one auth directive at the front. auth send authentication using this method-id Then, list the method and key that go with that method-id Available methods: crc sha1, md5. Crc doesn't need/want a key. You normally only have one authentication method-id listed in this file Put more than one to make a smooth transition when changing auth methods and/or keys. sha1 is believed to be the "best", md5 next best. crc adds no security, except from packet corruption. Use only on physically secure networks. Select method 2: auth 2 1 crc 2 sha1 This_is_my_SHA_key 3 md5 This_is_my_MD5_key This file is pretty straightforward select an authentication method, and set the key. Ha.cf The ha.cf files holds all of the program configuration details: Note on logging: If any of debugfile, logfile and logfacility are defined then they will be used. If debugfile and/or logfile are not defined and logfacility is defined then the respective logging and debug messages will be loged to syslog. If logfacility is not defined then debugfile and logfile will be used to log messges. If logfacility is not defined and debugfile and/or logfile are not defined then defaults will be used for debugfile and logfile as required and messages will be sent there. File to wirte debug messages to debugfile /var/log/ha-debug File to write other messages to logfile /var/log/ha-log

6 Facility to use for syslog()/logger logfacility local0 keepalive: how many seconds between heartbeats keepalive 2 deadtime: seconds-to-declare-host-dead deadtime 10 Very first dead time (initdead) On some machines/oses, etc. the network takes a while to come up and start working right after you've been rebooted. As a result we have a separate dead time for when things first come up. It should be at least twice the normal dead time. initdead 120 hopfudge maximum hop count minus number of nodes in config hopfudge 1 serial serialportname... serial /dev/ttys0 Baud rate for serial ports... baud What UDP port to use for communication? udpport 694 What interfaces to heartbeat over? udp eth0 udp eth1 Tell what machines are in the cluster node nodename must match uname -n node VPNGW1 node VPNGW2 The important things to set in here are the interfaces to heartbeat on, and the node names. The node names must match the output of a uname -a on the node. Haresources The haresources file holds the configuration information for all the clustered resources that are shared between them as well as the how to configure them. This is a list of resources that move from machine to machine as nodes go down and come up in the cluster. Do not include

7 "administrative" or fixed IP addresses in this file. We refer to this file when we're coming up, and when a machine is being taken over after going down. You need to make this right for your installation, then install it in /etc/ha.d These resources in this file are either IP addresses, or the name of scripts to run to "start" or "stop" the given resource. The format is like this: node-name resource1 resource2... resourcen If the resource name contains an :: in the middle of it, the part after the :: is passed to the resource script as an argument. Multiple arguments are separated by the :: delimeter In the case of IP addresses, the resource script name IPaddr is implied. For example, the IP address could also be represented as IPaddr:: THIS IS IMPORTANT vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv The given IP address is directed to an interface which has a route to the given address. This means you have to have a net route set up outside of the High-Availability structure. We don't set it up here -- we key off of it. The broadcast address for the IP alias that is created to support an IP address defaults to the highest address on the subnet. The netmask for the interface that is brought up on this IP address defaults to the same netmask as the route that it selected in in the step above. If you want to specify that this IP address is to be brought up on a subnet with a netmask of , you would specify this as IPaddr:: /8. If you wished to tell it that the broadcast address for this subnet was , then you would specify that this way: IPaddr:: /8/ The IP addresses you list in this file are called "service" addresses, since they're they're the publicly advertised addresses that clients use to get at highly available services. For a hot/standby (non load-sharing) 2-node system with only a single service address, you will probably only put one system name and one IP address in here. The name you give the address to is the name of the default "hot" system. Where the nodename is the name of the node which "normally" owns the resource. If this machine is up, it will always have the resource it is shown as owning. The string you put in for nodename must match the uname -n name of your machine. Depending on how you have it administered, it could be a short name or a FQDN

8 We have 2 IP addresses to takeover, the outside ( ) and inside ( ) And 1 service (ipsec) to takeover VPNGW ipsec 2.4.Program Control Heartbeat uses the standard System V startup method. Under RedHat, SuSE and most other distributions, to control the program you use the service command. To bring a cluster up from you must start heartbeat on both machines in the cluster. service heartbeat [start stop status] Basic control commands service heartbeat start service heartbeat stop : This starts the service : This stops the service 2.5.Troubleshooting Heartbeat is a robust and verbose program, and uses the standard unix syslog as well as its' own details logs to store messages. All of the messages produced by Heartbeat are stored in /var/log/ha-log. For any troubleshooting for Heartbeat, start with the log file. The log file has plain English log entries that are easy to read and understand. When a single node on the cluster detects an error it will either move the services over if there is a program error or shut the cluster service down if it detects a hardware error. The log file will contain all the information to see what happened to the node. 2.6.External References and Documentation Linux-HA Website Website:

9

10 3.FreeS/WAN 3.1.Introduction This document will only cover the relavent changes required to the ipsec.conf file to enable Heartbeat to manage IPSec properly. Our example network - eth0 is connected to the Internal, and eth1 is connected internally: External Service Address: External Real IP of VPNGW1: External Real IP of VPNGW2: Internal Service Address: Internal Real IP of VPNGW1: Internal Real IP of VPNGW2: In your FreeS/WAN /etc/ipsec.conf file, you will need to make sure all tunnels are configured to use the external Service Access, and not the Real Addresses. As well, you'll need to configure the ipsec0 interface to bind to eth0:0. This is accomplished in the config setup section: config setup interfaces="ipsec0=eth0:0" klipsdebug=none plutodebug=none plutoload=%search plutostart=%search uniqueids=yes You will, of course, need to copy ipsec.conf and ipsec.secrets (and any other file - ie: certs if you are using X.509) from the primary node to the backup somehow. I recommend a crobjob with SSH + DSA keys to do this - there's plenty of documenation on how to do this available. That's it for the FreeS/WAN configuration.

11 4.Zebra 4.1.Introduction Zebra is an implementation of the RIP, RIPv6, OSPF and BGP dynamic routing protocols for Linux. If your configuration for HA FreeS/WAN is only 2 interfaces, then you probably don't need it. If you have other network interfaces (ie: a DMZ connected to the VPNGW's) or you wish to peer directly with a service provider to better redundancy, you'll need to install and configure Zebra. 4.2.Installation Zebra is available from You will need at least version 0.93a. Prior versions had problems with OSPF on virtual (ie: eth0:0) interfaces. tar -xzvf zebra-0.93a.tar.gz cd zebra-0.93a./configure make make install 4.3.Program Control Zebra runs each protocol (RIP, OSPF, BGP) as a seperate process, as well as one master zebra progress. Binaries are in /usr/local/sbin. For all examples, we'll be using OSPF as our dynamic routing procotol. To start zebra's master process, run zebra -d To start the ospf daemon, run ospfd -d To stop zebra, just kill the process ID's assocated with ospfd and zebra. 4.4.Basic Configuration Zebra's configuration system is nearly identical to Cisco routers. If you're confortable configuring a Cisco or Foundry router, you'll feel right at home.

12 In order to access the configs, there's 2 methods - edit the config files in /usr/local/etc/ (zebra.conf and ospfd.conf), or telnet localhost [ ] to get an interactive shell. Before you begin, copy the zebra.conf.sample and ospfd.conf.sample files to zebra.conf and ospfd.conf. Then edit them and change the passwords. For more security, firewall out incoming connections to these ports. Since every network is vastly different, I won't explain in detail how to configure zebra, since my config only applies to my network. I'll include general configs that should work for most people. At this point, you're into advanced network design and configuration, so you should be comfortable with basic router configurations. Example zebra.conf: Zebra configuration saved from vty 2002/02/01 07:14:37 hostname vpngw1 password your_password enable password your_enable_password interface lo interface ipsec0 interface eth0 interface eth1 interface eth2 interface eth3 interface eth4 ip route / line vty 4.5.GRE Tunneling GRE (General Routing Encapsulation) is a protocol that allows you to generically encapsulate other protocols (IP, IPX, etc...) in IP, and tunnel it across this internet. GRE can be used quite well with IPSec you set up a host to host IPSec tunnel, and then add a GRE tunnel over top of it. This allows you to forward other protocols over the tunnel, as FreeS/WAN sees the IP traffic from host A to host B and encrypts it.

13 We use GRE to get around IPSec's limitations about Multicast and Broadcast traffic. Since OSPF uses Multicast traffic to send routing updates, we need to set up a GRE tunnel between our two sites to forward this traffic. This also has the side effect of letting us forward any ip packets over the GRE tunnel, ignoring FreeS/WAN's routing policies. Note: You'll need to modify your iptables rules to reflect this, and probably set rules that normally would have been applied to ipsec to apply to the new GRE interface. You'll also need to have GRE enabled in your kernel (under Network Options) JuanJo Ciarlante has provided a copy of his script he uses to create the GRE tunnels. You'll have to run this after your IPSec tunnel is established - it figures out if it is left or right and configures the correct side of the tunnel - you can use the same script on both gateways. /bin/sh left is "18", right is "20" diff DEVnames are not needed, but make things more clear DEV_LEFT=tun18 DEV_RIGHT=tun20 LEFT_IP= LEFT_NET= /32 RIGHT_IP= RIGHT_NET= /32 setup_left() { DEV=$DEV_LEFT local_ip=$left_ip local_net=$left_net remote_ip=$right_ip remote_net=$right_net } setup_right() { DEV=$DEV_RIGHT local_ip=$right_ip local_net=$right_net remote_ip=$left_ip remote_net=$left_net } case "`/sbin/ip -4 -o addr show dev eth0`" in * *) setup_left ;; * *) setup_right ;; *) echo "ERROR";; esac case "$1" in start) modprobe ip_gre (set -x ip tunnel add $DEV mode gre remote $remote_ip local $local_ip ttl 255 ip link set $DEV up multicast on DOESNT SEEMS to work for zebra :(

14 ip addr add $local_net peer $remote_net dev $DEV ip route add $remote_net dev $DEV Needed if you run BGP instead of OSPF ) ;; stop) (set -x ip tunnel del $DEV ) modprobe -r ip_gre ;; esac 4.6.Advanced Configuration Example ospfd.conf files from JuanJo Ciarlante. These detail a 3 area OSPF network, with 2 Secure Gateways (SGA, SGB), running a GRE tunnel overtop of them. There are 2 files here, ospfd.conf for ospfd.conf for : -*- ospf -*- *** THIS ospfd.conf *** tun20 tun18 IPSec (*alt. link*) /16 --[ SGA ]================[ SGB ] /16 : : ========//======== /24 (*main eth link*) < >< >< > area 20 area 0 area 18 hostname ospfd password jtest enable password jtest Actually a renamed gre interface: interface tun18 not needed, read by kernel: ip ospf network point-to-point ip ospf authentication null ip ospf cost 50 router ospf ospf router-id not needed, zebra seems to get p-to-p peer from iface neighbor network /32 area 0 redistribute kernel redistribute connected network /32 area 0 network /24 area 0 network /24 area 18 network /16 area 18

15 log stdout log file /var/log/zebra/ospfd.log line vty ************************************************************************************** -*- ospf -*- *** THIS ospfd.conf *** tun20 tun18 IPSec (*alt. Link*) /16 --[ SGA ]================[ SGB ] /16 : : ========//======== /24 (*main eth link*) < >< >< > area 20 area 0 area 18 hostname ospfd password jtest enable password jtest Actually a renamed gre interface: interface tun20 not needed, read by kernel: ip ospf network point-to-point ip ospf authentication null ip ospf cost 50 router ospf ospf router-id not needed, zebra seems to get p-to-p peer from iface neighbor network /32 area 0 redistribute kernel redistribute connected network /32 area 0 network /24 area 0 network /24 area 20 network /16 area 20 network /16 area distribute-list OUT_FILTER out connected distribute-list OUT_FILTER out kernel access-list OUT_FILTER permit /8 access-list OUT_FILTER deny any log stdout log file /var/log/zebra/ospfd.log The key elements of the above config are the redistribute commands. they inject the FreeS/WAN IPSec eroutes into OSPF to be redistributed to other routers (including your backup VPN gateway).

16 This document was created with Win2PDF available at The unregistered version of Win2PDF is for evaluation or non-commercial use only.