High Availability Cluster Solutions for Ubuntu14.04 on Power Author: Iranna D. Ankad (iranna.ankad@in.ibm.com) PowerLinux Test Architect Abstract: In this article we focus on how to set up a two-node apache web servers and two-node load balancers in an active/passive configuration using HAProxy and keepalived on Ubuntu14.04 nodes running under PowerKVM environment. Ubuntu 14.04 is a first Little Endian Linux distribution which enjoys long term support (LTS) on IBM Power8 hardware. This Ubuntu version runs as a guest operating system on IBM PowerKVM, a hypervisor based on open source Kernel Virtual Machine (KVM) technology. Ubuntu and IBM PowerKVM hypervisor are integral part of OpenPower initiative from IBM. Introduction: HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. In this configuration, the load balancer sits between the user and two (or more) backend Apache web servers that hold the same content. The load balancer not only distribute the requests to the two backend Apache servers, it also does check the health of the backend servers. If one of them is down, all requests will automatically be redirected to the remaining backend server. In addition to that, the two load balancer nodes monitor each other using keepalived, and if the master fails, the slave becomes the master, which means the users will not notice any disruption of the service.
The Setup: In my setup I have four Ubuntu14.04 guests running under PowerKVM, which are configured as following: haproxy1: Our first proxy (master)/load-balancer [IP: 9.3.189.211] haproxy2: Our second proxy (failover)/load-balancer [IP: 9.3.189.220] webserver1: Primary apache web server node [IP: 9.3.189.139] webserver2: Secondary apache web server node [IP: 9.3.189.198] Virtual or Shared IP: 9.3.189.219 (this is configured inside haproxy1 and it will be shared with haproxy2) OS version on guest nodes: 3.13.0-30-generic #54-Ubuntu SMP Mon Jun 9 22:46:02 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux OS version on PowerKVM host: 3.10.23-1702.pkvm2_1.4.ppc64 #1 SMP Thu Jun 19 11:32:18 CDT 2014 ppc64 ppc64 ppc64 GNU/Linux Fig. Pictorial view of HA Solution
Preparing Web Servers (Web Server1 and WebServer2) : Install apache packages on both web server nodes #sudo apt-get install apache* Configure HAProxy as a transparent proxy, i.e. it will pass on the original user's IP address in a field called X-Forwarded-For to the backend web servers. Modify the LogFormat line in /etc/apache2/apache2.conf and replace %h with %{X-Forwarded-Fori: i.e [...] #LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Refereri\" \"%{User-Agenti\"" combined LogFormat "%{X-Forwarded-Fori %l %u %t \"%r\" %>s %b \"%{Refereri\" \"%{User- Agenti\"" combined [...] Also, we will configure HAProxy to check the backend servers' health by continuously requesting the file check.txt (translates to /var/www/check.txt if /var/www is your document root) from the backend servers. vi /etc/apache2/sites-available/000-default.conf and put these two lines into it (comment out all other CustomLog directives in your vhost configuration): [...] SetEnvIf Request_URI "^/check\.txt$" dontlog CustomLog /var/log/apache2/access.log combined env=!dontlog [...] Next.. we restart Apache: #/etc/init.d/apache2 restart create the file check.txt (this can be an empty file): #touch /var/www/check.txt Now..we are done with two web server nodes configuration. Next, let us configure the two load balancer nodes. Preparing Load balancer1 and Load balancer2: Install HAProxy packages on both nodes. #sudo apt-get install haproxy
Configure HAProxy on both load balancers Let us back up the original /etc/haproxy/haproxy.cfg and create a new one like this: sudo cat /etc/haproxy/haproxy.cfg global log 127.0.0.1 local0 log 127.0.0.1 local1 notice #log loghost local0 info maxconn 4096 #debug #quiet user haproxy group haproxy defaults log global mode http option httplog option dontlognull retries 3 option redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000 listen webfarm 9.3.189.219:80 mode http stats enable stats auth haproxyuser:passw0rd balance roundrobin cookie SERVERID insert indirect nocache option httpclose option forwardfor option httpchk HEAD /html HTTP/1.1 server webserver1-9.3.189.139 9.3.189.139:80 cookie 1 check inter 2000 rise 2 fall 3 server webserver2-9.3.189.198 9.3.189.198:80 cookie 2 check inter 2000 rise 2 fall 3 backup Afterwards, we set ENABLED to 1 in /etc/default/haproxy:
i.e sudo cat /etc/default/haproxy # Set ENABLED to 1 if you want the init script to start haproxy. ENABLED=1 # Add extra flags here. #EXTRAOPTS="-de -m 16" Install and set up keepalived #sudo apt-get install keepalived To allow HAProxy to bind to the shared IP address, we add the following line to /etc/sysctl.conf: i.e [...] net.ipv4.ip_nonlocal_bind=1... and run: #sysctl -p Next, we configure keepalived on both load balancer nodes, using the configuration file /etc/keepalived/keepalived.conf. Load balancer1: Load balancer1 will be our active (or master) load balancer, so we use this configuration: ubuntu@haproxy1:~$ sudo cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { # Requires keepalived-1.1.13 script "killall -0 haproxy" # cheaper than pidof interval 2 # check every 2 seconds weight 2 # add 2 points of prio if OK vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 101 virtual_ipaddress { 9.3.189.219 track_script { chk_haproxy ubuntu@haproxy1:~$ # 101 on master, 100 on backup
Then we start keepalived: #/etc/init.d/keepalived start Then run: #ip addr sh eth0 you should find that load balancer1 is now listening on the shared IP address, too: i.e ubuntu@haproxy1:~$ sudo ip addr sh eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:c0:a8:7a:89 brd ff:ff:ff:ff:ff:ff inet 9.3.189.211/24 brd 9.3.189.255 scope global eth0 inet 9.3.189.219/24 brd 9.3.189.255 scope global secondary eth0:0 inet6 fe80::5054:c0ff:fea8:7a89/64 scope link ubuntu@haproxy1:~$ Load balancer2: Repeat same steps on load balancer2 as well except that we use priority 100 instead of priority 101 in /etc/keepalived/keepalived.conf which makes lb2 the passive (slave) load balancer: i.e sudo cat /etc/keepalived/keepalived.conf [sudo] password for ubuntu: vrrp_script chk_haproxy { # Requires keepalived-1.1.13 script "killall -0 haproxy" # cheaper than pidof interval 2 # check every 2 seconds weight 2 # add 2 points of prio if OK vrrp_instance VI_1 { interface eth0 state MASTER virtual_router_id 51 priority 100 virtual_ipaddress { 9.3.189.219 track_script { chk_haproxy # 101 on master, 100 on backup
Then we start keepalived: #/etc/init.d/keepalived start Since this node is configured as passive load balancer, it should not be listening on the virtual IP address as long as load balancer1 is up. We can check that with: #ip addr sh eth0 i.e sudo ip addr sh eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:5f:ef:76 brd ff:ff:ff:ff:ff:ff inet 9.3.189.220/24 brd 9.3.189.255 scope global eth0 inet6 fe80::5054:ff:fe5f:ef76/64 scope link Now..start HAProxy service on both load balancers. # /etc/init.d/haproxy start Testing it out.. I am hosting a simple http application from both web servers at /var/www/html/index.html You can now make HTTP requests to that page using virtual IP address 9.3.189.219 (or to any domain/hostname that is pointing to the virtual IP address), and you should be able to load the http application from the backend web servers (below is an example of my http web page)
HAProxy Statistics: You can access HAProxy statistics under the URL http://9.3.189.219/haproxy?stats. [Login to HAProxy using the appropriate user credentials. In this case I am using haproxyuser and passw0rd, you might have noticed that we have set these credentials in the haproxy.cfg file] Below screen shots demonstrate working of HAProxy: Note: Initially all nodes (load balancers & web servers) are up and running Test Scenario 1: Let us test high-availability / failover capabilities by switching off primary web server and load balancers The load balancer should then redirect all requests to the remaining backend web server. Afterwards, switch off the primary load balancer1 then load balancer2 (slave) should take over immediately. You can check that by running: Web server1: ubuntu@webserver1:~$ sudo /etc/init.d/apache2 stop * Stopping web server apache2 * ubuntu@webserver1:~$ Load balancer1: ubuntu@haproxy1:~$ sudo /etc/init.d/haproxy stop * Stopping haproxy haproxy [ OK ] ubuntu@haproxy1:~$
Load balancer2: Observe the virtual IP address in the output on load balancer2. This indicates secondary load balancer has taken over the master role. sudo ip addr sh eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:5f:ef:76 brd ff:ff:ff:ff:ff:ff inet 9.3.189.220/24 brd 9.3.189.255 scope global eth0 inet 9.3.189.219/32 scope global eth0 inet6 fe80::5054:ff:fe5f:ef76/64 scope link HAProxy statistics: Notice primary web server is down. Application can be accessed via virtual IP (9.3.189.219), which in turn pulls this page from backup web server.
Test Scenario 2: Now.. let us bring up primary web server and load balancers (both should now take over the master roles respectively), then bring down respective secondary web server and load balancers. Web server1: ubuntu@webserver1:$ sudo /etc/init.d/apache2 start *Starting web server apache2 ubuntu@webserver1:$ Web server2: ubuntu@webserver1:$ sudo /etc/init.d/apache2 stop sudo: unable to resolve host webserver2 * Stopping web server apache2 * ubuntu@webserver1:$ Load balancer1: ubuntu@haproxy1:~$ sudo /etc/init.d/keepalived start * Starting keepalived keepalived [ OK ] ubuntu@haproxy1:~$ sudo /etc/init.d/haproxy start * Starting haproxy haproxy [ OK ] ubuntu@haproxy1:~$ Load balancer2: sudo /etc/init.d/haproxy stop * Stopping haproxy haproxy [ OK ] sudo /etc/init.d/keepalived stop * Stopping keepalived keepalived [ OK ] sudo ip addr sh eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:5f:ef:76 brd ff:ff:ff:ff:ff:ff inet 9.3.189.220/24 brd 9.3.189.255 scope global eth0 inet6 fe80::5054:ff:fe5f:ef76/64 scope link Notice the absence of virtual IP in the above output, which is now taken over by primary load balancer1
HAProxy statistics: Now..notice secondary web server is down and primary web server has taken over as master role. And... the application can still be accessed without experiencing any downtime. That's it...!