FILECLOUD HIGH AVAILABILITY

FILECLOUD HIGH AVAILABILITY ARCHITECTURE VERSION 1.0 FileCloud HA Architecture... 2 Load Balancers... 3 FileCloud Component: App server node... 3 FileCloud Component: Mongo DB Replica set... 3 Instructions for HA deployment in Ubuntu 14.04... 5 Setting up Ha-Proxy:... 6 Setting up FileCloud Appserver nodes:... 8 MongoDB Replica set setup... 9 Verification of service... 10 Setting Managed Storage... 11 1

FileCloud HA Architecture FileCloud servers can be configured for HA environment to improve service reliability and reduce downtime in your IT environment. External Network Client s Firewal l DM Z Corporate Network Filecloud components Load Balancer(s) App server Nodes Mongo DB Replica set File Servers LDAP/AD This deployment uses the classic 3-tier approach with Load balancer handling the client traffic, Application server nodes to serve requests and redundant database servers to store application data. 2

Load Balancers The Load balancer will route the traffic to the FileCloud Application nodes. Load balancers (LB) provide many advantages to serving requests from your FC servers because they allow you to better control how the traffic is handled in order to provide the best performance.. If one or more App server nodes fail, the load balancer will automatically reroute traffic to other App server nodes. Typically there is no need to scale the number of load balancers because these servers can handle a very large amount of traffic. However, more than one load balancers can be used to provide additional reliability in the event of one of the load balancer failure. In order to protect against load balancer hardware failure, multiple A records for the load balancer host name in the DNS service can be used. The idea here is that different clients will get different ordered lists of IP addresses corresponding to your domain name. This has the effect of distributing requests across the group of IPs in a specific manner. If an IP address is does not respond in an appropriate amount of time, the client will time out on that request and move on to the next IP address until the list is exhausted or it finds a connection that's valid. FileCloud Component: App server node The FileCloud App server node consists of the Apache webserver as well as the FileCloud Application code to server the client requests. The FileCloud app server nodes do not contain any application specific data. The data is retrieved from the MondoDB replica sets. Because of this, the FileCloud Appserver nodes can be added or removed without disrupting the service. FileCloud Component: Mongo DB Replica set The MongoDB database replica sets provide high availability with automatic failover support. Failover allows a secondary member to become primary in the event of failure to the primary DB node. The minimum number of DB nodes needed for MongoDB is three. All app server nodes will connect to primary and in the event of primary node failure, a new primary is elected and all the app server nodes will switch to the new primary. 3

Primary Secondary Heartbeat Secondary 4

Instructions for HA deployment in Ubuntu 14.04 The following instructions apply for deploying FileCloud HA in Ubuntu 14.04 environment. This can easily be adapted to other Linux flavors as well. This example is showing HTTP and can be expanded easily to use HTTPS as well. Some of the instructions will have to be adapted to your specific environment. Load Balancer: The load balancer of choice is HaProxy (http://www.haproxy.org/). HaProxy is a high performance and battle tested load balancer and allows you to scale your FileCloud deployment quickly as well. For this set of instructions, we will use the following names. NOTE: Before starting the install, ensure the servers are already available and their IP addresses are known. Ha-LB Ha-WS1 Ha-WS2 Ha-DB1 Ha-DB2 Heartbeat Ha-DB3 Component Type Load balancer App server node 1 App server node 2 Name Ha-LB Ha-WS1 Ha-WS2 5

Mongo DB server 1 Mongo DB server 2 Mongo DB server 3 Ha-DB1 Ha-DB2 Ha-DB3 Setting up Ha-Proxy: Use apt-get command to install HAProxy apt-get install haproxy Enable HAProxy to be started by the init script vi /etc/default/haproxy set the ENABLED option to 1 ENABLED=1 Move the default config file to create a new default configuration file mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.save vi /etc/haproxy/haproxy.cfg Add the following in the empty haproxy.cfg file global log 127.0.0.1 local0 notice maxconn 2000 user haproxy group haproxy Notes: The log directive mentions a syslog server to which log messages will be sent. On Ubuntu rsyslog is already installed and running but it doesn't listen on any IP address. We'll modify the config files of rsyslog later. The maxconn directive specifies the number of concurrent connections on the frontend. The default value is 2000 and should be tuned according to your VPS' configuration. The user and group directives changes the HAProxy process to the specified user/group. These shouldn't be changed. defaults log global mode http 6

option httplog option dontlognull retries 3 option redispatch timeout connect 5000 timeout client 10000 timeout server 10000 Notes: We're specifying default values in this section. The values to be modified are the various timeout directives. The connect option specifies the maximum time to wait for a connection attempt to a VPS to succeed. The client and server timeouts apply when the client or server is expected to acknowledge or send data during the TCP process. HAProxy recommends setting the client and server timeouts to the same value. The retries directive sets the number of retries to perform on a VPS after a connection failure. The option redispatch enables session redistribution in case of connection failures. So session stickness is overriden if a VPS goes down. Notes: listen filecloud 0.0.0.0:80 mode http stats enable stats uri /haproxy?stats stats realm Strictly\ Private stats auth proxyuser:proxypassword balance roundrobin option http-server-close timeout http-keep-alive 3000 option forwardfor server Ha-WS1 xx.xx.xx.xx:80 check server Ha-WS2 xx.xx.xx.xx:80 check This contains configuration for both the frontend and backend. We are configuring HAProxy to listen on port 80 for filecloud which is just a name for identifying an application. The stats directives enable the connection statistics page and protect it with HTTP Basic authentication using the credentials specified by the stats auth directive. 7

This page can be viewed with the URL mentioned in stats uri so in this case, it is http://ha- LB/haproxy?stats; The balance directive specifies the load balancing algorithm to use. Options available are Round Robin (roundrobin), Static Round Robin (static-rr), Least Connections (leastconn), Source (source), URI (uri) and URL parameter (url_param). Information about each algorithm can be obtained from the official documentation. The server directive declares a backend server, the syntax is: server <name> <address>[:port] [param*] In the directive server Ha-WS1 xx.xx.xx.xx:80, replace xx.xx.xx.xx with the actual IP address of the app server nodes. Start HAProxy service service haproxy start Setting up FileCloud Appserver nodes: The following steps have to be done for each of the FileCloud app server nodes (Ha-WS1 and Ha-WS2) Download and install FileCloud debian package (The actual URL may not be the same as shown below) cd /tmp wget http://patch.codelathe.com/tonidocloud/live/installer/file_cloud_deb.t gz tar xzvf file_cloud_deb.tgz chmod +x install.sh./install.sh During the install, you can continue the install as if it is a standalone installation and provide a dummy cloud storage (something like /tmp). This will be changed later. Stop apache and mongodb service apache2 stop service mongodb stop Edit the configuration file and locate and replace the following (if those lines are not found, add it) 8

vi /var/www/config/cloudconfig.php Add/replace the following keys //... Cloud Database define("tonidocloud_dbserver", "mongodb://ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaset=rs0"); //... Audit Database define("tonidocloud_audit_dbserver", "mongodb:// ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaset=rs0"); // Settings Database define("tonidocloud_settings_dbserver", "mongodb:// ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaset=rs0"); Edit Cloud config and add/replace the following keys vi /var/www/config/localstorageconfig.php define("tonido_localstorage_dbserver", "mongodb:// ip of ws-db1,ip of ws-db2,ip of ws-db3/?replicaset=rs0"); Start web server service apache2 start MongoDB Replica set setup The following should be done on each of the database servers (HA-DB1, HA-DB2 and HA-DB3) Download and install FileCloud debian package cd /tmp wget http://patch.codelathe.com/tonidocloud/live/installer/file_cloud_deb.t gz tar xzvf file_cloud_deb.tgz chmod +x install.sh./install.sh 9

During the install, you can continue the install as if it is a standalone installation and provide a dummy cloud storage (something like /tmp). This will be changed later. Stop apache and mongodb service apache2 stop service mongodb stop update-rc.d mongodb defaults #Start mongodb automatically update-rc.d apache2 disable #Do not start apache on DB nodes In each of the DB server (HA-DB1, HA-DB2 and HA-DB3) Open mongo configuration file vi /etc/mongodb.conf Uncomment replset and set it like the following replset = rs0 Save the config file and start Mongo DB server service mongodb start Open mongo shell mongo Initiate the replica set rs.initiate() Add the IP address of the other Database server (For example, when setting up WS-DB1, add the IPs of WS-DB2 and WS-DB3) rs.add( ip address of other WS-DB2 ) rs.add( ip address of other WS-DB3 ) Verification of service Once all the service is set, Connect to the Haproxy web page ( http:// <loadbalancer IP>/haproxy?stats ) to ensure that Haproxy is able to see all the FileCloud app server nodes. In each of the three database server nodes, connect to mongo shell and enter rs.status() to see the actual value rs.status() 10

It should show something like rs.status() { "set" : "rs0", "date" : ISODate("2014-09-03T20:52:14Z"), "mystate" : 2, "members" : [ { "_id" : 0, "name" : "<ip of other DB>:27017", "health" : 0, "state" : 8, "statestr" : "(not reachable/healthy)", "uptime" : 0, "optime" : Timestamp(1409777412, 1), "optimedate" : ISODate("2014-09- 03T20:50:12Z"), "lastheartbeat" : ISODate("2014-09- 03T20:52:13Z"), "lastheartbeatrecv" : ISODate("2014-09- 03T20:51:54Z"), "pingms" : 0 }, { "_id" : 1, "name" : "<Ip of other DB>:27017", "health" : 1, "state" : 2, "statestr" : "SECONDARY", "uptime" : 749, "optime" : Timestamp(1409777412, 1), "optimedate" : ISODate("2014-09- 03T20:50:12Z"), "errmsg" : "syncing to: hadb1.codelathe.com:27017", "self" : true }, Setting Managed Storage Since the FileCloud Appserver nodes do not store any of the application data, the managed storage must be an external location (A NAS, ISCSI, SAN, Amazon S3 or Open stack) 11

IN THIS EXAMPLE, WE ASSUME THAT EITHER NAS, NFS MOUNT IS ALREADY AVAILABLE AND MOUNTED ON EACH OF THE WEBSERVER NODES. Open the FileCloud Admin portal by opening http:// <load balancer IP/ui/admin/index.html and log into the Administration portal. Navigate to Settings->Storage Tab and set the mounted path and click save Once the setup is complete, create user accounts by connecting to the admin portal and then log into the user account using the load balancer IP (which will route the traffic to one of the app server node). To test app server HA, Turn off one of App server by logging into the app server (say Ha-WS1) and stop Apache (using service apache2 stop). The service will be accessible because HaProxy will reroute traffic to Ha-WS2. To test database HA, Connect to the primary database server (Check the role by typing mongo in the database server and the prompt will indicate if it is a PRIMARY or SECONDARY ) and turn off the mongodb server (service mongodb stop). This will cause a failover and one of the secondary will become the primary. 12

The service will be disrupted during the failover (It can take up to a minute) and the service will resume automatically. 13