Dual Server Hot Standby Architecture for Disaster Recovery

Transcription

1 Dual Server Hot Standby Architecture for Disaster Recovery Team Members Bharadwaj S bharadwaj.s@iiitb.org Joya Neema joyav.neema@iiitb.org Salini S salini.s@iiitb.org Shefali Das shefali.das@iiitb.org Udita Gangwal udita.gangwal@iiitb.org Technical Report IIITB-TR April 2012

2 Abstract Of companies that had a major loss of business data, 43% never reopen, 51% close within two years, and only 6% will survive long-term [1]. Minimizing the impact of unforeseen events becomes a paramount for any organization to ensure smooth working of business operations. We propose a dual server hot standby system wherein initially, the main server takes care of implementing the business operations. At the time of a catastrophic event the hot standby server, which has information of all valid transactions until time of disruption of service, will takeover services of main server with minimal reduction user experience. The proposed solution will ensure to restore the services running on a disaster affected server with transaction information no older than 30 seconds prior to crash is lost and secure that there is no more than 10% reduction in performance of normal running of server Bharadwaj Sreenivasan, Joya V Neema, Salini Sasidharan, Shefali Das, Udita Gangwal. This material is available under the Creative Commons Attribution- Noncommercial-Share Alike License. See by-nc-sa/3.0/ for details.

3 Contents 1 Introduction 5 2 System Architecture Main Server Hot Standby Server Implementation Details SQL Replication Apache State Management And State Replication Client Re-routing Performance Metrics 20 References 27 Appendix:Performance Metrics Data 28 3

4 List of Tables 1 Data for CPU utilization with disaster recovery software working Data for CPU utilization without disaster recovery software working Data for response time (threads varying) without disaster recovery software working Data for response time (threads varying) with disaster recovery software working Data for response time (requests varying) without disaster recovery software working Data for response time (requests varying) with disaster recovery software working Data for throughput without disaster recovery software working Data for throughput with disaster recovery software working List of Figures 1 System architecture Commenting the bind address in main server my.cnf file Changing the database name (to exam) in main server my.cnf file The master status on main server Changing the server-id at the slave s side Transaction failure recovery scenario Number of requests per second v/s throughput Number of requests per second v/s response time Number of threads v/s CPU utilization Number of threads v/s response time

5 1 Introduction This document describes an approach towards disaster recovery on disaster affected systems. Disasters, by their very nature, cannot be predicted, in either their intensity, timing, or effects [2]. However, all enterprises should be prepared to protect themselves against loss of data, that in the worst case, might lead to the failure of their entire business. It is necessary for services running on enterprise computing systems affected by major disasters (major earthquake, flood, tsunami, 9/11-type terrorist attack etc.) to be recovered with minimal disruption. The steps that need to be taken before and immediately after a disaster are operating system dependent. For disaster recovery, it is important to have an off-site backup data, in case of any physical damage to the site where the server resides. The increased demand in recent years for continuous operation of databases has created interest for remote backups, which allow systems to continue their operation, even in the presence of disasters. Objective: To create a utility that can be used to restore services running on a disaster-affected system within minutes, from elsewhere. The state of the system must be recovered to be as close to current as possible (certainly no older than 30 seconds prior to crash), and the utility must not slow the normal running of the system (preferably, a performance penalty of 5% or less, but certainly no more than 10% is required). Problem Description: Disaster recovery systems could be effectively implemented using a hot standby server [3]. Hot standby server means a method of redundancy in which the primary and secondary (, i.e., backup) systems run simultaneously. The data is mirrored to the hot standby server in real time so that both systems contain identical information. An issue that needs to be addressed is the time delay for data transfer from the main server to the hot standby server. This delay would cause inconsistencies between the databases in the hot standby and the main server. Client interaction with the hot standby database is enabled only in the event of failure in the main server and after a proper recovery procedure is executed. Another challenge in continuous data replication is to introduce replication without affecting the performance of the main server [4]. Apart from the data replication, the state of the services running on the main server also needs to be communicated in order to restore them effectively. Approach Towards Solution: The project aims at implementing a scaled down version of a typical disaster recovery system. In this project, we have selected a single application, running on the main server as the test application. The application has client-server model and multiple users are interacting with the application. During the event of a failure at the main server, a mechanism is triggered to restore application in the hot standby server. All the clients would be routed to the hot standby server seamlessly. 5

6 2 System Architecture We have considered a web application scenario for providing hot standby functionality. A web application hosted on a web server will facilitate on-line transactions for users with their private login accounts. The initial interaction of the user will be with the main server, which has high risk of damage due to natural calamities. Information regarding transactions (either current or completed) has to be stored in the main server. A hot standby server is the one which is physically separated from the main server, located at a safe place (safe with respect to probability of occurrence of a natural disaster), with all the information about the ongoing and past transactions transferred from the main server, via the Internet or a Secure line. Figure 1: System architecture With this kind of a setup, if a disaster occurs, the clients connected to main server can be re-routed to the hot standby server and thereby resume their transactions from the same state as it was when the switch-over took place. To facilitate this resumability of transactions, software modules are required in the main server and the hot standby server. The architecture consists of a main server and a hot standby server as shown in Figure 1. 6

7 2.1 Main Server 1. Application-Server Side: This refers to the application that runs on the main server and interacts with the clients. This application will provide certain services (,eg., on-line ticket booking, on-line exam) for clients via the Internet. 2. Application Data Storage: This refers to the database that is present in the main server. This provides persistent data storage for all services provided by the application. Whenever the clients interact with the application, the client s transactions will be recorded in this database. 3. Recovery Module: The Recovery Module in the Main server side consists of the following modules: (a) Database Replication Software: The Database Replication Software is used to carry out a synchronous replication of the data present in the database. This replication takes place as and when the database gets updated due to the client s interaction with the application. This replicated data is then handed over to the Data Transfer Module for transfer across the network. (b) Data Transfer Module: Data Transfer Module is present on the main server allows data to be transferred to the Hot Standby Server via a network infrastructure. The Data Transfer Module transfers the data, that is obtained from the Database Replication Software as well as the Application State Mirroring Software. The data on the main server takes some time to reach the hot standby server resulting in a measurable delay which is determined by the network infrastructure. (c) Application State Mirroring Software: The Application State Mirroring Software [5] is used to carry out a synchronous replication of the state of the application. By state of the application, we mean an instance of the application that is being executed. This module will capture the application state in (nearly) real time. This is essential because at the time of failure of the main server, the application in the hot standby server is expected to resume to the state of the application in the the main server just before failure. The state of the application is saved in the database. 7

8 2.2 Hot Standby Server 1. Application-Server Side: This refers to the application that is pre-installed in the hot standby server and has the same functionality as the application running on the main server. However, this application gets invoked only on the failure of the main server. The Application Restore Module on the hot standby reconstitute this application to the most recent state. 2. Application Data Storage: The Application Data Storage is the replica of the database on the main server. Database Restore module takes care of the database replication. 3. Recovery Module: The Recovery Module in the Hot Standby Server side consists of the following modules: (a) Database Restore Software: This software will update the database in the hot standby server with the data received from the main server via Data Replication Software. Thus, the data at the hot standby server will always be consistent with the data at the main server. (b) Application Restore Software: The Application Restore Software gets the state information from the Application State Mirroring Software present in the main server. After failure of the main server, this module will restore the application to the most recent state on the hot standby server. (c) Main Server Monitor: This module is used to check whether the main server is active or not. This is achieved through mutually transmitted messages between the main server and the hot standby server. When the main server crashes, this module will trigger the Recovery Mechanism [6]. (d) Client Re-Routing Module: After the main server crashes, the network switching has to take place. Once a disaster occurs at the main server, the Client Re-Routing Module will trigger a mechanism to reroute all the clients to the hot standby server (which were originally connected to the main server). 3 Implementation Details The main modules required for implementation of disaster recovery is as follows. SQL Replication Apache State Management And State Replication Client Re-routing 8

9 3.1 SQL Replication Replication is a process of copying and distributing data and database objects from one database to another and then synchronizing between databases to maintain consistency. Using replication, we can distribute data to different locations and to remote or mobile users over local and wide area networks, dial-up connections, wireless connections, and the Internet. There are many reasons why SQL replication exists. In our case, replication is done to ensure redundancy, i.e., when a business runs a mission-critical database, it is wise to have a replication of this database on a separate physical database server, which can take over in case of failure in the primary database (software corruption, hardware failure, etc.). To understand how SQL replication works, we define two replication terms: 1. Publisher 2. Subscriber In a SQL replication the Publisher is the part that offers the data for distribution. The Subscriber is the part that consumes (gets updates) the data from the Publisher. In our project, the publisher is the main server through which the clients communicate directly, and the Subscriber is the hot standby machine, which would take up the whole control in case of a failure. Types of SQL Replication: MS SQL Server has three main replication models Merge Replication Transactional Replication Snapshot Replication The Merge replication is used in server-to-client environments when both Publisher and Subscriber need to make changes to their respective databases. In this case, multiple subscribers might update the same data at various times and propagate those changes to the Publisher and to other Subscribers. Also subscribers need to receive data, make changes off-line, and later synchronize changes with the Publisher and other Subscribers. The replication merges the changes made in all the locations. The application requires net data change rather than access to intermediate data states. For example, if a row changes five times at a Subscriber before it synchronizes with a Publisher, the row changes only once at the Publisher to reflect the net data change (, i.e., the fifth value). When using merge replication, conflicts might occur. For example, duplicated primary keys. If there is a conflict, then the merge replication follows predetermined conflict resolution plan to correct the issue. Merge replication allows various sites to work autonomously and later 9

10 merge updates into a single, uniform result. The Transactional replication is typically used in server-to-server environments and is appropriate when incremental changes are required to be propagated to Subscribers as they occur. It is usually used with databases where data changes frequently and there is need for constant refreshing of the data. The replication process watches the publisher s database for any changes. If there are changes, it distributes them over to the replication subscribers. The application requires access to intermediate data states. For example, if a row changes five times, transactional replication allows an application to respond to each change (such as firing a trigger), not simply the net data change to the row. The Snapshot replication is relatively rarely used, and as its name implies it does a snapshot of the publisher database at certain point of time and distributes it to the subscribers. The snapshot process is commonly used to provide the initial set of data and database objects for transactional and merge publications, but snapshot replication can also be used by itself. Snapshot replication by itself is used when data changes infrequently and it is acceptable to have copies of data that are out of date with respect to the Publisher for a period of time. Snapshot replication is most appropriate when data changes are substantial but infrequent. For example, if a sales organization maintains a product price list and the prices are updated at the same time once or twice each year, replicating the entire snapshot of data after it has changed is recommended. The Snapshot replication is time and resource consuming compared to the other two SQL replication types and is used when you need to fully overwrite the database at the subscriber or when you do an initial database replication update, which will be refreshed using one of the other two SQL replication types after the initial copy. In our project, we are making use of Transactional replication, in which any changes occurring on the main server are replicated back to the hot standby server. How to setup database replication in MySQL: MySQL replication allows you to have an exact copy of a database from a master server on another server (hot standby server), and all updates to the database on the master server are immediately replicated to the database on the slave server so that both databases are in sync. This is not a backup policy because an accidentally issued DELETE command will also be carried out on the slave; but replication can help protect against hardware failures though. Database being replicated - exam main server s IP address hot standby s IP address Both the servers are present in different networks in order to simulate the scenario 10

11 that the hot standby server must be geographically separated from the main server. The database exam with tables and data is already existing on the master server, but not on the slave (hot standby server). Setting up replication in the master/main server: 1. Install MySQL server and client on each node: The following command would install the mysql-server mysql-client: sudo apt-get install mysql-server mysql-client 2. Configure the master server: To make sure that the replication works, we must make MySQL listen on all interfaces on the master server, therefore we comment out the line bindaddress = in /etc/mysql/my.cnf (as shown in Figure 2) by using the following command: vi /etc/mysql/my.cnf Figure 2: Commenting the bind address in main server my.cnf file 3. Restart MySQL afterwards: The following command restarts the mysql. /etc/init.d/mysql restart Now we set up a replication user, slave user, that can be used by hot standby server to access the MySQL database on Main server: mysql -u root -p On the MySQL shell, run the following commands: GRANT REPLICATION SLAVE ON *.* TO slave slave IP IDENTIFIED BY slave password ; FLUSH PRIVILEGES; quit; 4. Furthermore we have to tell MySQL for which database it should write logs (these logs are used by the slave to see what has changed on the master), 11

12 which log file it should use, and we have to specify that this MySQL server is the master. We want to replicate the database exam, so we add/enable the lines shown in Figure 3 in my.cnf file: vi /etc/mysql/my.cnf Figure 3: Changing the database name (to exam) in main server my.cnf file 5. Then MySQL is restarted: /etc/init.d/mysql restart 6. Next we lock the exam database on main server, find out about the master status of main server, create an SQL dump of exam (that we will import into exam on hot standby server so that both databases contain the same data), and unlock the database so that it can be used again: mysql -u root -p On the MySQL shell, run the following commands: USE exam; FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS; The last command should show something as shown in Figure 4: 7. Leaving MySQL shell will cause the database lock will be removed, and this is not what we want right now because we must create a database dump now. While the MySQL shell is still open, we open a second command line window where we create the SQL dump snapshot.sql and transfer it to main server (using scp): Main Server: cd /tmp// mysqldump -u root -pyourrootsqlpassword opt exam > snapshot.sql scp snapshot.sql root@slave s IP:/tmp 12

13 Figure 4: The master status on main server 8. Afterwards, we can close the second command line window. On the first command line window, we can now unlock the database and leave the MySQL shell: UNLOCK TABLES; quit; Setting up replication in the hot standby server/slave: 1. Open my.cnf file on the hot standby server and make the following settings as shown in Figure 5: vi /etc/mysql/my.cnf Figure 5: Changing the server-id at the slave s side The value of the server-id must be unique and thus different from the one on the master. 2. Restart mysql: /etc/init.d/mysql restart 3. Before setting up the replication, create an empty database exam on the hot standby server: mysql -u root -p CREATE DATABASE exampledb; quit; 4. Now we import the SQL dump snapshot.sql by the following command: /usr/bin/mysqladmin user=root password=yourrootsqlpassword stop-slave 13

14 cd /tmp mysql -u root -pyourrootsqlpassword exampledb snapshot.sql 5. Now we connect to mysql again mysql -u root -p and run the following command to make the hot standby server a slave of the master server: CHANGE MASTER TO MASTER HOST= , MASTER USER= slave user, MASTER PASSWORD= slave password, MASTER LOG FILE= mysql-bin , MASTER LOG POS=106; MASTER HOST is the IP address of the master. MASTER USER is the user we granted replication privileges on the master. MASTER PASSWORD is the password of MASTER USER on the master. MASTER LOG FILE is the file MySQL gave back when you ran SHOW MASTER STATUS; on the master. MASTER LOG POS is the position MySQL gave back when you ran SHOW MASTER STATUS; on the master. 6. Finally we start the slave: START SLAVE; 7. then slave status is checked: SHOW SLAVE STATUS \G; It is important that both Slave IO Running and Slave SQL Running have the value Yes in the output. 3.2 Apache State Management And State Replication 1. Background Hyper Text Transfer Protocol (HTTP) protocol is inherently stateless. An Apache server which runs the http protocol does not maintain any information about the status of each user and the requests they are making. Whenever a new request comes from a user, HTTP considers the request as a new request. When the activity of a user has to be tracked, the burden is usually on the web application to track the progress of the user from page to page. A web application can use many ways to store the state information. Following are some of widely used methods[7]. (a) Store the state within hidden fields of the web page (eg: hidden fields of a fill out form). 14

15 (b) Store the state in cookies. (c) Store the state in Uniform Resource Identifier (URI). (d) Store the state in web server process memory. (e) Store the state files within web server. (f) Store state in database within the web server. The first three methods are client-side techniques where the state information is saved on the browser side of the connection. The later three techniques are server-side techniques where state information is saved on the web server. Client side techniques are not advisable when when the state information should not be modified by the user. 2. Generic-application state maintenance module Design In most of the web applications the state information is sensitive so that it should not be manipulated by the user, hence we have chosen a server side state storage method. We are using a database within the web server to store the state of all the users of the web server. Usually the web developers has to do some work to incorporate the state maintenance in their web applications. The greatest advantage of this implementation is it takes away the burden of state maintenance from the web developers and provides a pluggable module to maintain the state of the multiple users. Basic concepts: The below two concepts are the corner stones of the module. (a) Unique identification of the user: To store the data pertaining to a particular user, it is required to uniquely identify a user and this unique identification should be preserved across the different requests by the same user. This unique identification is named as the session id. The session id acts as the primary key of the database tuple which stores the information about the user. (b) User transactions: Transaction is the collection of actions, which is done between the http request and the response of the corresponding request. The transaction starts when the web server receives an http request. Upon receiving the http request, the web server performs few actions. They may include the following: i. If its a POST type of request, then use the data in the post (like save it in a database, or use it for calculation). ii. Verify the user by validating the user name and password (may be passed in the form of cookie). 15

16 iii. If its a GET request, then render the page requested by the client. Once a transaction starts there may be two conditions: (a) Successful transaction: The transaction is carried out successfully and the response is send to web user. (b) Failed transaction : A transaction failure is when the client has send the http request and the web server goes down before the transaction could complete. The web server might have partially processed the request, but without completing the processing and sending the response message. Under this condition the web server should either seamlessly recover from failure or it should show a failure message to the user and quit. Figure 6: Transaction failure recovery scenario 16

17 Implementation (a) Assigning unique identification for users [8] Session support in Hypertext Processor (PHP) is used to allocate a unique identification for all the web users. As soon as the web server receives a request, it runs a php script, which will use the following php function to generate a unique session id bool session start ( void ) session start() creates a session or resumes the current one based on a session identifier passed via a GET or POST request, or passed via a cookie. As result of this function call a unique session id is created for a user session. (b) Tracking and Storing the user transactions The transaction details are stored in the mysql database of the web server. The table which stores the transaction details consist of the following fields. 1. session id - This is the primary key of the table. This field contains the unique session id of the user currently logged in. 2. session obj - - This field contains a serialized version of the session object. The session object consist of the requested Universal Resource Locator (URL) and the Post parameters of the current request. 3. transaction flag - This field contains a boolean value if the value is 0 indicates that the transaction has started and in progress, if the transaction flag is 1, that indicates that last transaction has successfully completed and there are no transactions in progress (Assuming that one user can have only one transaction at a time). (c) Marking the start of the transaction When the web server receives an http request from a particular user, the first action is to run txn init php script, which will mark the transaction as started. The txn init php script makes appropriate changes in the database to track the user request. i. The tuple which corresponds to the current user is retrieved from the database using the session id (primary key), then the script retrieves the URL of the current request from the $ SERVER [9] global variable of php. The post variables are retrieved from the $ POST global variable of the php. These values are serialized into a session object and the session object is updated in the database. ii. The transaction flag field is set to 0 to mark that the transaction has started. To make sure that the web server will run the php script just after receiving the http request, the auto prepend file [8] directive of PHP is used. The php script which does the transaction initiation function is mentioned as the auto prepend file [8] in the php.ini. As a result this file will be run by the web server, before the main file of the http request. 17

18 After the above php script complete the functionality, the http request is served as usual manner. (d) Marking the end of the transaction As with the beginning of the transaction, the end of transaction is marked by txn end php script. This php script name is mentioned as the value of auto append file in the php.ini. As a result the web server executes this file just after the main php file is executed. The txn end.php file does the following things. i. Retrieves the database tuple which stores the state information of the current user. ii. Update the above retrieved tuple with transaction flag as 1 indicating the current transaction has ended. (e) Replication of the transaction state As and when a transaction started, the url of the current request and the post parameters are saved into the main server s mysql database. By the mysql replication, the same transaction data is replicated to the hot standby. (f) Recognizing the transaction failure If the transaction fails to complete successfully, the transaction flag field of the database will remain 0. The corresponding session obj field, gives the details of the request that has not been completed successfully. (g) Action on transaction failure Whenever the client requests are re-routed to the hot standby, hot standby checks the transaction flag of the last transaction, if the transaction flag is 0, that means the last transaction was a failure, hence the hot standby runs a script which retrieves the last requested URL and sends that request to the web server once more. This requested is auto generated without the intervention/knowledge of the user.curl [10] is used to make this kind of request. Hence, the failed request is run once more and the transaction is completed successfully. After this transaction is completed, the normal user transactions can take place. 3.3 Client Re-routing Initially, all the clients will be connected to the main server. On the event of a failure at the main server, the clients have to be re-routed to the hot standby server. The aim of this module is that clients should be oblivious of the fact the there has been a switch over of authority of management of transactions from main server to hot standby server. The re-routing from the main server to the hot standby server could be done in different ways. The two most appropriate approaches are: 1. By making use of Virtual IP (VIP) 2. By Domain Name System (DNS) translation 18

19 Virtual IP (VIP) is an IP address that is shared among multiple servers. A Virtual IP address eliminates a host s dependency upon individual network interfaces. Incoming packets are sent to the system s VIP address, but all packets travel through the real network interfaces. In our setup, the main server and the hot standby server could share the same VIP and the clients could get re-routed to the hot standby server seamlessly (based on the VIP) on the failure of the main server. There are two major drawbacks of this method of re-routing: The VIP cannot be used over multiple networks. The servers sharing the VIP should be present in the same network. But in reality, the main server and the hot standby server are geographically separated. Secondly, the very well-known problem of Split Brain could happen with this approach. Split Brain is a scenario wherein a momentary loss of connection with the Main Server might confuse the Hot standby into believing that a disaster has occurred. This results in both Main server and the Hot standby accept transactions.thus leading to incomplete or invalid transactions being made. Hence we do not go with this way of re-routing the clients to the hot standby server. Hence, the Domain Name Server (DNS) based re-routing is used in order to route the clients to the hot standby server on the event of a disaster in the main server. DNS is a hierarchical distributed naming system for computers, services, or any resource connected to the Internet or a private network. It associates various information with domain names assigned to each of the participating entities. DNS associates host names with their respective IP addresses, so that when users want to connect to other machines on the network, they can refer to them by name, without having to remember IP addresses. The application in the main server and the hot standby server share the same domain name (online-exam.com); both the servers have different IP addresses. The router acts as the name-server in the setup. This name-server translates the domain name to the respective IP address. Clients use this name-server to translate the application s domain name. Before the disaster, the mapping in the name-server would be: DOMAIN NAME > MAIN SERVER After the disaster, the DNS changes its translation to: DOMAIN NAME > HOT STANDBY SERVER The Linux DNS server BIND (Berkeley Internet Name Domain) service [11] has been used for resolving host names to IP addresses. 19

20 The following are the steps to configure the BIND service: 1. The bind9 name-server has been installed in the Linux machine that acts as the router. After this is done, a bind directory could be seen in the path /etc. 2. Next a bind DNS zone file is created for the domain name exam-online.com. This file contains a DNS record to assist a name-server resolve a fully qualified domain name to an IP address. 3. The next step is to train the name-server to resolve a host from an IP address. For this, another file is created. At this point, we have two files ready. 4. After this, we need to add an IP address of a stable DNS server to a named.conf.options file. This IP address is used in case that a local DNS server do not know the answer the a name resolution query. For example, we could change to (Google IP). But we have done away with this step as we do not need any authoritative Server to resolve the domain name. The bind name server is started in the beginning and when the disaster occurs, it is restarted with the new mapping. This is done with the help of shell scripts. 4 Performance Metrics Performance of a server is an important issue when we are doing lot of transaction of real time data. For checking the performance of server, several performance metrics are used for monitoring the server. There are many tools through which performance of a server can be measured: ab Munin JMeter Nagois For testing the performance metrics, we are using JMeter [12]. JMeter is an Apache tool and is an open source software. Typically, JMeter is used for analysing and measuring the performance of a variety of services, with a focus on web applications. Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, File Transfer Protocol (FTP) Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyse overall performance under different load types. We can use it to make a graphical analysis of performance or to test your server/script/object behaviour under heavy concurrent load. 20

21 How JMeter is used in our project: We created a new thread group where we are setting number of threads and ramp up time. We added config element, HttpRequest, used for giving server name or IP address and path to the web page. All the parameters are also given here which is required for session. We added one more Config Element, CSV Data Set Config, using which we are simulating 100 concurrent users. Here, we have to give path to that.csv file which is generating 100 users. We also have to mention the variables names which a user is giving to the website. In our case, it is USER NAME and PASSWORD. For observation, we used Listener Element, Graph Results which will generate the graph and provide all the information about the performance of the server. In our project, we try to simulate concurrent users with the help of ramp up time and the number of threads. We consider two scenarios: 1. When Disaster Recovery Software is not running in the main server. 2. When Disaster Recovery Software is running in the main server. For both the above cases, we measure the performance of the main server with performance metrics. The Apache MPM Part of the Apache web server installation is the MPM [13], which stands for Multi-Processing Method. The MPM determines the mechanism Apache uses to handle multiple connections. Through Apache MPM we can tell Apache to stay within the memory available to the server. The performance of the Apache server will depend upon the machine hardware. There are various parameter in MPM: 1. StartServers 2. MinSpareServers 3. MaxSpareServers 4. MaxClients 5. MaxRequestsPerChild MaxClients sets the maximum number of threads used by the Apache server. Any number of requests greater than this value will be queued until a thread is free to action the request. MaxClients is not the same as the maximum number of visitors the server can have. It is the maximum number of requests that can be fielded at the same time. 21

22 We did two kinds of experiments in the performance measurements Type-1 Experiment (Keeping the resources fixed): In the first type of experiment, we keep the MaxClients as constant and vary the number of request per second. As MaxClients directly translates to the number of threads that the Apache server using, it is a measure of the system resources used by the Apache server. In this experiment, we keep the resources available to Apache server as constant, i.e., MaxClients=150. With the limited amount of resources, we are measuring the performance of Apache server by varying the number of requests per second. We are considering throughput and Response time as performance metrics and we try to figure out the behaviour of these performance metrics with varying number of requests per second. Graphs: 1. between number of requests per second and throughput (Figure 7) 2. between number of requests per second and response time (Figure 8) Type-2 Experiment (Varying the resources): In the second type of experiment, varied the MaxClients parameter of the Apache server, the number of requests was kept the same (number of requests=100). The aim of the experiment is to measure how well the Apache server can perform, given unlimited amount of resources. In this experiment we observed that increasing the MaxClients in the server will improve the response time. Graphs: 1. between number of threads and CPU Utilization (Figure 9) 2. between number of threads and Response Time (Figure 10) Observations All the below results are obtained by running the two experiments on following configuration of the main server: 4 GB RAM, Intel i3 processor, Dual Core 2.10Ghz. 22

23 Results of the Type 1 experiment: Graph between Number of requests per second and Throughput: Figure 7: Number of requests per second v/s throughput As the number of request per second increases, there is a linear growth in the throughput observed. However, as the number of requests crosses a threshold value of 140, the throughput starts declining. The behaviour is directly connected to the above stated MaxClient parameter of the Apache server. Also in the following graph, we have summarized the behaviour of the throughput, with and without the disaster recovery software. The performance degradation observed in throughput with the introduction of disaster recovery software is 5.2%. 23

24 Graph between Number of Requests per Second and Response Time: Figure 8: Number of requests per second v/s response time As the number of requests per second increases, the response time remains a constant to a threshold value. As the requests per second increases above the threshold value, the response time shoots up and then fluctuates unpredictably. This behaviour is a direct effect of the MaxClient parameter. Till the number of request reaches the MaxClient parameter value, each request is allocated a separate thread by the Apache server, hence the response time is consists of only the processing time of the page. However, the number of requests per second grows above the MaxClients value, the Apache server is unable to allocate exclusive threads to each requests, hence the queuing up of requests happen and it results in the sudden increase of the average response time. The performance degradation observed in response time with the introduction of disaster recovery software is 7.51%. 24

25 Results of the type 2 experiment For Graph between Number of Threads and CPU Utilization: Figure 9: Number of threads v/s CPU utilization As the resource is increasing for constant number of users, CPU Utilization is rising. We are getting 7.42% performance degradation in CPU Utilization when the disaster recovery software is installed. 25

26 For Graph between Number of Threads and Response Time: Figure 10: Number of threads v/s response time As resources is increasing Response Time decreases, i.e., better response time we get for large resource. Here, the performance metric, i.e., Response time is degraded by 7.40% when disaster recovery software is installed. For simulating these graph results for very large number of users, we can fix any of the two performance metrics and check what resources (may be hardware) more we require to get performance metrics. 26

27 References [1] Hoffer, Jim. Backing Up Business - Industry Trend or Event. Health Management Technology, m0dud/is 1 22/ ai /, [Online;accessed Mar 9, 2012]. [2] Disaster Recovery Strategies with Tivoli Storage Management, redbooks.ibm.com/abstracts/sg html, [Online;accessed Mar 15, 2012]. [3] C. Zhong, L. Zhang, H. Li, and L. Tian, Research and implementation of dualserver hot-standby of configuration software, in The Sixth World Congress on Intelligent Control and Automation, vol. 2, 2006, pp [4] M. Wiesmann, F. Pedone, A. Schiper, B. Kemme, and G. Alonso, Database replication techniques: a three parameter classification, in Proceedings The 19th IEEE Symposium on Reliable Distributed Systems, 2000, pp [5] L. Casale, F. Farina, and E. M. Maffione, Disaster recovery architecture, Patent EP , December, [6] C. Chen, Y. Ting, W. Lu, and G. Wang, Recovery mechanism design for hot standby computer system, in IEEE International Conference on Systems, Man and Cybernetics, vol. 3, Oct 2003, pp [7] Maintaining State of Apache Server, mod/52.htm, [Online;accessed Mar 9, 2012]. [8] session start, [Online;accessed Mar 10, 2012]. [9] $ SERVER, [Online;accessed Mar 16, 2012]. [10] Client URL Library, [Online;accessed Mar 20, 2012]. [11] Linux DNS server BIND configuration, linux-dns-server-bind-configuration, [Online;accessed Feb 14, 2012]. [12] Apache JMeter, [Online;accessed Mar 17, 2012]. [13] Apache MPM, [Online;accessed Feb 20, 2012]. 27

28 Appendix:Performance Metrics Data The data collected from the performance Measurements are given below. Data for CPU utilization, when the number of threads varied from 5 to 100. Experiment conducted without the disaster recovery software. Number of threads CPU utilization (%) Table 1: Data for CPU utilization with disaster recovery software working 28

29 Data for CPU utilization, when the number of threads varied from 5 to 100. Experiment conducted with the disaster recovery software. Number of threads CPU utilization (%) Table 2: Data for CPU utilization without disaster recovery software working 29

30 Data for response time, when the number of threads varied from 5 to 100. Experiment conducted without the disaster recovery software. Number of threads Response time (sec) Table 3: Data for response time (threads varying) without disaster recovery software working 30

31 Data for response time, when the number of threads varied from 5 to 100. Experiment conducted with the disaster recovery software. Number of threads Response time (sec) Table 4: Data for response time (threads varying) with disaster recovery software working 31

32 Data for response time, when the number of requests per second varied from 10 to 400. Experiment conducted without the disaster recovery software. Table 5: Data for response time (requests varying) without disaster recovery software working Request per second Response time Continued on next page 32

33 Table 5 Continued from previous page Request per second Response time Data for response time, when the number of requests per second varied from 10 to 400. Experiment conducted with the disaster recovery software. Table 6: Data for response time (requests varying) with disaster recovery software working Request per second Response Time Continued on next page 33

34 Table 6 Continued from previous page Request per second Response time

35 Data for throughput, when the number of requests per second varied from 10 to 400. Experiment conducted without the disaster recovery software. Table 7: Data for throughput without disaster recovery software working Request per second Throughput Continued in the next page 35

36 Table 7 Continued from previous page Request per second Throughput Data for throughput, when the number of requests per second varied from 10 to 400. Experiment conducted with the disaster recovery software. Table 8: Data for throughput with disaster recovery software working Request per second Throughput Continued in the next page 36

37 Table 8 Continued from previous page Request per second Throughput