Autonomous Fault Detection and Recovery System in Large-scale Networks

Transcription

1 Autonomous Fault Detection and Recovery System in Large-scale Networks Raheel Ahmed Memon 1, Yeonseung Ryu, Abdul Qadir Rahimoo Abstract In networks like Combat ship data network, the Ethernet is required to provide fault tolerance to all of its constituents for smooth operations. The analysis in this paper is done on the basis of our already proposed scheme Recursive Scalable Autonomous Fault tolerant Ethernet. Where, the large scale network is divided into small subnets by limiting the number of nodes in each. The nodes in RSAFE contain two Network Interface Cards for dual connectivity in network. The results show that failure can be detected by missing of 2 consecutive Heartbeat Messages, which is acceptable performance in a network, along with failover, RSFAE is also cost effective solution, it is capable to even extend itself without interrupting the existing structure. Index Terms Network topology, Fault tolerant Ethernet, RSAFE, Fault detection in network, Fault recovery in network. I. INTRODUCTION The most important capability in any system is its robustness, how a system behaves in critical situations and how a system can prevent it from fault occurrence. Though Ethernet was not specially designed to provide such robustness, but scientists started thinking in that direction because several mission critical systems such as; Combat Ship data network, unmanned vehicles, military weapons are network based mission critical systems. In response to such needs Fault tolerant Ethernet (FTE) introduced [1]. FTE provides multiple paths to all the connected equipments in the network, so that if primary path would fail then standby path can recover that fault. The primary goals of FTE are: 1) No loss of communication on single failure 2) Fault must be recovered within specified time 3) Use of existing protocols and 4) No proprietary hardware should be used. The FTE can be achieved by two different approaches either software or hardware; in software approach it works quite fine with small network but as the network size grows, it starts consuming huge bandwidth. In the hardware approach a proprietary hardware (Multiport Network Interface Card) is needed. [3] In our previously proposed scheme Scalable Autonomous Fault Tolerant Ethernet (SAFE) [2], a software approach has been used, according to OSI Model SAFE lies in Data-Link Layer and Network Layer. In SAFE a large network is divided into several manageable subnets and one of those 1 Corresponding author, he is currently working with the Department of Computer Science at Sukkur Institute of Business Administration, Sindh, Pakistan ( raheelmemon@iba-suk.edu.pk). subnets would perform as central subnet. The limitation with that system is; if the central subnet will fail then all over the communication would become dead. Recursive Scalable Autonomous Fault tolerant Ethernet (RSAFE) is an enhanced version of SAFE, where all the subnets are connected recursively to form a FTE in which no failure can interrupt the communication of other nodes or subnets. This also works on same principles of SAFE, it lies in Data-link layer and network layer, it divides large network into several manageable subnets but it doesn t select any central subnet rather the architecture of RSAFE scheme has been build in recursive manner. Where if any failures will occur then no other subnet would get affected, all subnets are formed recursively. This work is about the theoretical analysis of RSAFE for fail-over functionality in RSAFE. As RSAFE works on Heartbeat mechanism [11], where every node broadcast a heartbeat to the all other nodes within a subnet, and if any node can t deliver heartbeat messages successfully then that node would be reported as faulty node and recovery algorithm will execute. The time consumed between the fault detection and fault recovery has been theoretically analyzed in this paper. Section 2, is about background where discussion is focused about some of popular fault tolerant network architectures, section 3 is about architecture details and building algorithm for scheme. Section 4 is about methodologies used to detect and recover the fault. II. BACKGROUND Apart from the performance and power, the reliability is utmost requirement in mission critical systems. A system must perform fault free without any interruption. Reliability of a system is defined as inverse of fault occurrence (very low or no failure). So to achieve reliability of networks in such systems, extensive research has been conducted by different scientist, but almost the research done for Fault tolerant Ethernet (FTE) is for small networks, FTE in large scale network is a bit harder than the FTE for a network where number of nodes is limited. In our previous work [2] SAFE was proposed, which uses the single network with redundant cables to form a subnet, those created subnets are further connected to central subnet as shown in Figure1. Each node in the subnet has two ports (i.e; Port A and Port B) and both ports are connected to two different switches in a subnet, it is then called as Dual Fault Tolerant Ethernet (DFTE). In SAFE architecture the large number of nodes is divided into several manageable subnets, as shown in Figure 1 that 5 subnets are created where one subnet is selected as central subnet, all other subnets are connecting to that central subnet.

2 Each node in subnet have two network interface cards (NICs) one named as Port A and other named as Port B, both are connected to two switches named as Switch A and Switch B respectively. There are four possible paths for routing the data packets if any undesirable condition will happen, one of these paths will work as primary path for data traveling, while others one will remain in standby. So that, in response of fault detection, the standby path can recover the faults. As shown in Figure 2, DCell is constructed by combining several low-level DCells and DCells at the same level are fully connected with one another. DCell have several good features, like; DCell is a fully fault tolerant approach, which doesn t contain a single point of failure, it uses shortest-path routing protocol, DCell have capacity to facilitate new servers with exponential growth these days, and also capable to provide them a robust system [5]. But the cabling cost in DCell is higher; as number of servers increases the overall cost also grows exponentially. A. Architecture: III. RSAFE RSAFE dare to survive in the situations where the network is large and also expected to expand. To detect the fault in RSFAE architecture Heartbeat Message (HBM) is used, and to report that fault to other subnets (inter-subnet communication) is achieved by using Primary Master Node method, one of the nodes from every subnet serves as PMN, which keeps the updates of whole network. The recursive architecture of RSAFE has several advantages: 1) Failover within specified time 2) Multiple routing paths 3) Easily Scalable 4) No dependant subnet 5) Use of TCP/IP Protocols 6) No Proprietary Hardware needed: Figure 1. SAFE Architecture The limitation of this system is the connectivity of all subnets to central subnet, in case if both switches (Switch A and Switch B) of center subnet will fail then whole the network will stop communicating. For exponentially growth in number of servers at data centers, Chaunxiong Guo [3] introduced a new structure and called it as DCell. DCell uses a recursively-defined structure to interconnect servers. Each server connects to different levels of DCell via multiple links. 1) Failover within specified time: RSAFE work on HBMs, and every HBM is sent and received in 1.8/sec, if any node is unable to successfully deliver 2 consecutive HBMs then it would be assumed that the current path for that node has got some issues, and standby path will replace it. 2) Multiple routing paths: Within the subnet there are four possible paths to transfer data from one node to another node. Let s suppose in Figure 3. node N 1 wants to send the data to N 2 then possible paths are: (PP 1 ) (PP 2 ) (PP 3 ) (PP 4 ) Figure 2. DCell The Data Center Network Architecture 3) Easily Scalable: Scalability in RSAFE is easier, because adding new subnets wouldn t disturb the existing system. 4) No dependant Subnet: As in SAFE there was central subnet where all the subnets were connected, therefore in some conditions if both switches of central subnet would fail then all over network will stop working. 5) Use of TCP/IP Protocols: RSAFE works on already existing protocols like TCP/IP. 6) No Proprietary Hardware needed: In hardware approach [4] the failover time is very low, but on the other side it requires implementing a

3 proprietary hardware, so it is very expensive approach to implement, or to update the existing network. Figure 3. Shows the single subnet construction, all nodes have 2 Network interface Cards named as Port A and Port B, both are connected to Switch A and Switch B respectively. One of the nodes is selected as primary node, which is used to spread the subnet fault to other all subnets. Figure 3. Single Subnet There are three basic building blocks of RSAFE as shown in Figure 4, 1) Subnets 2) Groups and 3) Levels. Together these components form RSAFE architecture. 7) Subnets: Figure 4 (a) Composed of limited number of nodes and two switches (Switch A and Switch B ), each node in a subnet contains 2 NICs to connect with these 2 switches. One node is selected as Primary Node (PMN), while others are treated as normal nodes in subnet. This dually connected subnet has 4 possible paths to send and receive data. Figure 4. RSAFE Architecture 1) Groups: A limited number of subnets can form a group, where all the subnets connected to each other. Figure 4 (b) shows that the connectivity is formed as Second Switch of Subnet 0 is connected to first Switch of Subnet 1 and Second Switch of Subnet 1 is connected to first Switch of Subnet 2, for last subnet the Second Switch of Subnet n is connected to first Switch of Subnet 0. 2) Levels: Is the higher level than groups, it combines all the constructed groups and forms the complete levels. As shown in example Figure 4 (C) this is the highest level where no subnets are remaining back; all groups are constructed; now those groups are combined here to form the level. This is the last level, and the highest level. All the highest levels are constructed recursively from the lower levels. Whole the connectivity is based on subnets, subnets are used to connect and form the groups, further selected subnets from groups connected to form levels after all required levels were created randomly selected subnets are combined to form a complete level which is a complete large scale network. B. Fault- tolerant in Network: There are two modules used for fault tolerance in network; first is Fault detection (FD) which helps to detect the fault within some specific time, second is Fault Recovery (FR), this is followed by fault detection, once a fault has been detected the recovery module will execute to activate one of the standby paths. Now, fault detection is further divided into two sub categories, 1) Subnet FD 2) Inter-subnet FD. 1) Subnet FD: Fault detection module is continuously monitoring systems Heartbeat messages (HBMs); those are frequently sent and received by each node within a subnet. The sending and receiving time of HBMs is 1.8/sec. If any node is unable to receive 2 consecutive HBMs from other node then FD would be assumed that the fault has been occurred on the primary path, and FD will activate FR which then update the routing table and activate the standby path. The reason for using 2 consecutive HBMs to detect the fault is to confirm the connection failure, because due to several reasons it is possible to experience some losses in data receiving. 2) Inter-subnet FD: The primary node is selected in every subnet to exchange the network status with the other subnets. The inter-subnet communication about status updates is achieved by PMN. On any fault detection within a subnet a data packet is released between subnets and PMN of all subnet are responsible to share that information with all over the networks. IV. ANALYSIS There are two main contributions of RSAFE. First one is; RSAFE brought a less expensive fault tolerant scheme for large-scale networks. Thus scheme supports available hardware in market, and less expensive in cabling. The Second contribution is its performance, as this is scheme for mission critical systems, so performance very important factor to analyze. RSAFE gives acceptable performance in large networks.

4 Eq. 1 The failover time is Time fo in a subnet, and Time HBM is heartbeat repetition interval and T LAT is delay latency. "#$ " = 2 "#$ "# + "#$ "# (1) Time marker show that on 45 seconds, where HBM 21 was not received but still there was no fall in communication. Thus the number of Nodes in a subnet is a big factor for poor performance in fault detection, that s why we give 64 as a predefined limit to the number of nodes in a subnet. Figure 5, shows the connections needed to create a subnet. Maximum 64 nodes can form a subnet. Equation 2 is for calculating requirements: Figure 6. Fault Detection & Recovery time That is only because missing of one HBM is not a parameter to measure fault occurrence. One packet loss can be caused by several reasons. Figure 5. Number of Connections required for building a subnet "#$%& = "#$ (2) Where N total is total number of nodes and each node contains two Network Interface Cards (NICs) that are connected to two switches using straight through cables. To connect two switches with each other one crossover cable is needed. The graph shown in Figure 5 is estimation of required cables to create a subnet with 4 to 64 nodes. Table 1. Comparison Table RSAFE SAFE DCell 4 Possible Paths 4 Possible Paths Only 1 route Recursive structure, no central failure point in network linear One subnet works as center of communication linear Large no of alternative paths because of direct connectivity. Quadratic V. CONCLUSION In this paper we have presented an approach to provide fault tolerance in large scale networks for perform smooth operation in critical conditions without any failure. Our proposed scheme RSAFE is analyzed for two effective factors 1) cost effectiveness and 2) failover time. The cost of effectiveness is achieved by minimizing number of cables required in building a subnet, comparison is shown in Table 1, it shows that RSAFE is reasonable approach which can be adopted to save huge required price for large number of cabling and proprietary hardware development. For failover in RSAFE consecutively two heartbeat messages are required to detect fault occurrence. And recovery process can be done within one heartbeat. Where, the interval between each heartbeat message is 1.8 second. This analysis shows the feasibility of RSAFE for several mission critical systems like Combat Ship Data Networks, Unmanned vehicles, military weapons. Where the failure of data transmission can leads to a big loss. So we can conclude that RSAFE is good scheme to provide fault tolerance in huge networks where expansion in existing structure is also expected, because RSAFE allows adding or eliminating any subnet/node at any time without interrupting the working network. Though the size of a subnet is limited to 64 nodes, because more nodes can reduce the quality of performance, delay time would increase by increasing number of nodes. The fault detection analysis is done in Figure 6. As already stated that, loss of two consecutive HBMs is the sign of link-down. So we can see that y-axis is representing number of HBMs and x-axis is the time line (Seconds). The interval in between HBMs is 1.8 second. For 0 to12.6 Seconds the communication worked fine because 5 heartbeats received successfully. But, as HB 6 and 7 were not received up to expected time that s why on time 16.2 activating the standby node restored the communication. REFERENCES [1] Honeywell Automation and Control Solutions rionpks/faulttolerantethernet/default.htm [2] K.Y Kim, Y.S Ryu, J.M Rhee, and D.H Lee, SAFE: Scalable Autonomous Fault-tolerant Ethernet, Proc. of the 11th International Conference on Advanced Communication Technology (ICACT 2009), pp , Feb [3] Chuanxiong Guo, Haito Wu, Kun Tan, Lei Shei, Yongguang Zhang, Songwu Lu. Dcell: a scalable and fault-tolerant network structure for data centers ACM SIGCOMM 2008.

5 [4] J. Huang, S. Song, L. Li, P. Kappler, R. Freimark, and T. Kozlik, An Open Solution to Fault-Tolerant Ethernet : Design, Prototyping, and Evaluation, Proc. IEEE International Performance, Computing, and Communications Conference, Feb. 1999, pp [5] C. Guo, H. Wu, K. Tan, L. Shi, Y. Zhang, and S. Lu, DCell: A Scalable and Fault-Tolerant Network Structure for Data Centers, Proc. ACM SIGCOMM 2008, Aug [6] G. Hector, Elections in a Distributed Computing System, IEEE Transaction on Computer, Vol. C-31, No. 1, 1982, pp [7] P. Stelling, I. Foster, C. Kesselman, C. Lee, and G. Laszewski, A Fault Detection Service for Wide Area Distributed Computations, Cluster Computing, Vol. 2, No. 2, 1999, pp [8] T.D. Chandra and S. Toueg, Unreliable Failure Detectors for Reliable Distributed Systems, Journal of the ACM, Vol. 43, No. 2, Mar [9] J.B. Dugan, S.J. Bavuso, and M.A. Boyd, Dynamic fault-tree models for fault-tolerant computer systems, IEEE Transactions on Reliability, Vol. 41, Issue. 3, Sep. 1992, pp [10] S. Varadarajan and T. Chiueh, Automatic Fault Detection and Recovery in Real Time Switched Ethernet Networks, Proc. IEEE INFOCOM '99, 1999, pp [11] Pham, H. A., Rhee, J. M., Ryu, Y., & Lee, D. H. (2010, November). An adaptive and reliable data-path determination for Fault-Tolerant Ethernet using heartbeat mechanism. In Computer Sciences and Convergence Information Technology (ICCIT), th International Conference on (pp ). IEEE. Raheel Ahmed Memon Mr. Memon received his Masters degree in Computer Engineering from Myongji University South Korea in year He is currently working as Assistant Professor with Sukkur Institute of Business Administration. His research interests includes, Fault tolerant networking, Android Mobile computers, Flash Memories.