PAPER Flow-level multipath load balancing in MPLS network

Transcription

1 IEICE TRANS.?? VOL.Exx?? NO.xx XXXX 2x PAPER Flow-level multipath load balancing in MPLS network Zenghua ZHAO a) Yantai SHU Lianfang ZHANG and Oliver YANG b) Nonmembers SUMMARY Multi-Protocol Label Switching (MPLS) efficiently supports the explicit routes setup by the use of Label Switched Paths (LSPs) between the Label Switched Router (LSR) and the LSR. Hence it is possible to distribute the network among several paths to achieve load balancing thus improving the network utilization and minimizing the congestion. The packet-level characteristics in the Internet are so complex that it is natural to do engineering (TE) and control at the flow level. The emerging Multi-Protocol Label Switching (MPLS) has introduced an attractive solution to TE in IP networks. The main objective of this paper is to balance at the flow level among the parallel Label Switched Paths (LSPs) in MPLS networks. We introduce a multipath load-balancing model at the flow level. In this model each LSP is modeled as an M/G/ processor-sharing queue. The load-balancing problem is then considered as an optimization problem. Based on the analysis of the model we propose a heuristic but efficient mechanism. It makes good use of the characteristics at the flow level. The packets belonging to one flow will be dispatched to the same path. Therefore packets disorder can be avoided effectively. We discussed the allocation granularity and the implementation issues in details. In addition this mechanism is implemented in the LSRs and the LSRs while the intermediate LSRs only forward the packets. Extensive simulations using NS2 have been performed with MPLS modules. The simulation results show that the load through the network is well balanced so that the network throughput is improved and the delay is decreased efficiently. key words: Flow level model multipath load balancing MPLS network. INTRODUCTION The purpose of engineering (TE) is to improve network performance through the optimization of network resources []. The emerging Multi-Protocol Label Switching (MPLS) technology has introduced an attractive solution to TE in IP networks. MPLS can efficiently support the explicit routes setup through the use of Label Switched Paths (LSPs) between the Label Switched Router (LSR) and the LSR [2]. Manuscript received January?. Manuscript revised January 23. Final manuscript received January 23. The author is with the Faculty Department of Computer Science School of Electronics and Information Engineering Tianjin University Tianjin 372 China The author is with the Faculty CCNR Lab School of Information Technology and Engineering University of Ottawa Ottawa Ontario Canada KN 6N5 a) zenghua@tju.edu.cn b) yang@site.uottawa.ca Hence it is possible to balance the through the network thus improving the network utilization and minimizing the congestion. Several researchers have proposed some solutions to balance the load in MPLS networks. An analytical framework is presented in [3] where different models with different objective functions for best-effort and expedited-forwarding respectively are established. However it is too complex to implement in operational networks. Moreover it is impossible to obtain the delay contributed by the which is used to calculate how much should be shifted among the LSPs. Another load balancing mechanism called MATE (MPLS adaptive engineering) [4] develops its algorithm based on the gradient project method and hence unavoidably inherits its disadvantages namely high complexity and slow convergence. The ECMP (Equal Cost Multi-Path)algorithm [5] attempts to distribute the as equally as possible among the equal-cost paths. Since the link costs are static and bandwidth constraints are not considered the allocation is independent of the congestion status of each path. Therefore it is possible that one of the paths will be more congested than the other. Traffic distribution is an important part of the implementation of a load-balancing algorithm. The two pieces of work [3] [4] above have adopted the hash function method. In this method is first distributed into N bins according to a hash function where the number of bins determines the minimum amount of the that can be shifted. If the total incoming is of rate R bps each bin would receive an amount of r = R/N bps. The from the N bins is then mapped into several LSPs according to their loadbalancing algorithms. Using the hash function on the IP fields the incoming packets with same destination can be distributed into the same bin. It seems that it can reduce the possibilities of having packets arrive at the destination out of order as described in [4]. But unfortunately it does not work well. In order to achieve load balancing the should be shifted dynamically among the multiple LSPs in the unit of one bin (i.e. /N of the total ). Therefore the from one bin will be shifted from one LSP to another during the load balance phase. In other words the packets belonging to one flow may end up going along different LSPs and therefore it is difficult to avoid packet disor-

2 2 IEICE TRANS.?? VOL.Exx?? NO.xx XXXX 2x dering that can impact on the performance of TCP. It is in general difficult to model the performance of the Internet with a self-similar characteristic at the packet level [6]. Therefore for control purpose it is natural to characterize at the flow level instead. Flows may be broadly categorized as streaming or elastic according to their nature [7]. It has been pointed out [8] that the arrival of flows in the Internet could be modeled as a Poisson process. Therefore a new service model at the flow level would shed some light on network engineering issues including flow-aware routing pricing admission control and bandwidth sharing. When many flows (fluid) share a single link evenly an M/G/ processor sharing (PS) queuing model [8] can be used. Also since a symmetric queue is quasi reversible for a general service time distribution [9] we can exploit these properties to be explained in Section II Finally when compared with the Kleinrock s Independence Assumption [] an M/G/ PS queue can approximate the network of transmission links more realistically [] [chap 3]. In view of the above we have adopted a PS (processor sharing) queueing network to model the MPLS networks according to characteristics at the flow level. The main objective of this paper is to balance the at the flow level among multiple parallel LSPs in an MPLS network. A per-flow distribution that will avoid packet disorder as much as possible is employed. Based on the analytical model we propose a heuristic but efficient algorithm. Unlike the ECMP we propose to measure the link status (such as delay)dynamically according to where the is distributed. Our simulation results demonstrate that our algorithm can distribute the to different LSPs and balance their loads approximately. The rest of the paper is organized as follows. Section 2 gives the flow-level analytical model. Section 3 proposes a heuristic algorithm to balance the load in the MPLS networks based on the analytical model. Section 4 describes the implementation issues of the load-balancing mechanism. In Section 5 simulation is performed to evaluate the mechanism and the results are shown. Section 6 concludes this paper. 2. THE ANALYTICAL MODEL AT THE FLOW LEVEL IN MPLS NETWORKS In this section we first introduce the characteristics at the flow level before we give our multipath loadbalancing model in MPLS networks. Using a simple case study we want to shed some light on the heuristic load-balancing algorithm proposed in the next section. 2. Flow-level characteristics We define a flow to be a sequence of packets having the same identifier (source address destination address port number etc.). Of the two broad categories of flows we only consider the elastic type since elastic nowadays makes up the majority of the Internet. Flow arrivals are assumed to be Poisson and the flow size has a Pareto distribution given by P [size x] = x /x β for x x > with < β 2. () Note that this distribution has a finite mean and infinite variance. Like [8] we model a single bottleneck link fairly shared by elastic flows as an M/G/ processor sharing a (PS) queue. Let ρ be the link utilization. Then for ρ < the number of flows X(t) in progress has a geometric distribution 2.2 The model P [X(t) = n] = ρ n ( ρ). We consider an MPLS network where multiple parallel LSPs exist between any given LSR and LSR pair. The main objective is to distribute the at each LSR among the multiple LSPs so as to balance the load through the network and thus improving the network performance. Given an objective function we can view this as an optimization problem [3] [4] []. Now the MPLS network considered could be approximated as a PS queueing network where each PS queue corresponds to a link and an arrival flow as a customer. According to the basic theorem of quasireversible queueing networks [9] the stationary number of flows on the links is independent and their invariant distribution has a product-form. For simplicity we assume further that every link bandwidth is equal to. Let L denote the link set and R the set of LSPs. We assume that all the links are identical. Let ν be the total flow arrival rate entering the networkν r the flow arrival rate on LSP r and µ r be the average flow size. Then ρ r = ν r /µ r ν r = ν. Let l L ρ l = r L ρ r then according to the stationary product-form distribution of customer number on the quasi-reversible queueing network [9][Theorem ]P (n... n L ) = l L ρn l l ( ρ l ) and the average flows numbers on link l can be obtained via E [n l ] = ρ l for all l L ρ l. Based on the above discussion on network operation and assumptions we can now pose our loadbalancing problem as follows. Minimize ρ l ρ l ρ l < l L subject to (2) ν r = ν. r R

3 ZHAO et al.: FLOW-LEVEL MULTIPATH LOAD BALANCING IN MPLS NETWORK 3 This is an optimization problem and the solution could be given by a gradient-projection algorithm. But such an algorithm is complex and not robust. For example it is difficult to determine the convergence step size of the algorithm. To gain more insights into our model we shall first conduct a simple network case study with only one - nodes pair with two LSPs between them. We shall analyze it in details in the section below. Our experience will allow us to extend our model to several LSPs (and even to the whole MPLS network) and give us guidance in deducing a heuristic equation for distribution in Section A simple case study Since the average response time T of M/G/ PS system is: x T = ρ dg(x) = µ ρ where x is the flow size with a general distribution G(x). Therefore the average system response time before the engineered arrives is T i = µ ρ i = µ ν i (i = 2). (5) By substituting (5) in (4) and rewriting it we can obtain the equilibrium point condition: ν ν 2 = T T 2. (6) A case of flow level multipath load-balancing model traf- Fig. fic LSR LSP LSP2 engress LSR In order to further make the following algorithm robust we have relaxed the equilibrium condition to ( ν 2 = ν ν 2 ). (7) T T 2 For the N paths case the minimization problem becomes Minimize N i= ρ i ρ i subject to ν + + ν N = ν In Fig. we consider only one - LSR pair with two LSPs between them in an MPLS network. There are two types of : engineered and. The engineered is the that needs to be balanced and the is the background that we have no control over. Engineered flows traverse the network from the LSR to the LSR. We assume that the total engineered flow rate is ν. According to the splitting mechanism discussed below is distributed into the two LSPs each with ν and ν 2 respectively. We let ν and ν 2 be the on LSP and on LSP2. The average size of flows is assumed to be µ. Our optimal load balancing should therefore minimize ρ + ρ ρ ρ + ρ2+ ρ2 ρ 2 ρ 2 subject to ν + ν 2 = ν. (3) where ρ i = ν i /µ and ρ i = ν i /µ (i = 2). Using the Lagrangian Multiplier method it is easy to obtain the minimized condition of ( ρ ρ ) 2 = ( ρ 2 ρ 2 ) 2 After further substitution of ρ i = ν i /µ (i = 2) to the above equation we can obtain: =. (4) µ ν ν µ ν 2 ν 2 which can be solved by the Lagrangian method and the equations similar to the above can be obtained. That is µ ν i ν i = µ ν j ν j for all i j =... N. In the same way we obtain the following relaxed conditions: ( ) ν ij = ν i ν j T i T j (8) i j = 2... N i j. 3. THE HEURISTICS From the above discussion we can deduce the method to distribute among LSPs sharing the same - LSR pair. Eq.(8) shows that the difference of distributed on any two LSPs should be proportional to the difference of the reciprocal of the average packet delays of the corresponding LSPs. However in order to obtain a sensible deduction with a simple form we only take the queueing delay into considerations in the theoretical analysis. Unfortunately the propagation delay of LSPs cannot be neglected especially in core network such as MPLS. While we load we prefer load more on the LSP with lower queueing delay and propagation delay. In addition the above information obtained is static and it is difficult to be applied directly in real networks. In order to make the distribution method discussed above more feasible

4 4 IEICE TRANS.?? VOL.Exx?? NO.xx XXXX 2x we propose a heuristic load balancing equation: w i = /d i (9) /d k k where w i is the weight of LSPi which determines how much should be distributed to LSPi and d i is the total delay (including propagation delay) of LSPi which can be approximated by the round-trip time (RTT) measured dynamically. The difference between the weights of LSPi and LSPj is ( ) ( ) w ij = w i w j = /d k d i d j ( ) k () i.e. w ij d i d j. Since () is in accordance with (8) (9) is a feasible solution. 4. IMPLEMENTATION ISSUES In this section we will discuss further the implementation issues involved in our mechanism. Flow -state table Average delay estimation to the LSP the existing flow the packet belongs to. The flow-state table is maintained by the flow distribution module including the flow state information described below. The average delay estimation is used to measure the average delay of LSPs between the LSR and the LSR. 4.2 Discussion of The Traffic Allocation Granularity For the sake of simplicity we only consider the elastic flows that occupy most of the on the Internet. The allocation granularity is larger at the flow level than that at the packet level. Although Krishnan et al. [2] advocated per-packet allocation granularity we would argue that per-flow allocation granularity is fine enough for distribution. The majority of elastic flows are very short (they are less than Kbytes) and only a few of them are long-lived flows [3]. Therefore for any given LSR and LSR pair the potential number of flows between the two LSRs is typically much higher than the number of the parallel LSPs connecting them. This case is similar to what was suggested by Lai [4]. Moreover the direct advantage of per-flow distribution is that it decreases the disorder phenomenon whereas the per-packet mechanism does not. This is due to the fact that in the per-flow distribution all packets in a flow will go along the same path as long as the path is determined. Traffic Fig. 2 Flow distr ibution The mechanism in an LSR Traffic 4.3 Flow State Table In order to implement per-flow distribution we maintain a flow-state table in LSR as mentioned above. The items in the flow-state table are now shown in Table. 4. Overview of Flow-Level Load-Balancing Mechanism in MPLS Networks Our mechanism is implemented only in the LSRs and LSRs in MPLS networks. The LSRs distribute the input to the network at the flow levels according to our algorithm. The intermediate LSRs do nothing but forward the through them. This adheres to the philosophy of the backbone network s design: keep complexity at the edge while make the middle as simple as possible. Moreover the mechanism is asynchronous and it is unnecessary for the LSRs to exchange information. Our mechanism functions implemented in the LSR are shown in Fig.2. When a packet arrives the flow distribution module in the LSR first decides if it belongs to a new flow. If so the flow will be assigned to a LSP according to our algorithm discussed above. Otherwise the LSR will search the flow-state table and distribute the packet Table Flow-state table maintained by LSR index flow identification LSP# last packet arrival time The item flow identification records the 5-tuples of a flow by which a flow can be identified. In order to speed up the searching process an index item is added. Index is a short word and can be obtained by transforming flow identification through a particular function. The LSP# is the number of LSPs along which the flow should go. Last packet arrival time records the arrival time of the flow s last packet. The flow-state table is maintained only in the LSR; it s unnecessary for the core LSRs and LSRs to do so. In additionthe length of the table is a linear function of the number of the current active flows through the LSR. In fact in order to save memory the flow identification can be removed without any impact on the process. The flow-state table length can thus be shortened. On the other hand

5 ZHAO et al.: FLOW-LEVEL MULTIPATH LOAD BALANCING IN MPLS NETWORK 5 the speed of the CPU nowadays is faster and faster and the memory is cheaper and cheaper therefore we think it s worth improving the performance of the MPLS network at the cost of the maintenance of the flow-state table. 4.4 Flow Distribution When a packet arrives the LSR first searches its flow-state table to see if this packet belongs to an existing flow. If so the LSR will access the corresponding entry distribute the packet to LSP# LSP and update last packet arrival time with the packet arrival time. Otherwise a new flow begins. Then the LSR records the flow identification obtains the index through transformation sets last packet arrival time to the current time (i.e. the packet arrival time) and determines the LSP# according to our algorithm. At last a new entry filled with all the above information is inserted into the flow-state table. When a flow exists the corresponding entry should be deleted from the flow-state table in time. The Last packet arrival time is used to do so. The LSR will verify each flow s Last packet arrival time in its flow-state table. If the difference between this item and the current time is more than the threshold T that is to say the flow has existed then this entry will be deleted from the flow-state table. 4.5 Delay Estimation In order to monitor the real-time information of each LSP we have adopted a probing mechanism. We send probing packets periodically to each LSP and measure their round-trip time (RTT) then estimate the path delay using an exponential weighted moving average shown below.[5] ED i = ( α) ED i + α SRT T i connection. Flow arrivals are Poisson arrival process with a mean of 2 flows per second. The flow size has a Pareto distribution with x = 4 β =.5 (cf. ()). 5. Simulation scenario The network topology is shown in Fig.3. Each link is set to be 6Mbps with ms delay. There is only one LSR and LSR pair with two parallel LSPs (numbered as LSP and LSP2) between them. Flows enter the MPLS network from the LSR and exit from the LSR. In order to verify the efficiency of our mechanism we have introduced to change the link bandwidth available to flows entering the network. The uses the on-off mode CBR (Constant Bit Rate) and goes through the intermediate LSRs. flows LSR LSR LSR2 5.. Simulation results LSR3 LSR4 Fig. 3 Network topology LSR flows Figure 4 is used as a reference to show the offered load distributed on LSP and LSP2 without in the network. where ED i is the ith estimated delay SRT T i is the ith measured RTT. The flow distribution module will use the estimated delay to distribute the according to our algorithm discussed in the above section. 5. PERFORMANCE EVALUATION.6.2 LSP LSP We use NS-2 [6] with an MPLS module [7] to implement our simulation. To illustrate our method feasible and general two simulation scenarios are established. The first scenario is the simplest case with two paths between one IE (Ingress LSR - Egress LSR) pair. while in the second scenario there are four IE pairs where the intermediate links are shared by the LSPs from different IE pairs. In the simulation A flow is simply a TCP Fig. 4 distributed on LSP and LSP2 without For comparison Fig.5 allows some running at a data rate of 3Mbps. It starts at 5s and ends at 5s. During this period the bandwidth available on LSP has decreased. As can be seen from Fig. 6 the offered load is adaptively shifted from LSP to

6 6 IEICE TRANS.?? VOL.Exx?? NO.xx XXXX 2x Fig LSP LSP distributed on LSP and LSP2 with Throughput (Kbps) (a) Flow sequence number same as generated 2 Delay (s) Fig. 6 End-to-end delay of flows Throughput (Kbps) (b) Flow sequence number rearranged according to throughput Fig. 7 End-to-end throughput of flows LSP2 and the load is balanced between the two LSPs very well. Fig.6 and Fig.7 illustrate the end-to-end delay and throughput of each flow respectively where Fig.7(b) rearrange the flow sequence according to the throughput in order to show the results clearly. The end-to-end throughput is increased and the end-to-end delay is decreased obviously. The results show the network performance is improved significantly compared with that in the original MPLS network. 5.2 Simulation scenario 2 flows LSR LSR2 LSR3 LSR LSR2 link LSR3 LSR4 link2 LSR5 link3 LSR6 LSR LSR2 LSR3 flows In the sumulation above the network topology is the simplest case which we used to deduce load balancing equation in theory. The simulation results say that the equation works well. In this section we design a more practical MPLS network topology presented in Fig.8. There are four IE pairs with two parallel LSPs between each of them. The LSPs from different IE pairs share some intermediate links. For example link is shared by LSP ( LSR-LSR-LSR2- LSR) and LSP ( LSR2-LSR-LSR2- LSR2). The with different rate and duration are added to link and link4 as shown in Fig LSR4 LSR7 link4 LSR8 Fig. 8 Network topology 2 LSR4..on.link..on.link simulation results In this simulation we collected the offered load of the four links. Fig. shows the are allocated evenly to the four links. During 2s and 7s the Fig. 9 The added to the network

7 ZHAO et al.: FLOW-LEVEL MULTIPATH LOAD BALANCING IN MPLS NETWORK 7 occupying about 5% bandwidth is added to link the on link are then shifted partly to other links as shown in Fig.. Similar result can be observed during 9s and 3s. It s worth to note that the is shifted adaptively according to the strength of the (in other words the available bandwidth). That s to say our method balance the load very well. The end-to-end delay and throughput of each flow are shown in Fig. 2 and Fig. 3 respectively. Compared with the original MPLS network our method improves flows performance. That is the end-to-end throughput is increased and the end-to-end delay is decreased. Throughput (Kbps) (a) Flow sequence number same as generated 2.2. Fig..2 link link2 link3 link distributed on links without Throughput (Kbps) 5 5 without.loadbalancing (b) Flow sequence number rearranged according to throughput Fig. 3 End-to-end throughput of flows. link link2 link3 link Fig. distributed on links with.6 performance and to minimize the congestion. Our analysis allows us to propose a flow-level mechanism. Although a heuristic our mechanism is efficient because it employs per-flow distribution to avoid the packet disorder. It is also easy to implement in the LSRs and LSRs. Our simulation results show that the load through the network is well balanced so that the network throughput is improved and the delay is decreased significantly. Acknowlegement Delay (s) This research is supported in part by the National Natural Science Foundation of China (NSFC) under grant No and No. 945 by the Cisco CCRP grant by the NSF of Tianjin under grant No. 236 and No and by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant No. OGP Thanks to Hongmei Wang for her work on simulation. Fig. 2 End-to-end delay of flows References 6. CONCLUSIONS In this paper we have analyzed a flow-level model in order to balance the load among parallel LSPs in the MPLS networks. The purpose is to improve network [] D. Awduche et al. Requirements for engineering over MPLS Internet RFC 272 September [2] Rosen E. Viswanathan A. and R. Callon Multiprotocol label switching architecture Internet RFC 33 January 2 [3] E. Dinan D. Awduche B. Jabbari Analytical framework for dynamic partitioning in MPLS networks IEEE ICC NEW ORLEANS pp June 2.

8 8 IEICE TRANS.?? VOL.Exx?? NO.xx XXXX 2x [4] A. Elwalid C. Jin S. Low I. Widjaja Mate: MPLS adaptive engineering in Proceedings of IEEE IN- FOCOM Anchorage Alaska pp April 2. [5] D. Awduche A. ElwalidI. WidjajaX. Xiao Overview and Principles of Internet Traffic Engineering RFC3272May 22 [6] Feldmann A. Huang P. Gilbert A.C. Willinger Dynamics of IP : a study of the role of variability and the impact of control. Computer communication review Sigcomm 29 no [7] J. W. Roberts et al. Traffic Theory and the Internet IEEE Communications Magazine pp January 2 [8] S.B. Fredj T. Bonald A.Proutiere G. Regnie J.W. Roberts Statistical bandwidth sharing: a study of congestion at flow level In Proceeding of ACM SIGCOMM San Diego California U.S.A. pp.27-3 August 2 [9] Kelly Reversibility and Stochastic NetworksWiley Chichester 979 [] Kleinrock L. Queueing Systems vol.2. Wiley New York 976 [] D. Bertsekas R. Gallager Data Networks Prentice Hall 992 [2] R. Krishnan and J. Silvester Choice of Allocation Granularity in Multipath Source Routing Schemes Proceedings of the IEEE Conference on Computer Commuincations San Francisco CA pp March 993. [3] M.F. Arilitt C. Williamson. Web server workload characterization: The search for invariants. In Proceeding of the ACM SIGMETRICS 96 Conference Philadelphia pp April 996 [4] W.S. Lai. Bifurcated routing in computer networks. computer communications review vol.5 no.3 pp [5] P. Karn and C. Partridge Improving Round-Trip Time Estimates in Reliable Transport Protocols Computer Communicators review vol. 7 no.5 pp.2-7 August 987. [6] NS-2 Network Simulator [7] Gaeil Ahn and Woojik Chun Overview of MPLS networks simulator: Design and Implementation Chungnam National University Korea Yantai Shu was born in 942. He received his Ph.D at Tianjin University China in 968. Currently he is a Professor of Computer Science at Tianjin University China. His current interests are focussed on computer networks real-time systems modeling and simulation. Lianfang Zhang was born in 946. He received his M.S. in Institute of Mathematics Academia Sinica in 98. His research interests are in performance evaluation of computer networks. Oliver Yang is a Full Professor in the School of Information Technology and Engineering at University of Ottawa Ontario Canada. Dr. Yang received his Ph.D. degree in Electrical Engineering from the University of Waterloo Ont. Canada in 988. His research interests are in modeling analysis and performance evaluation of computer communication networks queueing theory simulations computational algorithms and their applications such as reliability and analysis. Zenghua Zhao was born in 974. She received her M.S. in Computer Science Tianjin University China in 2. Currently she is a Assistant Professor of Computer Science at Tianjin University China. Her current interests are focussed on computer networks modeling and simulation Wireless Multimedia network.