On Traffic Fairness in Data Center Fabrics

On Traffic Fairness in Data Center Fabrics Dallal Belabed, Stefano Secci, Guy Pujolle, Dee Medhi Sorbonne Universities, UPMC Univ Paris 6, UMR 766, LIP6, F-755, Paris, France. Email: firstname.lastname@umc.fr Univ. of Missouri Kansas City, 5 Rockhill Road, Kansas City, MO 64 USA. Email: dmedhi@umkc.edu Abstract A resent challenge in data center networks (DCN) is to better understand the imact of novel flattened and modular DCN architectures on congestion control rotocols, and viceversa. One of the major concerns in congestion control being the fairness in the offered throughut, the imact of the additional ath diversity and forwarding features, brought by the novel DCN architectures and rotocols, on the throughut of individual endoints (servers) and aggregation oints (edge switches) is unclear. This aer attemts to answer these oen questions. Secifically, how best is the allocation of the cometing elastic demand flows and how is this allocation imacted with the increase in caacity? We rovide an otimization formulation of the roblem based on the roortional fairness rincile of TCP. We conducted a series of test scenarios on the fat-tree data center toology by considering load balancing and link caacities for different network cases in order to resent our analysis on the results. We observed that the traffic allocation fairness is rimarily imacted by the weights associated with the TCP imlementation in use. Index Terms Data Center Networks, Resource fairness, TCP throughut, Fat-tree toology I. INTRODUCTION The emergence of network virtualization solutions, such as Infrastructure as a Service (IaaS), offers several advantages to organizations in terms of both oerational and caital exenditures [9]. The transition from hysical indeendent networks to virtual de-localized networks oerated in the Cloud can be facilitated if, besides security concerns, connectivity erformance is at an accetable level and shows desirable fairness roerties. With the growth in customer volumes, service differentiation and elastic demands, avoiding bottlenecks is a critical oint in Data Center Network (DCN) architectures. With the de-facto dominating trend of deloying services using virtualization servers, a non negligible ratio of the traffic is horizontal traffic between virtualization servers, in suort of virtual machine migration and storage synchronisation. The amount of intra- DC horizontal traffic can overcome the access vertical traffic volume []. This has eventually favored the emergence of novel DCN architectures that exose additional horizontal caacity between server racks and clusters of racks such as fat-tree [], and BCube [5]. An oen question is: how best is the traffic allocation of the cometing elastic demand flows for horizontal traffic between edge servers in data center fabrics, and how is this allocation imacted with the increase in caacity? To address this question, we assume that all traffic uses TCP allowing multiath forwarding. More secifically, we are interested in understanding this imact in equilibrium. It has been shown that several variants of TCP are roortionally fair in the equilibrium, which have been verified through simulation [8], [9]. We, therefore, use a roortional fairness model to understand the allocation for cometing demands in data center fabrics. Our study is focused on a fat-tree data center toology, one of the oular data center network architectures, as this allows us to understand the imact between intra-od and inter-od traffic among all the horizontal traffic. The rest of the aer is organized as follows. Section II resents the background of our work. Our otimization model is formulated in Section III and the study results are resented in Section IV. Section V concludes the aer. II. BACKGROUND In this section, we resent an overview of the TCP roortional fairness model and of existing multiath forwarding rotocols. A. TCP Proortional Fairness Model In a network with multile cometing TCP sessions sharing links, several studies [4], [8], [8], [9] have shown that TCP imlicitly solves a utility roblem in equilibrium. This utility roblem is formally described as a imization of an aggregate utility subject to caacity constraints: subject to X U(X j ) () j J δ je X j c e, e =,,..., E () j J The above model imizes the utility function U(X j ) of each session j J where X j denotes the rate of session j while δ je is the indicator that takes the value if session j uses link e, otherwise. For the Proortional Fairness (PF) [], [6] allocation that is alicable to TCP, utility U(x j ) is set to ω j log x j, where ω j is the weight of the session j. Hence, the resource allocation corresonding to this utility function is commonly referred to as weighted roortionally fair, or, if all ω d are equal to one, as roortionally fair. Thus, () for PF becomes: X ω j log X j (3) j J

B. Multiath forwarding rotocols There are many recent rotocols designed in the last decade that include forms of multiath forwarding, also referred to as acket load-balancing techniques. They can act either at the data-link, network, or transort levels. At the data-link layer, a rotocol has been designed secifically for data-center networks, the Transarent Interconnection of a Lot of Links Protocol (TRILL) [7]. It allows a switch and even a virtualization server, acting as virtual bridge, to balance the load over multile destination TRILL bridges for the same air of nodes. However, no forms of congestion control are imlemented here as the evolution of IP networks is such that this has been left to the transort layer. At the network layer, Equal Cost MultiPath (ECMP) [6] is adoted in the Oen Shortest Path First (OSPF) and Intermediate System to Intermediate System (ISIS) rotocols []: it allows balancing the load over multile next hos. ECMP can also be imlemented in TRILL. However, this is tyically erformed in such a way that for a secific TCP flow, only one ath is used in order to avoid acket disordering and buffer exlosion at TCP endoints. At the transort layer, there have been two major roositions. Stream Control Transmission Protocol (SCTP) [5] allows end hosts to use several aths concurrently, when devices are multihomed. The way it has been designed, however, makes SCTP weak against the de-facto ervasive resence of middleboxes in the Internet such as firewalls, erformance otimizers, load balancers at lower layers and interfaces. In many cases, SCTP connections cannot be oened or maintained. More recently, the Multiath TCP (MPTCP) [4] has been designed with retrocomatibility and incremental deloyability as the first design requirements so that using multile aths simultaneously is made ossible, falling back to standard TCP in case of middlebox blocking. Major attention has also be given to congestion control and fairness. An imortant requirement is that an MPTCP connection over a given link should not take more resources than legacy TCP connections running on the same link. However, as shown in [7], [], the congestion control algorithm is a key choice when fairness with resect to other connections needs to be guaranteed as it is a major concern of network oerators. Our study is agnostic about the secific multiath forwarding rotocol that could be adoted in the DCN fabric, and the related analysis is concetually alicable to any configuration including multiath forwarding and congestion control in whatever layer. III. PROBLEM FORMULATION Following [3], we now generalize the basic roortional fairness model for DCN allowing multiath forwarding for elastic demands that use TCP. First, while the actual TCP sessions are between edge servers in a DCN, we can consider the model in terms of elastic demands between a air of edge switches since all such sessions must enter and exit through edge switches (see Figure ). Thus, moving away from sessions (identified by j earlier), we identify a demand between Indices d =,,... D =,,... P d e =,,... E Variables x d X d Parameters δ ed = α c e ω d Fig.. TABLE I MATHEMATICAL NOTATIONS demands associated with airs of edge switchs candidates aths for demand d links amount allocated to ath of demand d amount allocated to d if link e belongs to ath of demand d;, otherwise a minimum sub-flow ratio allocated to each ath available to a demand d caacity of link e weight of demand d (constant) Fat-tree toology with four ods. a air of edge switches by d with the elastic demand denoted by X d. Secondly, due to multiath forwarding, we identify traffic flow along each ath associated with demand d by using x d (notations are summarized in Table I). Therefore, for a given demand, the sum of traffic amount allocated to the aths is equal to the total elastic demand X d given by: x d = X d d =,,..., D (4) Next, the sum of all the flows using a articular link e must satisfy the link caacity constraint: δ ed x d c e e =,,..., E (5) d The goal is to imize the utility objective: X,x F (X) = d ω d log X d (6) where ω d is weight for demand d, which is discussed further in Section III-B. To summarize, our model is to address the goal given by (6) subject to constraints (4) and (5). It should be noted that while elastic demand X d is non-negative, the logarithm function ensures that no elastic demand takes the value zero, i.e., every demand must get its share according to roortional fairness subject to caacity constraints and any influence due to ω d. In addition to the above model, we are also interested in understanding the imact when we enforce at least some traffic to be carried on each ath of a demand d, which can be imosed using the following additional constraint (7): x d αx d d =,,..., D =,,..., P (7)

TABLE II LINEAR APPROXIMATION Indices k =,,..., K Consecutive ieces of the aroximation of log x. Variables f d aroximation of log X d. Parameters a k, b k coefficients of the linear ieces of the linear aroximation of log x. Global throughut 8 6 4 Here, each candidate ath has to carry at least αx d, i.e., the minimum of rate allocated to demand d on each candidate ath. A. Linear aroximation of the objective We note that in the revious formulation, the objective function is non-linear due to the logarithm function. We use a linearization aroximation [3] of the logarithm function as follows: log X d = min {a kx d + b k }. (8) k=,,...,k Then, the otimization roblem becomes subject to: X,x,f F = d ω d f d (9) x d = X d d =,,..., D () δ ed x d c e e =,,..., E () d f d a k X d + b k d =,,..., D k =,,..., K () The advantage of this aroximation is that it is a linear rogramming roblem that can be solved using a well-known software ackage such as CPLEX. B. On weights w d We now elaborate on w d taking into consideration two valid TCP imlementations [9]. This was a result of different interretations of TCP Vegas [3]: the one based on bytes er round tri time and the other based on bytes er second leading to utility functions U(X d ) = log X d (3) U(X d ) = ω d log X d, (4) resectively. Here ω d corresonds to the roagation delay of session d. The first situation (3) does not give any weight to the session, and we name it the fixed-delay case. The second situation (4) gives weight to the roagation delay through ω d for session d and we name it the weighted-delay case. Besides the two valid imlementations of TCP Vegas, FAST TCP follows the weighted-delay case [8]. For comarison uroses, we use a simlification for the weighted-delay case by setting ω d to be based on the number of hos between the source and the destination to serve as a rough aroximation of the delay being the number of hos []. Fig.. Global throughut Fig. 3. 8 7 64 56 48 4 3 4 6 8 3 4 5 6 7 8 9 Uniform caacity case: global throughut (All-to-All) 3 4 5 6 7 8 9 Asymmetric caacity case: global throughut (All-to-All) IV. PERFORMANCE EVALUATION We evaluate our roortional fairness model to understand the fair allocation for cometing demands, focusing on a secific data center toology and studying cases to understand the imact of the DCN caacity on traffic fairness. We imlemented our study set u using C++ and CPLEX as the solver for the linear rogramming formulation given by ((9)) - (()). In the following, we resent the study framework and discuss the simulation results. A. DCN toology We run our study cases on the fat-tree toology [], a oular novel DCN architecture, deicted in Figure. It interconnects commodity Ethernet switches as a k-ary network where all switches are identical and organized in two layers: core layer and od layer. At the od layer, there is an aggregation stage and an edge stage. There are k ods, each one containing two layers of k switches. Each k-ort switch in the lower layer is directly connected to k hosts. Each of the remaining k orts is connected to k of the k orts in the aggregation stage. There are ( k ) k-ort core switches. Each core switch has one ort connected to each of the k ods. The i th ort of any core switch is connected to od i. Figure shows a fat-tree examle for k = 4 that was used in our study. The advantage of considering this toology is that it has intra-od traffic and inter-od traffic. Secondly, the caacity

.5.5.5.5.5.5 3 4 5 6 7 8 9 3 4 5 6 7 8 9 (a) All-to-All (b) One-to-All Fig. 4. Uniform caacity case: intra-to-inter od 8 7 6 5 4 3 All to all One to all 8 6 4 6 5 4 3 3 4 5 6 7 8 9 3 4 (a) One-to-All and All-to-All (b) All-to-All (c) One-to-All Fig. 5. Asymmetric caacity case: intra-to-inter od may be set different for links with ods comared to the links that connect aggregation switches to core switches. B. Study cases In order to assess traffic fairness with different levels of the horizontal DCN caacities, we consider the two following DCN dimensioning cases: Uniform caacity: all link caacities are set equally. In this case, we consider different caacity configurations, increasing the caacity on all the links from to units in increments of units. Asymmetric caacity: the starting configuration has an equal caacity of units er link. We then increase the caacity only on the intra-od links (link between edge and aggregation switches) from to units by increments of units. The caacity of links between the aggregation and cored switches ( extra-od links ) remains fixed at units. We run our cases for different values of α, i.e., the minimum sub-flow allocated to each ath available to a demand d. We consider the following cases: Unbounded MultiPath (UMP) case, with α =, so that multiath forwarding is not forced for any demand, but can be used; Bounded MultiPath (BMP) case, with α =.5, so that multiath forwarding is lightly forced on all available aths for all demands, and can be freely used; Equi-distribution MultiPath (EMP) case, with α being relaced by α d = /N d in (7), where N d is the number of aths available to demand d, so that traffic distribution is forced to be even over the aths available to each demand. It is worth noting that for the fat-tree toology, intra-od traffic can have two aths, while inter-od traffic can use u to four aths. Moreover, we evaluate the results for both the utility functions resented in Section III: the fixed-delay situation given by (3) and the weighted-delay situation ( ω d = ho count) given by (4). These two otions allow us to see how fairness is guaranteed for intra-od and inter-od traffic. More imortantly, a data center rovider can decide to deloy its referred TCP imlementation (as they own the servers) by taking advantage of the lessons learned from this study, and accordingly allocate jobs to servers to target traffic fairness. In other words, this study also hels the Cloud rovider to decide on fine-grained scheduling of jobs that meets traffic fairness requirements. In order to show the imact on throughut allocation between intra-od and inter-od edge switches, we measure the intra-to-inter-od traffic allocation ratio. Finally, we measure the ath diversity of the solution for the UMP case for all the edge-to-edge demands ( All-to-All ) and from the oint of view of a single edge switch ( One-to-All ).

.8.8 Used aths.6.4 Used aths.6.4.. 3 4 5 6 7 8 9 3 4 5 6 7 8 9 (a) All-to-All (b) One-to-All Fig. 6. Uniform caacity case: used aths ratio.8.8 Used aths.6.4 Used aths.6.4.. 3 4 5 6 7 8 9 3 4 5 6 7 8 9 (a) All-to-All (b) One-to-All Fig. 7. Asymmetric caacity case: used aths ratio C. Results This subsection illustrates the results of the roortional fairness model concentrating the analysis around three key asects: the throughut allocation, the traffic distribution within and across ods, and the achieved ath diversity. ) Throughut allocation: First, consider the global throughut when traffic between all air of edge switches are allowed ( All-to-All ). Next, assume that all extra-od links are droed, i.e., there are only intra-od links with a caacity of each. It is easy to see that each od is isolated in this case (and there is no inter-od traffic). Thus, the traffic throughut between the two edge switches in a od is limited by the caacity of the intra-od links. Since two links form a ath, we can see that the throughut within a od between its two edge switches is. Thus, for the 4-od fat-tree toology, the total throughut is 8. In this case, also the traffic allocation between intra-od and inter-od is most skewed. It is interesting to note that when extra-od links have a ositive caacity, the total throughut still remains at 8 as long as the caacity of the intra-od links are at units each. In Figures and 3, the global throughut is lotted as the caacity is increased. We find that it grows linearly as dictated by the caacity of intra-od links, irresective of the caacity of the links connecting aggregation and core switches. More imortantly, it is not affected by the multiath case (UMP, BMP, EMP) nor the tye of the utility function (fixed-delay vs. weighted delay). It is not so trivial to observe that the global throughut aears as being directly roortional to the fixed link caacity at eight times the link caacity for the All-to-All case. ) The traffic distribution: Next we investigate how the traffic distribution is affected between intra-od and inter-od edge switches, for which we use the metric intra-to-inter od. We characterize the traffic distribution sensibility with resect to the various cases focusing on the intra-to-inter od and on the between neighboring ods and between non-neighboring ods. For the uniform caacity case with regards to the intra-tointer od for all-to-all demands (Figure 4), we find that on average the allocation between intra-od and inter-od traffic is similar with the fixed-delay situation. On the other hand, with the weighted-delay situation, the inter-od traffic has a traffic roortion that is almost twice that of the intraod traffic. This can also be exlained since the ath ho count of an inter-od demand (4 hos) is twice that of an intra-od demand ( hos) this is reflected in the weights for the weighted-delay situation. For the asymmetric caacity case, the observation is strik-

ingly different than the uniform case. The intra-od demands have 6 times more throughut than inter-od demands (Figure 5-a) for the all-to-all traffic case when the caacity of the intra-od links reaches, while the extra-od links caacity was ket fixed at. This gain is in alignment with the secial case we discussed earlier when there is no caacity on extraod links, the most skewed case. From Figures 5b and 5c, we note that the fixed-delay situation also favors intra-od traffic and becomes steady when the extra-od links caacity is three times higher than the intraod link caacity. When we have only one source, it becomes steady when the extra-ods link caacity is twice the caacity of the intra-od links (for the two cases the curves converge when the ratio is equal to 6). 3) Path Diversity: Figures 6 and 7 illustrate ath diversity for the UMP case, measured as the ratio of the overall used aths to the number of overall available aths. We also lot the line corresonding to the single-ath situation. It is worth remembering that for the BMP and EMP cases, all the aths are used (so it would be a to line at a ratio equal to ). Any ath diversity of the solution does not seem to be affected by the secific utility function (TCP behavior). We can see that although ath diversity was allowed, the unconstrained multiath case did not take full advantage of it. This seems to imly that ath diversity is not necessary to maintain the highest throughut. V. CONCLUSION Data center networking is a challenging field of alications of old and new technologies and concets. In this aer, we investigated DCN caacity sharing among cometing greedy demands from a roortional fairness ersective rovided by TCP utility functions in the equilibrium. We resented a generalized formulation of the basic roortional fairness model for DCN allowing multiath forwarding for elastic demands. We also described and evaluated our model under two different TCP utility functions: fixed-delay and weighted-delay. Through a series of scenarios studied on the 4-od fattree toology, we discovered a number of interesting results. In articular, we found out that the weighted-delay utility function gives twice as much imortance to the intra-od traffic, which may be exloited by the data center rovider for high-level scheduling of traffic intensive alications. Another imlication is that for a very large IaaS comosed of numerous virtual machines needing to san more than a od, this bias towards intra-od traffic may be an undesirable behavior, while for a small IaaS this could be a desirable behavior. In our oinion, this should influence the cloud orchestration logic and IaaS management algorithms in VM lacements, to be roerly designed. We also measured the ath diversity of the solutions in the case in which a systematic multiath mode over all demands was not forced, and multile ath selection was left to the congestion control. We found that only a fraction of the aths was eventually chosen for demands. Our study, to the best of our knowledge, was the first one to address the imact of DCN toology design, caacity lanning and multiath forwarding in traffic fairness in DCN fabrics. We believe the results are interesting and deserve further study, esecially grounded on real data as soon as this becomes ublicly accessible to researchers. ACKNOWLEDGEMENT This work was artially suorted by the Systematic FUI 5 roject RAVIR (htt://www.ravir.io), by the EU FP7 IRSES MobileCloud Project (Grant No. 6), and by National Science Foundation grant CNS-9655. REFERENCES [] M. Al-Fares, A. Loukissas, and A. Vahdat, A scalable, commodity data center network architecture, in ACM SIGCOMM Comuter Communication Review, vol. 38, no. 4. ACM, 8,. 63 74. [] T. Benson, A. Akella, and D. Maltz, Network traffic characteristics of data centers in the wild, in Proceedings of the th annual conference on Internet measurement. ACM,,. 67 8. [3] L. S. Brakmo and L. L. Peterson, TCP vegas: End to end congestion avoidance on a global internet, IEEE Journal on Selected Areas in Communications, vol. 3, no. 8,. 465 48, 995. [4] A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, Architectural guidelines for multiath TCP develoment, RFC68 (March ), www. ietf. ort/rfc/68,. [5] C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, and S. Lu, BCube: a high erformance, server-centric network architecture for modular data centers, in ACM SIGCOMM Comuter Communication Review, vol. 39, no. 4. ACM, 9,. 63 74. [6] F. Kelly, A. Maulloo, and D. Tan, Rate control for communication networks: shadow rices, roortional fairness and stability, Journal of the Oerational Research society, vol. 49, no. 3,. 37 5, 998. [7] R. Khalili, N. Gast, M. Poovic, U. Uadhyay, and J.-Y. Le Boudec, MPTCP is not areto-otimal: erformance issues and a ossible solution, in Proceedings of the 8th international conference on Emerging networking exeriments and technologies. ACM,,.. [8] S. Kunniyur and R. Srikant, End-to-end congestion control schemes: Utility functions, random losses and ecn marks, IEEE/ACM Transactions on Networking, vol., no. 5,. 689 7, 3. [9] S. H. Low, L. L. Peterson, and L. Wang, Understanding TCP Vegas: a duality model, J. ACM, vol. 49, no.,. 7 35, Mar.. [] R. Mazumdar, L. Mason, and C. Douligeris, Fairness in network otimal flow control: otimality of roduct forms, IEEE Transactions on Communications, vol. 39, no. 5,. 775 78, 99. [] D. Medhi, Alications with multile arallel flows: Assessing their unfair advantage with roortional fair sharing TCP, in Proc. of IEEE International Conference on Communications (ICC 4), Sydney, Australia, June 4. [] D. Oran, OSI IS-IS intra-domain routing rotocol, 99. [3] M. Pióro and D. Medhi, Routing, flow, and caacity design in communication and comuter networks. Elsevier/Morgan Kaufmann, 4. [4] R. Srikant, The mathematics of Internet congestion control. Birkhauser, 4. [5] R. Stewart, Stream control transmission rotocol, 7. [6] D. Thaler and C. Hos, Multiath issues in unicast and multicast next-ho selection, RFC 99, November, Tech. Re.,. [7] J. Touch and R. Perlman, Transarent interconnection of lots of links (TRILL): Problem and alicability statement, 9. [8] D. Wei, C. Jin, S. Low, and S. Hegde, FAST TCP: Motivation, architecture, algorithms, erformance, IEEE/ACM Transactions on Networking, vol. 4, no. 6,. 46 59, 6. [9] Q. Zhang, L. Cheng, and R. Boutaba, Cloud comuting: state-of-the-art and research challenges, Journal of Internet Services and Alications, vol., no.,. 7 8,.