[Yawen comments]: The first author Ruoyan Liu is a visting student coming from our collaborator, A/Prof. Huaxi Gu s research group, Xidian

Transcription

1 [Yawen comments]: The first author Ruoyan Liu is a visting student coming from our collaborator, A/Prof. Huaxi Gu s research group, Xidian Universtity. He stays in the University of Otago for 2 mongths under the supervison of Yawen Chen and Haibo Zhang working on a collabrated research topic of data center networks.

2 Analyzing Packet-level Routing in Data Centers Ruoyan Liu and Huaxi Gu State Key Laboratory of Integrated Service Networks Xidian University, China Yawen Chen and Haibo Zhang Department of Computer Science University of Otago, New Zealand Abstract Data centers host diverse applications with stringent QoS requirements. The key issue is to eliminate network congestions which severely degrade application performance. One effective solution is to load balance the traffic in the datacenter regular topologies. Many previous strategies focused on optimized flow routing. Limited by practicality, these solutions can hardly achieve ideal load balance while guaranteeing QoS of different traffic flows. In this paper, we discuss packet-level routing and analyze its merit for fine-grained load balance in data centers. Though packet-level routing interacts poorly with TCP in traditional network settings, we prove that it suits datacenter environment. Motived by the work by Dixit, we assert that packet-level routing is the right choice for data centers. We also make a minor change to TCP for better compatibility with packet-level routing in data centers. Finally, we demonstrate by simulations that packet-level routing better fulfills datacenter requirements. Index Terms data center, routing, TCP I. INTRODUCTION In the last few years, 10 Gigabit Ethernet has proven its capability to provide a unified infrastructure for data centers comprising LAN, SAN and clustering network. Low cost, high compatibility and easy management have made Ethernet the first choice for constructing large scale data centers. It can be expected that the impending 40 and 100 Gigabit Ethernet will further reinforce the status of Ethernet in the future. However, one inherent drawback of traditional Ethernet is that packet delivery follows a best-effort manner, which gives rise to packet losses when network traffic overwhelms switch buffers. This is not tolerated in data centers, since Fiber Channel (typically for constructing SAN) consolidated over Ethernet requires reliable packet transmission. As one popular solution for guaranteeing packet lossless, link-level flow control (LL-FC) like credit-based or on/off flow control is widely employed. However, this leads to another undesirable side-effect that network congestion cannot be detected by TCP until the timer of one packet expires. What is worse, LL-FC may rapidly spread local congestion to neighboring switches which are typically commodity components with shallow buffers. Since the rapid growth of data centers brings booming traffic with different stringent QoS requirements, congestion avoidance is increasingly important to datacenter performance. One big challenge is to design a well-behaved routing strategy which perfectly balances the traffic evenly in the datacenter network. Many previous proposals focused on optimized flow routing [3][4][5]. However, limited by overhead and scalability, these proposals can only effectively manage a small number of flows, achieving undesirable load balance while leaving the rest of flows unguaranteed. In this paper, we move to packet-level routing for seeking better solutions. Per-packet load balance in data centers is firstly researched by Dixit et al. [10], who analyzed three simple strategies splitting traffic evenly among all available paths. These strategies achieve fine-grained load balance and eliminate network congestion. However, packet-level routing has long been abandoned in practice (especially in Internet), since it will induce severe packet disorder which drastically degrades TCP throughput. Despite this, packet-level routing is proved by [10] to perform unexpectedly well in data centers. However, Dixit et al. did not perform full analysis of the reason why packet-level routing specially suits datacenter environment. Motived by the work of [10], we continue researching packet-level routing in data centers. We aim to minimize the adverse impact of packet disorder while maximize the advantage of fine-grained load balance. Since multi-rooted topologies begin to be widely employed by data centers, packet-level routing can largely utilize the path diversity of the topologies and eliminate network congestions. Also, packetlevel routing is static with low overhead, it is easy to be implemented in current switch. The two main contributions of this paper are: 1. We analyze the advantages and practicality of packet-level routing in data centers. We assert that packetlevel routing is the right choice for implementing fine-grained load balance. 2. We make a minor change to TCP for better compatibility with packet-level routing in data centers. Finally, our simulations demonstrate that packet-level routing with modified TCP better fulfill datacenter requirements. II. BACKGROUND AND RELATED WORK A. Traffic pattern and network congestion Data centers have been dominating computing services such as web searching, social networking, data mining and scientific computing. The heterogeneous applications require strictly high QoS for either low latency or high throughput [6]. For example, soft-real-time applications (discussed by [6] and [7]) like web searching and social working, produce query traffic which stems from partition-aggregate pattern. An aggregator partitions queries to sub-aggregators which in turn partition each query over a number of worker nodes. The responses from the workers have tight deadlines which are typically between 10ms to 100ms. Responses after the deadline will be deleted, violating service SLA and thus revenue. Besides

3 query traffic, the applications also create a mix of background flows, which are typically used for updating data structures. These flows are much larger, requiring high throughput and burst tolerance. With the mixture of the variety of traffic in data centers, network congestions can be caused by many reasons. Incast [8], for example, is a common problem in partition-aggregate traffic pattern: servers respond almost simultaneously to the aggregator, easily causing a sharp rise of buffer occupancy at the aggregators port. Incast may happen even each response is very small, since a number of responses in a short time may quickly exhaust the shallow buffer of a commodity switch. Another major reason of network congestions comes from large flows collisions: two or more bandwidth-thirsty flows happen to compete for the same output port on a switch. When their bandwidth demands exceed the link capacity, packets will soon saturate the switch buffer. When network congestion happens, the performance of either latency-sensitive or throughput-sensitive traffic will degrade drastically. In addition, since datacenter topologies are always designed with sufficient bandwidth for the expected peak load, the traffic imbalance will cause a lot of network resources underutilized. Therefore, load balance is always one important concern in data centers. B. Datacenter topology As intra-datacenter traffic begins to dominate, todays data centers are organized in the form of multi-rooted tree. Fattree [1] and BCube [2] are typical multi-rooted topologies which are dense interconnect structures. These topologies use identical, commodity components for providing high bisection bandwidth with high scalability and fault tolerance. More importantly, multi-rooted topologies are abundant in path diversity, which endows the network the potential of load balance. For example, in a fat-tree (as shown in Fig. 1) with K pods (a complete bipartite graph of edge and aggregation switches), each edge switch or aggregation switch has K/2 equal-cost uplinks, which translate to (K/2) 2 equal-cost paths between any two server pairs in different pods. C. Flow-level routing Routing in data centers have long been based on flow level: packets in each traffic flow can only follow one route, ensuring that packets will arrive in order. Todays data centers typically use hashed based flow routing like Equal-Cost Multi-Path (ECMP) [9] which forwards each flow according to the TCP 5-tuples in the packet header. This strategy achieves equivalent performance to per-flow random load balancing (RLB), which randomly selects one of the equal-cost paths for each flow. However, neither ECMP nor per-flow RLB achieves ideal performance, for large flows may happen to collide and result in congestions. Recently, several optimized flow scheduling strategies have been proposed. For example, in [3], a central controller with global visibility greedily schedules each large flow to the current least-congested path. This strategy yields much higher throughput than ECMP. However, too much overhead is generated during large flow detection, link state Fig. 1. Equal-cost paths between communicating servers in a 4-pod fat-tree. collection and path probing. Limited by current controller with NOX system [11], this strategy achieves poor scalability. III. PACKET-LEVEL ROUTING Motivated by the work of [10], we discover that packetlevel routing can better fulfill datacenter requirements. The following part will analyze the details of advantages and implementation of packet-level routing. A. Disadvantage of flow-level routing The root problem of flow-level routing is that only one of the equal-cost paths is selected for forwarding each flow, resulting traffic imbalance inevitably due to diversity of flows. As far as we know, all current optimized flow routing strategies can schedule only large flows, since it is far from trivial to schedule each of the numerous short, latency-sensitive flows in large scale data centers. However, even buffer overflows are avoided by these sophisticated strategies, the latency-sensitive flows can still be deadly delayed if they are in queue behind packets from large flows, or there is little space buffering them when large flows occupy too much shared buffer of the switch. The two phenomena are termed as queue-buildup and buffer pressure [6] respectively, which are inherent problems of flow routing and cannot be avoided. Another problem stems from the link-level flow control (LL-FC) which prevents packet losses in data centers. Packet dropping is an effective means for TCP rapidly detecting and reacting to network congestions, however, this function is blocked in data centers. Since flow routing can easily result network congestions which may spread quickly to neighboring nodes and cause disastrous performance degradation. B. Packet-level routing in Internet and data centers Compared with flow-level routing, packet-level routing can achieve per-packet load balance which fully utilizes path diversity. By fully dispersing the packets from each flow to multiple available paths, bandwidth contention between large flows will be eliminated. The problems of queue-buildup, buffer pressure and congestion spreading caused by LL-FC will also be solved, thus the QoS of short while latencysensitive flows can be ensured. Despite the obvious advantages, packet-level routing is long abandoned in Internet. This is because the severe packet disorder caused by packet-level routing interacts poorly with TCP (TCP must transfer in-order segments to upper-layer

4 applications) in Internet where diverse subnets have large differences in latency. Therefore, flow routing is set by default in Internet and TCP is designed by assuming the packets from a flow will arrive at the receiver in-order under normal circumstances. Previous researches have demonstrated the differences between datacenter network and Internet. Despite unwelcomed in Internet, packet-level routing is possible for datacenter setting for the reasons below: 1. Data centers consist of homogeneous components. As shown in Fig. 1, all the switches and links are identical in a 3-tier fat-tree. Parallel paths between any server pair contain same number of switches with same processing capabilities. Therefore, they are equal-cost paths with no difference with each other. It is quite different with the structure of Internet where packet disorder is much more severe caused by packet-level routing. 2. Multi-rooted topologies in data centers are high in scalability but low in network diameter. In a fat-tree of an arbitrary scale, the hops between any server pair are no more than five. This means that it is possible to control packet disorder at a low level. 3. Link level protocols and standards have evolved in data centers. For example, group has defined standards of flow control and congestion control for data center bridging, overlapping some similar mechanisms of TCP. Since these TCP mechanisms interact poorly with packetlevel routing, they can be modified and even cancelled in datacenter environment. The following part will discuss the details. C. Two packet-level routing strategies for data centers In this part, we introduce two simple strategies of packetlevel routing for data centers: per-packet random load balancing (per-packet RLB) and cycle-based round robin (cyclebased RR). 1. Per-packet random load balancing This simple strategy is also introduced in [10]. At each layer, per-packet RLB randomly selects one of the equal-cost next hops for forwarding each packet. From the perspective of probability, packets from a single flow will evenly distribute on each equal-cost path. In a fat-tree, for example, packets are fully dispersed on the upward half of the route. Even the downward half of the route is deterministic, it also achieves fine-grained load balance. 2. Cycle-based round robin In per-packet RLB, packets from two or more input ports may happen to contend the same output port simultaneously. This results minor latency that may cause packet disorder. Cycle-based RR is designed for alleviating this problem. Assume the situation that n input ports share k equal-cost output ports in a switch. For each polling cycle of the switch, cycle-based RR changes the mapping relationship between the input and output ports in a round-robin fashion. For an arbitrary input port C i, the output port S i assigned to C i in the current cycle is: S i =(S 0 i + 1) modulo k, where Si 0 is the assigned output port in the last cycle. The strategy initially lets the input and output ports form the mapping relationship: S init i = i modulo k, for each i 2 [0,n 1]. Obviously, if n is larger than k, it will be a many-toone mapping and packet contention is evitable when there are packets waiting to be allocated in each input buffer. Some topologies, such as fat-tree where n equals to k, this strategy will form a one-to-one mapping, eliminating packet contention. As a result, each new-arriving packet will be transferred immediately. D. TCP modification For packet-level routing, packet disorder cannot be completely eliminated. Packets may contend the same determined output port (typically when destination-based routing is used for packet forwarding) and result in packet delay. As traditionally designed, when receiving a packet whose sequence number is not expected, the TCP receiver will return a duplicate ACK of the last received in-order packet to the TCP sender. TCP will regard three consecutive duplicate ACKs acknowledging the same packet as a sign of packet loss. The considered lost packet will be retransmitted immediately (fast retransmission stage) and the congestion window will be halved. This will in turn reduce the sending rate to alleviate network congestion. Though the TCP reactions make little influence to latency-sensitive flows (typically very short), large flows will suffer from throughput degradation severely. As packet-level routing fully utilizes high bandwidth in data centers, it is not likely to create network congestion unless the aggregate traffic demand from one server surpasses link or NIC capacity. This traffic imbalance seldom happens in data centers, since virtual machines can migrate to other physical servers for a balanced traffic matrix [12]. Therefore, traditional congestion mechanism of TCP improperly handles disordered packets in data centers and causes unnecessary throughput degradation. As a simple modification, we propose that TCP tolerates appropriate more disordered packets before triggering fast retransmission. Note that we dont abandon the congestion control mechanism for considering the situation: a packet may be unexpectedly delayed for a relative long time or dropped for some unknown reasons. If fast retransmission is not allowed for this case, the sending window will grow very slowly until the ACK of the packet returns or the timer expires. As a consequence, the sending rate will also degrade severely. Therefore, an appropriate number of disordered packets that TCP can tolerate (we call it TCP-DP) is needed in real implementation. In the following section, we experiment different TCP-DP with simulations.

5 A. Simulation model IV. SIMULATION Our simulations are based on OPNET We use a 6- pod fat-tree hosting 54 servers as our testbed, and employ the traffic pattern of common soft-real-applications. In our testbed, we simulate aggregate-partition workload by designating one fixed server as the aggregator, and 18 other servers which are scattered in each pod as the worker nodes. The rest servers produce background traffic and the targets are uniformly distributed. Our traffic parameters follow the measurement by [6]: each query flow is 2KB, comprising two packets (we set fixed packet size of 1KB), while the size of each background flow is 25MB. For simplicity, each server generates only one flow at one time and receives at most three background flows simultaneously. We use an identical 10Gbps interconnection with shallow buffered switches (buffer of 160 packets for each port). The polling cycle of each switch is initially set to 50ns. TCP Reno is used as the transport protocol and credit-based flow control is implemented to guarantee packet lossless. Also, each server is configured with infinite storage for buffering delayed packets. B. Performance comparison We compare flow-level ECMP, per-packet RLB and cyclebased RR for query traffic completion and network goodput. During each simulation, the statistics are collected after the system becomes totally stable. 1) Query traffic completion: We simplify the process of query partition by letting the aggregator inform each worker node through remote interruption. A timer is set up when queries are sent out. After getting informed, each worker node immediately generates a response of two packets. We set a tight deadline at the aggregator of 10ms, the responses arriving after the deadline will be shipped out. The aggregator repeats sending queries after each deadline and we count the number of completed responses for each time. As shown in Fig. 2 (left), ECMP only completely finishes 3 times out of 15 independent experiments, while per-packet RLB and cycle-based RR finish each time. This proves that packet-level routing has good and stable performance in delivering latency-sensitive responses. In contrast, flow-level routing unexpectedly delays some of the responses, for bursty flows easily cause buildup which deadly blocks latency-sensitive flows. 2) Network goodput: For this evaluation, we use goodput instead of throughput, since goodput reflects the actual rate the servers receiving valid data. As shown in Fig. 2 (right), either per-packet RLB or cycle-based RR outperforms ECMP even TCP is not modified. At low TCP-DP, cycle-based RR achieves higher goodput than per-packet RLB, since it reduces the degree of packet contention. As we increase TCP-DP, the two packet-level routing strategies achieve higher goodput, for unnecessary retransmission and throughput degradation are prevented. When the number of tolerated disordered packets is bigger than 7, the two packet-level routing strategies achieve similar performance. This is because the discrepancy of packet disorder in two strategies has already been eliminated by Fig. 2. Comparison of query traffic completion and network goodput between flow-level and packet-level routing tolerating more disordered packets. Enlarging TCP-DP over 9 does little improvement, since most disordered packets are within a small range and the performance has reached its limit. However, if the network unexpected delays or drops a packet, a too big TCP-DP may be very harmful to the network performance. In our testbed, we recommend setting TCP-DP to 9. V. FUTURE WORK We give a simple analysis and evaluation of how modified TCP upgrades the performance of packet-level routing in data centers. As for this paper, we still need an accurate mathematical analysis to seek an optimal number of tolerated disordered packets when taking diverse topologies, traffic patterns etc. into consideration. We believe packet-level routing is promising in data center setting, and a dynamic mechanism may better promote the network performance. Also, more sophisticated modifications to TCP or other transport protocols are necessary for better compatibility with packet-level routing in data centers. REFERENCES [1] M. Al-Fares, A. Loukissas, and A. Vahdat, A Scalable, Commodity Data Center Network Architecture, SIGCOMM, [2] Chuanxiong, Guohan Lu and Dan Li, BCube: A High Performance, Server-centric Network Achitecture for Modular Data Centers, SIG- COMM, [3] M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat, Hedera: Dynamic Flow Scheduling for Data Center Networks, NSDI, [4] A. R. Curtis, W. Kim, and P. Yalagandula, Mahout: Low-Overhead Datacenter Traffic Management using End-Host-Based Elephant Detection, INFOCOM, [5] Xin Wu, Xiaowei Yang, DARD: Distributed Adaptive Routing for Datacenter Networks, ICDCS, [6] Mohammad Alizadeh, Albert Greenberg, Data Center TCP (DCTCP), SIGCOMM, [7] Christo Wilson, Hitesh Ballani, Thomas Karagiannis, Ant Rowstron,Better Never than Late: Meeting Deadlines in Datacenter Networks, SIGCOMM, [8] V. Vasudevan, Amar Phanishayee, Hiral Shah et al., Safe and Effective Fine-Grained TCP Retransmissions for Datacenter Communication, SIGCOMM, [9] C. Hopps. Analysis of an Equal-Cost Multi-Path Algorithm. RFC 2992 (Informational), Nov [10] Advait Dixit, Pawan Prakash, Ramana Rao Kompella, On the Efficacy of Fine-Grained Traffic Splitting Protocols in Data Center Networks, SIGCOMM, [11] A. Tavakoli, M. Casado, T. Koponen, and S. Shenker, Data Center TCP (DCTCP)Applying NOX to the datacenter, HotNets-VIII, [12] Navendu Jain, Ishai Menache, Joseph Naor and F. Bruce Shepherd, Topology-Aware VM Migration in Bandwidth Oversubscribed Datacenter Networks, ICALP, 2012.