GreenTE: Power-Aware Traffic Engineering

GreenTE: Power-Aware Traffic Engineering Mingui Zhang zmg6@mais.tsinghua.edu.cn Cheng Yi yic@emai.arizona.edu Bin Liu iub@tsinghua.edu.cn Beichuan Zhang bzhang@arizona.edu Abstract Current network infrastructures exhibit poor power efficiency, running network devices at fu capacity a the time regardess of the traffic demand and distribution over the network. Most research on router power management are at component eve or ink eve, treating routers as isoated devices. A compementary approach is to faciitate power management at network eve by routing traffic through different paths to adjust the workoad on individua routers or inks. Given the high path redundancy and ow ink utiization in today s arge networks, this approach can potentiay aow more network devices or components to go into power saving mode. This paper proposes an intra-domain traffic engineering mechanism, GreenTE, which maximizes the number of inks that can be put into seep under given performance constraints such as ink utiization and packet deay. Using network topoogies and traffic data from severa widearea networks, our evauation shows that GreenTE can reduce ine-cards power consumption by 27% to 42% under constraints that the maximum ink utiization is beow 5% and the network diameter remains the same as in shortest path routing. I. INTRODUCTION The Internet as an indispensabe communication system in our society aso has its share in energy consumption. Research on energy management has traditionay focused on battery-operated devices, and more recenty, stand-aone servers and server custers in data centers. The underying network infrastructure, namey routers, switches and other network devices, sti acks effective energy management soutions. Epps et a. [1] from Cisco report that a high-end router CRS- 1 with maximum configuration can consume as much as one MegaWatt. The same report aso points out that driven by exponentia growth of Internet traffic, router system requirements are outpacing siicon and cooing technoogies. In addition to eectricity bis, the arge power consumption by network devices aso puts a ot of stress on power deivery to and heat remova from router components as we as the hosting faciity. With the advent of coud computing and arge data centers, the probem wi ony get worse. In short, router power consumption has become an increasing concern for Internet Service Providers (ISPs), Internet Exchange Points (IXPs), and data centers. Existing research on router power management treats routers as isoated devices and focuses on reducing power consumption at hardware component eve. Recenty Gupta et a. [2] Department of Computer Science and Technoogy, Tsinghua University, Beijing 184, China. Computer Science Department, The University of Arizona, Tucson, AZ 85721, USA. This work was done during Mingui Zhang s visit to the University of Arizona. He wi join Huawei Technoogies Co. Ltd. This work is partiay supported by US NSF award CNS-721863, China NSFC (662521, 687325), 973 project (27CB3171) and Tsinghua University Initiative Scientific Research Program. suggested to consider routers in the network context and create more power saving opportunities by adjusting the amount of traffic going through routers, but they did not propose specific soutions. There are ink-eve soutions which put ine-cards to seep when there is no traffic on the ink [3], however, the power saving from opportunistic seeping is imited by the inter-arriva time of packets. Compementary to component-eve and ink-eve soutions are network-eve soutions. Today s networks are designed and operated to carry the most traffic in the most reiabe way without considerations of energy efficiency. A network usuay buids many redundant inks and aggressivey over-provisions ink bandwidth to accommodate potentia ink faiures and traffic bursts. Whie these redundant inks and bandwidth greaty increase the network reiabiity, they aso greaty reduce the network s energy efficiency as a network devices are powered on at fu capacity 24x7 but highy under-utiized most of the time. Rue of thumb states that today s backbone inks are used by 4% or ower [4] in their capacity. The high path redundancy and ow ink utiization provide unique opportunities for poweraware traffic engineering. Intuitivey, when there are mutipe paths between the same origin-destination (OD) pair, and the traffic voume on each path is ow, one can move the traffic to a fewer number of paths so that the other paths do not carry any traffic for an extended period of time. Routers that have ide inks can then put the inks to seep for energy conservation. This approach can be combined with component-eve and inkeve soutions to achieve higher network energy efficiency. Network-eve soutions require network-wide coordination of routers. The chaenges are two-fod, namey how to manipuate the routing paths to make as many ide inks as possibe to maximize the power conservation, and how to achieve power conservation without significanty affecting network performance and reiabiity. Since power-aware traffic engineering uses fewer number of inks at any moment, it is important to make sure that inks are not overoaded and packets do not experience extra ong deays. This paper proposes GreenTE, a power-aware traffic engineering mechanism that reduces network power consumption whie sti maintaining network performance at desired eves. GreenTE is formuated as a Mixed Integer Programming (MIP) probem with the tota power saving as the objective to be maximized. Performance requirements such as maximum ink utiization (MLU) and network deay are considered as constraints in the probem. Whie the probem formuation bears simiarity to that of traditiona traffic engineering research, the main contribution of this work is the soution resuts. Traditiona traffic engineering and power-aware traffic engineering

Fig. 1. Link Utiization (%) 6 5 Maximum 4 3 2 Average 1 2 3 4 5 6 7 8 9 Days (2/9/24, 8/9/24) Maximum and average ink utiization in the Abiene network TABLE I THE CONFIGURATION OF A CISCO 12 ROUTER [5] Sot Cardtype Watts 1 OC3-4-POS-X 9 2 GE-4 16 6 OC3-POS-16 1 7 OC12-ATM-4 122 8 OC3-4-POS-X 9 5, 9 GSRP 38 16, 17 CSC1 19 18 22 SFC1 64 24, 25 ALARM1 33 29 BLOWER16 178 have two opposite optimization goas: the former tries to spread traffic eveny to a the inks, whie the atter tries to concentrate traffic to a subset of the inks. It is uncear whether one can achieve significant power saving whie sti maintaining acceptabe ink utiization in rea networks. We sove the poweraware traffic engineering probem using rea network topoogies and traffic data, demonstrating that it is both promising and feasibe. Using network topoogies and traffic data from two widearea research networks, Abiene and GÉANT, we evauate GreenTE in terms of power saving, ink utiization, packet deay and routing stabiity. Resuts show that GreenTE can achieve 27% to 42% power saving on ine-cards under the constraints that maximum ink utiization is beow 5% and the network diameter remains the same as that in pure OSPF routing. The number of MPLS tunnes needed is sma compared with fu mesh, and routing is argey stabe as more than 7% of the MPLS tunnes remain unchanged from one adjustment period to the next one. We aso show that GreenTE can be appied to arge commercia networks such as Sprint and AT&T and achieve simiar power savings too. The rest of the paper is organized as foows. Section II gives an overview of the basic idea and its assumptions. Section III formuates the power saving probem as a traffic engineering probem and presents the GreenTE mode. Section IV discusses potentia impementation issues. Section V evauates GreenTE using topoogies and traffic matrices from severa rea networks with coected traces and synthesized data. Section VI reviews the previous work and Section VII concudes the paper. II. BASIC IDEA AND ASSUMPTIONS Today s wide-area networks usuay have redundant and overprovisioned inks, resuting in ow ink utiization during most of the time. Figure 1 shows the maximum and average ink utiization under OSPF routing in Abiene, a arge US education backbone, during a typica week. The average ink utiization is ony about 2%, the maximum fuctuates mosty between 1% and 2%, and ony one rare event pushes the maximum over 5%. Such behavior is common in arge commercia networks as we. High path redundancy and ow ink utiization combined aso provide a unique opportunity for power-aware traffic engineering as iustrated by the exampe in Figure 2. Traditiona TABLE II THE POWER BUDGET OF A CISCO 12 ROUTER [5] Sot Category Watts 1, 2, 6, 7, 8 ine cards 58 W 5, 9 Route processors 76 W 16 22, 24, 25, 29 Chassis components 62 W Tota inuse power 1186 W traffic engineering spreads the traffic eveny in a network (Figure 2(a)), trying to minimize the chance of congestion induced by traffic bursts. However, in power-aware traffic engineering (Figure 2(b)), one can free some inks by moving their traffic onto other inks, so that the inks without traffic can go seep for an extended period of time. This shoud resut in more power saving than pure opportunistic ink seeping because the seep mode is much ess ikey to be interrupted by traffic. In this paper, we focus on saving power by turning off inks, or interchangeaby, putting ine-cards (or their ports) into seep mode. Line-cards contribute a significant portion to the tota power consumption of a router. Tabe I shows a typica configuration of a Cisco 12 series router with ow to medium interface rates and Tabe II shows its budget of inuse power consumption. A the ine-cards together consume 58 Watts, about 43% of the router s tota power budget. This particuar configuration uses reativey ow rate interfaces (ess than 1Gb/s) and the router is aso of an od mode. With faster interfaces (1Gb/s or even 4Gb/s) in newer routers, ine-cards power consumption wi constitute an even arger part of the entire system s power consumption. Besides direct power savings, turning off inks may aso give indirect savings, e.g., one of the router s bowers may be abe to shut down due to ess heat. There can aso be different ways to reduce power consumptions of ine-cards, e.g., using sower ine rates for ess traffic, but they are out of the scope of this paper. GreenTE, ike any other power saving mechanisms, needs support from the underying hardware. We make the foowing assumptions based on today s typica router architectures and hardware in designing GreenTE. However, most of them can be reaxed to take advantage of better hardware support in the future without impacting the basic GreenTE probem formuation and soution. First, a ine-card can have mutipe ports, and each port may connect to a ink. The mutipe ports of one ine-card may

2% 2% 4% 4% TABLE III SUMMARY OF NOTATION USED IN THIS PAPER Ingress/origin 2% 2% (a) traditiona Fig. 2. egress/destination Ingress/origin Different traffic engineering goas (b) power-aware egress/destination connect to the same remote router, making it a bunded ink, or connect to different remote routers. When a ink is put to seep, the port that connects to the ink can go seep; when a ports on a ine-card are aseep, the entire ine-card can be put to seep, resuting in more power saving due to the inecard s base power consumption. A comprehensive anaysis of the power consumption of different network components can be found in [6]. Second, a ink can be put to seep ony when there is no traffic in both inbound and outbound directions, which is based on the fact that the transceivers of inbound and outbound traffic do not have separate power contro in most hardware. The GreenTE mode can be easiy adjusted to aow turning on/off inks unidirectionay shoud hardware supports it, which wi bring even more power saving than what we show in this paper. Third, a port or a ine-card can go into seep or wake up quicky, in the order of miiseconds, controed by its host router [3]. III. GREENTE MODEL To generaize the basic idea iustrated in Figure 2(b), we deveop the GreenTE mode, which, given the network topoogy and traffic matrix, finds a routing soution (i.e., the inks to be used and the traffic voume to be carried on each ink) that maximizes the power saving from turning off ine-cards as we as satisfying performance constraints incuding ink utiization and packet deay. A. The Genera Probem Formuation We mode the network as a directed graph G = (V, E), where V is the set of nodes (i.e., routers) and E is the set of inks. A port can be put to seep if there is no traffic on the ink, and a ine-card can be put to seep if a its ports are aseep. Let M be the set of ine-cards in the network. For a singe ine-card m M, its base power consumption is B m, its set of ports is S m, and each port S m consumes power P, then the power saving from turning of one port is P, and the power saving from turning off the entire ine-card is B m + S m P. The objective is to find a routing that maximizes the tota power saving in the network. This genera power-aware traffic engineering probem can be formuated based on the Muti-Commodity Fow (MCF) mode as foows. Pease see Tabe III for the notation used in this paper. Equation 1 computes the objective, the tota power saving in the network. Equation 2 states the fow conservation constraints. Let S m be the cardinaity of S m, then equation 3 ensures that a ine-card is put to seep ony when a its ports are aseep. Equation 4 cacuates the ink utiization. Equation 5 ensures that inks are put to seep in pairs, i.e., there is no inbound traffic nor outbound traffic. Equation 6 states that a ink can Notation Meaning S m Set of inks connected to ine-card m P Power consumption of the port connected to ink B m Base power consumption of ine-card m x 1 if ink is seeping, otherwise y m 1 if ine-card m is seeping, otherwise f s,t Traffic demand from s to t that is routed through ink H s head node T s tai node I v 1 if v is the head node of ink, otherwise O v 1 if v is the tai node of ink, otherwise D s,t Traffic demand from s to t C Capacity of ink u Utiization of ink r() Reverse ink of k Maximum number of candidate paths for each OD pair U T Threshod for the MLU Q s,t i () 1 if the ith candidate path from s to t contains ink, otherwise α s,t i Ratio of traffic demand from s to t that is routed through the ith candidate path be put to seep ony if there is no traffic on it, and when it is on, it does not carry traffic more than its capacity. Soving this probem gives which inks to be turned off, and how much traffic each remaining ink shoud carry. maximize s.t. B m y m (1) P x + E m M f s,t O i f s,t I i = E E D s,t, i = t D s,t, i = s, s, t, i V, s t (2), i s, t S m y m x (3) S m u = 1 C s,t V,s t f s,t, E (4) x = x r(), E (5) x + u 1, E (6) The binary (integer) variabes x and y m that denote the power state of ink and ine-card m make the mode a MIP probem. Generay speaking, MIP probems are NP-Hard, thus its computation time for networks with medium and arge sizes is a concern. This mode, though maximizes power saving in the network, does not consider some practica constraints. For exampe, packet deay coud be much onger than that of current shortest path routing, and inks may operate at unacceptaby high ink utiization, making them vunerabe to any traffic bursts. B. A Practica Heuristic To consider the practica constraints and reduce computation time, we refine the probem formuation as foows.

maximize s.t. f s,t P x + E m M = i<k B m y m (7) Q s,t i ()D s,t α s,t i, s, t V, E, s t (8) α s,t i = 1, s, t V, s t (9) i<k S m y m u = 1 C x (1) S m f s,t, E (11) s,t V,s t x = x r(), E (12) x + u 1, E (13) u U T, E (14) One change is the addition of the bound on maximum ink utiization in a network. Equation 14 states that MLU must be no greater than a configured threshod U T. In this paper, we use 5% as the defaut vaue of U T. Another change is the use of candidate paths instead of searching the soution in a possibe paths. The candidate paths are chosen based on the k-shortest paths; therefore each OD pair has at most k candidate paths. Equation 8 and 9 are equivaent to the fow conservation constraints under this change. It reduces overa computation time as we as adds path ength as another constraint. The genera mode introduced in the previous subsection considers a possibe paths for each OD pair, making the search space extremey arge. To reduce search space and computation time, for each OD pair, we pre-compute its set of candidate paths and ony search soutions within this set. Since the k-shortest paths are pre-computed with network topoogy as the ony input, they do not change with the traffic matrix and the computation does not add run-time overhead. Note that when k is set to be arge enough, we can actuay consider a possibe paths for each OD pair, which wi give the maxima power saving under the MLU constraint. However, the computation time increases with the vaue of k; therefore there is a tradeoff between the precision of the heuristic and the computation time. Our evauation ater wi show that a reasonaby arge k can achieve near optima resuts. Searching soutions ony within the candidate paths aso avoids very ong paths. In practice, network operators can have their own definitions of ink deays and path engths, and choose the set of candidate paths accordingy. In this paper we add up ink propagation deays to get path engths, and consider two different constraints in seecting the candidate paths. One is that any candidate path shoud not be onger than the diameter of the network, i.e., the ength of the shortest path between the farthest pair of nodes in the topoogy. The other is that between any OD pair, a candidate path s ength shoud not be greater than twice that of the shortest path. Depending on how the candidate paths are chosen, in this paper, we wi evauate three different combinations: basic: The candidate paths are the k-shortest paths. MLU bound is appied. basic+nd: The candidate paths are the k-shortest paths which aso satisfy the network diameter constraint. MLU bound is appied. basic+e2e: The candidate paths are the k-shortest paths which aso conform to the OD-pair end-to-end deay constraint. MLU bound is appied. With these changes, the GreenTE mode now has practica constraints on ink utiization and path ength, and aso can be soved within reasonabe time. C. Load Baancing In conventiona traffic engineering, oad baancing is the main objective, usuay formuated as minimizing the MLU in a network. Though it may not be a good idea to combine it with power-aware traffic engineering in the same probem formuation, we can sti do oad baancing on top of the routing resuted from power-aware traffic engineering. From soving the probem formuated in the previous subsection we can obtain the set of inks to be put to seep. Excuding paths containing these inks from the origina set of candidate paths, we get a new set of paths Q, onto which the traffic oad wi be baanced by soving the foowing probem: minimize s.t. f s,t max u (15) E = i<k Q s,t i ()D s,t α s,t i, s, t V, E, s t (16) α s,t i = 1, s, t V, s t (17) i<k u = 1 C s,t V,s t f s,t, E (18) The above formuation minimizes the MLU over inks that remain on, and soving it gives the traffic oad that each such ink shoud carry. We use basic+b, basic+nd+b and basic+e2e+b to denote the modes under different performance constraints after performing oad baancing. In summary, the GreenTE mode maximizes power saving, considers constraints on ink utiization and path ength, and aso baances oad over the inks. IV. IMPLEMENTATION ISSUES Reaizing GreenTE in operationa networks requires coordination among a routers in the network. In this section we outine such coordination and discuss its different aspects. Our basic principe in GreenTE design is to use existing protocos and mechanisms as much as possibe for the benefits of compatibiity and depoyabiity. We aso assume that networks run both OSPF (or any ink state routing protoco) and MPLS.

A. Overview As in conventiona traffic engineering, GreenTE reies on a ogicay centraized controer in the Network Operation Center (NOC) to make decisions on traffic engineering. The physica impementation of such a controer can have hardware redundancy and/or repica in different ocations for better reiabiity. The controer coects input information (i.e., network topoogy and traffic matrix) from routers, soves the GreenTE probem to get new routing configurations (i.e., which inks are up and how much traffic on each ink), and disseminates the resuts to routers. Each router wi then turn on/off some ine-cards or ports according to the GreenTE soution and set up MPLS tunnes for data forwarding if needed. As traffic demand changes over time and sometimes unpredictaby, the process described above needs to be done periodicay. The frequency of such routing adjustment depends on how often the traffic matrix changes and by how much. Adjusting routing too often wi resut in more contro overhead and more disturbance to data forwarding (e.g., packet oss and re-order), but too few wi eave the routing non-optimized for too ong as the traffic matrix may have changed significanty. In our experiments, we adjust the routing every 5-15 minutes. However, our resut shows that route seection by GreenTE is reativey stabe, i.e., most MPLS tunnes remain unchanged from one routing configuration to the next, which means that the negative impacts of routing adjustment is rather imited each time. B. Gathering Input Information for GreenTE The controer coects network topoogy and traffic matrix from OSPF s Link State Advertisements (LSAs). In OSPF, each router foods its LSAs whenever its ink state changes. Thus the controer can readiy coect a the ink state information and compie the up-to-date network topoogy. Directy measuring traffic matrix in rea-time is sti expensive in arge networks. Instead, the GreenTE controer coects ink oad information from routers and computes the traffic matrix ocay. The ink oad information is part of the Traffic Engineering Link State Advertisement (TE-LSA) defined in RFC363 [7]. As an extension to the basic OSPF LSA, TE-LSA is aso fooded in the network. TE-LSA reports a ink s maximum bandwidth and unreserved bandwidth, and the difference between them is the ink oad. A router sends out TE-LSA when there is a significant change in its bandwidth usage. Once the ink oad information is coected, the controer computes the network traffic matrix using the tomogravity method, which is ightweight, accurate, and can be done within a few seconds for arge ISP networks [8]. Both the network topoogy and ink oad information are coected by the controer passivey. The controer does not po any specific router, nor has any expicit point-to-point conversation with any individua router. A information is announced via LSAs. This design choice is compatibe with existing mechanisms, simpifies operations, and aso inherits the deivery reiabiity provided by LSA fooding. C. Distributing GreenTE Resuts With the network topoogy and traffic matrix, the controer soves the GreenTE probem to get which inks to be turned on or off, and distributes this information to routers via the Traffic Engineering Metric (TE-Metric) attribute, another extension to OSPF defined in RFC 363 [7]. The GreenTE convention is that if a ink s TE-Metric is set to be equa to its OSPF weight, the ink shoud to be turned on; if a ink s TE-Metric is set to be the maximum vaue aowed, it shoud be turned off. Note that both TE-LSA and TE-Metric messages are fooded in the network as reguar OSPF LSAs; therefore they can reach a routers as ong as the network is connected and require no separate interfaces or inks to be reserved. The fow conservation constraints in GreenTE formuation guarantee that the soution does not partition the network. To minimize packet oss during routing transitions, extra care is needed when routers are turning on/off inks. When a router has a ink to turn off, it shoud not do so immediatey, because otherwise some on-the-fy packets may be ost. Ideay it shoud wait for a the aternative paths have been set up before actuay turning off a ink. In practice a router may turn off a ink after the ink has been ide for more than a certain period of time. The network diameter can serve as a rough threshod for this purpose. When a router has a ink to wake up, it shoud turn on the ink immediatey but not transmit data onto this ink unti both ends of the ink are ready. Two routers can exchange messages to confirm that they are ready. Such messages can be MPLS signaing messages, OSPF Heo messages, or simpe messages designed specificay for this purpose. An aternative is to simpy use a timer. D. Data Forwarding under GreenTE In GreenTE, data packets are forwarded aong either OSPF paths or MPLS tunnes (i.e., Labe Switching Path, LSP). Soving the GreenTE mode gives the paths that data traffic shoud take. If a GreenTE path happens to be the shortest path according to OSPF, the traffic is simpy transmitted as native IP packets; otherwise an LSP is set up, by either the Constraint-based Routing Labe Distribution Protoco (CR- LDP) [9] or Resource Reservation Protoco-Traffic Engineering (RSVP-TE) [1], to impement the non-shortest path to carry traffic. In the case that the traffic between an OD pair takes mutipe paths in GreenTE soution, the traffic spit ratio among the mutipe paths wi aso be part of the soution. Such traffic spit is usuay supported in today s commercia routers by a hash-based mechanism [11]. Basicay at the ingress routers, regardess of whether its from MPLS or OSPF, one or mutipe next-hop interfaces wi be associated with a destination prefix in the forwarding tabe. When there are mutipe next-hop interfaces, a hashing mechanism is empoyed to determine which fow of traffic wi take which next-hop, so that traffic of the same fow wi aways take the same path. The hashing mechanism can be configured with different weights to reaize different traffic spit ratios.

The OSPF/MPLS hybrid approach has been shown in previous work (e.g., [12]) to have two main advantages. First, the number of MPLS tunnes is much ess than what woud be required in a fu-mesh configuration because the majority of the traffic actuay takes the shortest path. Second, it causes much fewer OSPF convergence than pure-ospf traffic engineering because each time routing adjustment is achieved by changing a few MPLS tunnes and the traffic spit ratio instead of changing OSPF ink weights. These advantages are confirmed in our evauations. E. Impacts on Other Protocos In conventiona networking, a ink has two states, either up or down. An up ink is abe to transmit packets whie a down ink cannot. With power-aware networking, a ink has a third state: seeping. A seeping ink is not used to transmit packets for the moment but can do so if needed. Introducing seeping inks has no or minima impacts on end-to-end protocos such as TCP and UDP since GreenTE has set up aternative paths for data deivery, but it may affect the operation of protocos that depend on ink-eve information. A typica exampe is OSPF, which uses periodic HELLO messages to detect the existence of a ink and its state. Simpy putting a ink into seep wi make OSPF beieve that the ink is down, which wi trigger LSAs and network-wide OSPF convergence process. Generay speaking there are three different approaches that a protoco can use: Expicity handing seeping inks. For exampe, in GreenTE, the information of seeping inks is fooded via TE-Metric attribute in LSAs. A router, after missing a few HELLO messages from a ink, can check whether this ink is supposed to be seeping, and if yes, it can abe this ink as seeping and hande it differenty from down inks. This is the ceanest way to support poweraware networking, but it requires changes to protoco specification or impementation. Adjusting protoco parameters. For exampe, one can use a arger interva for OSPF HELLO messages to avoid a seeping ink from being detected as a down ink. This does not require any changes to the protoco, but may discover actua ink faiures much ater than it woud with origina parameters. Waking up the ink. The ink is awaken on-demand whenever there is a packet for it. This is fuy compatibe with current networking but resuts in ess power saving. Exacty which approach to take is a tradeoff between backward compatibiity and network energy efficiency, and may evove over time as we see different soutions take pace. The impact of GreenTE on OSPF convergence time is imited. Most of the time GreenTE does not change OSPF ink weights or its routing paths; it ony adjusts the traffic spit ratio and/or MPLS tunnes. Thus GreenTE usuay does not trigger OSPF convergence. When OSPF convergence does happen, one factor in its convergence time is how quick the LSA is fooded to reach a routers. Seeping inks may make this time onger since there are ess inks for fooding. However, as we have deay bound buit in GreenTE, such propagation deay of routing messages shoud not be much onger than in non-greente networks. For exampe, under the network diameter constraint, GreenTE network maintains the same network diameter, thus it shoud take about the same time for a routing message to reach a routers in the network. Another issue with seeping inks is network robustness to ink faiures and traffic bursts. When a ink fais or a burst of traffic arrives, the network needs to find aternative paths to accommodate the traffic if the traffic being affected is of very high voume. This probem exists in power-aware networking as we as in any IP networks, athough in the former the probem can be more severe as the seeping inks are not readiy avaiabe. In this case some seeping inks need to wake up ondemand in order to hande the extra traffic demand. One of our future work is to further investigate this issue. V. EVALUATION In this section, we evauate GreenTE and show that it is abe to achieve considerabe power savings in rea networks with minor impact on network performance. A. Experiment Setup TABLE IV NETWORK TOPOLOGIES USED IN THE EVALUATION Network Usage Location Nodes Links Abiene Research US 12 3 GÉANT Research Europe 23 74 Sprint Commercia US 52 168 AT&T Commercia US 115 296 We use different network topoogies in the evauation, incuding Abiene, GÉANT and seected topoogies from Rocketfue [13] as isted in Tabe IV. These topoogies vary in size and usage. For Abiene, the router-eve topoogy (i.e., ink connectivity, weights, engths and capacities) and measured traffic matrices are avaiabe at [14]. The non-anonymized topoogy and traffic matrices of GÉANT are provided by the authors of [15]. The traffic matrices are measured every 5 minutes for Abiene and every 15 minutes for GÉANT. Whie Abiene and GÉANT are both research networks, Rocketfue provides PoP-eve topoogies of commercia ISPs. We assume that each node in the topoogy corresponds to a router. Since ink capacities and traffic matrices are not avaiabe for the Rocketfue topoogies, we assign capacities to inks using the method described in [16] and generate traffic matrices using the gravity mode [17] [18]. Given the above information, we are abe to pre-compute the candidate paths for each OD pair and sove the GreenTE mode using CPLEX [19]. From the soution of the mode, we can obtain the power saving for the network as we as the utiization for each ink. In addition, the soution aso gives which paths to use for each OD pair and how to spit traffic among these paths.

Power Saving Potentia (%) 1 8 6 4 2 basic basic+nd basic+e2e : 6: 12: 18: 24: Time Power Saving Potentia (%) 1 8 6 4 2 basic+nd,k=1 basic+e2e,k=1 : 6: 12: 18: 24: Time Power Saving Potentia (%) 1 8 6 4 2 Sprint,basic+nd,k=1 Sprint,basic+e2e,k=1 AT&T,basic+nd,k=2 AT&T,basic+e2e,k=2 2 4 6 8 1 Maximum Link Utiization under OSPF (%) Fig. 3. Power saving potentia of Abiene on Sep. 5th, 24 Fig. 4. Power saving potentia of GÉANT on May 5th, 25 Fig. 5. Power saving potentia of Sprint and AT&T In the evauation, we assume that each ine-card is connected to a singe ink; therefore a ine-card can be put to seep when there is no traffic on the ink. We make this assumption because the information of physica connections among ine-cards is not avaiabe in the data set. However, the GreenTE mode can be appied to the situation when ine-cards have mutipe ports. We reconfigure the network every 5-15 minutes based on the avaiabiity of traffic matrices. We assume that traffic matrix does not change significanty within 5-15 minutes, and this is confirmed by rea traffic data as foows. We anayze the traffic matrices of Abiene from one typica day (Sep. 4th, 24), and evauate how traffic between each OD pair changes over time. Resuts show that for more than 74% of the OD pairs, the change of traffic voume within 5 minutes is ess than 3%. To evauate the impact of GreenTE on queuing deay, we impement GreenTE in ns2 [2] for packet-eve simuations. We set up OSPF paths as we as MPLS tunnes, assign traffic spit ratio to the paths, and generate traffic based on measured traffic matrices. Specificay, we generate sef-simiar traffic for each OD pair using a mix of Pareto fows to simuate rea Internet traffic [21]. A the experiments are conducted on machines with 8 GB of RAM and a Quad-Core Inte Q965 processor (3. GHz). B. Power Savings TABLE V POWER CONSUMPTIONS OF LINE-CARDS [5] ine-card Speed (Mbps) Power (Watts) 1-Port OC3 155.52 6 8-Port OC3 1244.16 1 1-Port OC48 2488.32 14 1-Port OC192 9953.28 174 We expore the power saving potentia under GreenTE using different network topoogies and traffic matrices. We compute the power saving ratio as the tota power of seeping ine-cards over the tota power of a ine-cards in the network. As noted in Section II, ine-cards a together account for more than 4% of a router s tota power budget; therefore it is meaningfu to measure the power saving ratio of ine-cards. The power consumption of ine-cards we use in the evauation is specified in Tabe V. 1) Abiene: Figure 3 shows the power saving potentia of Abiene on Sep. 5th, 24 under different performance constraints. The power saving ratio under basic OSPF is not shown here because it is aways zero. We sti use 5% as the MLU threshod. In this experiment, we set k to be arge enough so that a paths that satisfy the deay constraints are incuded; therefore the resuts shown in the figure are actuay optima. GreenTE is abe to achieve about 27% power savings under basic and basic+nd. The two curves overap because basic+nd incudes sufficient candidate paths to achieve the maxima power saving. The power saving ratio under basic+e2e is about 2%, ower than the other two because ess number of candidate paths are considered. The power saving ratio does not change over time because the traffic voume is reativey sma throughout the day so that GreenTE is aways abe to put the maximum number of inks to seep whie conforming to the constraints. 2) GÉANT: Figure 4 presents the power saving potentia of GÉANT on May 5th, 25. We set k = 1 in this experiment to incude most of the candidate paths that conform to the deay constraints. We show how the vaue of k affects the power saving potentia in Section V-G. The resut for basic is not shown here because the probem cannot be soved within reasonabe time, and we expect basic+nd and basic+e2e to be more commony used in practice. Simiar to Abiene, GreenTE achieves more power savings under basic+nd because it is ess restrictive than basic+e2e; but the power saving ratio under basic+e2e is sti considerabe (more than 2%). 3) Rocketfue Topoogies: For arge topoogies such as Sprint or AT&T, CPLEX is unabe to get the optima soution within reasonabe time. To resove this probem, we force CPLEX to terminate after 3 seconds. The resuts, though not optima, are guaranteed to be correct as they satisfy a the constraints. We show in Section V-G that this method actuay yieds satisfactory resuts within acceptabe time imit. Figure 5 presents the power saving potentia of Sprint and AT&T under various traffic conditions. We generate one traffic matrix for each network using the gravity mode, which is then scaed to obtain different traffic oads. The figure shows that the power saving ratio decreases as the traffic oad increases; but in a normay operated network where the MLU is beow 4%, GreenTE is abe to achieve stabe power savings. Sprint

Percent of Points (%) 1 8 6 4 OSPF MCFTE 2 basic+b basic+nd+b basic+e2e+b 5 1 15 2 Maximum Link Utiization (%) Percent of Points (%) 1 8 6 4 OSPF 2 MCFTE basic+nd+b basic+e2e+b 2 4 6 8 1 Maximum Link Utiization (%) Percent of Points (%) 1 8 6 4 2 basic+nd basic+nd+b basic+e2e basic+e2e+b 1 2 3 4 5 6 Maximum Link Utiization (%) Fig. 6. CDF of MLU of Abiene on Sep. 5th, 24 Fig. 7. CDF of MLU of GÉANT on May 5th, 25 (k = 1) Fig. 8. CDF of MLU of GÉANT on May 5th, 25 (k = 1), before and after oad baancing Percent of Traffic (%) 1 8 6 4 2 OSPF basic+nd+b basic+e2e+b 1 2 3 4 5 6 7 8 Packet Deay (ms) Percent of Traffic (%) 1 8 6 4 2 OSPF basic+nd+b basic+e2e+b 2 4 6 8 1 12 14 Packet Deay (ms) Queue Length (Bytes) 1 8 6 4 2 median,ospf 99th percentie,ospf median,basic+e2e+b 99th percentie,basic+e2e+b 1 2 3 4 5 6 Maximum Link Utiization under OSPF (%) Fig. 9. CDF of Packet Deay for GÉANT (k = 1) Fig. 1. CDF of Packet Deay for Sprint (k = 1) Fig. 11. Queue ength for Abiene under different traffic conditions exhibits higher power saving potentia because it has reativey higher ink redundancy than AT&T (3.23 vs. 2.57 inks per node). C. Link Utiization Intuitivey, GreenTE woud affect the utiization of inks as fewer inks are used to carry traffic. In this subsection, we evauate the impact of GreenTE on ink utiization. Specificay, we show how the maximum ink utiization of the network is affected by different routing mechanisms. 1) Abiene: We draw the CDF of MLU of Abiene throughout one day in Figure 6. Since the traffic oad is ight, the MLU is aways under 2% for a routing mechanisms. MCFTE achieves optima oad baancing and thus acts as the ower bound for the MLU [12] [22]. basic+b is very cose to MCFTE because it has sufficient candidate paths for oad baancing. The figure aso shows that basic+nd+b and basic+e2e+b are abe to achieve simiar MLU as OSPF when the network is ighty oaded. 2) GÉANT: The CDF of MLU of GÉANT throughout one day is shown in Figure 7. The MLU as high as about 9% under OSPF is caused by a singe hot-spot ink, and GreenTE is abe to shift traffic away from that ink to avoid congestion. GreenTE achieves simiar MLU as MCFTE whie obtaining considerabe power savings. Figure 8 shows the comparison of MLU before and after the oad baancing optimization. Before oad baancing is performed, more than 4% of inks have utiization of 5%. This is because the sover ony focuses on putting inks to seep as ong as the MLU is no greater than 5%. The oad baancing optimization effectivey reduces the MLU of the network. D. Deay Since part of the traffic is routed through non-shortest paths, GreenTE may aso increase the packet deay. In this subsection, we evauate propagation deay which dominates packet deay when the network is not congested. Queuing deay wi be considered in the next subsection. Figure 9 and 1 show the CDF of packet deay for GÉANT and Sprint. The resuts for other topoogies are simiar and thus not shown here. For GÉANT, we choose a traffic matrix whose MLU under OSPF is about 5%; for Sprint we scae the generated traffic matrix so that the MLU under OSPF is about 5%. Since ink weight refect ink ength in a the experiments, OSPF actuay gives a ower bound for packet deay. The figures show that GreenTE is abe to effectivey bound the packet deay within a desired eve. For GÉANT, basic+e2e+b is coser to OSPF than basic+nd+b for most of the traffic, whie basic+nd+b successfuy bounds the worst case to be no greater than the ongest deay in OSPF. For Sprint, the three curves are coser to each other because the topoogy is arger; hence there are more paths for each OD pair that have deay cose to the shortest path. E. Queue Length As GreenTE uses fewer number of inks to carry the same amount of traffic, queuing deay experienced by packets is aso

5 4 3 2 1 od new 9: 9:15 9:3 9:45 1: 4 35 3 25 2 15 1 5 od new 1: 11: 12: 13: Power Saving Potentia (%) 6 5 4 3 2 1 basic+nd,k=5 basic+nd,k=2 basic+nd,k=1 : 6: 12: 18: 24: Time Fig. 12. Number of MPLS tunnes for Abiene under basic+nd+b Fig. 13. Number of MPLS tunnes for GÉANT under basic+nd+b Fig. 14. Power saving potentia of GÉANT with different k vaues ikey to be affected. We use ns2 to evauate the queuing deay under OSPF and GreenTE. We choose the same traffic matrix from Abiene as in Section V-D and scae it to produce different traffic oads. We run each experiment for 5 minutes, and coect the average queue ength of each ink per second. Figure 11 shows the median and 99th percentie of queue engths for Abiene under different traffic conditions. The queue engths under GreenTE and OSPF are very cose when the MLU under OSPF is ower than 3%. As the network becomes more heaviy oaded, the queue ength under GreenTE becomes obviousy arger than that under OSPF. However, since the absoute vaues of queue engths are actuay very sma, the impact of GreenTE on queuing deay is minor. F. Routing Stabiity Transition from one route configuration to another may cause probems such as packet oss and re-ordering. In this subsection, we show that route seection by GreenTE is reativey stabe so that negative impacts caused by routing adjustments can be imited. We choose traffic matrices from peak hours during the day (9:-1: for Abiene and 1:-13: for GÉANT). Figure 12 and 13 show that on average more than 7% of MPLS tunnes stay unchanged during routing transitions under basic+nd+b. This is because traffic matrix does not change significanty between two contiguous routing configurations. The figures aso show that the number of MPLS tunnes needed in the network is much ess than that in a fu mesh. The same concusion aso appies to other topoogies and constraints. G. Precision of Heuristics TABLE VI POWER SAVING POTENTIAL OF AT&T UNDER basic+e2e WITH 21% MLU UNDER OSPF k vaue Computation Time Status Power Saving Potentia 5 65s Optima 11.9% 1 5747s Optima 17.54% 2 1892s Optima 19.79% 2 3s Non-optima 18.99% Figure 14 shows that the power saving potentia grows as the vaue of k increases. However, increasing k aso increases the computation time. When k is arge enough (2 in this exampe), increasing k ony improves the power saving potentia by a negigibe amount. Therefore, GreenTE is abe to achieve near optima power savings as ong as k is reasonaby arge. As the computation time is too ong for arge topoogies such as Sprint and AT&T, we force CPLEX to stop after 3 seconds. Tabe VI shows that computation time increases dramaticay as the vaue of k grows. However, when k = 2, we can obtain about 96% of the optima power saving if we imit the computation time to be 3 seconds. VI. RELATED WORK Gupta et a. identify the power saving probem in the Internet, and propose seeping as the approach to conserve energy [2]. Specificay, they suggest two options - uncoordinated seeping which works at ink eve and coordinated seeping which operates at network eve. In their foow-up works [23] [24] [25], the authors study uncoordinated seeping in Loca Area Networks (LANs). This approach works effectivey in LANs because of its specific traffic patterns; however, it might not be appicabe to backbone networks where inter-packet time is too short for the inks to seep. In [3], Nedevschi et a. propose the buffer-and-burst approach which shapes traffic into sma bursts to create greater opportunities for network components to seep. The same work aso brings up the idea of rate-adaptation, which adjusts operating rates of inks according to the traffic condition. This work is aso focused on ink eve soutions. Chabarek et a. expore power-awareness in the design of networks and routing protocos in wire-ine networks [26]. The authors revea the significant power saving potentia in operationa networks by incuding power-awareness, but they do not come up with a specific power-aware routing design. In this paper, we propose GreenTE to achieve power-aware routing through traffic engineering. Heer et a. propose EasticTree [27], which optimizes the energy consumption of Data Center Networks by turning off unnecessary inks and switches during off-peak hours. Eastic- Tree aso modes the probem based on the MCF mode, but is focused on Fat-Tree or simiar tree-based topoogies. EasticTree takes ink utiization and redundancy into consideration when cacuating the minimum-power network subset, and is impemented using OpenFow.

Vasić et a. propose EATe [28] to reduce Internet power consumption through traffic engineering. EATe considers seeping of inks and routers as we as ink rate adaption. EATe achieves its routing decisions in a distributed fashion via router coordination and thus requires routers to be abe to send announcement and feedback to each other. In contrast, GreenTE is mosty compatibe with current operation practice. Internet traffic engineering is a widey studied topic. Fortz and Thorup first propose the idea of IGP ink weight optimization for the purpose of traffic engineering [29] [3]. However, frequent changes to ink weights woud cause probems such as network-wide routing convergence and traffic shift. MATE [31] and TeXCP [16] perform traffic engineering by spitting traffic among mutipe MPLS paths. MPLS-based traffic engineering can achieve optima routing, but does not scae we as the size of network grows. In [12], Zhang et a. propose MCFTE, which performs traffic engineering through hybrid OSPF/MPLS routing. MCFTE achieves optima routing with ony a sma number of MPLS tunnes, and thus aeviates the scaabiity probem. Other works on hybrid routing incude [32] [33] [34]. VII. CONCLUSION High path redundancy and ow ink utiization in today s arge networks provide unique opportunities for power-aware traffic engineering. By switching traffic onto fewer number of paths, one can free some inks from carrying data traffic and put them to seep for energy conservation. The GreenTE mode maximizes the number of inks that can be put to seep under the constraints of ink utiization and path ength, and aso baances the network oad afterwards. Evauations based on rea network topoogies and traffic matrices show that GreenTE is abe to achieve considerabe power savings with minor impacts on the network performance. Our future work incudes investigating the impication of mutipe ports per inecard on power savings, handing ink faiures and sudden traffic bursts, taking advantages of recent advances in hardware energy management mechanisms. REFERENCES [1] G. Epps, D. Tsiang, and T. Boures, System Power Chaenges, 26. [Onine]. Avaiabe: http://www.sidefinder.net/c/cisco routing research/ seminar august 29/156216 [2] M. Gupta and S. Singh, Greening of the Internet, in Proceedings of ACM SIGCOMM, 23. [3] S. Nedevschi, L. Popa, G. Iannaccone, S. Ratnasamy, and D. Wethera, Reducing Network Energy Consumption via Seeping and Rate- Adaptation, in Proceedings of USENIX NSDI, 28. [4] J. Guichard, F. L. Faucheur, and J.-P. Vasseur, Definitive MPLS Network Designs. Cisco Press, 25. [5] Power Management for the Cisco 12 Series Router. [Onine]. Avaiabe: http://www.cisco.com/en/us/docs/ios/12 s/feature/ guide/12spower.htm [6] P. Mahadevan, P. Sharma, S. Banerjee, and P. Ranganathan, A Power Benchmarking Framework for Network Devices, in Proceedings of the 8th Internationa IFIP-TC 6 Networking Conference, 29. [7] D. Katz, K. Kompea, and D. Yeung, RFC 363: Traffic Engineering (TE) Extensions to OSPF Version 2, Sept 23. [Onine]. Avaiabe: http://www.ietf.org/rfc/rfc363.txt [8] Y. Zhang, M. Roughan, N. Duffied, and A. Greenberg, Fast Accurate Computation of Large-Scae IP Traffic Matrices from Link Loads, in Proceedings of ACM SIGMETRICS, 23. [9] B. Jamoussi et a., RFC 3212: Constraint-based LSP Setup Using LDP, Jan 22. [Onine]. Avaiabe: http://www.ietf.org/rfc/rfc3212.txt [1] D. Awduche et a., RFC 329: RSVP-TE: Extensions to RSVP for LSP tunnes, Dec 21. [Onine]. Avaiabe: http://www.ietf.org/rfc/ rfc329.txt [11] Using Hash-Key Load Baancing for LSP Traffic. [Onine]. Avaiabe: https://www.juniper.net/techpubs/en US/junos1./information-products/ topic-coections/nog-mps-frr/mps-oad-baancing-hash-key.htm [12] M. Zhang, B. Liu, and B. Zhang, Muti-Commodity Fow Traffic Engineering with Hybrid MPLS/OSPF Routing, in Proceedings of IEEE GLOBECOM, 29. [13] N. Spring, R. Mahajan, D. Wethera, and T. Anderson, Measuring ISP Topoogies with Rocketfue, IEEE/ACM Transactions on Networking, vo. 12, no. 1, pp. 2 16, February 24. [14] Yin Zhang s Abiene TM. [Onine]. Avaiabe: http://www.cs.utexas. edu/ yzhang/research/abienetm/ [15] S. Uhig, B. Quoitin, J. Lepropre, and S. Baon, Providing Pubic Intradomain Traffic Matrices to the Research Community, ACM SIGCOMM Computer Communication Review, vo. 36, no. 1, pp. 83 86, January 26. [16] S. Kandua, D. Katabi, B. Davie, and A. Charny, Waking the Tightrope: Responsive Yet Stabe Traffic Engineering, in Proceedings of ACM SIGCOMM, 25. [17] M. Roughan, A. Greenberg, C. Kamanek, M. Rumsewicz, J. Yates, and Y. Zhang, Experience in Measuring Backbone Traffic Variabiity: Modes, Metrics, Measurements and Meaning, in in ACM SIGCOMM Internet Measurement Workshop, 22. [18] D. Appegate and E. Cohen, Making Intra-domain Routing Robust to Changing and Uncertain Traffic Demands: Understanding Fundamenta Tradeoffs, in Proceedings of ACM SIGCOMM, 23. [19] CPLEX. [Onine]. Avaiabe: http://www.iog.com/products/cpex/ [2] Network Simuator ns-2. [Onine]. Avaiabe: http://www.isi.edu/ nsnam/ns/ [21] M. Yukse, B. Sikdar, K. S. Vastoa, and B. Szymanski, Workoad Generation for NS Simuations of Wide Area Networks and the Internet, in Proceedings of Communication Networks and Distributed Systems Modeing and Simuation Conference, 2. [22] M. Roughan, M. Thorup, and Y. Zhang, Traffic Engineering with Estimated Traffic Matrices, in Proceedings of ACM IMC, 23. [23] M. Gupta, S. Grover, and S. Singh, A Feasibiity Study for Power Management in LAN Switches, in Proceedings of IEEE ICNP, 24. [24] M. Gupta and S. Singh, Dynamic Ethernet Link Shutdown for Energy Conservation on Ethernet Links, in Proceedings of IEEE ICC, 27. [25], Using Low-Power Modes for Energy Conservation in Ethernet LANs, in Proceedings of IEEE INFOCOM, 27. [26] J. Chabarek, J. Sommers, P. Barford, C. Estan, D. Tsiang, and S. Wright, Power Awareness in Network Design and Routing, in Proceedings of IEEE INFOCOM, 28. [27] B. Heer, S. Seetharaman, P. Mahadevan, Y. Yiakoumis, P. Sharma, S. Banerjee, and N. McKeown, EasticTree: Saving Energy in Data Center Networks, in Proceedings of USENIX NSDI, 21. [28] N. Vasić and D. Kostić, Energy-Aware Traffic Engineering, in Proceedings of the 1st Internationa Conference on Energy-Efficient Computing and Networking, 21. [29] B. Fortz and M. Thorup, Internet Traffic Engineering by Optimizing OSPF Weights, in Proceedings of IEEE INFOCOM, 2. [3], Optimizing OSPF/IS-IS Weights in a Changing Word, IEEE Journa On Seected Areas In Communications, vo. 2, no. 4, pp. 756 767, May 22. [31] A. Ewaid, C. Jin, S. Low, and I. Widjaja, MATE: MPLS Adaptive Traffic Engineering, in Proceedings of IEEE INFOCOM, 21. [32] E. Muyana and U. Kiat, An Offine Hybrid IGP/MPLS Traffic Engineering Approach under LSP Constraints, in Proceedings of the 1st Internationa Network Optimization Conference, INOC, 23. [33] H. Pham and B. Lavery, Hybrid Routing for Scaabe IP/MPLS Traffic Engineering, in Proceedings of IEEE ICC, 23. [34] F. Skivée, S. Baon, and G. Leduc, A Scaabe Heuristic for Hybrid IGP/MPLS Traffic Engineering - Case Study on an Operationa Network, in Proceedings of IEEE ICON, 26.