Cooperative Content Distribution and Traffic Engineering in an ISP Network

Transcription

1 Cooperative Content Distribution and Traffic Engineering in an ISP Network Wenjie Jiang, Rui Zhang-Shen, Jennifer Rexford, Mung Chiang Department of Computer Science, and Department of Eectrica Engineering Princeton University {wenjiej, rz, jrex, ABSTRACT Traditionay, Internet Service Providers (ISPs) make profit by providing Internet connectivity, whie content providers (CPs) pay the more ucrative roe of deivering content to users. As network connectivity is increasingy a commodity, ISPs have a strong incentive to offer content to their subscribers by depoying their own content distribution infrastructure. Providing content services in an ISP network presents new opportunities for coordination between traffic engineering (to seect efficient routes for the traffic) and server seection (to match servers with subscribers). In this work, we deveop a mathematica framework that considers three modes with an increasing amount of cooperation between the ISP and the CP. We show that separating server seection and traffic engineering eads to sub-optima equiibria, even when the CP is given accurate and timey information about the ISP s network in a partia cooperation. More surprisingy, extra visibiity may resut in a ess efficient outcome and such performance degradation can be unbounded. Leveraging ideas from cooperative game theory, we propose an architecture based on the concept of Nash bargaining soution. Simuations on reaistic backbone topoogies are performed to quantify the performance differences among the three modes. Our resuts appy both when a network provider attempts to provide content, and when separate ISP and CP entities wish to cooperate. This study is a step toward a systematic understanding of the interactions between those who provide and operate networks and those who generate and distribute content. Categories and Subject Descriptors C.4 [Performance of Systems]: [Performance attributes] Genera Terms Design, Economics, Performance 1. INTRODUCTION Internet Service Providers (ISPs) and content providers (CPs) are traditionay independent entities. ISPs ony provide connectivity, Permission to make digita or hard copies of a or part of this work for persona or cassroom use is granted without fee provided that copies are not made or distributed for profit or commercia advantage and that copies bear this notice and the fu citation on the first page. To copy otherwise, to repubish, to post on servers or to redistribute to ists, requires prior specific permission and/or a fee. SIGMETRICS/Performance 9, June 15 19, 29, Seatte, WA, USA. Copyright 29 ACM /9/6...$5.. or the pipes to transport content. As in most transportation businesses, connectivity and bandwidth are becoming commodities and ISPs find their profit margin shrinking [1]. At the same time, content providers generate revenue by utiizing existing connectivity to deiver content to ISPs customers. This motivates ISPs to host and distribute content to their customers. Content can be enterpriseoriented, ike web-based services, or residentia-based, ike tripe pay as in AT&T s U-Verse [2] and Verizon FiOS [3] depoyments. When ISPs and CPs operate independenty, they optimize their performance without much cooperation, even though they infuence each other indirecty. When ISPs depoy content services or seek cooperation with CP, they face the question of how much can be gained from such cooperation and what kind of cooperation shoud be pursued. A traditiona service provider s primary roe is to depoy infrastructure, manage connectivity, and baance traffic oad inside its network. In particuar, an ISP soves the traffic engineering (TE) probem, i.e., adjusting the routing configuration to the prevaiing traffic. The goa of TE is to ensure efficient routing to minimize congestion, so that users experience ow packet oss, high throughput, and ow atency, and that the network can gracefuy absorb fash crowds. To offer its own content service, an ISP repicates content over a number of strategicay-paced servers and directs requests to different servers. The CP, whether as a separate business entity or as a new part of an ISP, soves the server seection (SS) probem, i.e., determining which servers shoud deiver content to each end user. The goa of SS is to meet user demand, minimize network atency to reduce user waiting time, and baance server oad to increase throughput. To offer both network connectivity and content deivery, an ISP is faced with couped TE and SS probems, as shown in Figure 1. TE and SS interact because TE affects the routes that carry the CP s traffic, and SS affects the offered oad seen by the network. Actuay, the degrees of freedom are aso the mirror-image of each other: the ISP contros routing matrix, which is the constant parameter in the SS probem, whie the CP contros traffic matrix, which is the constant parameter in the TE probem. In this paper, we study severa approaches an ISP coud take in managing traffic engineering and server seection, ranging from running the two systems independenty to designing a joint system. We refer to CP as the part of the system that manages server seection, whether it is performed directy by the ISP or by a separate company that cooperates with the ISP. This study aows us to expore a migration path from the status-quo to different modes of synergistic traffic management. In particuar, we consider three scenarios with increasing amounts of cooperation between traffic engineering and server seection: 1

2 Optimaity Gap Information Exchange Fairness Architectura Change Mode I Large, not Pareto-optima No exchange No Current practice Measurement ony Mode II Improved, not Pareto-optima Topoogy, routes, capacity No Minor CP changes Socia-optima in specia case Background traffic eve Better SS agorithm More info. may hurt the CP Mode III Pareto-optima Topoogy Yes Cean-sate design 5-3% performance improvement Link prices Incrementay depoyabe CP given more contro Tabe 1: Summary of resuts and engineering impications. TE: Minimize Congestion Non CP Traffic Routes CP Traffic SS: Minimize Deay Figure 1: The interaction between traffic engineering (TE) and server seection (SS). Mode I: no cooperation (current practice). Mode II: improved visibiity (sharing information). Mode III: a joint design (sharing contro). Mode I. Content services coud be provided by a CDN that runs independenty on the ISP network. However, the CP has imited visibiity into the underying network topoogy and routing, and therefore has imited abiity to predict user performance in a timey and accurate manner. We mode a scenario where the CP measures the end-to-end atency of the network and greediy assigns each user to the servers with the owest atency to the user, a strategy some CPs empoy today [4]. We ca this SS with e2e info. In addition, TE assumes the offered traffic is unaffected by its routing decisions, despite the fact that routing changes can affect path atencies and therefore the CP s traffic. When the TE probem and the SS probem are soved separatey, their interaction can be modeed as a game in which they take turns to optimize their own networks and sette in a Nash equiibrium, which may not be Pareto optima. Not surprisingy, performing TE and SS independenty is often sub-optima because (i) server seection is based on incompete (and perhaps inaccurate) information about network conditions and (ii) the two systems, acting aone, may miss opportunities for a joint seection of servers and routes. Modes II and III capture these two issues, aowing us to understand which factor is more important in practice. Mode II. Greater visibiity into network conditions shoud enabe the CP to make better decisions. There are, in genera, four types of information that coud be shared: (i) physica topoogy information [5, 6, 7], (ii) ogica connectivity information, e.g., routing in the ISP network, (iii) dynamic properties of inks, e.g., OSPF ink weights, background traffic, and congestion eve, and (iv) dynamic properties of nodes, e.g., bandwidth and processing power that can be shared. Our work focuses on a combination of these types of information, i.e., (i)-(iii), so that the CP is abe to sove the SS probem more efficienty, i.e., to find the optima server seection. Sharing information requires minima extensions to existing soutions for TE and SS, making it amenabe to incrementa depoyment. Simiar to the resuts in the parae work [8], we observe and prove that TE and SS separatey optimizing over their own variabes is abe to converge to a goba optima soution, when the two systems share the same objectives with the absence of background traffic. However, when the two systems have different or even conficting performance objectives (e.g., SS minimizes endto-end atency and TE minimizes congestion), the equiibrium is not optima. In addition, we find that mode II sometimes performs worse than mode I that is, extra visibiity into network conditions sometimes eads to a ess efficient outcome and the CP s atency degradation can be unbounded. The facts that both Mode I and Mode II in genera do not achieve optimaity, and that extra information (Mode II) sometimes hurts the performance, motivate us to consider a cean-sate joint design for seecting servers and routes next. Mode III. A joint design shoud achieve Pareto optimaity for TE and SS. In particuar, our joint design s objective function gives rise to Nash Bargaining Soution [9]. The soution not ony guarantees efficiency, but aso fairness between synergistic or even conficting objectives of two payers. It is a point on the Pareto optima curve where both TE and SS have better performance compared to the Nash equiibrium. We then appy the optimization decomposition technique [1] so that the joint design can be impemented in a distributed fashion with a imited amount of information exchange. The anaytica and numerica evauation of these three modes aows us to gain insights for designing a cooperative TE and SS system, summarized in Tabe 1. The conventiona approach of Mode I requires minimum information passing, but suffers from sub-optimaity and unfairness. Mode II requires ony minor changes to the CP s server seection agorithm, but the resut is sti not Pareto optima and performance is not guaranteed to improve, even possiby degrading in some cases. Mode III ensures optimaity and fairness through a distributed protoco, requires a moderate increase in information exchange, and is incrementay depoyabe. Our resuts show that etting CP have some contro over network routing is the key to effective TE and SS cooperation. We perform numerica simuations on reaistic ISP topoogies, which aow us to observe the performance gains and oss over a wide range of traffic conditions. The joint design shows significant improvement for both the ISP and the CP. The simuation resuts further revea the impact of topoogies on the efficiencies and fairness of the three system modes. Our resuts appy both when a network provider attempts to provide content, and when separate ISP and CP entities wish to cooperate. For instance, an ISP paying both roes woud find the optimaity anaysis usefu such that a ow efficiency operating region can be avoided. And cooperative ISP and CP woud appreciate the 2

3 Notation Description G : Network graph G = (V,E). V set of nodes, E set of inks S : S V, the set of CP servers T : T V, the set of users C : Capacity of ink : Proportion of fow i j traversing ink R : The routing matrix R : { }, TE variabe R bg Background routing matrix R : { } (i, j)/ S T X : Traffic matrix of a communication pairs X = {x i j } (i, j) V V : Traffic rate from server s to user t X cp : X cp = { } (s,t) S T, SS variabe M t : User t s demand rate for content B s : Service capacity of server s : The amount of traffic for (s,t) pair on ink ˆX cp : ˆX cp = { } (s,t) S T, the generaized SS variabe f cp : CP s traffic on ink f bg : Background traffic on ink f : f = f cp + f bg, tota traffic on ink. f = { f } E D p : Deay of path p D : Deay of ink g( ) : Cost function used in ISP traffic engineering h( ) : Cost function used in CP server seection Tabe 2: Summary of key notation. distributed impementation of Nash bargaining soution that aows for an incrementa depoyment. The rest of the paper is organized as foows. Section 2 presents a standard mode for traffic engineering. Section 3 presents our two modes for server seection, when given minima information (i.e., Mode I) and more information (i.e., Mode II) about the underying network. Section 4 studies the interaction between TE and SS as a game and shows that they reach a Nash equiibrium. Section 5 anayzes the efficiency oss of Mode I and Mode II in genera. We show that the Nash equiibria achieved in both modes are not Pareto optima. In particuar, we show that more information is not aways hepfu. Section 6 discusses how to jointy optimize TE and SS by impementing a Nash bargaining soution. We propose an agorithm that aows practica and incrementa impementation. We perform arge-scae numerica simuations on reaistic ISP topoogies in Section 7. Finay, Section 8 presents reated work, and Section 9 concudes the paper and discusses our future work. 2. TRAFFIC ENGINEERING (TE) MODEL In this section, we describe the network mode and formuate the optimization probem that the standard TE mode soves. We aso start introducing the notation used in this paper, which is summarized in Tabe 2. Consider a network represented by graph G = (V,E), where V denotes the set of nodes and E denotes the set of directed physica inks. A node can be a router, a host, or a server. Let x i j denote the rate of fow (i, j), from node i to node j, where i, j V. Fows are carried on end-to-end paths consisting of some inks. One way of modeing routing is W = {w p }, i.e., w p = 1 if ink is on path p, and otherwise. We do not imit the number of paths so W can incude a possibe paths, but in practice it is often pruned to incude ony paths that actuay carry traffic. The capacity of a ink E is C >. Given the traffic demand, traffic engineering changes routing to minimize network congestion. In practice, network operators contro routing either by changing OSPF ink weights [11] or by estabishing MPLS abe-switched paths [12]. In this paper we use the muti-commodity fow soution to route traffic, because a) it is optima, i.e., it gives the routing with minimum congestion cost, and b) it can be reaized by routing protocos that use MPLS tunneing, or as recenty shown, in a distributed fashion by a new ink-state routing protoco PEFT [13]. Let [, 1] denote the proportion of traffic of fow (i, j) that traverses ink. To reaize the muticommodity fow soution, the network spits each fow over a number of paths. Let R = { } be the routing matrix. Let f denote the tota traffic traversing ink, and we have f = (i, j) x i j. Now traffic engineering can be formuated as the foowing optimization probem: TE(R X): minimize T E = g ( f ) (1) subject to f = x i j C, (i, j) : In(v) : Out(v) = I v= j, (i, j), v V \{i} variabes 1, (i, j), where g ( ) represents a ink s congestion cost as a function of the oad, I v= j is an indicator function which equas 1 if v = j and otherwise, In(v) denotes the set of incoming inks to node v, and Out(v) denotes the set of outgoing inks from node v. In this mode, TE does not differentiate between the CP s traffic and background traffic. In fact, TE assumes a constant traffic matrix X, i.e., the offered oad between each pair of nodes, which can either be a point-to-point background traffic fow, or a fow from a CP s server to a user. As we wi see ater, this common assumption is undermined when the CP performs dynamic server seection. We ony consider the case where cost function g ( ) is a convex, continuous, and non-decreasing function of f. By using such an objective, TE penaizes high ink utiization and baances oad inside the network. We wi discuss ater the anaytica form of g ( ) which the ISP may use in practice. 3. SERVER SELECTION (SS) MODELS Whie traffic engineering usuay assumes that traffic matrix is point-to-point and constant, both assumptions are vioated when some or a of the traffic is generated by the CP. A CP usuay has many servers that offer the same content, and the servers seected for each user depend on the network conditions. In this section, we present two nove CP modes which correspond to modes I and II introduced in Section 1. The first one modes the current CP operation, where the CP reies on end-to-end measurement of the network condition in order to make server seection decisions; the second one modes the situation when the CP obtains enough information from the ISP to cacuate the effect of its actions. 3.1 Server Seection Probem The CP soves the server seection probem to optimize the perceived performance of a of its users. We first introduce the notation used in modeing server seection. In the ISP s network, et S V denote the set of CP s servers, which are strategicay paced at different ocations in the network. For simpicity we assume that a content is dupicated at a servers, and our resuts can be extended to the genera case. Let T V denote the set of users who 3

4 request content from the servers. A user t T has a demand for content at rate M t, which we assume to be constant during the time a CP optimizes its server section. We aow a user to simutaneousy downoad content from mutipe servers, because node t can be viewed as an edge router in the ISP s network that aggregates the traffic of many endhosts, which may be served by different servers. To differentiate the CP s traffic from background traffic, we denote as the traffic rate from server s to user t. To satisfy the traffic demand, we need = M t. s S In addition, the tota amount of traffic aggregated at a server s is imited by its service capacity B s, i.e., B s. t T We denote X cp = { } s S,t T as the CP s decision variabe. One of the goas in server seection is to optimize the overa performance of the CP s customers. We use an additive ink cost for the CP based on atency modes, i.e., each ink has a cost, and the end-to-end path cost is the sum of the ink costs aong the way. As an exampe, suppose the content is deay-sensitive (e.g., IPTV), and the CP woud ike to minimize the average or tota end-to-end deay of a its users. Let D p denote the end-to-end atency of a path p, and D ( f ) denote the atency of ink, modeed as a convex, non-decreasing, and continuous function of the amount of fow f on the ink. By definition, D p = p D ( f ). Then the overa atency experienced by a CP s users is SS = p D p( f) (s,t) p P(s,t) = p D ( f ) (s,t) p P(s,t) p = D ( f ) p (s,t) p P(s,t): p = f cp D ( f ) where P(s,t) is the set of paths serving fow (s,t) and p is the amount of fow (s,t) traversing path p P(s,t). Let h ( ) represent the cost of ink, which we assume is convex, non-decreasing, and continuous. In this exampe, h ( f cp, f )= f cp D ( f ). Thus, the ink cost h ( ) is a function of the CP s tota traffic f cp on the ink, as we as the ink s tota traffic f, which aso incudes background traffic. Expression (2) provides a simpe way to cacuate the tota userexperienced end-to-end deay simpy sum over a the inks, but it requires the knowedge of the oad on each ink, which is possibe ony in Mode II. Without such knowedge (Mode I), the CP can rey ony on end-to-end measurement of deay. 3.2 Server Seection with E2E Info: Mode I In today s Internet architecture, a CP does not have access to an ISP s network information, such as topoogy, routing, ink capacity, or background traffic. Therefore a CP reies on measured or inferred information to optimize its performance. To minimize its users atencies, for instance, a CP can assign each user to servers with the owest (measured) end-to-end atency to the user. In practice, content distribution networks ike Akamai s server seection agorithm is based on this principe [4]. We ca it SS with e2e info and use it as our first mode. CP monitors the atencies from a servers to a users, and makes server seection decisions to minimize users tota deay. Since the (2) demand of a user can be arbitrariy divided among the servers, we can think of the CP as greediy assigning each infinitesima demand to the best server. The pacement of this traffic may change the path atency, which is monitored by the CP. Thus, at the equiibrium, the servers which send (nonzero) traffic to a user shoud have the same end-to-end atency to the user, because otherwise the server with ower atency wi be assigned more demand, causing its atency to increase, and the servers not sending traffic to a user shoud have higher atency than those that serve the user. This is sometimes caed the Wardrop equiibrium [14]. The SS mode with e2e info is very simiar to sefish routing [15, 16], where each fow tries to minimize its average atency over mutipe paths without coordinating with other fows. It is known that the equiibrium point in sefish routing can be viewed as the soution to a goba convex optimization probem [15]. Therefore, SS with e2e info has a unique equiibrium point under mid assumptions. Athough the equiibrium point is we-defined and is the soution to a convex optimization probem, in genera it is hard to compute the soution anayticay. Thus we everage the idea of Q- earning [17] to impement a distributed iterative agorithm to find the equiibrium of SS with e2e info. The agorithm is guaranteed to converge even under dynamic network environments with cross traffic and ink faiures, and hence can be used in practice by the CPs. The detaied description and impementation can be found in [18]. As we wi show, SS with e2e info is not optima. We use it as a baseine for how we a CP can do with ony the end-to-end atency measurements. 3.3 Server Seection with Improved Visibiity: Mode II We now describe how a CP can optimize server seection given compete visibiity into the underying network, but not into the ISP objective. That is, this is the best the CP can do without changing the routing in the network. We aso present an optimization formuation that aows us to anayticay study its performance. Suppose that content providers are abe to either obtain information on network conditions directy from the ISP, or infer it by its measurement infrastructure. In the best case, the CP is abe to obtain the compete information about the network, i.e., routing decision and ink atency. This situation is characterized by probem (3). To optimize the overa user experience, the CP soves the foowing cost minimization probem: SS(X cp R): minimize subject to variabes SS = h ( f cp, f ) (3) f cp = r st, (s,t) f = f cp + f bg C, = M t, t s S B s, s t T, (s,t) where we denote f bg = (i, j) (s,t) x i j as the non-cp traffic on ink, which is a parameter to the optimization probem. If the cost function h ( ) is increasing and convex on the variabe f cp, one can verify that (3) is a convex optimization probem, hence has a unique goba optima vaue. In the remainder of this paper, we ignore the server capacity constraint by assuming sufficienty arge server capacities, since we are more interested in the impact 4

5 of SS on the network than the decision of oad baancing among congested servers. SS with improved visibiity (3) is amenabe to an efficient impementation. The probem can either be soved centray, e.g., at the CP s centra coordinator, or via a distributed agorithm simiar to that used for Mode I. We sove (3) centray in our simuations, since we are more interested in the performance improvement brought by compete information than any particuar agorithm for impementing it. 4. ANALYZING TE-SS INTERACTION In this section, we study the interaction between the ISP and the CP when they operate independenty without coordination in both Mode I and Mode II, using a game-theoretic mode. The game formuation aows us to anayze the stabiity condition, i.e., we show that aternating TE and SS optimizations wi reach an equiibrium point. In addition, we find that when the ISP and the CP optimize the same system objective, their interaction achieves goba optimaity under Mode II. Resuts in this section are aso found in a parae work [8]. 4.1 TE-SS Game and Nash Equiibrium We start with the formuation of a two-payer non-cooperative Nash game that characterizes the TE-SS interaction. Definition 1. The TE-SS game consists of a tupe [N, A,U]. The payer set N = {isp,cp}. The action set A isp = {R} and A cp = {X cp }, where the feasibe set of R and X cp are defined by the constraints in (1) and (3) respectivey. The utiity functions are U isp = T E and U cp = SS. Figure 1 shows the interaction between SS and TE. In both Mode I and Mode II, the ISP pays the best response strategy, i.e., the ISP aways optimizes (3) given the CP s strategy X cp. Simiary, the CP pays the best response strategy in Mode II when given fu information. However, the CP s strategy in Mode I is not the best response, since it does not optimize (3) due to the ack of network visibiity. Indeed, the utiity the CP impicity optimizes in SS with e2e info is [15] f U cp = D (u)du E This ater heps us understand the stabiity conditions of the game. Consider a particuar game procedure in which the ISP and the CP take turns to optimize their own objectives by varying their own decision variabes, treating that of the other payer as constant. Specificay, in the (k+ 1)-th iteration, we have R (k+1) = argmin R T E(X (k) cp ) X cp (k+1) = argmin SS(R (k+1) ) X cp Note that the two optimization probems may be soved on different timescaes. The ISP runs traffic engineering at the timescae of hours, athough it coud run on a much smaer timescae. Depending on the CP s design choices, server seection is optimized a few times a day, or at a smaer timescae ike seconds or minutes of a typica content transfer duration. We assume that each payer has fuy soved its optimization probem before the other one starts. Next we prove the existence of Nash equiibrium of the TE- SS game. We estabish the stabiity condition when two payers use genera cost functions g ( ) and h ( ) that are continuous, nondecreasing, and convex. Whie TE s formuation is the same in (4) Mode I and Mode II, we consider the two SS modes, i.e., SS with e2e info and SS with improved visibiity. Theorem 1. The TE-SS game has a Nash equiibrium for both Mode I and Mode II. Proof Sketch: It suffices to show that (i) each payer s strategy space is a nonempty compact convex subset, and (ii) each payer s utiity function is continuous and quasi-concave on its strategy space, and foow the standard proof in [19]. The ISP s strategy space is defined by the constraint set of (1), which are affine equaities and inequaities, hence a convex compact set. Since g ( ) is continuous and convex, we can easiy verify that the objective of (1) is quasi-convex on R = { }. CP s strategy space is defined by the constraint set of (3), which is aso convex and compact. Simiary, if h ( f cp ) is continuous and convex, the objective of (3) is quasi-convex on X cp. In particuar, consider the specia case in which CP minimizes atency (2). When CP soves SS with e2e info, h ( f ) = f D (u)du. When CP is soves SS with improved visibiity, h ( f cp ) = f cp D ( f ). In both cases, if D ( ) is continuous, non-decreasing, and convex, so is h ( ). One can again verify the quasi-convexity of the objective in (3). The existence of a Nash equiibrium does not guarantee that the trajectory (4) eads to one. In Section 7 we demonstrate the convergence of iterative payer optimization in simuation. In genera, the Nash equiibrium may not be unique, in terms of both decision variabes and objective vaues. Next, we discuss a specia case where the Nash equiibrium is unique and can be attained by aternating payer moves (4). 4.2 Goba Optimaity under Same Objective and Absence of Background Traffic In the foowing, we consider a specia case of the TE-SS game, in which the ISP and the CP optimize the same objective function, i.e., g ( ) = h ( ), so T E = SS = Φ ( f ), (5) when there is no background traffic. One exampe is when the network carries ony the CP traffic, and both the ISP and the CP aim to minimize the average traffic atency, i.e., Φ ( f ) = f D ( f ). An interesting question that naturay arises is whether the two payers aternating best response to each other s decision can ead to a sociay optima point. Define a notion of goba optimum, which is the optima point to the foowing optimization probem. TE-SS-Specia( ˆX cp ): minimize subject to variabes Φ ( f ) (6) f = C, (s,t) s S : In(v), (s,t), : Out(v) = M t I v=t, v / S, t T where denotes the traffic rate for fow (s,t) deivered on ink. The variabe aows a goba coordinator to route a user s demand from any server in any way it wants, thus probem (6) estabishes an upper-bound on how we one can do to minimize the traffic atency. Note that captures both R and X cp, which offers more degrees of freedom for a joint routing and server-seection 5

6 D F E ink 1 : BD 2 : BE 3 : CD 4 : CE C 1+ε 1+ε 1+ε 1+ε D ( f ) f ε f ε f 3 f 4 g (x) g 1 ( ) = g 2 ( ) = g 3 ( ) = g 4 ( ) B A C Tabe 3: Link capacities, ISP s and CP s ink cost functions in the exampe of Paradox of Extra Information. Figure 2: An Exampe of the Paradox of Extra Information probem. Its mathematica properties wi be further discussed in Section 5.2. The specia case TE-SS game (5) has a Nash equiibrium, as shown in Theorem 1. Nash equiibrium may not be unique in genera. This is because when there is no traffic between a server-user pair, the TE routing decision for this pair can be arbitrary without affecting its utiity. In the worst case, a Nash equiibrium can be arbitrariy suboptima to the goba optimum. Now assume that there exists non-zero traffic demand between any server-user pair as in [8]. Then the aternating payer moves in (4) reach a unique Nash equiibrium, which is aso goba optimum to (6). TE-SS interaction does not sacrifice any efficiency in this specia case, and the optima operating point can be achieved by aternating best response uniateray, without the need of a goba coordination. This resut is shown in [8] where the idea is to prove the equivaence of Nash equiibrium and the goba optimum. An aternative proof is to first transform probem (6) and consider aternating projections of variabes onto a convex set [18]. The specia case anaysis estabishes a ower bound on the efficiency oss of TE-SS interaction. In genera, there are three sources of mis-aignment between the optimization probems of TE and SS: (1) different shapes of the cost functions, (2) different types of deay modeed by the cost functions, and (3) existence of background traffic in the TE probem. The above specia case iustrates what might happen if differences (1) and (3) are avoided, whie the evauation resuts in Section 7 highight the impact of difference (2). As we wi show next, difference in objective functions and presence of background traffic can ead to significant efficiency oss. 5. EFFICIENCY LOSS We next study the efficiency oss in the genera case of the TE-SS game, which may be caused by incompete information, or uniatera actions that miss the opportunity to achieve a jointy attainabe optima point. We present two case studies that iustrate these two sources of suboptima performance. We first present a toy network and show that under certain conditions the CP performs even worse in Mode II than Mode I, despite having more information about underying network conditions. We next propose the notion of Pareto-optimaity as the performance benchmark, and quantify the efficiency oss in both Mode I and Mode II. 5.1 The Paradox of Extra Information Consider an ISP network iustrated in Figure 2. We designate an end user node, T = {F}, and two CP servers, S = {B,C}. The end user has a content demand of M F = 2. We aso aow two background traffic fows, A D and A E, each of which has one unit of traffic demand. Edge directions are noted on the figure, so one can figure out the possibe routes, i.e., there are two paths for each traffic fow (cockwise and counter-cockwise). To simpify the anaysis and deiver the most essentia message from this exampe, suppose that both TE and SS costs on the four thin inks are negigibe so the four bod inks constitute the botteneck of the network. In Tabe 3, we ist the ink capacities, ISP s cost function g ( ), and ink atency function D ( ). Suppose the CP aims to minimize the average atency of its traffic. We compare the Nash equiibrium of two situations when the CP optimizes its network by SS with e2e info and SS with improved visibiity. The stabiity condition for the ISP at Nash equiibrium is g 1 ( f 1) = g 2 ( f 2) = g 3 ( f 3) = g 4 ( f 4). Since the ISP s ink cost functions are identica, the tota traffic on each ink must be identica. On the other hand, the stabiity condition for the CP at Nash equiibrium is that (B, F) and (C, F) have the same margina atency. Based on the observations, we can derive two Nash equiibrium points. When the CP takes the strategy of SS with e2e info, et { } X CP : x BF = 1, x CF = 1 { Mode I: R : r1 BF = 1 α, r2 BF = α, r CF 3 = α, r CF 4 = 1 α, } r1 AD = α, r3 AD = 1 α, r2 AE = 1 α, r4 AE = α One can check that this is indeed a Nash equiibrium soution, where f 1 = f 2 = f 3 = f 4 = 1, and D BF = D CF = 1 α + α/ε. The CP s objective SS I = 2(1 α + α/ε). When the CP takes the strategy of SS with improved visibiity, et { } X CP : x BF = 1, x CF = 1 { Mode II: R : r1 BF = α, r2 BF = 1 α, r CF 3 = 1 α, r CF 4 = α, } r1 AD = 1 α, r3 AD = α, r2 AE = α, r4 AE = 1 α This is a Nash equiibrium point, where f 1 = f 2 = f 3 = f 4 = 1, and d BF = d CF = α(1 + α) + (1 α)(1/ε + (1 α)/ε 2 ). The CP s objective SS II = 2(α +(1 α)/ε). When < ε < 1, α < 1/2, we have the counter-intuitive SS I < SS II : more information may hurt the CP s performance. In the worst case, SS im II = α,ε SS I i.e., the performance degradation can be unbounded. This is not surprising, since the Nash equiibrium is generay non-unique, both in terms of equiibrium soutions and equiibrium objectives. When ISP and CP s objectives are mis-aigned, the ISP s decision may route CP s traffic on bad paths from the CP s perspective. In this exampe, the paradox happens when the ISP privieges the CP traffic in Mode I (athough SS reies on e2e info ony) by assigning it to good paths, and when the ISP misroutes the CP traffic to bad paths in Mode II (athough SS gains 6

7 improved visibiity). In practice, such a scenario is ikey to happen, since the ISP cares about ink congestion (ink utiization), whie the CP cares about atency, which depends not ony on ink oad, but aso on propagation deay. Thus ISP and CP s partia coaboration by ony passing information is not sufficient to achieve goba optimaity. 5.2 Pareto Optimaity and Iustration of Sub- Optimaity As in the above exampe, one of the causes of sub-optimaity is that TE and SS s objectives are not necessariy aigned. To measure efficiency in a system with mutipe objectives, a common approach is to expore the Pareto curve. For points on the Pareto curve, we cannot improve one objective further without hurting the other. The Pareto curve characterizes the tradeoff of potentiay conficting goas of different parties. One way to trace the tradeoff curve is to optimize a weighted sum of the objectives: minimize T E + γ SS (7) variabes R R, X cp X cp where γ is a scaar representing the reative weight of the two objectives. R and X cp are the feasibe regions defined by the constraints in (1) and (3): { R X cp =, 1,, : In(v) f = x i j (i, j) : Out(v) = I v= j, v V \{i}, } C, = M t s S The formuation of probem (7) is not easy to sove. In fact, the objective of (7) is no onger convex in variabes {r st,x st}, and the feasibe region defined by constraints of (7) is not convex. One way to overcome this probem is to consider a reaxed decision space that is a superset of the origina soution space. Instead of restricting each payer to its own operating domain, i.e., ISP contros routing and CP contros server seection, we introduce a joint routing and content deivery probem. Let denote the rate of traffic carried on ink that beongs to fow (s,t). Such a convexification of the origina probem (7) gives more freedom to joint TE and SS probem. Denote the generaized CP decision variabe as ˆX cp = { } s S,t T, and R bg = { } (i, j)/ S T as background routing matrix. Consider the foowing optimization probem: TE-SS-weighted( ˆX cp,r bg ) minimize T E + γ SS (8) subject to f cp =, (s,t) variabes f = f cp + x i j C, (i, j)/ S T : In(v) : Out(v) s S : In(v), 1 : Out(v) = I v= j, (i, j) / S T, v V \{i} = M t I v=t, v / S, t T Denote the feasibe space of the joint variabe as A = { ˆX cp,r bg }. SS cost Measure of efficiency oss operating region Pareto Curve Mode I Mode II TE cost Figure 3: A numerica exampe iustrating sub-optimaity. If we vary γ and pot the achieved TE objectives versus SS objectives, we obtain the Pareto curve. To iustrate the Pareto curve and efficiency oss in Mode I and Mode II, we pot in Figure 3 the Pareto curve and the Nash equiibria in the two-dimensiona objective space (TE,SS) for the network shown in Figure 2. The simuation shows that when the CP everages the compete information to optimize (3), it is abe to achieve ower deay, but the TE cost suffers. Though it is not cear which operating point is better, both equiibria are away from the Pareto curve, which shows that there is room for performance improvement in both dimensions. 6. A JOINT DESIGN: MODEL III Motivated by the need for a joint TE and SS design, we propose the Nash bargaining soution to reduce the efficiency oss observed above. Using the theory of optimization decomposition, we derive a distributed agorithm by which the ISP and the CP can act separatey and communicate with a imited amount of information exchange. 6.1 Motivation An ISP providing content distribution service in its own network has contro over both routing and server seection. So the ISP can consider the characteristics of both types of traffic (background and CP) and jointy optimize a carefuy chosen objective. The jointy optimized system shoud meet at east two goas: (i) optimaity, i.e., it shoud achieve Pareto optimaity so the network resources are efficienty utiized, and (ii) fairness, i.e., the tradeoff between two non-synergistic objectives shoud be baanced so both parties benefit from the cooperation. One natura design choice is to optimize the weighted sum of the traffic engineering goa and server seection goa as shown in (8). However, soving (8) for each γ and adaptivey tuning γ in a tria-and-error fashion is impractica and inefficient. First, it is not straightforward to weigh the tradeoff between the two objectives. Second, one needs to compute an appropriate weight parameter γ for every combination of background oad and CP traffic demand. In addition, the offine computation does not adapt to dynamic changes of network conditions, such as cross traffic or ink faiures. Last, tuning γ to expore a broad region of system operating points is computationay expensive. Besides the system considerations above, the economic perspective requires a fair soution. Namey, the joint design shoud benefit both TE and SS. In addition, such a mode aso appies to a more genera case when the ISP and the CP are different business enti- 7

8 ties. They cooperate ony when the cooperation eads to a win-win situation, and the division of the benefits shoud be fair, i.e., one who makes greater contribution to the coaboration shoud be abe to receive more reward, even when their goas are conficting. Whie the joint system is designed from a cean state, it shoud accept an incrementa depoyment from the existing infrastructure. In particuar, we prefer that the functionaities of routing and server seection be separated, with minor changes to each component. The moduarized design aows us to manage each optimization independenty, with a judicious amount of information exchange. Designing for scaabiity and moduarity is beneficia to both the ISP and the CP, and aows their cooperation either as a singe entity or as different ones. Based on a the above considerations, we appy the concept of Nash bargaining soution [9, 2] from cooperative game theory. It ensures that the joint system achieves an efficient and fair operating point. The soution structure aso aows a moduar impementation. 6.2 Nash Bargaining Soution Consider a Nash bargaining soution which soves the foowing optimization probem: maximize (T E T E)(SS SS) (9) variabes { ˆX cp,r bg } A where (T E,SS ) is a constant caed the disagreement point, which represents the starting point of their negotiation. Namey, (T E,SS ) is the status-quo we observe before any cooperation. For instance, one can view the Nash equiibrium in Mode I as a disagreement point, since it is the operating point the system woud reach without any further optimization. By optimizing the product of performance improvements of TE and SS, the Nash bargaining soution guarantees the joint system is optima and fair. A Nash bargaining soution is defined by the foowing axioms, and is the ony soution that satisfies a of four axioms [9, 2]: Pareto optimaity. A Pareto optima soution ensures efficiency. Symmetry. The two payers shoud get equa share of the gains through cooperation, if the two payers probems are symmetric, i.e., they have the same cost functions, and have the same objective vaue at the disagreement point. Expected utiity axiom. The Nash bargaining soution is invariant under affine transformations. Intuitivey, this axiom suggests that the Nash bargaining soution is insensitive to different units used in the objective and can be efficienty computed by affine projection. Independence of irreevant aternatives. This means that adding extra constraints in the feasibe operating region does not change the soution, as ong as the soution itsef is feasibe. The choice of the disagreement point is subject to different economic considerations. For a singe network provider who wishes to provide both services, it can optimize the product of improvement ratio by setting the disagreement point to be the origin, i.e., equivaent to T E SS/(T E SS ). For two separate ISP and CP entities who wish to cooperate, the Nash equiibrium of Mode I may be a natura choice, since it represents the benchmark performance of current practice, which is the baseine for any future cooperation. It can be obtained from the empirica observations of their average performance. Aternativey, they can choose their preferred performance eve as the disagreement point, written into the contract. In this work, we use the Nash equiibrium of Mode I as the disagreement point to compare the performances of our three modes. 6.3 COST Agorithm In this section, we show how Nash bargaining soution can be impemented in a moduarized manner, i.e., keeping SS and TE functionaities separate. This is important because moduarized design increases the re-usabiity of egacy systems with minor changes, ike existing CDNs depoyment. In terms of cooperation between two independent financia entities, the moduarized structure presents the possibiity of cooperation without reveaing confidentia interna information to each other. We next deveop COST (COoperative Server seection and Traffic engineering), a protoco that impements NBS by separate TE and SS optimizations and communication between them. We appy the theory of optimization decomposition [1] to decompose probem (9) into subprobems. ISP soves a new routing probem, which contros the routing of background traffic ony. The CP soves a new server seection probem, given the network topoogy information. The ISP aso passes the routing contro of content traffic to the CP, offering more freedom to how content can be deivered on the network. They communicate via underying ink prices, which are computed ocay using traffic eves on each ink. Consider the objective of (9), which can be converted to maximize og(t E T E)+og(SS SS) since the og function is monotonic and the feasibe soution space is unaffected. The introduction of the og functions hep revea the decomposition structure of the origina probem. Two auxiiary variabe f cp and f bg are introduced to refect the preferred CP traffic eve from the ISP s perspective and the preferred background traffic eve from the CP s perspective. (9) can be rewritten as max. s.t. var. og(t E f cp f cp g ( f bg + f cp ))+og(ss =, f bg = x i j, (s,t) (i, j)/ S T = f cp, f bg : In(v) = f bg : Out(v) s S : In(v),, f cp : Out(v) + f bg C, h ( f cp + f bg )) (1) = I v= j, (i, j) / S T, v V \{i} = M t I v=t, v / S, t T 1, (i, j) / S T, f cp, f bg The consistency constraint on the auxiiary variabe and the origina variabe ensures that the soution equivaent to probem (9). We take a partia Lagrangian of (1) as L(,ri j, f cp, f bg,λ, µ,ν ) = og(t E g ( f bg + f cp ))+ + og(ss h ( f cp + f bg ))+ + λ (C f bg f cp ) µ ( f bg f bg ) ν ( f cp f cp ) 8

9 λ is the ink price, which refects the cost of overshooting the ink capacity, and µ,ν are the consistency prices, which refect the cost of disagreement between ISP and CP on the preferred ink resource aocation. Observe that f cp and f bg can be separated in the Lagrangian function. We take a dua decomposition approach, and (1) is decomposed into two subprobems: SS-NBS(, f bg ): max. s.t. var. and og(ss f cp = (s,t) h ( f cp, s S : In(v) TE-NBS(, f cp ): max. s.t. var. + f bg : Out(v), (s,t) S T, f bg og(t E g ( f bg + f cp f bg = x i j, (i, j)/ S T : In(v) : Out(v) ))+ ))+ 1, (i, j) / S T, f cp (ν f cp µ f bg λ f cp ) (11) = M t I v=t, v / S, t T (µ f bg ν f cp λ f bg ) (12) = I v= j, (i, j) / S T, v V \{i} The optima soutions of (11) and (12) for a given set of prices µ,ν, and λ define the dua function Dua(µ,ν,λ ). The dua probem is given as: minimize Dua(µ,ν,λ ) (13) variabe λ, µ,ν We can sove the dua probem with the foowing price updates: [ ( λ (t + 1) = λ (t) β λ C f bg f cp ) ] +, (14) µ (t + 1) = µ (t) β µ ( f bg f bg ), (15) ν (t + 1) = ν (t) β ν ( f cp f cp ), (16) where β s are diminishing step sizes or sma constant step sizes often used in practice [21]. Tabe 4 presents the COST agorithm that impements the Nash bargaining soution distributivey. In COST, the ISP soves the new version TE, i.e., TE-NBS, and the CP soves the new version SS, i.e., SS-NBS. In terms of information sharing, the CP earns the network topoogy from the ISP. They do not directy exchange information with each other. Instead, they report f cp and f bg information to underying inks, which pass the computed price information back to TE and SS. It is possibe to further impement TE or SS in a distributed manner, such as on the user/server eves. There are two main chaenges on practica impementation of COST. First, TE needs to adapt quicky to network dynamics. Fast timescae TE has recenty been proposed in various works. Second, an extra price update component is required on each ink, which COST (COoperative SS and TE) ISP: TE agorithm (i) Receives ink price λ and consistency price µ,ν from physica inks E (ii) ISP soves (12) and computes R bg for background traffic (iii) ISP passes f bg, f cp information to each ink (iv) Go back to (i) CP: SS agorithm (i) Receives ink price λ and consistency price µ,ν from physica inks E (ii) CP soves (11) and computes X cp for content traffic. (iii) CP passes f cp, f bg information to each ink (iv) Go back to (i) Link: price update agorithm (i) Initiaization step: set λ, and µ,ν arbitrariy (ii) Updates ink price λ according to (14) (iii) Updates consistency prices µ,ν according to (15)(16) (iv) Passes λ, µ,ν information to TE and SS (v) Go back to (ii) Tabe 4: Distributed agorithm for soving probem (9). invoves price computation and message passing between TE and SS. This functionaity can be potentiay impemented in routers. The COST agorithm is precisey captured by the decomposition method described above. Certain choice of step sizes, such as β(t) = β /t, where β >, guarantees that the agorithm converges to a goba optimum [22]. Theorem 2. The distributed agorithm COST converges to the optimum of (9) for sufficienty sma step sizes β λ,β µ and β ν. 7. PERFORMANCE EVALUATION In this section, we use simuations to demonstrate the efficiency oss that may occur for rea network topoogies and traffic modes. We aso compare the performance of the three modes. We sove the Nash bargaining soution centray, without using the COST agorithm, since we are primariy interested in its performance. Compementary to the theoretica anaysis, the simuation resuts aow us to gain a better understanding of the efficiency oss under reaistic network environments. These simuation resuts aso provide guidance to network operators who need to decide which approach to take, sharing information or sharing contro. 7.1 Simuation Setup We evauate our modes under ISP topoogies obtained from Rocketfue [23]. We use the backbone topoogy of the research network Abiene [24] and severa major tier-1 ISPs in north America. The choice of these topoogies aso refects different geometric properties of the graph. For instance, Abiene is the simpest graph with two botteneck paths horizontay. The backbones of AT&T and Exodus have a hub-and-spoke structure with some shortcuts between nodes pairs. The topoogy of Leve 3 is amost a compete mesh, whie Sprint is in between these two kinds. We simuate the traffic demand using a gravity mode [25], which refects the pairwise communication pattern on the Internet. The content demand of a CP user is assumed to be proportiona to the node popuation. The TE cost function g( ) and the SS cost function h( ) are chosen as foows. ISPs usuay mode congestion cost with a convex increasing function of the ink oad. The exact shape of the function g ( f ) is not important, and we use the same piecewise inear 9

10 cost function as in [11], given beow: f f /C < 1/3 3 f 2/3C 1/3 f /C < 2/3 1 f 16/3C 2/3 f /C < 9/1 g ( f,c ) = 7 f 178/3C 9/1 f /C < 1 5 f 1468/3C 1 f /C < 11/1 5 f 16318/3C 11/1 f /C < The CP s cost function can be the performance cost ike atency, financia cost charged by ISPs. We consider the case where atency is the primary performance metric, i.e., the content traffic is deay sensitive ike video conferencing or ive streaming. So we et the CP s cost function h ( ) be of the form given by (2), i.e., h ( f ) = f cp D ( f ). A ink s atency D ( ) consists of queuing deay and propagation deay. The propagation deay is proportiona to the geographica distances between nodes. The queuing deay is approximated by the M/M/1 mode, i.e., D queue = 1 C f, f < C with a inear approximation when the ink utiization is over 99%. We reax the ink capacity constraints in both TE and SS and penaize traffic overshooting the ink capacity with high costs. The shapes of the TE ink cost function and queuing deay function are iustrated in Figure 4. We intensionay choose the cost functions of TE and SS to be simiar in shape. This aows us to quantify the efficiency oss of Mode I and Mode II even when their objectives are reativey we aigned, as we as the improvement brought by Mode III. g (f ) TE ink cost function, C = f /C (a) TE ink cost deay Queuing deay function, C = f /C (b) Link queuing deay Figure 4: ISP and CP cost functions. 7.2 Evauation Resuts Tusse between background and CP s traffic We first demonstrate how CP s traffic intensity affects the overa network performance. We fix the tota amount of traffic and tune the ratio between background traffic and CP s traffic. We evauate the performance of different modes when CP traffic grows from 1% to 1% of the tota traffic. Figure 5 iustrates the resuts on Abiene topoogy. The genera trend of both TE and SS objectives for a three modes is that the cost first decreases as CP traffic percentage grows, and ater increases as CP s traffic dominates the network. The decreasing trend is due to the fact that CP s traffic is sef-optimized by seecting servers cose to a user, thus offoading the network. The increasing trend is more interesting, suggesting that when a higher TE cost TE cost v.s. CP traffic voume Mode I (No cooperation) Mode II (Sharing Info) Mode III (Sharing Contro) CP traffic percentage (a) SS cost SS cost v.s. CP traffic voume Mode III (Sharing Contro) Mode II (Sharing Info) Mode I (No cooperation) CP traffic percentage Figure 5: The TE-SS tusse v.s. CP s traffic intensity (Abiene topoogy) percentage of tota traffic is CP-generated, the negative effect of TE-SS interaction is ampified, even when the ISP and the CP share simiar cost functions. Low ink congestion usuay means ow endto-end atency, and vice versa. However, they differ in the foowing: (i) TE might penaize high utiization before queueing deay becomes significant in order to eave as much room as possibe to accommodate changes in traffic, and (ii) CP considers both propagation deay and queueing deay so it may choose a moderateycongested short path over a ighty-oaded ong path. This expains why the optimization efforts of two payers are at odds Network congestion v.s. performance improvement We now study the network conditions under which more performance improvement is possibe. We evauate the three modes on the Abiene topoogy. Again, we fix the tota amount of traffic and vary the CP s traffic percentage. Now we change ink capacities and evauate two scenarios: when the network is moderatey congested and when the network is highy congested. We show the performance improvement of Mode II and Mode III over Mode I (in percentages) and pot the resuts in Figure 6. Figures 6(a-b) show the improvement of the ISP and the CP when the network is under ow oad. Generay, Mode II and Mode III improve both TE and SS, and Mode III outperforms Mode II in most cases, with the exception that Mode II is biased towards SS sometimes. However, both ISP and CP s improvement are not substantia (note the different scaes of y-axes), except when CP traffic is insignificant (1%). This is because when the network is under ow oad, the sopes of TE and SS cost functions are fat, thus eaving itte space for improvement. Figure 6(c-d) show the resuts when the network is under high oad. Improvement becomes more significant, especiay at the two extremes: when CP s traffic is insignificant or prevaent. This again suggests that when CP traffic is dominant, there is a arge space eft for improvement even when two objectives are simiar in shape. However, observe that whie mode III aways improves TE and SS, Mode II coud sometimes perform worse than Mode I Impact of ISP topoogies We evauate our three modes on different ISP topoogies. The topoogica properties of different graphs are discussed earier. The CP s traffic is 8% of the tota traffic and ink capacities are set such that networks are under high traffic oad. Our findings are depicted in Figure 7. Note that performance improvement is rea- (b) 1

11 percentage of SS cost saving CP s performance improvement Mode II (Sharing Info) Mode III (Sharing Contro) percentage of TE cost saving ISP s performance improvement Mode II (Sharing Info) Mode III (Sharing Contro) percentage of SS cost saving CP s performance improvement Mode II (Sharing Info) Mode III (Sharing Contro) percentage of TE cost saving ISP s performance improvement Mode II (Sharing Info) Mode III (Sharing Contro).5 1 CP traffic percentage.5 1 CP traffic percentage CP traffic percentage CP traffic percentage (a) (b) (c) (d) Figure 6: TE and SS performance improvement of Mode II and III over Mode I. (a-b) Abiene network under ow traffic oad: moderate improvement; (c-d) Abiene network under high traffic oad: more significant improvement, but more information (in Mode II) does not necessariy benefit the CP and the ISP (the paradox of extra information). percentage of TE cost saving ISP improvement on different networks Mode II (Sharing Info) Mode III (Sharing Contro) Abiene AT&T Exodus Leve3 Sprint percentage of SS cost saving CP improvement on different networks Mode II (Sharing Info) Mode III (Sharing Contro) Abiene AT&T Exodus Leve3 Sprint max ink utiization Maximum ink utiization on different networks 1 Mode I (No Cooperation).95 Mode II (Sharing Info) Mode III (Sharing Contro) Abiene AT&T Exodus Leve3 Sprint (a) (b) (c) Figure 7: Performance evauation over different ISP topoogies. Abiene: sma cut graph; AT&T, Exodus: hub-and-spoke with shortcuts; Leve 3: compete mesh; Sprint: in between. tivey more significant in more compex graphs. Simpe topoogies with sma min-cut sizes are networks where the apparent paradox of more (incompete) information is ikey to happen. Besides the TE and SS objectives, we aso pot the maximum ink utiization to iustrate the eve of congestion in the network. Higher network oad shows more space for potentia improvement. Aso, mode III improves this metric generay, which might be another important consideration for network providers. 8. RELATED WORK This paper is an extension of our earier workshop paper [26]. Additions in this paper incude the foowing: a more genera CP mode, anaysis of optimaity conditions in three cooperation modes, paradox of extra information, impementation of Nash bargaining soution, and arge scae evauation. The most simiar work is a parae work [8], which studied the interaction between content distribution and traffic engineering. It shows the optimaity conditions for two separate probems to converge to a sociay optima point, as discussed in Section 4.2. It aso provides a theoretica bound on efficiency oss and discusses generaizations to mutipe ISPs and overay networks. Some earier work studied the sef-interaction within ISPs or CPs themseves. In [16], the authors used simuation to show that sefish routing is cose to optima in Internet-ike environments without sacrificing much performance degradation. [27] studied the probem of oad baancing by overay routing, and how to aeviate race conditions among mutipe co-existing overays. [28] studied the resource aocation probem at inter-as eve where ISPs compete to maximize their revenues. [29] appied Nash bargaining soution to sove an inter-domain ISP peering probem. The need for cooperation between content providers and network providers is raising much discussion in both the research community and industry. [3] used price theory to reconcie the tusse between peer-assisted content distribution and ISP s resource management. [6] proposed a communication porta between ISPs and P2P appications, which P2P appications can consut for ISPbiased network information to reduce network providers cost without sacrificing their performances. [5] proposed an orace service run by the ISP, so P2P users can query for the ranked neighbor ist according to certain performance metrics. [7] utiized existing network views coected from content distribution networks to drive biased peer seection in BitTorrent, so cross-isp traffic can be significanty reduced and downoad-rate improved. [31] studied the interaction between underay routing and overay routing, which can be thought of as a generaization of server seection. The authors studied the equiibrium behaviors when two 11

12 CP no change CP change ISP no change current practice partia coaboration ISP change partia coaboration joint system design Tabe 5: To cooperate or not: possibe strategies for content provider (CP) and network provider (ISP) probems have conficting goas. Our work expores when and why sub-optimaity appears, and proposes a cooperative soution to address these issues. [32] studied the economic aspects of traditiona transit providers and content providers, and appied cooperative game theory to derive an optima settement between these entities. 9. CONCLUSION AND FUTURE WORK We examine the interpay between traffic engineering and content distribution. Whie the probem has ong existed, the dramaticay increased amount of content-centric traffic, e.g., CDN and P2P traffic, makes it more significant. With the strong motivation for ISPs to provide content services, they are faced with the question of whether to stay with the current design or to start sharing information or contro. This work sheds ight on ways ISPs and CPs can cooperate. This paper serves as a starting point to better understand the interaction between those that operate networks and those that distribute content. Traditionay, ISPs provide and operate the pipes, whie content providers distribute content over the pipes. In terms of what information can be shared between ISPs and CPs and what contro can be jointy performed, there are four genera categories as summarized in Tabe 5. The top eft corner is the current practice, which may give an undesirabe Nash equiibrium. The bottom right corner is the joint design, which achieves optima operation points. The top right corner is the case where the CP receives extra information and adapts contro accordingy, and the bottom eft corner is the case of content-aware networking. This paper studies three of the four corners in the tabe. Starting from the current practice, to move towards the bottom right corner of the tabe, whie the two parties remain separate business entities, requires uniateray-actionabe, backward-compatibe, and incrementaydepoyabe migration paths yet to be discovered. 1. ACKNOWLEDGMENTS We are gratefu for Dan Rubenstein s shepherding the paper and for Ramesh Johari s sharing the preprint [6] that has the resuts cited in Section 4.2. This work is in part supported by NSF grants CNS and CNS REFERENCES [1] W. B. Norton, Video Internet: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem, Sept 26. Eqinix white paper. [2] AT&T, U-verse. [3] Verizon, FiOS. [4] A.-J. Su, D. R. Choffnes, A. Kuzmanovic, and F. E. Bustamante, Drafting behind Akamai (Traveocity-based detouring), in Proc. ACM SIGCOMM, 26. [5] V. Aggarwa, A. Fedmann, and C. Scheideer, Can ISPs and P2P users cooperate for improved performance?, ACM SIGCOMM Computer Communication Review, vo. 37, no. 3, pp. 29 4, 27. [6] H. Xie, Y. R. Yang, A. Krishnamurthy, Y. Liu, and A. Siberschatz, P4P: Provider Porta for (P2P) Appications, in Proc. ACM SIGCOMM, 28. [7] D. R. Choffnes and F. E. Bustamante, Taming the torrent: a practica approach to reducing cross-isp traffic in peer-to-peer systems, ACM SIGCOMM Computer Communication Review, vo. 38, no. 4, pp , 28. [8] D. DiPaantino and R. Johari, Traffic engineering versus content distribution: A game theoretic perspective, in Proc. IEEE INFOCOM, 29. [9] J. F. Nash, The bargaining probem, Econometrica, vo. 28, pp , 195. [1] D. P. Paomar and M. Chiang, A tutoria on decomposition methods for network utiity maximization, IEEE J. on Seected Areas in Communications, vo. 24, no. 8, pp , 26. [11] B. Fortz and M. Thorup, Internet traffic engineering by optimizing OSPF weights, in Proc. IEEE INFOCOM, pp , 2. [12] D. Awduche, J. Macom, J. Agogbua, M. O De, and J.McManus, RFC 272: Requirements for Traffic Engineering Over MPLS, September [13] D. Xu, M. Chiang, and J. Rexford, Like-state routing with hop-by-hop forwarding can achieve optima traffic engineering, in Proc. IEEE INFOCOM, 28. [14] J. Wardrop, Some theoretica aspects of road traffic research, the Institute of Civi Engineers, vo. 1, no. 2, pp , [15] T. Roughgarden and Éva Tardos, How bad is sefish routing?, J. of the ACM, vo. 49, no. 2, 22. [16] L. Qiu, Y. R. Yang, Y. Zhang, and S. Shenker, On sefish routing in Internet-ike environments, in Proc. ACM SIGCOMM, 23. [17] M. Littman and J. Boyan, A distributed reinforcement earning scheme for network routing, Tech. Rep. CMU-CS , Robotics Institute, Carnegie Meon University, [18] W. Jiang, R. Zhang-Shen, J. Rexford, and M. Chiang, Cooperative content distribution and traffic engineering in a provider network, Tech. Rep. TR-846-8, Department of Computer Science, Princeton University, 28. [19] M. J. Osborne and A. Rubinstein, A Course in Game Theory. MIT Press, [2] K. Binmore, A. Rubinstein, and A. Woinsky, The Nash bargaining soution in economic modeing, RAND Journa of Economics, vo. 17, pp , [21] J. He, R. Zhang-Shen, Y. Li, C.-Y. Lee, J. Rexford, and M. Chiang, DaVinci: Dynamicay Adaptive Virtua Networks for a Customized Internet, in Proc. CoNEXT, Dec 28. [22] D. P. Bertsekas, Noninear Programming. Athena Scientific, [23] N. Spring, R. Mahajan, D. Wethera, and T. Anderson, Measuring ISP topoogies with Rocketfue, IEEE/ACM Trans. Networking, vo. 12, no. 1, pp. 2 16, 24. [24] Abiene. [25] M. Roughan, M. Thorup, and Y. Zhang, Performance of estimated traffic matrices in traffic engineering, SIGMETRICS Perform. Eva. Rev., vo. 31, no. 1, pp , 23. [26] W. Jiang, R. Zhang-Shen, J. Rexford, and M. Chiang, Cooperative content distribution and traffic engineering, in NetEcon 8, August 28. [27] W. Jiang, D.-M. Chiu, and J. C. S. Lui, On the interaction of mutipe overay routing, Perform. Eva., vo. 62, no. 1-4, pp , 25. [28] S. C. Lee, W. Jiang, D.-M. C. Chiu, and J. C. Lui, Interaction of ISPs: Distributed resource aocation and revenue maximization, IEEE Transactions on Parae and Distributed Systems, vo. 19, no. 2, pp , 28. [29] G. Shrimai, A. Akea, and A. Mutapcic, Cooperative interdomain traffic engineering using Nash bargaining and decomposition, in Proc. IEEE INFOCOM, 27. [3] M. J. Freedman, C. Aperjis, and R. Johari, Prices are right: Managing resources and incentives in peer-assisted content distribution, in IPTPS 8, February 28. [31] Y. Liu, H. Zhang, W. Gong, and D. Towsey, On the interaction between overay routing and underay routing, in Proc. IEEE INFOCOM, pp , 25. [32] R. T. Ma, D. Chiu, J. C. Lui, V. Misra, and D. Rubenstein, On cooperative settement between content, transit and eyeba internet service providers, in Proc. CoNEXT, December