264 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014

Transcription

1 264 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014 Energy Management n Cross-Doman Content Delvery Networks: A Theoretcal Perspectve Chang Ge, Student Member, IEEE, Zhl Sun, Member, IEEE, Nng Wang, Member, IEEE, Ke Xu, Senor Member, IEEE, and Jnsong Wu, Senor Member, IEEE Abstract In a content delvery network (CDN), the energy cost s domnated by ts geographcally dstrbuted data centers (DCs). Generally wthn a DC, the energy consumpton s domnated by ts server nfrastructure and coolng system, wth each contrbutng approxmately half. However, exstng research work has been addressng energy effcency on these two sdes separately. In ths paper, we jontly optmze the energy consumpton of both server nfrastructures and coolng systems n a holstc manner. Such an objectve s acheved through both strateges of: 1) puttng dle servers to sleep wthn ndvdual DCs; and 2) shuttng down dle DCs entrely durng off-peak hours. Based on these strateges, we develop a heurstc algorthm, whch concentrates user request resoluton to fewer DCs, so that some DCs may become completely dle and hence have the opportunty to be shut down to reduce ther coolng energy consumpton. Meanwhle, QoS constrants are respected n the algorthm to assure servce avalablty and end-to-end delay. Through smulatons under realstc scenaros, our algorthm s able to acheve an energy-savng gan of up to 62.1% over an exstng CDN energy-savng scheme. Ths result s bound to be near-optmal by our theoretcally-derved lower bound on energy-savng performance. Index Terms Content delvery network, data center, energy management. I. INTRODUCTION GENERALLY, a content delvery network (CDN) operator strategcally establshes a number of data centers (DC) at multple dstrbuted stes towards the edge of the Internet. Web contents are replcated and cached at these DCs, so that end-users can experence reduced end-to-end delay and enhanced servce avalablty when requestng contents. Such an nfrastructure typcally nvolves dozens of large DCs (each wth thousands of servers) or hundreds to thousands of smaller DCs (each wth hundreds of servers), whch are geographcally dstrbuted across large geographcal areas. It s adopted by major CDN operators such as Akama and Lmelght [1]. The contrbuton of DCs worldwde to the global energy consump- Manuscrpt receved May 7, 2014; revsed August 1, 2014; accepted August 6, Date of publcaton August 13, 2014; date of current verson September 5, Ths work s supported by EU FP7 EVANS Project (PIRSES-GA ). The assocate edtor coordnatng the revew of ths paper and approvng t for publcaton was P. Owezarsk. C. Ge, Z. Sun, and N. Wang are wth the Centre for Communcaton Systems Research, Unversty of Surrey, Guldford GU2 7XH, U.K. (e-mal: [email protected]; [email protected]; [email protected]). K. Xu s wth Tsnghua Unversty, Bejng , Chna (e-mal: xuke@ tsnghua.edu.cn). J. Wu s wth Bell Laboratores, Shangha , Chna (e-mal: wujs@ eee.org). Color versons of one or more of the fgures n ths paper are avalable onlne at Dgtal Object Identfer /TNSM ton has ncreased from 1% n 2005 to around 1.5% to 2010, whch s expected to grow by 15% every year n the foreseeable future [2]. Therefore, t s crucal to develop effectve energysavng schemes for CDNs. To reduce CDN energy consumpton, the common practce n the lterature s to dynamcally provson fewer actve servers wthn ndvdual DCs durng off-peak hours, so that the dle servers are put nto the sleep mode [3] [5]. Furthermore, energy management schemes of coolng systems n ndvdual DCs have been proposed [6], [7]. Through these strateges, most exstng schemes focused on managng energy costs of servers or coolng systems separately. Snce coolng systems n modern DCs contrbute around 44.4% to 47% to the overall energy cost on average [2], [8], to holstcally maxmze energy savng n CDNs, energy consumpton of servers and coolng systems need to be optmzed smultaneously by takng nto account ther dependency. In ths paper, our goal s to jontly mnmze both server s and coolng system s energy consumpton among geographcallydstrbuted DCs. We employ two strateges smultaneously to acheve ths goal. Frstly, wthn each ndvdual DC, server energy consumpton s reduced through concentratng loads to fewer servers, so that the remanng dle servers can be put to sleep [3] [5]. Secondly, among multple DCs n a CDN, we am to concentrate request resoluton to fewer DCs, so that some DCs wthout any mapped request can be shut down entrely. Such an dea s based on the fact that an dle coolng system typcally consumes around 40% of ts peak power consumpton n an actve DC [9], whle t consumes zero power n a shut down DC [10]. Furthermore, DC shutdown operatons are not uncommon n the DC ndustry, whch are performed manly for hardware mantenance etc. Therefore, durng offpeak hours, t s lkely that controlled DC shutdown can produce substantal energy-savng gans over exstng server-sleepngonly schemes. Despte the very promsng potental, there are dstnct challenges assocated wth such jont optmzaton. When managng server sleepng and DC shutdown n a CDN, the trade-off between end-to-end QoS performance and energy-savng gan must be well balanced. Intutvely, after some DCs are shut down, some requests mght have to be resolved to alternatve DCs n further remote locatons. Specfcally, n a crossdoman CDN, ths mght even lead to nter-doman request redrecton whch would cause poorer servce avalablty and user-perceved end-to-end delay. Ths challenge can be tackled through strategc plannng on user-to-dc request resoluton. When resolvng end-users requests to DCs, nstead of IEEE. Personal use s permtted, but republcaton/redstrbuton requres IEEE permsson. See for more nformaton.

2 GE et al.: ENERGY MANAGEMENT IN CROSS-DOMAIN CONTENT DELIVERY NETWORKS 265 resolvng requests to the fewest DCs and shuttng down as many DCs as possble, the CDN operator can assure better QoS through e.g., prohbtng nter-doman request resoluton. Although energy savng s traded off n ths way, the energy- QoS trade-off s balanced better, whch leads to both effectve energy savng and assured QoS performance. Our contrbutons n ths paper are as follows. Frstly, we establsh a holstc optmzaton formulaton for the energy mnmzaton problem, whose objectve s to jontly optmze servers and coolng systems energy consumpton among DCs n a CDN. To the best of authors knowledge, t s the frst n the lterature to perform such a jont optmzaton through both server sleepng and DC shutdown smultaneously. Secondly, we develop a practcal algorthm named Mn- DC-LD for the formulated problem. Mn-DC-LD s a greedy heurstc that resolves requests to as few DCs as possble, whch creates opportuntes for some DCs to become dle and get shut down (especally durng off-peak hours). Meanwhle, t enforces QoS constrants to mantan a well-balanced energy- QoS trade-off. Frstly, t restrcts all requests to be resolved wthn ther local domans to avod unnecessary nter-doman content traffc. Secondly, t allows the CDN operator to specfy the maxmum dstance between a request s source node and ts desgnated DC, whch suts a varety of QoS requrements by web applcatons wth dfferent tolerances of end-to-end delay. Furthermore, we show our algorthm s practcalty n modern CDN nfrastructures through explanng ts partcpaton n the request resoluton process. Thrdly, we analytcally derve a theoretcal lower bound to the formulated optmzaton problem. Such a lower bound can serve as a benchmark for Mn-DC-LD and other algorthm s energy-savng performance. Snce a problem s optmal objectve s bound to be between ts lower bound and any polynomal-tme algorthm s performance, f the gap between lower bound s and Mn-DC-LD s result s suffcently small, our algorthm s performance s guaranteed to be near-optmal. We carred out smulatons based on real cross-doman network topology and workload trace, and the results are as follows. 1) Mn-DC-LD s capable of achevng an energysavng gan of up to 62.1% over exstng CDN energy-savng schemes [4], and such gan s guaranteed to be near-optmal by our derved lower bound. 2) Whle producng ts maxmum energy-savng gan, Mn-DC-LD s end-to-end delay meets the requrements of typcal real-tme web applcatons. Moreover, better latency and server response tme can be acheved through trade-off n energy-savng performance. 3) The gap between our derved lower bound and Mn-DC-LD s results s always less than 24.8% wth medan values between 6.2% and 11.7%, whch makes the lower bound a sutable benchmark for other polynomal-tme algorthms for the problem. II. PROBLEM FORMULATION Consder a CDN nfrastructure that covers multple ISP autonomous domans wth n Pont-of-Presence (PoP) nodes n total. PoP nodes are where end-users requests are aggregated n ISP backbone networks. A set of DCs are deployed at locatons close to a subset of PoP nodes, and each DC contans one or more server clusters to resolve user requests. The ncomng request volume (averaged per second) at each PoP node j (j = 1...n) s denoted by r j n request/s. Each request from PoP j s resolved to some DC ( =1...m) desgnated by the CDN s request mappng system, and the overall request volume resolved from any PoP j to any DC s represented by x j (n request/s). For each DC, ts servce capablty s denoted by Y, whch refers to the maxmum number of requests t can handle concurrently when all of ts servers are actve. Furthermore, ts utlzaton, u, s defned as u Δ = n j=1 x j Y ( =1,...,m). (1) Snce we consder the opton of shuttng down entre DCs, a bnary varable δ s ntroduced to ndcate the on/off status of each DC.DC has zero utlzaton f t s shut down (.e., δ = 0). If DC s actve (.e., δ =1), u s value must be between 0 and 1 to avod overloadng DC. δ and u s relatonshp s summarzed as { 0 u 1, f δ =1 ( =1,...,m). (2) u =0, f δ =0 We vew each DC as a set of servers when consderng ts request resoluton and power consumpton. Frstly, request resoluton are performed at PoP-to-DC level only, and we assume that requests resolved to a DC are automatcally assgned nternally to a sutable server through exstng DC load balancng mechansm [11]. Furthermore, we assume that each DC automatcally puts ts dle servers to the sleep mode wth respect to ts utlzaton, and u ndcates the proporton of servers that are actve n DC [4]. In practce, a modern CDN nfrastructure contans two man types of DCs,.e., back-end and surrogate DCs. There are typcally a few back-end DCs and many geographcally dstrbuted surrogate DCs. The man reason for such setup s that content popularty generally follow Zpf dstrbuton wth a long Pareto tal [12]. Typcally, more than half of content objects are requested by users nfrequently, whle a small subset of them are frequently requested [13]. Therefore, n practce, only the popular contents are cached at dstrbuted surrogate DCs for reduced latency [14]. In contrast, the unpopular ones are stored at back-end DCs for nfrequent access [15]. In ths paper, we assume that each surrogate DC stores a full copy of all popular content objects [16], and all unpopular content objects are stored at only the back-end DC [15]. It can be nferred that back-end DCs can never be shut down under ths crcumstance, snce they are the only avalable DCs to resolve requests for unpopular contents. Therefore, from energy savng s perspectve, we do not nclude back-end DCs n our optmzaton problem. For the rest of ths paper, we use the term DC to refer to surrogate DCs only. Based on the assumptons above, we calculate power consumpton of servers and coolng systems of each DC (denoted by P svr and P cool Frstly, P svr respectvely) as follows. s defned by P svr Δ = P peak,svr u + P slp,svr (1 u ) (3)

3 266 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014 where P slp peak,svr and P,svr refer to aggregated power consumpton of servers n DC when all of them are sleepng or fully-loaded respectvely. Note that unlke other related work n the lterature [4], [10], we do not explctly model power consumpton of ndvdual servers n each DC. Instead, we model the entre DC s power consumpton as a whole. Ths s due to the followng reasons. Frstly, content servers n CDN DCs are typcally homogeneous wth dentcal capactes and power characterstcs [10]. Secondly, based on our energy-savng strategy, the actve servers n each DC wll have very hgh utlzaton snce the sleepng servers loads have been mgrated to them. Hence, they are expected to consume near-peak power consumpton. 1 Due to the two reasons above, P peak,svr u and P slp,svr (1 u ) can represent the power consumpton of actve and sleepng servers respectvely at DC wth utlzaton u. Secondly, P cool s defned by P cool Δ = P peak (,cool A δ + B u + C u 2 ) + P slp,cool (1 δ ) (4) where P peak,cool and P slp,cool refers to the power consumpton of coolng system n DC when t s fully loaded or shut down respectvely, and coeffcents A, B, and C are regresson parameters calculated through the functons n [9] gven the DC outdoor and ndoor dry bulb temperature. In a typcal modern DC, ts coolng power consumpton s domnated by ts chller plant s compressor [17], whose power consumpton s a quadratc functon of ts utlzaton [9]. We estmate a DC s chller utlzaton as follows. Frstly, the amount of heat that s removed by the coolng system roughly equals the amount of heat dsspated by servers n a DC (for smplcty, we do not consder the heat exchange caused by dfferent DC layout). Secondly, although most DC operators tend to over-provson coolng system s capacty, we conservatvely assume that a coolng system s maxmum heat removal capacty matches the maxmum overall heat dsspaton of all servers n that DC. Hence, a DC s chller utlzaton can be estmated to equal ts server utlzaton u. In each DC, the followng relatonshp between P peak,cool and P peak,svr holds: PUE = P peak,svr + P peak,cool P peak,svr where Power Usage Effectveness (PUE) s an ndustry standard metrc that ndcates a DC s energy effcency. A DC ndustry survey has reported average values of 1.8 (2012) and 1.67 (2013) [8], whle another ndustral source reported the fgures of 2.8 (2012) and 2.9 (2013) n North Amerca [18]. Note that snce servers and coolng systems domnate a DC s power consumpton, we do not consder other DC components n ths paper [19]. We now present the formulaton of the energy optmzaton problem: 1 In practce, to avod poor server response tme caused by overloadng, an utlzaton threshold lmt can be appled to the actve servers. (5) Problem (P): Gven r =(r j ) Z n + and Y =(Y ) R m +, fnd X =(x j ) Z m n 0 and δ =(δ ) Z m such that ( mn P svr + P cool ) (6) x j,δ subject to: x j = r j (j =1,...,n) (7) u δ ( =1,...,m) (8) δ {0, 1} ( =1,...,m). (9) Equaton (6) s the objectve of mnmzng overall server and coolng power consumpton among all DCs. Constrant (7) guarantees 100% servce avalablty through resolvng all requests from each PoP node to one or more DCs. Constrants (8) and (9) enforce the relatonshp between δ and u as n (2). Substtutng (1), (3), and (4) nto (6), the orgnal objectve functon (6) s expressed as: mn x j,δ where a δ + b x j + c j=1 x j j=1 a = P peak,cool A P slp,cool b = 1 ( ) P peak,svr Y P slp,svr + P peak,cool B 2 (10) c = 1 Y 2 P peak,cool C. Smlarly, we substtute (1) nto constrant (8) to get: 1 x j δ ( =1,...,m) (11) Y j=1 whch replaces constrant (8) n the orgnal formulaton. Problem (P) has been shown to be NP-hard n [20] snce t can be reduced from a smooth non-convex nonlnear programmng problem. Therefore, we derve a lower bound to the problem s optmal objectve. III. DERIVING A LOWER BOUND TO (P) In ths secton, we employ Lagrangan relaxaton to derve a lower bound to (P). Among the constrants n (P), we choose to apply Lagrangan relaxaton on constrant (11). The resultng problem after relaxaton s: Problem (LP): Gven r =(r j ) Z n + and Y =(Y ) R m +, fnd X =(x j ) Z m n 0, δ =(δ ) Z m and λ =(λ ) R m + such that 2 mn x j,δ,λ a δ + b x j + c j=1 + λ 1 Y x j j=1 n j=1 x j δ (12)

4 GE et al.: ENERGY MANAGEMENT IN CROSS-DOMAIN CONTENT DELIVERY NETWORKS 267 subject to: x j = r j (j =1,...,n) (13) δ {0, 1} ( =1,...,m) (14) where λ s a set of m Lagrangan multplers. For notatonal convenence, we defne φ as below: Defnton 1: Gven any optmzaton problem (OP), the value of ts optmal objectve s defned as φ(op). It can be nferred from constrant (11) that snce (λ ) R m +, φ(lp) s lower bound to φ(p). Followng the relaxaton, t s further observed that (LP) s the sum of the followng two sub-problems: Problem (LP1): subject to: Problem (LP2): mn x j,λ subject to: j=1 mn δ,λ (a λ )δ (15) δ {0, 1} ( =1,...,m). (16) ( b + λ ) x j + Y c x j j=1 2 (17) x j = r j (j =1,...,n). (18) Frstly, (LP1) s a smple unconstraned 0 1 optmzaton problem that can be solved through { 1 f a <λ δ = ( =1,...,m). (19) 0 otherwse Secondly, (LP2) s a quadratc knapsack problem whch s NP-hard as t can reduced from the clque problem [21]. Hence, nstead of solvng φ(lp2) drectly, we try to fnd a lower bound to φ(lp2) (denoted by LB(LP2)), snce φ(lp1) + LB(LP2) s also a lower bound to φ(p). To fnd LB(LP2), consder the problem wth the objectve below: Problem (LB-LP2): mn x j,λ j=1 ( b + λ ) x j + Y j=1 c x 2 j (20) subject to (18). We clam that φ(lb LP2) s a vald lower bound to φ(lp2). Note the dfference between (20) and (17), whch equals m n j=1 c x 2 j m c ( n j=1 x j) 2, s always negatve snce (x j ) Z 0. Hence, φ(lb LP2) <φ(lp2) holds. A. Solvng φ(lb-lp2) n Polynomal Tme Problem (LB-LP2) s n the form of a specal separable quadratc convex problem wth a fxed number of lnear constrants, whose optmal objectve can be found n polynomal tme [22]. The process of solvng φ(lb-lp2) s as follows. Frstly, (LB-LP2) can be decomposed nto the sum of n optmzaton problems, and each sub-problem has ts objectve as the nth part of the decomposed (21). Each sub-problem, denoted by (LB-LP2 j )(j =1,...,n), s descrbed as below: Problem (LB-LP 2 j ): subject to: mn x j,λ ( b + λ ) x j + Y c x 2 j (21) x j = r j. (22) For each (LB-LP2 j ) and any fxed λ, wehave: Lemma 1: Wth j fxed, vector X j =(x j ) Z m 0 that satsfes (22) s an optmal soluton to (LB-LP2 j ) f and only f μ j R such that { μj (b +λ /Y ) x j = c f b + λ Y <μ j 0 f b + λ ( =1,...,m) Y μ j (23) Proof: Proof of Lemma 1 s gven n [22], [23]. It takes O(m) tme to search through the set of (b + λ /Y =1,...,m) to dentfy the ndces of such that b + λ /Y <μ j. Denotng such set of as I, we get from (21) and (22) that I μ j (b + λ /Y ) c = r j (24) Denote the optmal soluton to (LB-LP2 j ) as X j =(x j ). Substtutng (24) nto (23), we get that for each (LB-LP2 j ): x j = r j + I b +λ /Y c I 1/Y (b + λ /Y ) c (25) Note that (x j ) can be represented as a functon of solely λ. Therefore, f we can dentfy the set λ that produces (x j ), then ths set, denoted by λ, produces φ(lb-lp2) as well through (25). It s known that φ(λ ) s concave, and maxmzng φ(λ ) s equvalent to solvng φ(lb-lp2) [24]. Recall that φ(λ )= n j=1 φ j(λ ) where φ j (λ ) (φ j n short) s the optmal objectve value of (LB-LP2 j ). Hence, the next step s to dentfy the set λ for each φ j functon. Consder each φ j to have p j hyperplane equatons. For each φ j,we compute ts p j functons and all the respectve cells through computng ntersectons of components of φ j. Altogether, we have at most n j=1 p j = O(n) equatons and n j=1 O(pm j )= O(n) cells of quadracty of all φ j functons. We frstly ntroduce procedure Mult-Dm-Search, whch s used n dentfyng the cell contanng λ n each φ j va

5 268 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014 multdmensonal search [25]. We assume there exsts an oracle whch takes as nput any hyperplane equaton n R m, and outputs the poston of λ wth respect to ths equaton: Procedure Mult-Dm-Search (k, γ) Input: k hyperplane equatons n R m constant γ (0 <γ<1) Output: poston of λ wth respect to at least γk out of the gven k equatons 1: begn 2: repeat 3: after callng the oracle once: poston of λ wth respect to at least half of the hyperplanes n K can be dentfed 4: remove the hyperplanes whose poston to λ have been dentfed 5: untl poston of λ wth respect to at least γk out of gven k hyperplanes have been dentfed 6: end The man dea of Mult-Dm-Search s to repeatedly dentfy the locaton of λ wth respect to a certan proporton of the gven k hyperplanes that have not been vsted n prevous teratons. Wthn each teraton, the oracle s called once, and the poston of λ wth respect to at least half of the hyperplanes unvsted n prevous teratons can be dentfed. The procedure termnates when a gven threshold proporton (γ) of k hyperplanes locatons to λ has been dentfed. Through Mult-Dm-Search, we develop a mult-stage algorthm to dentfy the cells contanng λ wthn all φ j s. The key dea of the algorthm s to dentfy n each teraton the cells contanng λ n at least half of the canddate φ j s through ntellgent settng of γ j for each φ j. Snce the total number of p j functons at stage 1 s O(n), O(log n) stages are needed. Therefore, the algorthm can be fnshed n polynomal tme due to Mult-Dm-Search s polynomal complexty. After dentfyng the optmal λ for each problem (LB- LP2 j ), ts value can be substtuted nto (25) to calculate the optmal values of X j =(x j ). Correspondngly, the set of (x j ) can be used n (20) to calculate the optmal objectve value of (LB-LP2). Now we have a vald lower bound, φ(lp1) + φ(lb-lp2), to the orgnal problem (P). IV. HEURISTIC ALGORITHM FOR (P) In ths secton, we present a practcal polynomal-tme heurstc Mn-DC-LD (where LD stands for local doman). It employs two energy-savng strateges: 1) puttng dle servers to sleep wthn each DC; and 2) shuttng down dle DCs through load unbalancng among DCs n the same CDN doman. Mn-DC-LD has two stages. The frst stage performs localzed request resoluton and server sleepng wthn ndvdual DCs, where all requests are resolved to ther nearest avalable DC (subject to ther load capabltes). Afterwards, based on the frst stage s output, the second stage tres to redrect the ntally mapped requests to alternatve local-doman DCs whle subjectng to QoS constrants, so that opportuntes are created for some DCs to become dle and get shut down. We frstly ntroduce procedure Map-to-Best-DC, whch s called by both stages of Mn-DC-LD. Procedure Map-to-Best-DC(j, r j, max _dst) Algorthm 1 Identfyng cells contanng λ n all φ j s Input: φ =(φ j ): set of φ j functons (j =1,...,r s ) each φ j functon has p j hyperplane equatons Output: cells n each φ j that contan λ 1: begn 2: r s n // Intalzaton: all φ j s are put n φ 3: repeat 4: begn stage s 5: for all φ j φ do 6: γ j 1 1/(2p j ) 7: apply Mult-Dm-Search (p j,γ j ) to φ j 8: end for 9: for at least half of φ j s n φ: ther cells contanng λ are dentfed. 10: remove from φ: theφ j s whose cells contanng λ have been dentfed 11: r s φ 12: s s +1 13: untl the cells contanng λ are dentfed n all φ j s 14: end Input: j: PoP node r j : request volume at j max_dst: maxmum allowed dstance between a request s source PoP and ts desgnated DC Output: (x j ) Z m 0: request volume mapped from PoP j to each DC 1: begn 2: repeat through each DC n PoP j s local doman 3: dentfy DC that s unvsted and closest to j 4: f DC s avalable and the dstance between DC and PoP j<max_dst then 5: map r j to subject to s capacty, and denote mapped volume as x j 6: r j r j x j 7: end f 8: mark DC as vsted 9: untl r j == 0 10: end Bascally, procedure Map-to-Best-DC s a greedy heurstc that resolves r j requests at gven PoP node j to DCs whle subjectng to the followng crtera. Frstly, the desgnated DC

6 GE et al.: ENERGY MANAGEMENT IN CROSS-DOMAIN CONTENT DELIVERY NETWORKS 269 must be wthn local doman of PoP j, and ts dstance to j must be smaller than max _dst. Secondly, the desgnated DC s request-handlng capablty must not be volated. These crtera are honored to assure servce avalablty and end-toend delay durng DC shutdown operatons, whch effectvely avods deterorated QoS and unnecessary nter-doman content traffc. The procedure termnates when all r j requests have been resolved by one or more DCs, whch ensures 100% servce avalablty. Its runs n O(m) tme complexty. The reason for us to explctly specfy max _dst s that wthn a sngle CDN doman, the end-to-end delay between two nodes s approxmately lnear to ther physcal dstance n km [26]. Specfcally, an ntra-doman dstance of 300 and 500 km would lead to an end-to-end delay of around and 20 ms respectvely [10]. Dfferent web applcatons have varous requrements for latency. For example, most latency-senstve applcatons requre a latency of up to 30 ms [10], whle a delay of up to ms s specfed for typcal real-tme web servces [27]. These two values correspond to max _dst of about 800 km and 3000 km respectvely. Hence, the CDN operator wll be able to specfy dfferent max _dst values n our algorthm accordng to ther specfc latency requrements. The algorthm Mn-DC-LD s presented below. Algorthm 2 Mn-DC-LD Input: set of PoP nodes j, j =1,...,n r =(r j ) R n +: set of request volume at PoP j Output: X =(x j ) Z m n 0 : request volume mapped from each PoP j to each DC 1: begn // Stage 1 2: for all PoP node j (j =1,...,n) do 3: call Map-to-Best-DC(j, r j, max _dst) 4: end for // Stage 2 5: repeat through each DC ( =1,...,m) 6: dentfy DC that s lowest-utlzed and unvsted 7: f all requests currently mapped to DC can be redrected to alternatve DCs then 8: redrect requests and shut down DC 9: else keep DC actve 10: end f 11: mark DC as vsted 12: untl all DCs have been vsted 13: end In the frst stage, Mn-DC-LD terates through each PoP node j and resolve ts request volume r j to ts nearest avalable DCs by callng procedure Map-To-Best-DC (j, r j ). Ths stage produces localzed request resoluton between each PoP and each DC, whch runs n O(mn) tme (n teratons of callng Map-to-Best-DC that s O(m)). Then, n the second stage, the algorthm terates through each DC to check f all of ts currently-served requests can be redrected to alternatve DCs, whch s checked through applyng Map-to-Best-DC on each PoP s requests that are currently mapped to DC. If yes, then DC can be shut down after ts load has been completely redrected to alternatve DCs. It s worth notng that request redrecton durng ths stage wll not nterrupt ongong content delvery sessons, whch we wll cover n Secton V. The second stage runs n O(m 2 n) (m teratons at lne 5, O(m) for lne 6 and O(mn) for lne 7). Note that although we assumed each DC has homogeneous servers n Secton II, Mn-DC-LD does not requre such an assumpton. The reason s that Mn-DC-LD performs request resoluton at PoP-to-DC level only, whch does not requre specfc knowledge on each server s characterstc. Also, Mn- DC-LD requres future request volume at PoP nodes as nputs, whch needs to be accurately predcted based on hstorcal montorng results n a CDN. In practce, CDN workload normally follows a regular pattern [4], [28], [29], whch s commonly modeled wth queung theory [12], [15]. More smplfed approaches (lke movng average algorthm) were also employed n related work. In ths paper, we do not assume any specfc load predcton technque, and consder that request volume s already predcted and provded as nput. However, we plan to nvestgate and enhance Mn-DC-LD s performance under rregular traffc pattern, e.g., unexpected bursts n traffc volume n our future work. V. P RACTICALITY CONSIDERATIONS In ths secton, we dscuss some key practcalty ssues of our energy-aware scheme n modern CDN nfrastructures. A. Energy-Aware Request Resoluton and Redrecton In Secton I, we mentoned that strategc plannng of user-to- DC request resoluton s the key n balancng the energy-qos trade-off. In practce, a CDN nfrastructure s managed n a centralzed manner, and decsons on request resoluton from endusers to DCs are made by the request mappng system (RMS), whch s part of the CDN s central management platform. Tradtonally, many factors are consdered n the decson-makng process to ensure that users experence desred latency, whch nclude network condtons, server utlzaton and predcted user demand at PoP nodes [11]. After the RMS makes the decson on request resoluton, t ssues correspondng nstructons to DNS unts that are dstrbuted across ISP domans [11]. The above process s performed regularly, so that all DNS unts can always receve up-to-date user-to-dc mappng nformaton. Wth ths nformaton, when a user request reaches ts local DNS unt, t can be drected to ts desgnated DC for effcent content delvery. To realze energy-aware request resoluton, the RMS needs to take energy savng nto account durng the above decsonmakng process. Specfcally, nstead of followng the tradtonal QoS-orented approach, the RMS can execute the Mn-DC-LD algorthm to determne the desgnated DCs of each end-user s content request. As descrbed n Secton IV, our algorthm s able to make request resoluton decsons under the strategy of shuttng down as many DCs as possble whle respectng QoS constrants. Realzng energy awareness n the RMS also needs changes to be made to CDN nfrastructure,

7 270 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014 such as modfyng the RMS s components to execute our algorthm and deployng controllers to DCs to perform shutdown and startup operatons. Interested readers are referred to [30] for more detals. For each DC that s marked by the RMS elgble for shutdown, the followng actons need to be done by the RMS. Frstly, t needs to ensure that no upcomng new request s mapped to the DC. Secondly, regardng the ongong content delvery sessons on the DC, the operator has the optons of ether watng for them to fnsh or redrectng them to alternatve DCs. The RMS realzes these actons through ssung updated request resoluton nstructons (accordng to Mn-DC- LD s output) to dstrbuted DNS unts n the CDN beforehand. Through these actons, a DC wll have become fully dle before t s shut down, whch avods extra user-experenced delay ncurred by DC on/off state transton. Recall from Secton II that besdes DC shutdown, we also consder each DC to be able to automatcally put ts dle servers to the sleep mode. In ths crcumstance, some ongong content delvery sessons may need to redrected to alternatve servers wthn the same DC. Such redrecton can be acheved seamlessly by modern DC s nternal load balancng mechansm, whch nstructs alternatve servers to serve content requests on behalf of the server that s gong to sleep. Ths technque s wdely adopted n modern DCs [11]. B. Server Sleepng and DC Shutdown When savng CDN energy through server sleepng and DC shutdown, CDN operators often have the concerns over whether provsonng fewer actve servers and DCs wll deterorate CDN performance. Specfcally, the concerns are manly over the overhead/complexty of powerng on/off servers and DCs n terms of operatonal costs and the trade-off n QoS. Frstly, puttng dle servers n the sleep mode has become a common practce n energy effcent DCs, especally durng off-peak hours. Modern servers wth ndustry-standard energy effcency can be put to sleep or powered on wthn several mnutes [4]. However, there are a number of ssues that need attenton. Frstly, to cope wth sudden ncrease n user demand, t s common for DC operators to keep a small pool of dle servers actve n the DC. Furthermore, frequently turnng on/off a server should be avoded snce t can cause wear-and-tear effect on ts hardware, whch negatvely affects ts relablty. Snce CDN workload normally follows a regular pattern, t s relatvely easy to predct future user demand and accordngly plan the provsonng of actve servers. Secondly, although DC shutdown s a new strategy n CDN energy savng, t s not uncommon n the DC ndustry. For example, a DC operator can shut down a DC for mantenance purpose or durng power outage. In practce, there s a checklst that needs to be carefully followed when startng up or shuttng down a DC to avod any servce outage. Such a checklst s complcated and DC specfc, snce dfferent DCs have dfferent layout, hardware type and server vrtualzaton level etc. Typcally, t takes around 30 mnutes or less to fully shut down or start up a DC [31]. Ths makes DC shutdown sutable for energy-savng purpose. However, frequent DC shutdown operatons are not practcally feasble due to the tme duraton t takes. Furthermore, too-often on/off state transtons wll cause wear-and-tear on servers and other DC hardware components. Therefore, t s desred to lmt the number of DC shutdown operatons to e.g., maxmum once per day. Overall, both server sleepng and DC shutdown are practcal energy-savng technques n modern CDNs. Recall that normal CDN workload follows a regular pattern every 24 hours, whch enables a daly off-peak wndow for DC shutdown. Therefore, as long as the frequency and duraton of shutdown events are carefully orchestrated, and that future user request volume are accurately predcted, the overhead of server sleepng and DC shutdown operatons can be lmted to a low level. VI. PERFORMANCE EVALUATION In ths secton, we evaluate the energy-savng and QoS performances of Mn-DC-LD. Two reference schemes are used. Frstly, LB(P) s used to assess Mn-DC-LD s effcency (.e., close to optmal) n energy savng. Secondly, we use the scheme Mn-Dst as a benchmark to examne our scheme s energysavng gan and ts trade-off n QoS performance. Mn-Dst ams to save energy through server sleepng wthn DCs, but not va DC shutdown. Whle resolvng requests n a localzed manner for optmzed QoS, Mn-Dst reflects the state-of-theart n CDN energy savng [4]. To sut our needs of modelng DC servers and coolng system s energy consumpton, we developed a standalone CDN smulator wth Java and mplemented the three schemes above. A. Expermental Setup 1) CDN Topology: We use the real topologes from GEANT [32] and Internet2 [33] networks, whch are two nterconnected autonomous domans n Europe and U.S. respectvely. Altogether, there are 34 PoP nodes that are dstrbuted over Europe (25 nodes) and U.S. (9 nodes). We take each PoP s tme zone nto account Europe covers the tme zones from UTC to UTC+2, and U.S. covers UTC-8 to UTC-5. Overall, 9 DCs are deployed n the CDN (5 n GEANT and 4 n Internet2), whch s typcal consderng geographcal areas and populaton of the two domans [34]. Accordng to the conventonal CDN DC deployment strategy [14], these 9 DCs are deployed wthn proxmty to the 9 PoP nodes whose assocated ctes have the hghest local populatons. 2) Workload Informaton: We use the web traffc trace from ClarkNet (an ISP coverng Washngton DC metro area) [35], whch s very smlar to other traces that appeared n recent publcatons [4], [5], [10]. The trace covers 7 consecutve days and contans all content requests that orgnated from Washngton DC area durng the perod. Recall from Secton II that we only optmze energy consumpton of surrogate DCs, whch contan copes of popular content objects only. In the ClarkNet trace, 49.7% of the content objects are consdered as popular ones (havng been requested by more than once), whch contrbuted to 99.1% of the total request volume. Request volumes are averaged over dsjont 1-hour perods.

8 GE et al.: ENERGY MANAGEMENT IN CROSS-DOMAIN CONTENT DELIVERY NETWORKS 271 Fg day content traffc trace, derved from ClarkNet WWW server trace [35]. In our experments, the ClarkNet trace s approxmated and appled to all PoP nodes accordng to the followng approach. Frstly, we assume that end-users at all PoP nodes ntate content requests by followng the same pattern. Such an assumpton s based on observatons n the lterature [4], [36]. Afterwards, the number of requests at each PoP node s approxmated by applyng a PoP-specfc scalng factor to the request volume n the trace, whch s proportonal to each PoP s local populaton. Ths s necessary snce the populaton assocated to each PoP, whch affects ts request volume, vares substantally n realty [37]. The resulted traffc trace s shown n Fg. 1. Note that we approxmate user request volume n Europe and U.S. separately, where a tme zone dfference of 5 to 10 hours s present between them. As a result, ther peak and off-peak hours are experenced at dfferent tme everyday, whch needs to be taken nto account durng the jont energy optmzaton. 3) Expermental Scenaros: To reflect realstc CDN operatonal envronments, we consder the followng scenaros: Over-provsonng rato refers to the rato of total provsoned server capacty over requred capacty to handle user requests at peak hours. The ratos of 1 : 1 and 2 : 1 are consdered n our experments. A rato of 1 : 1 means that DCs n each doman have the servce capactes that exactly match peak user demand n that doman, whle a rato of 2 : 1 means that half of the servers n each doman are needed to handle ts peak user demand. PUE: Consderng the varaton n PUE values n the global DC ndustry [8], [18], we use the PUEs of 1.5, 2, and 3 n our experments. The scenaro wth PUE of 1.5 descrbes some exceptonally energy effcent DCs, and the PUEs of 2 and 3 should be able to reflect typcal DCs that are currently hostng CDN servers. Outdoor ar temperature: we consder the temperatures of 50 F(10 C) and 85 F (29.4 C), whch represent typcal ar temperatures n cold and hot areas respectvely. Fg. 2. Energy-savng performance gap between LB(P) and Mn-DC-LD (85 F). (a) 1 : 1 Over-Provsonng. (b) 2 : 1 Over-Provsonng. Through these settngs, we wll evaluate ar temperature s effect on our schemes energy-savng performance. 4) Parameters: The parameters values n the problem formulaton are specfed as follows. In (1), servce capablty Y of each DC s determned as below. Frstly, we calculate the peak aggregated request volume wthn each doman. Secondly, dfferent over-provsonng ratos are appled to calculate the total provsoned server capablty n each doman, whch s then evenly dstrbuted among ts DCs. Under 1 : 1 over-provsonng rato, each DC n Europe and US has the capablty of and request/s respectvely. In (3), P peak,svr slp and P,svr for each DC are respectvely set to be 92 Watts and 5 Watts multpled by the number of servers n DC [4]. Each DC s number of servers s calculated by dvdng ts servce capablty by 12 request/s (typcal capacty of modern servers [12], [15]). Under 1 : 1 over-provsonng rato, there are 2162 and 2728 servers wthn each DC n Europe and US respectvely. Such a fgure matches typcal CDN server deployment scenaro nowadays [4]. In (4), A, B, and C are determned by regresson equatons n [9] based on DC s ndoor and outdoor dry bulb temperature. Note that we do not consder dfferent ndoor temperature, snce t s normally set by e.g., ASHRAE recommendatons [38]. P peak,cool on (5). s calculated wth respect to P peak,svr B. Energy-Savng Performance through PUE based In ths subsecton, we evaluate the energy-savng performances of LB(P) and Mn-DC-LD, whch are compared aganst the reference scheme Mn-Dst. We frst evaluate Mn-DC-LD s effcency as a heurstc through assessng the gap between LB(P) s and ts results. As plotted n Fg. 2 wth 99.7-percentle results, the gaps between the two schemes are shown to be small n all scenaros. Specfcally, ther gap s always less than 14.3% and 21.7% under 1 : 1 and 2 : 1 over-provsonng settngs respectvely, whle the medan s up to 8.2% and 11.7% under the two settngs. Snce the optmal objectve of (P) s bound to be between LB(P) and Mn-DC-LD s results, the mplcatons of these results are two-fold. Frstly, LB(P) s suffcently close to ts optmal objectve value, whch means t can be used as an effectve benchmark to evaluate other algorthms performances. Secondly, Mn-DC-LD s energy-savng gan over

9 272 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014 Fg. 3. Overall DC energy consumpton per hour (1 : 1 over-provsonng, 85 F). (a) PUE =1.5. (b) PUE =2. (c) PUE =3. Fg. 4. Overall DC energy consumpton per hour (2 : 1 over-provsonng, 85 F). (a) PUE =1.5. (b) PUE =2. (c) PUE =3. Fg. 5. Mn-DC-LD s energy-savng gan over Mn-Dst (85 F). (a) 1 : 1 Over-Provsonng. (b) 2 : 1 Over-Provsonng. Mn-Dst s guaranteed to be near-optmal under all scenaros n our experments. Next, we evaluate Mn-DC-LD s energy-savng gan over Mn-Dst. The results of overall DC energy consumpton per hour are shown n Fgs. 3 and 4 respectvely, and Mn-DC-LD s 99.7-percentle energy-savng gan s plotted n Fg. 5. For now, we present the results under ar temperature of 85 F. Frstly, under 1 : 1 over-provsonng rato, Mn-DC-LD was able to acheve an energy-savng gan of up to 47.1% over Mn- Dst durng off-peak hours. However, durng peak hours, Mn- DC-LD s advantage over Mn-Dst was lmted. The reason s that that Mn-DC-LD s energy savng gan reles on shuttng down the dle DCs whose loads are shfted to alternatve DCs. However, durng peak hours, all DCs were heavly utlzed due to 1 : 1 over-provsonng. Hence, t was unlkely that any DC was able to become dle and get shut down, whch elmnated Mn-DC-LD s advantage over Mn-Dst. Ths s wthn our expectaton, because durng a CDN s peak hours, t s of hgher prorty to guarantee QoS n e.g., servce avalablty and endto-end delay, whch requres more actve servers and DCs to be provsoned. Secondly, under 2 : 1 over-provsonng rato, Mn-DC-LD was able to create more opportuntes of DC shutdown due to more spare server resources provsoned n the CDN. As a result, t produced sgnfcant energy-savng gans over Mn- Dst durng both peak and off-peak hours. Specfcally, the gans were 10.1% to 36% wth PUE of 1.5, and 28.6% to 62.1% wth PUE of 3. These results have shown Mn-DC-LD s capablty of savng DC energy through explotng spare server resources and shuttng down dle DCs. A common observaton under both scenaros above s that Mn-DC-LD s acheved a hgher energy-savng gan as PUE ncreased. Recall that the hgher PUE s, the hgher proporton that coolng system contrbutes to the overall DC energy consumpton. Therefore, the DC shutdown technque s able to save more energy on coolng systems when PUE s hgher (ndcatng poorer DC energy effcency). Such an observaton has mportant practcal mplcatons. For modern DCs wth average PUEs of 2 to 3, the technques of server sleepng and DC shutdown must be employed together to acheve

10 GE et al.: ENERGY MANAGEMENT IN CROSS-DOMAIN CONTENT DELIVERY NETWORKS 273 TABLE I NUMBER OF DCS THAT IS SHUT DOWN BY MIN-DC-LD DURING THE TRACE (PUE =2, 85 F) TABLE II REPRESENTATIVE SNAPSHOT SCENARIOS maxmum energy savng. However, for DCs whch have hgher energy effcency (wth PUE of 1.5 or lower), the beneft of DC shutdown depends on how much spare server resource s provsoned n that CDN. If a conservatve over-provsonng polcy s employed, the extra energy savng that DC shutdown can acheve over server sleepng wll be lmted. We further examne the number of DCs that Mn-DC-LD s able to shut down wth respect to dfferent request volume durng the trace, whch s shown n Table I. Under 1 : 1 overprovsonng settng, for 48.5% of the tme durng the 7-day trace, Mn-DC-LD was able to shut down at least four DCs. Moreover, for 99.5% of the tme, at least one DC was shut down. Under 2 : 1 over-provsonng rato, Mn-DC-LD was able to shut down at least fve DCs for 94.8% of the tme. These results complement our earler nference that gven the same request volume, the more spare server resources that s provsoned n a CDN (.e., hgher over-provsonng rato), the hgher energy-savng gan that Mn-DC-LD wll be able to acheve. Note that PUE does not affect the results here, snce DC shutdown decsons are determned by request resoluton, whch does not take coolng system nto account. So far, we have presented the results under outsde ar temperature of 85 F. The results under 50 F are very close to the ones under 85 F wth a less-than 1% gap. Therefore, we omt the results under 50 F due to pagnaton lmt. C. QoS Performance In ths subsecton, we evaluate the QoS performance of Mn- DC-LD wth respect to Mn-Dst as a reference scheme. We use two metrcs: 1) end-to-end delay experenced by content requests; and 2) server response tme n DCs. 1) End-to-End Delay: Under both Mn-DC-LD and Mn- Dst, all requests are resolved wthn the local doman as ther source PoP nodes. Therefore, a request s end-to-end delay can be approxmated to be proportonal to the network dstance that t travels between ts source PoP and ts desgnated DC [26]. In ths paper, we use IGP lnk weght to represent network dstance snce n GEANT and Internet2 networks, each lnk s weght s lnearly proportonal to ts actual end-to-end delay [32], [33]. Frst, Mn-Dst can acheve optmal end-to-end delay snce t resolves all content requests n a localzed manner [4]. Second, Mn-DC-LD ncurs extra network dstance traveled by requests, as t redrects a subset of requests to alternatve local-doman DCs to save more energy. To evaluate Mn-DC-LD s trade-off n network dstance, we pcked four snapshots n the trace (shown n Table II) that are representatve n user actvty varaton at each doman. For now, we do not lmt the maxmum network dstance traveled by requests n Mn-DC-LD. Wthn each scenaro, we statstcally analyzed and compared the network dstances traveled by each request under Mn-DC-LD and Mn-Dst respectvely. The 99.7-percentle results are shown n Fgs. 6 and 7 under 1 : 1 and 2 : 1 over-provsonng settngs respectvely. Note that we gnore the outlers n Fgs. 6 and 7 snce they represent less than 0.001% of all requests. It s observed from Fgs. 6 and 7 that Mn-DC-LD s network dstance results were up to 2.4 and 8 tmes of Mn-Dst s under 1 : 1 and 2 : 1 over-provsonng settngs respectvely. The gap between the two schemes results became larger as more domans were n off-peak hours, snce durng off-peak hours, more dle servers exst and more opportuntes for request concentraton and DC shutdown are created. As a result, more request redrecton takes place for load unbalancng, whch leads to hgher network dstances traveled. Moreover, hgher over-provsonng rato also mples more dle servers, whch explans the observatons here. We now quanttatvely evaluate Mn-DC-LD s end-to-end delay performance. Recall from earler ths secton that a lnk s network dstance (or weght) s lnearly proportonal to ts latency. Based on [10], an ntra-doman lnk wth weght of 300 or 500 has the latency of ms or 20 ms respectvely. Hence, we constructed a lnear relatonshp between a network path s dstance and ts end-to-end delay wth these data, whch was used to estmate the delay experenced by requests. Under 1 : 1 over-provsonng settng, the hghest end-toend delay under Mn-DC-LD was 70 ms, whch took place n scenaro #3 (both off-peak hours). In scenaros #1 and #2, Mn- Dst and Mn-DC-LD produced delays of 25 ms and 45 ms respectvely. Ther delays were both around 50 ms n scenaro #4. Under 2 : 1 over-provsonng settng, Mn-Dst s performance was mproved due to more server resources avalable. In contrast, Mn-DC-LD produced poorer end-to-end delay due to more request redrecton, whose values were 125 ms, 67 ms, 160 ms and 60 ms n scenaros #1 to #4. The results above have practcal mplcatons. Frstly, when at least one doman was not n off-peak hours, Mn-DC-LD s delay performance met the need of typcal real-tme web applcatons ( ms [27]). Secondly, when all domans were n off-peak hours (scenaro #3), although aggressve energy savng through DC shutdown obtaned near-optmal energy reducton,

11 274 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014 Fg. 6. Comparson on network dstance traveled by content requests (1 : 1 over-provsonng, PUE=2,85 F). (a) Snapshot #1. (b) Snapshot #2. (c) Snapshot #3. (d) Snapshot #4. Fg. 7. Comparson on network dstance traveled by content requests (2 : 1 over-provsonng, PUE =2,85 F).(a) Snapshot #1. (b) Snapshot #2. (c) Snapshot #3. (d) Snapshot #4. Fg. 8. Trade-off between Mn-DC-LD s energy-savng gan and QoS performance, ncludng end-to-end delay and server response tme (2 : 1 over-provsonng, PUE =2,85 F). (a) Trade-off: Max Net. Dstance vs Energy Savng. (b) Trade-off: Server Utl. vs Energy Savng. (c) Server Response Tme. the resultng QoS barely met 150 ms. Therefore, under these crcumstances, energy savng wll need to be compromsed so that the CDN s delay performance can meet requrements by CDN-hosted web applcatons. To quantfy such trade-off, we performed experments on Mn-DC-LD by specfyng a range of max _dst parameters and assessed the respectve trade-off n ts energy-savng gan. The results are shown n Fg. 8(a). Accordng to Fg. 8(a), to guarantee an end-to-end delay of 30 ms for latency-senstve web applcatons (where max _dst s set to 800), Mn-DC-LD s maxmum energy-savng gan was reduced from 51.1% to 20%. However, by lmtng max _dst to 3750 for an up-to-150 ms latency, Mn-DC-LD was able to acheve near-optmal energy savng. The results n Fg. 8(a) provde practcal gudelnes on balancng energy-qos tradeoff wth respect to latency requrements of dfferent web applcatons. 2) Server Response Tme: Typcally, the server behavor n a DC can be modeled by an M/M/1 queue where request arrval follows Posson dstrbuton [12], [39]. Hence, the average server response tme can be calculated by Lttle s Law of 1/μ(1 ρ), where μ and ρ are the server s servce rate and utlzaton respectvely. Recall from our expermental setup that each server has a servce rate of 12 req/s. Under both Mn-DC-LD and Mn-Dst, to maxmze server energy savng, each DC concentrates ts load to the fewest actve servers and put the remanng ones to sleep. Therefore, all servers wll be utlzed to the greatest extent that s allowed by the DC operator. On one hand, a server s response tme s negatvely correlated to ts utlzaton. On the other hand, hgher server utlzaton mples more energy savng gan, snce DC loads can be handled by fewer servers. Hence, we nvestgate the trade-off between allowed server utlzaton and energy

12 GE et al.: ENERGY MANAGEMENT IN CROSS-DOMAIN CONTENT DELIVERY NETWORKS 275 TABLE III REQUEST RESOLUTION DISTRIBUTION AMONG DATA CENTERS (2:1 OVER-PROVISIONING, PUE =2,85 F) savng gan n ths part. Due to pagnaton lmts, we only dscuss the scenaro of 2 : 1 over-provsonng and PUE =2. Frstly, through settng dfferent values of maxmum server utlzaton n our smulator, we obtan the 99.7-percentle results on energy savng gans as n Fg. 8(b). Note that settng maxmum server utlzaton to 1.0 gves the upper bound on energy savng gan, whch s hard to acheve n practce snce t would cause poor response tme. Secondly, we calculate server response tme wth a set of typcal server utlzaton values between 0.5 and 0.95 [12], whch s shown n Fg. 8(c). Note that we normalze the results wth respect to the response tme gven by 85% server utlzaton. The reason s that we observed n the smulaton that under 2 : 1 over-provsonng setup and Mn-Dst scheme, the peak DC utlzaton s 60% n Europe and 85% n U.S. Hence, a server utlzaton of 85% leads to the worst-case (.e., upper bound) response tme wthout server sleepng n our smulaton scenaro. It s drectly observed from Fg. 8(b) and (c) that at 85% server utlzaton lmt, the maxmum energy savng gan s 8.5% less than the upper bound. The trade-off n medan energy savng gan s 6.2%. If we set a hgher server utlzaton lmt (e.g., 95%), despte the response tme becomng 3 tmes hgher than under 85%, there would be hardly any ncrease n medan energy savng gan. In contrast, f we set the server utlzaton lmt to 70%, we can reduce the response tme by half wthout sacrfcng any energy savng performance. Furthermore, lmtng server utlzaton to any pont below 70% wll not lead to sgnfcantly-better response tme, but t wll cost a consderable amount of energy savng. D. Request Resoluton Strategy In ths subsecton, wth respect to the four snapshot scenaros mentoned above, we examne more detals on how content requests were resolved to DCs n each doman to realze the DC shutdown, whch wll help us better understand the tradeoff between energy savng and QoS performance. The results under 2 : 1 over-provsonng settng are shown n Table III. Frstly, t can be nferred that LB(P) s strategy was to concentrate request resoluton to as few DCs as possble. Moreover, t preferred mappng requests to DCs wth lower load capablty frst (European DCs n ths study). In scenaros #1, #3, and #4, no actve DC was provsoned n US. Such strateges contrbuted to LB(P) s advantage n energy-savng performance. However, ts dsadvantages are ntutve as well - requests from PoP nodes n US had be resolved to DCs n Europe, whch would lead to deterorated end-to-end QoS. Mn-DC-LD s total number of actve DCs was smlar to LB(P) s. Wthn each doman, Mn-DC-LD always provsoned as few number of DCs as possble wth respect to ts present request volume. In other words, when a doman s n off-peak hour, t wll stll have at least one actve DC. Although such a strategy ntroduces gap between LB(P) s and ts energy-savng gan, t has mportant practcal mplcatons. Frstly, unlke LB(P) s strategy, t enables all requests to be resolved wthn local doman, whch s crucal n assurng QoS performance. Secondly, n case a spke n request volume occurs, t would be useful for a doman to have at least one actve DC to absorb the load surge whle more DCs are beng powered up. VII. RELATED WORK Improvng CDN content delvery effcency s a tradtonal research topc, whch nvolves strategc management of request resoluton [12], [15], [34] and content cache placement [28], [40]. Snce a few years ago, the topc of CDN s energy effcency (especally on servers and DCs) started to emerge. Regardng savng energy n ndvdual DCs, there has been a sgnfcant number of research work [3], [19], [39], [41], on whch a comprehensve survey s avalable n [42]. In ths secton, we focus on the research that s related to savng energy costs among multple DCs n a CDN. So far, most work on CDN energy savng focused on the server part of DCs. Exstng schemes can be generally dvded nto two categores based on ther defntons of the term energy cost n Dollars or Joules/kWh. In both cases, energy cost reducton s acheved through strategc user-to-dc request resoluton under dfferent polces, so that utlzaton among multple DCs are coordnated towards the objectve of overall

13 276 IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT, VOL. 11, NO. 3, SEPTEMBER 2014 energy reducton. Our work falls nto the second category as we optmze energy n kwh. In the frst category, Quresh et al. [26] proposed that CDN energy bll can be reduced through explotng the dversty n electrcty prces at dfferent geographcal areas. The dea was to resolve requests to DC stes where local electrcty prces are cheaper, so that lower energy bll s ncurred under the same amount of DC load. A smlar approach as [26] was employed by Rao et al. n [43]. Later, a dstrbuted verson of the above strategy was developed by Xu et al. In [44], Lu et al. consdered the possblty of usng green renewable energy n some areas when performng request resoluton. We dd not compare our scheme aganst these work, snce a cheaper electrcty bll does not necessarly ndcate less energy consumpton and vce versa. In the second category, Islam et al. [45] dscussed the strategy of usng less surrogate servers to resolve user requests n a CDN. Charavglo et al. [46] developed a formulaton that jontly optmzes energy consumpton of CDN servers and ISP networkng elements. However, such a scheme requres full nformaton sharng between CDN operator and ISP, whch s usually not realstc. Mathew et al. [4] developed offlne and onlne algorthms to maxmze CDN server energy reducton, whch consdered servce avalablty and wear-and-tear on hardware caused by on/off state transtons. The algorthm n [4] was mplemented n our experments as the reference scheme Mn-Dst. Gao et al. [5] developed exact and approxmaton algorthms to optmze the trade-off between CO 2 emsson footprnt and QoS n terms of latency. In these schemes, only servers energy consumpton was optmzed n these schemes. In contrast, our scheme optmzed both servers and coolng systems energy smultaneously. Furthermore, our work explctly consdered cross-doman CDNs, whch made our study on energy-qos trade-off more realstc as modern CDNs often covers multple ISP domans. Mathew et al. [10] studed the strategy of shuttng down entre server clusters n DCs to save overall (server and coolng) energy costs, whch stated that a cluster s coolng energy consumpton can be elmnated when t s shut down. However, exstng server sleepng schemes were not taken nto account. In contrast, our work exploted both optons of server sleepng and DC shutdown, whch created more energy-savng opportuntes. Furthermore, whle only U.S. manland was consdered n [10], we studed cross-doman CDNs through consderng tme-zone dfference and nter-doman traffc reducton, whch had more practcal mplcatons. VIII. CONCLUSION In ths paper, we nvestgated the jont optmzaton problem of cross-doman CDN s overall energy consumpton, whch ncluded both server nfrastructures and coolng systems among dstrbuted DCs. We employed both strateges of server sleepng and DC shutdown, whch were realzed by our heurstc algorthm Mn-DC-LD through concentratng request resoluton to fewer DCs. QoS constrants were enforced n Mn-DC-LD to ensure 100% servce avalablty and that t meets end-toend delay requrements. Through smulatons under realstc scenaros, Mn-DC-LD was able to acheve an energy-savng gan of up to 62.1% over Mn-Dst (a server-sleepng-only scheme). These results were guaranteed to be near-optmal through our derved lower bound. We also studed the tradeoff between energy savng gan and 1) end-to-end delay and 2) response tme respectvely, and showed that our scheme can balance the energy-qos trade-off n a flexble manner. Consderng our algorthm s practcalty and low complexty, t s practcal to ntegrate our scheme n modern CDN s request mappng system to realze energy-aware request resoluton. REFERENCES [1] H. Yn, X. Lu, G. Mn, and C. Ln, Content delvery networks: A brdge between emergng applcatons and future IP networks, IEEE Netw., vol. 24, no. 4, pp , Jul./Aug [2] J. Koomey. (2011). Growth n Data Center Electrcty Use 2005 to [Onlne]. Avalable: [3] G. Chen et al., Energy-aware server provsonng and load dspatchng for connecton-ntensve nternet servces, n Proc. USENIX NSDI, Berkeley, CA, USA, 2008, pp [4] V. Mathew, R. K. Staraman, and P. Shenoy, Energy-aware load balancng n content delvery networks, n Proc. IEEE INFOCOM, Orlando, FL, USA, Mar. 2012, pp [5] P. X. Gao, A. R. Curts, B. Wong, and S. Keshav, It s not easy beng green, n Proc. SIGCOMM, 2012, pp [6] Y. Chen et al., Integrated management of applcaton performance, power and coolng n data centers, n Proc. IEEE NOMS, Apr. 2010, pp [7] R. Das, S. Yarlank, H. Hamann, J. Kephart, and V. Lopez, A unfed approach to coordnated energy-management n data centers, n Proc. CNSM, Oct. 2011, pp [8] M. Stansberry and J. Kudrtzk, Uptme Insttute 2012 data center ndustry survey, Uptme Inst., New York, NY, USA. [Onlne]. Avalable: Data_Industry_Survey.pdf [9] Nonresdental Alternatve Calculaton Method Approval Manual for the 2008 Buldng Energy Effcency Standards, Calforna Energy Commsson, Sacramento, CA, USA, Dec [10] V. Mathew, R. K. Staraman, and P. Shenoy, Energy-effcent content delvery networks usng cluster shutdown, n Proc. 4th IGCC, 2013, pp [11] E. Nygren, R. K. Staraman, and J. Sun, The Akama network: A platform for hgh-performance nternet applcatons, SIGOPS Oper. Syst. Rev., vol. 44, no. 3, pp. 2 19, Jul [12] T. Wu and D. Starobnsk, A comparatve analyss of server selecton n content replcaton networks, IEEE/ACM Trans. Netw., vol. 16, no. 6, pp , Dec [13] M. Busar and C. Wllamson, ProWGen: A synthetc workload generaton tool for smulaton evaluaton of web proxy caches, Comput. Netw., vol. 38, no. 6, pp , Apr [14] R. Krshnan et al., Movng beyond end-to-end path nformaton to optmze CDN performance, n Proc. IMC, 2009, pp [15] S. Manfred, F. Olvero, and S. Romano, A dstrbuted control law for load balancng n content delvery networks, IEEE/ACM Trans. Netw., vol. 21, no. 1, pp , Feb [16] G. Rodolaks, S. Sachalou, and L. Georgads, Replcated server placement wth QoS constrants, IEEE Trans. Parallel Dstrb. Syst., vol. 17, no. 10, pp , Oct [17] S. Pelley, D. Mesner, T. F. Wensch, and J. W. VanGlder, Understandng and abstractng total data center power, n Proc. WEED, Jun. 2009, pp [18] Dgtal Realty Trust, Colocaton and Data Center Case Studes, San Francsco, CA, USA, Jan [Onlne]. Avalable: [19] Z. Abbas, G. Varsamopoulos, and S. K. S. Gupta, Tacoma: Server and workload management n nternet data centers consderng coolngcomputng power trade-off and energy proportonalty, ACM Trans. Archt. Code Optm., vol. 9, no. 2, pp. 11:1 11:37, Jun [20] K. G. Murty and S. N. Kabad, Some NP-complete problems n quadratc and nonlnear programmng, Math. Programm., vol. 39, no. 2, pp , Nov [21] H. Kellerer, U. Pferschy, and D. Psnger, Knapsack Problems. Berln, Germany: Sprnger-Verlag, 2004.

14 GE et al.: ENERGY MANAGEMENT IN CROSS-DOMAIN CONTENT DELIVERY NETWORKS 277 [22] N. Megddo and A. Tamr, Lnear tme algorthms for some separable quadratc programmng problems, Oper. Res. Lett., vol. 13, no. 4, pp , May [23] T. Ibarak and N. Katoh, Resource Allocaton Problems: Algorthmc Approaches. Cambrdge, MA, USA: MIT Press, [24] Z.-Q. Luo and W. Yu, An ntroducton to convex optmzaton for communcatons and sgnal processng, IEEE J. Sel. Areas Commun., vol.24, no. 8, pp , Aug [25] N. Megddo, Lnear programmng n lnear tme when the dmenson s fxed, J. ACM, vol. 31, no. 1, pp , Jan [26] A. Quresh, R. Weber, H. Balakrshnan, J. Guttag, and B. Maggs, Cuttng the electrc bll for nternet-scale systems, SIGCOMM Comput. Commun. Rev., vol. 39, no. 4, pp , Oct [27] M. Chen, M. Ponec, S. Sengupta, J. L, and P. A. Chou, Utlty maxmzaton n peer-to-peer systems wth applcatons to vdeo conferencng, IEEE/ACM Trans. Netw., vol. 20, no. 6, pp , Dec [28] L. Qu, V. Padmanabhan, and G. Voelker, On the placement of web server replcas, n Proc. IEEE INFOCOM, 2001, vol. 3, pp [29] P. Gll, M. Arltt, Z. L, and A. Mahant, Youtube traffc characterzaton: A vew from the edge, n Proc. ACM IMC, 2007, pp [30] C. Ge, N. Wang, and Z. Sun, Energy-aware data center management n cross-doman content delvery networks, n Proc. IEEE Onlne Conf. GreenCom, Oct. 2013, pp [31] A Successful Data Center Mgraton, Infosys, Bangalore, Inda, [32] Geant Project Home. [Onlne]. Avalable: [33] The Internet2 Network. [Onlne]. Avalable: [34] J. M. Almeda, D. L. Eager, M. K. Vernon, and S. J. Wrght, Mnmzng delvery cost n scalable streamng content dstrbuton systems, IEEE Trans. Multmeda, vol. 6, no. 2, pp , Apr [35] Traces n the Internet Traffc Archve. [Onlne]. Avalable: gov/html/traces.html [36] L. Cherkasova and M. Gupta, Analyss of enterprse meda server workloads: Access patterns, localty, content evoluton, and rates of change, IEEE/ACM Trans. Netw., vol. 12, no. 5, pp , Oct [37] N. Kamyama, T. Mor, R. Kawahara, S. Harada, and H. Hasegawa, ISPoperated CDN, n Proc. IEEE INFOCOM Workshops, Ro de Janero, Brazl, 2009, pp [38] Thermal Gudelnes for Data Processng Envronments, Amercan Socety Heatng, Refrgeratng Ar-Condtonng Engneers, Inc., Atlanta, GA, USA, [39] M. Ln, A. Werman, L. Andrew, and E. Thereska, Dynamc rght-szng for power-proportonal data centers, IEEE/ACM Trans. Netw., vol. 21, no. 5, pp , Oct [40] T. Bektas, J.-F. Cordeau, E. Erkut, and G. Laporte, Exact algorthms for the jont object placement and request routng problem n content dstrbuton networks, Comput. Oper. Res., vol. 35, no. 12, pp , Dec [41] R. Urgaonkar, U. Kozat, K. Igarash, and M. Neely, Dynamc resource allocaton and power management n vrtualzed data centers, n Proc. IEEE NOMS, 2010, pp [42] C. Ge, Z. Sun, and N. Wang, A survey of power-savng technques on data centers and content delvery networks, IEEE Commun. Surveys Tuts., vol. 15, no. 3, pp , [43] L. Rao, X. Lu, L. Xe, and W. Lu, Mnmzng electrcty cost: Optmzaton of dstrbuted nternet data centers n a mult-electrcty-market envronment, n Proc. IEEE INFOCOM, 2010, pp [44] Z. Lu, M. Ln, A. Werman, S. H. Low, and L. L. Andrew, Greenng geographcal load balancng, n Proc. ACM SIGMETRICS, 2011, pp [45] S. ul Islam and J.-M. Person, Evaluatng energy consumpton n CDN servers, n ICT as Key Technology aganst Global Warmng. Berln, Germany: Sprnger-Verlag, 2012, pp [46] L. Charavglo and I. Matta, Greencoop: Cooperatve green routng wth energy-effcent servers, n Proc. ACM e-energy, 2010, pp Chang Ge (S 11) receved the B.Eng. (Honours) degree n telecommuncaton engneerng from Queen Mary, Unversty of London, n 2009, and the M.Sc. degree n electronc engneerng from the Unversty of Surrey, U.K. n He s currently workng toward the Ph.D. degree at the Centre for Communcaton Systems Research, Unversty of Surrey. Hs research nterests nclude energy effcency n data centers and content delvery networks, as well as cache and energy management n next-generaton Internet. Zhl Sun (M 99) receved the B.Sc. degree n mathematcs from Nanjng Unversty, Nanjng, Chna, n 1982 and the Ph.D. degree n computer scence from Lancaster Unversty, Lancaster, U.K., n He s currently the Char of Communcaton Networkng and has been wth the Centre for Communcaton Systems Research, Unversty of Surrey, Surrey, U.K., snce He was a Postdoctoral Research Fellow wth Queen Mary Unversty of London, London, U.K., from 1989 to He has been a Prncpal Investgator and a Techncal Cocoordnator on many projects wthn the EU Framework Programs, ESA, EPSRC, and ndustres. He has publshed more than 125 papers n nternatonal journals, book chapters, and conferences. He has publshed a book, as the sole author, enttled Satellte Networkng: Prncples and Protocols (Wley, 2005), a book, as a contrbutng edtor, enttled IP Networkng Over Next Generaton Satellte Systems (Sprnger, 2008), and another book, as a contrbutng edtor to the ffth edton of the textbook, enttled Satellte Communcatons Systems: Systems, Technques and Technology (Wley, 2009). Hs research nterests nclude wreless and sensor networks, satellte communcatons, moble operatng systems, traffc engneerng, Internet protocols and archtecture, qualty of servce, multcast, and securty. Nng Wang (M 12) receved the B.Eng. (honours) degree from the Changchun Unversty of Scence and Technology, Changchun, Chna, n 1996, the M.Eng. degree from Nanyang Unversty, Sngapore, n 2000, and the Ph.D. degree from the Unversty of Surrey, Surrey, U.K., n 2004, respectvely. He s also a Senor Lecturer at the Centre for Communcaton Systems Research, Unversty of Surrey, Surrey, U.K. Hs research nterests manly nclude energy-effcent networks, network resource management, nformaton-centrc networkng and QoS mechansms. Ke Xu (SM 09) receved the Ph.D. degree from the Department of Computer Scence, Tsnghua Unversty, Bejng, Chna, n Currently, he s a Full Professor wth the Department of Computer Scence, Tsnghua Unversty. He has publshed more than 100 techncal papers and holds 20 patents n the research areas of next generaton Internet, P2P systems, Internet of Thngs (IoT), network vrtualzaton and optmzaton. He s a member of ACM. He has guest edted several specal ssues n IEEE and Sprnger journals. Jnsong Wu (SM 11) s the Founder and Foundng Char of the Techncal Commttee on Green Communcatons and Computng (TCGCC), IEEE Communcatons Socety (establshed as Techncal Subcommttee on Green Communcatons and Computng (TSCGCC) n 2011, elevated to TCGCC n 2013). He s an Assocate Edtor of IEEE Communcatons Surveys & Tutorals, IEEE Systems Journal, and IEEE Access, and the Founder and a Seres Edtor of the IEEE Seres on Green Communcatons and Computng Networks for IEEE Communcatons Magazne. He has been a Guest Edtor for IEEE Communcatons Magazne, IEEE Systems Journal, IEEE Access, IEEE TRANSACTIONS ON CLOUD COMPUTING, and Elsever Computer Networks. He was the leadng Edtor of the comprehensve book Green Communcatons: Theoretcal Fundamentals, Algorthms, and Applcatons (CRC, 2012).