Temperature Aware Workload Management in Geo-distributed Datacenters

Transcription

1 Temperature Aware Workload Management n Geo-dstrbuted Datacenters Hong Xu, Chen Feng, Baochun L Department of Electrcal and Computer Engneerng Unversty of Toronto ABSTRACT For geo-dstrbuted datacenters, lately a workload management approach that routes user requests to locatons wth cheaper and cleaner electrcty has been shown promsng n reducng the energy cost. We consder two key aspects that have not been explored before. Frst, the energy-gobblng coolng systems are often modeled wth a locaton-ndependent effcency factor. Yet, through emprcal studes, we fnd that ther actual energy effcency depends drectly on the ambent temperature, whch exhbts a sgnfcant degree of geographcal dversty. Temperature dversty can be used to reduce the overall coolng energy overhead. Second, datacenters run not only nteractve workloads drven by user requests, but also delay tolerant batch workloads at the backend. The elastc nature of batch workloads can be exploted to further reduce the energy consumpton. In ths paper, we propose to make workload management for geo-dstrbuted datacenters temperature aware. We formulate the problem as a jont optmzaton of request routng for nteractve workloads and capacty allocaton for batch workloads. We develop a dstrbuted algorthm based on an m-block alternatng drecton method of multplers (ADMM) algorthm that extends the classcal 2-block algorthm. We prove the convergence of our algorthm under general assumptons. Through trace-drven smulatons wth real-world electrcty prces, hstorcal temperature data, and an emprcal coolng effcency model, we fnd that our approach s consstently capable of delverng a 15% 2% coolng energy reducton, and a 5% 2% overall cost reducton for geo-dstrbuted clouds. 1. INTRODUCTION Geo-dstrbuted datacenters operated by organzatons such as Google and Amazon are the powerhouses behnd many Internet-scale servces. They are deployed across the Internet to provde better latency and redundancy. These datacenters run hundreds of thousands of servers, consume megawatts of power wth massve carbon footprnt, and ncur electrcty blls of mllons of dollars [17,34]. Thus, the topc of reducng ther energy consumpton and cost has receved sgnfcant attenton [7, 11 13, 15, 17, 19, 26 29, 34, 35, 4]. Energy consumpton of ndvdual datacenters can be reduced wth more energy effcent hardware and ntegrated thermal management [7, 11, 15, 28, 4]. Recently, mportant progress has been made on a new workload management approach that nstead focuses on the overall energy cost of geodstrbuted datacenters. It explots the geographcal dversty of electrcty prces by optmzng the request routng algorthm to route user requests to locatons wth cheaper and cleaner electrcty [12, 17, 18, 26, 27, 29, 34, 35]. In ths paper, we consder two key aspects of geo-dstrbuted datacenters that have not been explored n the lterature. Frst, coolng systems, whch consume 3% to 5% of the total energy [33, 4], are often modeled wth a constant and locaton-ndependent energy effcency factor n exstng efforts. Ths tends to be an over-smplfcaton n realty. Through our study of a state-of-the-art producton coolng system (Sec. 2), we fnd that temperature has drect and profound mpact on coolng energy effcency. Ths s especally true wth outsde ar coolng technology, whch has seen ncreasng adopton n msson-crtcal datacenters [1 3]. As we wll show, ts partal PUE (power usage effectveness), defned as the sum of server power and coolng overhead dvded by server power, vares from 1.3 to 1.5 when temperature drops from 35 C (9 F) to -3.9 C (25 F). Through an extensve emprcal analyss of daly and hourly clmate data for 13 Google datacenters, we further fnd that temperature vares sgnfcantly across both tme and locaton, whch s ntutve to understand. These observatons suggest that datacenters at dfferent locatons have dstnct and tme-varyng coolng energy effcency. Ths establshes a strong case for makng workload management temperature aware, where such temperature dversty can be used along wth prce dversty n makng request routng decsons to reduce the overall coolng energy overhead for geodstrbuted datacenters. Second, energy consumpton comes not only from nteractve workloads drven by user requests, but also from delay tolerant batch workloads, such as ndexng and data mnng jobs, that run at the back-end. Exstng efforts focus manly on request routng to mnmze the energy cost of nteractve workloads, whch s only a part of the entre pcture. Such a mxed nature of datacenter workloads, verfed by measurement studes [36], provdes more opportuntes to utlze the 1 USENIX Assocaton 1th Internatonal Conference on Autonomc Computng (ICAC 13) 33

2 cost dversty of energy. The key observaton s that batch workloads are elastc to resource allocatons, whereas nteractve workloads are hghly senstve to latency and have more profound mpact on revenue [25]. Thus at tmes when one locaton s comparatvely cost effcent (n terms of dollar per unt energy), we can ncrease the capacty for nteractve workloads by reducng the resources for batch jobs. More requests can then be routed to and processed at ths locaton, and the cost savng can be more substantal. We thus advocate a holstc workload management approach, where capacty allocaton between nteractve and batch workloads s dynamcally optmzed wth request routng. Dynamc capacty allocaton s also techncally feasble because jobs run on hghly scalable systems such as MapReduce. Towards temperature aware workload management, we propose a general framework to capture the mportant tradeoffs nvolved (Sec. 3). We model both energy cost and utlty loss, whch correspond to performance-related revenue reducton. We develop an emprcal coolng effcency model based on a producton system. The problem s formulated as a jont optmzaton of request routng and capacty allocaton. The techncal challenge s then to develop a dstrbuted algorthm to solve the large-scale optmzaton wth tens of mllons of varables for a producton geo-dstrbuted cloud. Dual decomposton wth subgradent methods are often used to develop dstrbuted optmzaton algorthms. However they requre delcate adjustments of step szes that make convergence dffcult to acheve for large-scale problems. The method of multplers [22] acheves fast convergence, at the cost of tght couplng among varables. We rely on the alternatng drecton method of multplers (ADMM), a smple yet powerful algorthm that blends the advantages of the two approaches. ADMM recently has found practcal use n many large-scale dstrbuted convex optmzaton problems n machne learnng and data mnng [1]. It works for problems whose objectve and varables can be dvded nto two dsjont parts. It alternatvely optmzes part of the objectve wth one block of varables to teratvely reach the optmum. Our formulaton has three blocks of varables, yet lttle s known about the convergence of m-block (m 3) ADMM algorthms, wth two exceptons [2, 23] very recently. [2] establshes the convergence of m-block ADMM for strongly convex objectve functons, but not lnear convergence; [23] shows the lnear convergence of m-block ADMM under the assumpton that the relaton matrx s full column rank, whch s, however, not the case n our formaton. Ths motvates us to refne the framework n [23] so that t can be appled to our setup. In partcular, n Sec. 4 we show that by replacng the fullrank assumpton wth some mld assumptons on the objectve functons, we are not only able to obtan the same convergence and rate of convergence result, but also to smplfy the proof of [23]. The m-block ADMM algorthm s general and can be appled n other problem domans. For our case, we further develop a dstrbuted algorthm n Sec. 5, whch s amenable to a parallel mplementaton n datacenters. We conduct extensve trace-drven smulatons wth realworld electrcty prces, hstorcal temperature data, and an emprcal coolng effcency model to realstcally assess the potental of our approach (Sec. 6). We fnd that temperature aware workload management s consstently able to delver a 15% 2% coolng energy reducton and a 5% 2% overall cost reducton for geo-dstrbuted datacenters. The dstrbuted ADMM algorthm converges quckly wthn 7 teratons, whle a dual decomposton approach wth subgradent methods fals to converge wthn 2 teratons. We thus beleve our algorthm s practcal for large-scale realworld problems. 2. BACKGROUND AND MOTIVATION Before we make a case for temperature aware workload management, t s necessary to ntroduce some background of datacenter coolng, and emprcally assess the geographcal dversty of temperature. 2.1 Datacenter Coolng Datacenter coolng s provded by the computer room ar condtoners (CRACs) placed on the rased floor of the faclty. Hot ar exhausted from server racks travels through a coolng col n the CRACs. Heat s often extracted by chlled water n the coolng col, and the returned hot water s cooled through mechancal refrgeraton cycles n an outsde chller plant contnuously. The compressor of a chller consumes a massve amount of energy, and accounts for the majorty of the overall coolng cost [4]. The result s an energygobblng coolng system that typcally consumes a sgnfcant porton ( 3%) of the total datacenter power [4]. 2.2 Outsde Ar Coolng To mprove energy effcency, varous so-called free coolng technologes that operate wthout mechancal chllers have recently been adopted. In ths paper, we focus on a more economcally vable technology called outsde ar coolng. It uses an ar-sde economzer to drect cold outsde ar nto the datacenter to cool down servers. The hot exhaust ar s smply rejected out nstead of beng cooled and recrculated. The advantage of outsde ar coolng can be sgnfcant: Intel ran a 1-month experment usng 9 blade servers, and reported that 67% of the coolng energy can be saved wth only slghtly ncreased hardware falure rates [24]. Companes lke Google [1], Facebook [2], and HP [3] have been operatng ther datacenters wth up to 1% outsde ar coolng, whch brngs mllon dollars of savngs annually. The energy effcency of outsde ar coolng heavly depends on ambent temperature among other factors. When temperature s lower, less ar s needed for heat exchange, and the ar handler fan speed can be reduced to save energy. Thus, a CRAC wth an ar-sde economzer usually operates n three modes. When ambent temperature s hgh, outsde ar coolng cannot be used, and the CRAC falls back to me th Internatonal Conference on Autonomc Computng (ICAC 13) USENIX Assocaton

3 chancal coolng wth chllers. When temperature falls below a certan threshold, outsde ar coolng s utlzed to provde partal or entre coolng capacty. When temperature s too low, outsde ar s mxed wth exhaust ar to mantan a sutable supply ar temperature. In ths mode, CRAC energy effcency cannot be further mproved snce fans need to operate at a mnmum speed to mantan arflow. Table 1 shows the emprcal COP 1 and partal PUE (ppue) 2 data of a stateof-the-art CRAC wth an ar-sde economzer. Clearly, as the outdoor temperature drops, the CRAC swtches the operatng mode to use more outsde ar coolng. As a result the COP mproves sx-fold from 3.3 to 19.5, and the ppue decreases dramatcally from 1.3 to 1.5. Due to the sheer amount of energy a datacenter draws, the numbers mply huge monetary savngs for the energy bll. Outdoor ambent Coolng mode COP ppue 35 C(9 F) Mechancal C(7 F) Mechancal C(6 F) Mxed C(5 F) Outsde ar C(25 F) Outsde ar Table 1: Effcency of Emerson s DSE TM coolng system wth an EconoPhase ar-sde economzer [14]. Return ar s set at 29.4 C(85 F). Wth the ncreasng use of outsde ar coolng, ths fndng motvates our proposal to make workload management temperature aware. Intutvely, datacenters at colder and thus more energy effcent locatons should be better utlzed to reduce the overall energy consumpton and cost smultaneously. Our dea also apples to datacenters usng mechancal coolng, because contrary to prevous work s assumpton [28], as shown n Table 1, the chller energy effcency also depends on outsde temperature, albet mlder. 2.3 An Emprcal Clmate Study Our dea hnges upon a key assumpton: Temperatures are dverse and not well correlated at dfferent locatons. In ths secton, we make our case concrete by supportng t wth an emprcal analyss of hstorcal clmate data. We use Google s datacenter locatons for our study, as they represent a global producton nfrastructure and the locaton nformaton s publcly avalable [4]. Google has 6 datacenters n the U.S., 1 n South Amerca, 3 n Europe, and 3 n Asa. We acqure hstorcal temperature data from varous data repostores of the Natonal Clmate Data Center [6] for all 13 locatons, coverng the entre one-year perod of 211. It s useful to frst understand the clmate profles at ndvdual locatons. Fgure 1 plots the daly average temperatures for three select locatons n North Amerca, Europe, 1 COP, coeffcent of performance, s defned for a coolng devce as the rato between coolng capacty and power. 2 ppue s defned as the sum of coolng capacty and coolng power dvded by coolng capacty. Nearly all the power delvered to servers translates to heat, whch matches the CRAC coolng capacty. Temp. (C) Temp. (C) Temp. (C) The Dalles, OR Hamna, Fnland 4 2 Qulcura, Chle 2 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Fgure 1: Daly average temperature at three Google datacenter locatons. Data from the Global Daly Weather Data of the Natonal Clmate Data Center (NCDC) [6]. Tme s n UTC. and South Amerca, respectvely. Geographcal dversty exsts despte the clear seasonal pattern shared among all locatons. For example, Fnland appears to be especally favorable for coolng durng wnter months. Dversty s more salent for locatons n dfferent hemspheres (e.g. Chle). We also observe a sgnfcant amount of day-to-day volatlty, suggestng that the avalablty and capablty of outsde ar coolng constantly vares across regons, and there s no sngle locaton that s always coolng effcent. We then examne short-term temperature volatlty. As shown n Fgure 2, the hourly varatons are more dramatc and hghly correlated wth tme-of-day, whch s ntutve to understand. Further, the hghs and lows do not occur at the same tme for dfferent regons due to tme dfferences. Temp. (C) Temp. (C) Temp. (C) Councl bluffs, IA Dubln, Ireland 2 Tseung Kwan, Hong Kong 1 Apr 16 Apr 17 Apr 18 Apr 19 Apr 2 Apr 21 Apr 22 Fgure 2: Hourly temperature varatons at three Google datacenter locatons. Data from the Hourly Global Surface Data of NCDC [6]. Tme s n UTC. Our approach would fal f hourly temperatures are well correlated at dfferent locatons. However, we fnd that ths s not the case for datacenters that are usually far apart from each other. The parwse temperature correlaton coeffcents for all 13 locatons are mostly n between.6 and -.6. Due to space lmt, detals are omtted and can be found n Sec. 2.3 of our techncal report [39]. The analyss above reveals that for globally deployed datacenters, local temperature at ndvdual locatons exhbts both tme and geographcal dversty. Therefore, a carefully desgned workload management scheme s crtcally needed, 3 USENIX Assocaton 1th Internatonal Conference on Autonomc Computng (ICAC 13) 35

4 n order to dynamcally adjust datacenter operatons to the ambent condtons, and to save the overall energy costs ppue=7.175e 5 T T MODEL In ths secton, we ntroduce our model frst and then formulate the temperature aware workload management problem of jont request routng and capacty allocaton. 3.1 System Model We consder a dscrete tme model where the length of a tme slot matches the tme scale at whch request routng and capacty allocaton decsons are made, e.g., hourly. The jont optmzaton s perodcally solved at each tme slot. We therefore focus only on a sngle tme slot. We consder a provder that runs a set of datacenters J n dstnct geographcal regons. Each datacenter j 2 J has a fxed capacty C j n terms of the number of servers. To model datacenter operatng costs, we consder both the energy cost and utlty loss of request routng and capacty allocaton, whch are detaled below. 3.2 Energy Cost and Coolng Effcency We focus on servers and coolng system n our energy cost model. Other energy consumers, such as network swtches, power dstrbuton systems, etc., have constant power draw ndependent of workloads [15] and are not relevant. For servers, we adopt the emprcal model from [15] that calculates the ndvdual server power consumpton as an affne functon of CPU utlzaton, P dle +(P peak P dle ) u. P dle s the server power when dle, P peak s the server power when fully utlzed, and u s the CPU load. Ths model s especally accurate for calculatng the aggregated power of a large number of servers [15]. Thus, assumng workloads are perfectly dspatched and servers have a unform utlzaton as a result, the server power of datacenter j can be modeled as C j P dle +(P peak P dle ) W j, where W denotes the total workload n terms of the number of servers requred. For the coolng system, we take an emprcal approach based on producton CRACs to model ts energy consumpton. We choose not to rely on smplfyng models for the ndvdual components of a CRAC and ther nteractons [4], because of the dffculty nvolved n and the naccuracy resulted from the process, especally for hybrd CRACs wth both outsde ar and mechancal coolng. Therefore, we study CRACs as a black box, wth outsde temperature as the nput, and ts overall energy effcency as the output. Specfcally, we use partal PUE (ppue) to measure the CRAC energy effcency. As n Sec. 2.2, ppue s defned as Server power + Coolng power ppue =. Server power A smaller value ndcates a more energy effcent system. We apply regresson technques to the emprcal ppue data of the Emerson CRAC [14] ntroduced n Table 1. We fnd that the best fttng model descrbes ppue as a quadratc functon of the outsde temperature as plotted below. ppue Outsde temperature (C) Fgure 3: Model fttng of ppue as a functon of the outsde temperature T for Emerson s DSE TM CRAC [14]. Small crcles denote emprcal data ponts. The model can be calbrated gven more data from measurements. For the purpose of ths paper, our approach yelds a tractable model that captures the overall CRAC effcency for the entre spectrum of ts operatng modes. Our model s also useful for future studes on datacenter coolng energy. Gven the outsde temperature T j, the total datacenter energy as a functon of the workload W j can be expressed as E j (W j )=(C j P dle +(P peak P dle ) W j ) ppue(t j ). (1) Here we mplctly assume that T j s known a pror and do not nclude t as the functon varable. Ths s vald snce short-term weather forecast s farly accurate and accessble. A datacenter s electrcty prce s denoted as P j. The prce may addtonally ncorporate the envronmental cost of generatng electrcty [17], whch we do not consder here. In realty, electrcty can be purchased from local day-ahead or hour-ahead forward markets at a pre-determned prce [34]. Thus, we assume that P j s known a pror and remans fxed for the duraton of a tme slot. The total energy cost, ncludng server and coolng power, s smply P j E j (W j ). 3.3 Utlty Loss Request routng. The concept of utlty loss captures the lost revenue due to the user-perceved latency for request routng decsons. Latency s arguably the most mportant performance metrc for most nteractve servces. A small ncrease n the user-perceved latency can cause substantal revenue loss for the provder [25]. We focus on the end-toend propagaton latency, whch largely accounts for the userperceved latency compared to other factors such as request processng tmes at datacenters [31]. The provder obtans the propagaton latency L j between user and datacenter j through actve measurements [3] or other means. We use j to denote the volume of requests routed to datacenter j from user 2 I, and D to denote the demand of each user that can be predcted usng machne learnng [28, 32]. Here, a user s an aggregated group of customers from a common geographcal regon, whch may be dentfed by a unque IP prefx. The lost revenue from user then depends on the average propagaton latency P j jl j /D through a generc delay utlty loss functon U. U can take varous forms dependng on the nteractve servce. Our algorthm th Internatonal Conference on Autonomc Computng (ICAC 13) USENIX Assocaton

5 and proof work for general utlty loss functons as long as U s ncreasng, dfferentable, and convex. As a case study, here we use a quadratc functon to model user s ncreased tendency to leave the servce wth ncreased latency. 1 U ( X 2 j L j /D A, (2) j2j where q s the delay prce that translates latency to monetary terms, and =( 1,..., J ) T. Utlty loss s clearly zero when latency s zero between user and datacenter. Capacty allocaton. We denote the utlty loss of allocatng β j servers for batch workloads as a dfferentable, decreasng, and convex functon V j (β j ), snce allocatng more resources ncreases the performance of batch jobs. Unlke nteractve servces, batch jobs are delay tolerant and resource elastc. Utlty functons such as the log functon are often used to capture such elastcty. However, utlty functons model the beneft of resource allocaton. To model the utlty loss of resource allocaton, snce the loss s zero when the capacty s fully allocated to batch jobs, an ntutve defnton can be of the followng form: V j (β j )=r(log C j log β j ), (3) where r s the utlty prce that converts the loss to monetary terms. (3) captures the ntuton that ncreasng resources results n a decreasng margnal reducton of utlty loss. 3.4 Problem Formulaton We now formulate the temperature aware workload management problem. For a gven request routng decson, the total cost assocated wth nteractve workloads can be wrtten as X X E j j P j + X U ( ). (4) j2j 2I 2I For a gven capacty allocaton decson β, the total cost assocated wth batch workloads s: X E j (β j )P j + X V j (β j ). (5) j2j j2j Puttng everythng together, the optmzaton can be formulated as: mnmze (4) + (5) (6) subject to: 8 : X j2j j = D, (7) 8j : X 2I j apple C j β j, (8), β, (9) varables: 2 R I J, β 2 R J. (6) s the objectve functon that jontly consders the cost of request routng and capacty allocaton. (7) s the workload conservaton constrant to ensure the user demand s satsfed. (8) s the datacenter capacty constrant, and (9) s the nonnegatvty constrant. 3.5 Transformng to the ADMM Form Problem (6) s a large-scale convex optmzaton problem. The number of users,.e., unque IP prefxes, s typcally O(1 5 ) O(1 6 ) for producton systems. Hence, our problem can have tens of mllons of varables, and mllons of constrants. In such a settng, a dstrbuted algorthm s preferable to fully utlze the computng resources of datacenters. Tradtonally, dual decomposton wth subgradent methods [9] are often used to develop dstrbuted optmzaton algorthms. However, they suffer from the curse of step szes. For the fnal output to be close to the optmum, we need to strategcally pck the step sze at each teraton, leadng to well-known problems of slow convergence and performance oscllaton wth large-scale problems. Alternatng drecton method of multplers s a smple yet powerful algorthm that s able to overcome the drawbacks of dual decomposton methods, and s well suted to largescale dstrbuted convex optmzaton. Though developed n the 197s [8], ADMM has recently receved renewed nterest, and found practcal use n many large-scale dstrbuted convex optmzaton problems n statstcs, machne learnng, etc. [1]. Before llustratng our new convergence proof and dstrbuted algorthm that extend the classcal framework, we frst ntroduce the bascs of ADMM, followed by a transformaton of (6) to the ADMM form. ADMM solves problems n the form mn f 1 (x 1 )+f 2 (x 2 ) (1) s.t. A 1 x 1 + A 2 x 2 = b, x 1 2 C 1,x 2 2 C 2, wth varables x` 2 R n`, where A` 2 R p n`, b 2 R p, f` s are convex functons, and C` s are non-empty polyhedral sets. Thus, the objectve functon s separable over two sets of varables, whch are coupled through an equalty constrant. We can form the augmented Lagrangan [22] by ntroducng an extra L-2 norm term ka 1 x 1 + A 2 x 2 bk 2 2 to the objectve: L (x 1,x 2 ; y) =f 1 (x 1 )+f 2 (x 2 )+y T (A 1 x 1 +A 2 x 2 b) +( /2)kA 1 x 1 + A 2 x 2 bk 2 2. Here, > s the penalty parameter (L s the standard Lagrangan for the problem). The benefts of ntroducng the penalty term are mproved numercal stablty and faster convergence n practce [1]. Our formulaton (6) has a separable objectve functon due to the jont nature of the workload management problem. However, the request routng decson and capacty allocaton decson β are coupled by an nequalty constrant rather than an equalty constrant as n ADMM problems. Thus we 5 USENIX Assocaton 1th Internatonal Conference on Autonomc Computng (ICAC 13) 37

6 ntroduce a slack varable γ 2 R J, and transform (6) to the followng mnmze (4) + (5) + I J R (γ) (11) + subject to: (7), (9), 8j : X j + β j + γ j = C j, (12) varables: 2 R I J, β 2 R J, γ 2 R J. Here, I J R (γ) s an ndcator functon defned as + I J R (γ) = +, γ, +1, otherwse. (13) The new formulaton (11) s equvalent to (6), snce for any feasble and β, γ holds, and the ndcator functon n the objectve values to zero. Clearly, t s n the ADMM form, wth a key dfference that t has three sets of varables n the objectve functon and equalty constrant (12). The convergence of the generalzed m-block ADMM, where m 3, has long remaned an open queston. Though t seems natural to drectly extend the classcal 2-block algorthm to the m-block case, such an algorthm may not converge unless some addtonal back-substtuton step s taken [21]. Recently, some progresses have been made by [2, 23] that prove the convergence of m-block ADMM for strongly convex objectve functons and the lnear convergence of m-block ADMM under a full-column-rank relaton matrx. However, the relaton matrx n our setup s not full column rank. Thus, we need a new proof for the lnear convergence under a general relaton matrx, together wth a dstrbuted algorthm nspred by the proof. 4. THEORY Ths secton frst ntroduces a generalzed m-block ADMM algorthm nspred by [2, 23]. Then a new convergence proof s presented, whch replaces the full column rank assumpton wth some mld assumptons on the objectve functon, and further smplfes the proof n [23]. The notatons and dscussons n ths secton are made ntentonally ndependent of the other parts of the paper n order to present the proof n a mathematcally general way. 4.1 Algorthm We consder a convex optmzaton problem n the form mn f (x ) (14) s.t. =1 A x = b =1 wth varables x 2 R n ( =1,...,m), where f : R n! R ( =1,...,m) are closed proper convex functons; A 2 R l n ( =1,...,m) are gven matrces; and b 2 R l s a gven vector. We form the augmented Lagrangan L (x 1,...,x m ; y) = f (x )+y T ( A x b) =1 +( /2)k =1 A x bk 2 2. (15) =1 As n [23], a generalzed ADMM algorthm has the followng: x k+1 = argmn L (x k+1 1,...,x k+1 1,x,x k +1,...,x k m; y k ), x y k+1 = y k + %( A x k+1 =1 b), =1,...,m, where % > s the step sze for the dual update. Note that when m =2and the step sze % equals to the penalty parameter, the above algorthm s reduced to the standard ADMM algorthm presented n [8]. 4.2 Assumptons We present two assumptons on the objectve functons, based on whch we are able to show the convergence of the generalzed m-block ADMM algorthm. ASSUMPTION 1. The objectve functons f ( =1,...,m) are strongly convex. Note that strong convexty s qute reasonable n engneerng practce. Ths s because a convex functon f(x) can be always well-approxmated by a strongly convex functon f(x). For nstance, f we choose f(x) =f(x)+ kxk 2 2 for some suffcently small >, then f(x) s strongly convex. ASSUMPTION 2. The gradents rf ( =1,...,m) are Lpschtz contnuous. Assumpton 2 says that, for each, there exsts some constant apple > such that for all x 1,x 2 2 R n, krf (x 1 ) rf (x 2 )k 2 apple apple kx 1 x 2 k 2, whch s agan reasonable n practce, snce apple can be made suffcently large. 4.3 Convergence In ths secton, we outlne the proof for the convergence of the generalzed ADMM algorthm. The detaled proof can be found n Sec. 4.3 of our techncal report [39]. For convenence, we wrte 1 x = x 1. x m C A,f(x) = f (x ), and A =[A 1... A m ]. =1 Then the problem (14) can be rewrtten as mn s.t. f(x) Ax = b th Internatonal Conference on Autonomc Computng (ICAC 13) USENIX Assocaton

7 wth the optmal value p = nf{f(x) Ax = b}. Smlarly, the augmented Lagrangan can be rewrtten as L (x; y) =f(x)+y T (Ax b)+( /2)kAx bk 2 2, wth the assocated dual functon defned by d(y) =nf x L (x; y) and the optmal value d = sup{d(y)}. Now defne the prmal and dual optmalty gaps as k p = L (x k+1 ; y k ) d(y k ), k d = d d(y k ), respectvely. Clearly, we have k p and k d. Defne V k = k p + k d. We wll see that V k s a Lyapunov functon for the algorthm,.e., a nonnegatve quantty that decreases n each teraton. Our proof reles on three techncal lemmas. LEMMA 1. There exsts a constant # > such that V k apple V k 1 %ka x k+1 bk 2 2 #kx k+1 x k k 2 2, (16) n each teraton, where x k+1 = argmn x L (x; y k ). PROOF. See Appendx C n the techncal report [39]. LEMMA 2. For any gven δ >, there exsts a constant > (dependng on δ) such that for any (x, y) satsfyng kxk + kyk apple2δ, the followng nequalty holds kx x(y)k apple kr x L (x; y)k, (17) where x(y) = arg mn x L (x; y). PROOF. See Appendx B n the techncal report [39]. LEMMA 3. There exsts a constant > such that kr x L (x k ; y k )k 2 apple kx k x k+1 k 2. (18) PROOF. See Appendx A n the techncal report [39]. By Lemma 1, we have 1X %ka x k+1 bk #kx k+1 x k k 2 2 apple V. k= Hence, ka x k+1 bk 2 2! and kx k+1 x k k 2 2!, as k! 1. Suppose that the level set of p + d s bounded. Then by the Bolzano-Weerstrass theorem, the sequence {x k,y k } has a convergent subsequence,.e., lm k2r,k!1 (xk,y k )=( x, ỹ), for some subsequence R, where ( x, ỹ) denotes the lmt pont. By usng Lemma 2 and Lemma 3, we can show that the lmt pont ( x, ỹ) s an optmal prmal-dual soluton. Hence, lm V k = lm k2r,k!1 k2r,k!1 k p + k d =. Snce V k decreases n each teraton, the convergence of a subsequence of V k mples the convergence of V k, and we have lm k!1 k p + k d =. Ths further mples that both k p and k d converge to. To sum up, we have the followng convergence theorem for our generalzed ADMM algorthm. THEOREM 1. Suppose that Assumptons 1 and 2 hold and that the level set of p + d s bounded. Then both the prmal gap k p and the dual gap k d converge to. Due to space lmt, the rate of convergence s omtted and can be found n Sec. 4.3 of [39]. 5. A DISTRIBUTED ALGORITHM We now develop a dstrbuted soluton algorthm based on the generalzed ADMM algorthm n Sec Drectly applyng the algorthm to our problem (11) wll lead to a centralzed algorthm. The reason s that when the augmented Lagrangan s mnmzed over, the penalty term P P j j+ 2 β j + γ j C j couples j s across, and the utlty loss P U ( ) couples j s across j. The jont optmzaton of utlty loss and the quadratc penalty s partcularly dffcult to solve, especally when the number of users s large, snce U ( ) can take any general form. If they can be separated, then we wll have a dstrbuted algorthm where each U ( ) s optmzed n parallel, and the quadratc penalty term s optmzed effcently wth exstng methods. Towards ths end, we ntroduce a new set of auxlary varables a j = j, and re-formulate the problem (11): mnmze X E j ( X j subject to: (7), (9), a j )P j + X 8j : X a j + β j + γ j = C j, 8, j : a j = j, U ( )+(5) + I J R (γ) + varables: a, 2 R I J, β, γ 2 R J. (19) Ths s a 4-block ADMM problem, where a j replaces j n the objectve functon and constrant (12) when the couplng happens across users. Ths s the key step that enables the decomposton of the -mnmzaton problem. The augmented Lagrangan can then be readly obtaned from (15). By omttng the rrelevant terms, we can see that at each teraton k +1, the -mnmzaton problem s X mn U ( ) X X ' j j 2 ( 2 j 2 j a k j) j s.t. 8 : X j j = D,, (2) 7 USENIX Assocaton 1th Internatonal Conference on Autonomc Computng (ICAC 13) 39

8 where ' j s the dual varable for the equalty constrant a j = j. Ths s clearly decomposable over nto I per-user sub-problems snce the objectve functon and constrant are separable over. The per-user sub-problem s of a much smaller scale wth only J varables and J +1constrants, and s easy to solve even though t s a non-lnear problem for a general U. Some may now wonder f the auxlary varable a s hard to solve for. As t turns out, the a-mnmzaton problem s decomposable over j nto J per-datacenter sub-problems. Moreover, each per-datacenter sub-problem s a quadratc program. Though t s large-scale, t can be transformed nto a second-order cone program and solved effcently. More detals can be found n Sec. 5 n the techncal report [39]. β- and γ-mnmzaton steps are clearly decomposable over j. The entre procedure s summarzed below. Dstrbuted 4-block ADMM. Intalze a,, β, γ, λ, ' to. For k =, 1,...,repeat 1. -mnmzaton: Each user solves the followng subproblem for k+1 : mn U ( ) X ' j j 2 ( 2 j 2 j a k j) j X s.t. j = D,. (21) j 2. a-mnmzaton: Each datacenter solves the followng sub-problem for a k+1 j =(a k+1 1j,...,a k+1 I j )T : X mn E j X + a j P j + X a j (λ k j + ' k j)+ 2 (X a j (βj k + γj k C j +.5a j k+1 j ) s.t. a j. (22) 3. β-mnmzaton: Each datacenter solves the followng sub-problem for β k+1 j : mn E j (β j )P j + V j (β j )+λ k j β j + X 2 a k+1 j + β j + γj k C j 2 s.t. β j. 4. γ-mnmzaton: Each datacenter solves: ( γ k+1 j = max,c j λ j X a k+1 j β k+1 j ) a j ) 2, 8j. 5. Dual update: Each datacenter updates λ j for the capacty constrant (8): X λ k+1 j = λ k j + % a k+1 j + β k+1 j + γ k+1 j C j. Each user updates ' j for the equalty constrant a j = j : ' k+1 j = ' k j + %(a k+1 j k+1 j ), 8j. The dstrbuted nature of our algorthm allows for an effcent parallel mplementaton n datacenters wth a large number of servers. The per-user sub-problem (21) can be solved n parallel on each server. Snce (21) s a small-scale convex optmzaton as dscussed above, the complexty s low. A mult-threaded mplementaton can further speed up the algorthm wth mult-core hardware. The penalty parameter and utlty loss functon U can be confgured at each server before the algorthm runs. Step 2 and 3 nvolve solvng J per-datacenter sub-problems respectvely, whch can also be mplemented n parallel wth only J servers. 6. EVALUATION We perform trace-drven smulatons to realstcally assess the potental of temperature aware workload management. 6.1 Setup We rely on the Wkpeda request traces [38] to represent the nteractve workloads of a cloud servce. The dataset we use contans, among other thngs, 1% of all user requests ssued to Wkpeda from the 24-hour perod between January 1, 28 UTC to January 2, 28 UTC. The workloads are normalzed to a number of servers, assumng that each request requres 1% of a server s CPU. The traces reflect the durnal pattern of real-world nteractve workloads. The predcton of workloads can be done accurately as demonstrated by prevous work [28, 32], and we do not consder the effect of predcton error here. The optmzaton s solved hourly. We consder Google s nfrastructure [4] to represent a geodstrbuted cloud as dscussed n Sec Each datacenter s capacty C j s unformly dstrbuted between [1, 2] 1 5 servers. The emprcal CRAC effcency model developed n Sec. 3.2 s used to derve the total energy consumpton of all 13 locatons under dfferent temperatures. We use the 211 annual average day-ahead on peak prces [16] at the local markets as the power prces P j for the 6 U.S. locatons 3. For non-u.s. locatons, the power prce s calculated based on the retal ndustral power prce avalable on the local utlty company webstes wth a 5% wholesale dscount, whch s usually the case n realty [37]. The power prces at each locaton are shown n Table 2 n the techncal report [39]. The servers have peak power P peak = 2 W, and consume 5% power at dle. These numbers represent state-of-the-art datacenter hardware [15, 34]. To calculate the utlty loss of nteractve workloads, we obtan the latency matrx L from Plane [3], a system that collects wde-area network statstcs from Planetlab vantage ponts. Snce the Wkpeda traces do not contan clent sde nformaton, we emulate the geographcal dversty of user requests by splttng the total nteractve workloads among users followng a normal dstrbuton. We set the number of 3 The U.S. electrcty market s conssted of multple regonal markets. Each regonal market has several hubs wth ther own prcng. We thus use the prce of the specfc hub that each U.S. datacenter locates n th Internatonal Conference on Autonomc Computng (ICAC 13) USENIX Assocaton

9 Jont opt Capacty optmzed Coolng optmzed : 4: 8: 12: 16: 2:.2.1 Coolng cost ($1 3 ) Baselne Capacty optmzed Coolng optmzed Jont opt.5 : 4: 8: 12: 16: 2: (a) Overall mprovement. (b) Interactve workloads. Fgure 4: Coolng energy cost savngs. Tme s n UTC. Jont opt Capacty optmzed Coolng optmzed : 4: 8: 12: 16: 2: (a) Overall mprovement. users I = 1 5, and choose 1 5 IP prefxes from a Route- Vews [5] dump. Note that n our context, each user,.e. IP prefx, represents many customers accessng the servce. We then extract the correspondng round trp tmes from Plane logs, whch contan traceroutes made to IP addresses from Planetlab nodes. We only use latency measurements from Planetlab nodes that are close to our datacenter locatons to resemble the user-datacenter latency. We use utlty loss functons defned n (2) and (3). The delay prce q = 4 1 6, and the utlty loss prce for batch jobs r = 5. We nvestgate the performance of temperature aware workload management. We benchmark our ADMM algorthm, referred to as Jont opt, aganst three baselne strateges, whch use dfferent amounts of nformaton n managng workloads. The frst benchmark, called Baselne, s a temperature agnostc strategy that separately consders capacty allocaton and request routng of the workload management problem. It frst allocates capacty to batch jobs by mnmzng the backend total cost wth (5) as the objectve. The remanng capacty s used to solve the request routng optmzaton wth (4) as the objectve. Only the electrcty prce dversty s used, and coolng energy s calculated wth a constant ppue of 1.2 that corresponds to an ambent temperature of 2 C for the two cost mnmzaton problems. Though nave, such an approach s wdely used n current Internet-scale cloud servces. It also allows an mplct comparson wth pror work [17, 27, 29, 34, 35]. The second benchmark, called Capacty Optmzed, mproves upon Baselne by jontly solvng capacty allocaton and request routng, but stll gnores the coolng energy effcency dversty. Ths demonstrates the mpact of capacty allocaton n datacenter workload management. Utlty loss ($1 3 ) Baselne Capacty optmzed Coolng optmzed Jont opt 1 : 4: 8: 12: 16: 2: (b) Interactve workloads. Fgure 5: Utlty loss reductons. Tme s n UTC. Coolng cost ($1 3 ) Utlty loss ($1 3 ).8.6 Baselne.4 Capacty optmzed Coolng optmzed Jont opt.2 : 4: 8: 12: 16: 2: (c) Batch workloads. Baselne Capacty optmzed Coolng optmzed Jont opt 5 : 4: 8: 12: 16: 2: (c) Batch workloads. The thrd benchmark, called Coolng Optmzed, mproves upon Baselne by explotng the temperature and coolng effcency dversty n mnmzng cost, but does not adopt jont management of the nteractve and batch workloads. Ths demonstrates the mpact of beng temperature aware. We run the four benchmarks above wth our 24-hour traces at each day of January 211, usng the emprcal hourly temperature data we collected n Sec The dstrbuted ADMM algorthm s used to solve them untl convergence s acheved. The fgures show the average results over 31 runs. 6.2 Coolng energy savngs The central thess of ths paper s to save datacenter cost through temperature aware workload management that explots the coolng effcency dversty wth capacty allocaton. We examne the effectveness of our approach by comparng the coolng energy consumpton frst. Fgure 4 shows the results. In partcular, Fgure 4a shows that overall, Jont opt saves 15% 2% coolng energy compared to Baselne. A breakdown of the savng shown n the same fgure reveals that dynamc capacty allocaton provdes 1% 15% savng, and coolng effcency dversty provdes 5% 1% savng, respectvely. Note that the cost savng s acheved wth cuttngedge CRACs whose effcency s already substantally mproved wth outsde ar coolng capablty. The results confrm that our temperature aware workload management s able to further optmze the coolng effcency and cost of geo-dstrbuted datacenters. Fgure 4b and 4c show a detaled breakdown of coolng energy cost. Coolng cost attrbuted to nteractve workloads, as n Fgure 4b, exhbts a durnal pattern and peaks between 2: and 8: UTC (21: to 3: EST, 18: to : PST), mplyng that most of the Wkpeda traffc org- 9 USENIX Assocaton 1th Internatonal Conference on Autonomc Computng (ICAC 13) 311

10 nates from the U.S. The four strateges perform farly closely, whle Baselne and Capacty optmzed consstently ncur more coolng energy cost due to ther coolng agnostc nature that underestmates the overall energy cost. Coolng cost attrbuted to batch workloads s shown n Fgure 4c. Baselne ncurs the hghest cost snce t underestmates the energy cost, and runs more batch workloads than necessary. Coolng optmzed mproves Baselne by takng nto account coolng effcency dversty and reducng batch workloads as a result. Both strateges fal to explot the trade-off wth nteractve workloads. Thus ther coolng cost closely follows the daly temperature trend n that t gradually decreases from : to 12: UTC (19: to 7: EST) and then slowly ncreases from 12: to 2: UTC (7: to 15: EST). Capacty optmzed adjusts capacty allocaton wth request routng, and further reduces batch workloads n order to allocate more resources for nteractve workloads. Jont opt combnes temperature aware coolng optmzaton wth holstc workload management, and has the lowest coolng cost wth least batch workloads. Though ths ncreases the back-end utlty loss, the overall effect s a net reducton of total cost snce nteractve workloads enjoy lower latency as wll be observed soon. 6.3 Utlty loss reductons The other component of datacenter cost s utlty loss. From Fgure 5a, the relatve reducton follows the nteractve workloads and also has a vsble durnal pattern. Jont opt and Capacty optmzed provde the most sgnfcant utlty loss reductons from 5% to 25%, whle Coolng optmzed provdes a modest 5% reducton compared to Baselne. To study the reasons for the varyng degrees of reductons, Fgure 5b and 5c show the respectve utlty loss of nteractve and batch workloads. We observe that nteractve workloads ncur most of the utlty loss, reflectng ts mportance compared to batch workloads. Baselne and Coolng optmzed have much larger utlty loss from nteractve workloads as shown n Fgure 5b, because of the separate management of two workloads. The average latency performances under these two strateges are also worse as can be seen n Fgure 7 of our techncal report [39]. On the other hand, Capacty optmzed and Jont opt outperform the two by allocatng more capacty to nteractve workloads at cost-effcent locatons whle reducng batch workloads (recall Fgure 4c). Ths s especally effectve durng peak hours as shown n Fgure 5b. Capacty optmzed and Jont opt do have larger utlty loss from batch workloads as seen n Fgure 5c. However snce nteractve workloads attrbute to the majorty of the provder s utlty and revenue, the overall effect of jont workload management s postve. 6.4 Senstvty to seasonal changes One natural queston s, snce the results above are obtaned n wnter tmes (January), would the benefts be less sgnfcant durng summer tmes when coolng s more expensve? In other words, are the benefts senstve to the seasonal changes? We thus run our Jont opt wth Baselne at each day of May, whch represents typcal Sprng/Fall weather, and August, whch represents typcal Summer weather, respectvely. Fgure 6 shows the average overall cost savngs acheved n dfferent seasons. We observe that the cost savngs, rangng from 5% to 2%, are consstent and nsenstve to seasonal changes. The reason s that our approach depends on: 1) the geographcal dversty of temperature and coolng effcency; 2) the mxed nature of datacenter workloads, both of whch exst at all tmes of the year no matter whch coolng method s used. Temperature aware workload management s thus able to offer consstent cost benefts January May August : 4: 8: 12: 16: 2: Fgure 6: Overall cost savng s nsenstve to seasonal changes of the clmate. We also compare the convergence speed of our the dstrbuted ADMM algorthm wth the conventonal subgradent method. We have found that our algorthm converges wthn around 6 teratons, whle the subgradent method does not converge even after 2 teratons. Our dstrbuted ADMM algorthm s thus better suted to large-scale convex optmzaton problems. More detals can be found n Sec. 6.3 n the techncal report [39]. 7. CONCLUSION We propose temperature aware workload management, whch explores two key aspects of geo-dstrbuted datacenters that have not been well understood n the past. Frst, as we show emprcally, energy effcency of coolng systems, especally outsde ar coolng, vares wdely wth outsde temperature. The geographcal dversty of temperature s utlzed to reduce coolng energy consumpton. Second, the elastc nature of batch workloads s further captalzed by dynamcally adjustng capacty allocaton along wth the wdely studed request routng for nteractve workloads. We formulate the jont optmzaton under a general framework wth an emprcal coolng effcency model. To solve large-scale problems for producton systems, we rely on the ADMM algorthm. We provde a new convergence proof for a generalzed m- block ADMM algorthm. We further develop a novel dstrbuted ADMM algorthm for our problem. Extensve smulatons hghlght that temperature aware workload management saves 15% 2% coolng energy and 5% 2% overall energy cost and the dstrbuted ADMM algorthm s practcal to solve large-scale workload management problems wth only tens of teratons th Internatonal Conference on Autonomc Computng (ICAC 13) USENIX Assocaton

11 8. REFERENCES [1] [2] [3] [4] datacenters/nsde/locatons/. [5] [6] Natonal clmate data center (NCDC). [7] BASH, C., AND FORMAN, G. Cool job allocaton: Measurng the power savngs of placng jobs at coolng-effcent locatons n the data center. In Proc. USENIX ATC (27). [8] BERTSEKAS, D. P., AND TSITSIKLIS, J. N. Parallel and Dstrbuted Computaton: Numercal Methods. Athena Scentfc, [9] BOYD, S., AND MUTAPCIC, A. Subgradent methods. Lecture notes of EE364b, Stanford Unversty, Wnter Quarter http: // notes/subgrad_method_notes.pdf. [1] BOYD, S., PARIKH, N., CHU, E., PELEATO, B., AND ECKSTEIN, J. Dstrbuted optmzaton and statstcal learnng va the alternatng drecton method of multplers. Foundatons and Trends n Machne Learnng 3, 1 (21), [11] CHEN, Y.,GMACH, D., HYSER, C., WANG, Z., BASH, C., HOOVER, C., AND SINGHAL, S. Integrated management of applcaton performance, power and coolng n datacenters. In Proc. NOMS (21). [12] DENG, N., STEWART, C., GMACH, D., ARLITT, M., AND KELLEY, J. Adaptve green hostng. In Proc. ACM ICAC (212). [13] EL-SAYED, N., STEFANOVICI, I., AMVROSIADIS, G., AND HWANG, A. A. Temperature management n data centers: Why some (mght) lke t hot. In Proc. ACM Sgmetrcs (212). [14] EMERSON NETWORK POWER. Lebert R DSE TM precson coolng system sales brochure [15] FAN, X., WEBER, W.-D., AND BARROSO, L. A. Power provsonng for a warehouse-szed computer. In Proc. ACM/IEEE Intl. Symp. Computer Archtecture (ISCA) (27). [16] FEDERAL ENERGY REGULATORY COMMISSION. U.S. electrc power markets [17] GAO, P.X.,CURTIS, A. R., WONG, B., AND KESHAV, S. It s not easy beng green. In Proc. ACM SIGCOMM (212). [18] GOIRI, I. N., BEAUCHEA, R., LE, K., NGUYEN, T. D., HAQUE, M. E., GUITART, J., TORRES, J., AND BIANCHINI, R. Greenslot: Schedulng energy consumpton n green datacenters. In Proc. SC (211). [19] GOIRI, I. N., LE, K., NGUYEN, T. D., GUITART, J., TORRES, J., AND BIANCHINI, R. GreenHadoop: Leveragng green energy n data-processng frameworks. In Proc. ACM EuroSys (212). [2] HAN, D., AND YUAN, X. A note on the alternatng drecton method of multplers. J. Optm. Theory Appl. 155 (212), [21] HE, B. S., TAO, M., AND YUAN, X. M. Alternatng drecton method wth Gaussan back substtuton for separable convex programmng. SIAM J. Optm. 22 (212), [22] HESTENES, M. R. Multpler and gradent methods. Journal of Optmzaton Theory and Applcatons 4,5 (1969), [23] HONG, M., AND LUO, Z.-Q. On the lnear convergence of the alternatng drecton method of multplers, August 212. [24] INTEL INC. Reducng data center cost wth an ar economzer, August 28. [25] KOHAVI, R., HENNE, R. M., AND SOMMERFIELD, D. Practcal gude to controlled experments on the web: Lsten to your customers not to the hppo. In Proc. ACM SIGKDD (27). [26] LE, K., BIANCHINI, R., NGUYEN, T. D., BILGIR, O., AND MARTONOSI, M. Cappng the brown energy consumpton of Internet servces at low cost. In Proc. IGCC (21). [27] LIN, M., WIERMAN, A., ANDREW, L. L. H., AND THERESKA, E. Dynamc rght-szng for power-proportonal data centers. In Proc. IEEE INFOCOM (211). [28] LIU, Z., CHEN, Y.,BASH, C., WIERMAN, A., GMACH, D., WANG, Z., MARWAH, M., AND HYSER, C. Renewable and coolng aware workload management for sustanable data centers. In Proc. ACM Sgmetrcs (212). [29] LIU, Z., LIN, M., WIERMAN, A., LOW, S. H., AND ANDREW, L. L. Greenng geographcal load balancng. In Proc. ACM Sgmetrcs (211). [3] MADHYASTHA, H.V.,ISDAL, T.,PIATEK, M., DIXON, C., ANDERSON, T.,KRISHNAMURTHY, A., AND VENKATARAMANI, A. Plane: An nformaton plane for dstrbuted servces. In Proc. USENIX OSDI (26). [31] NARAYANA, S., JIANG, J.W.,REXFORD, J., AND CHIANG, M. To coordnate or not to coordnate? Wde-Area traffc management for data centers. Tech. rep., Prnceton Unversty, 212. [32] NIU, D., XU, H., LI, B., AND ZHAO, S. Qualty-assured cloud bandwdth auto-scalng for vdeo-on-demand applcatons. In Proc. IEEE INFOCOM (212). [33] PELLEY, S., MEISNER, D., WENISCH, T.F.,AND VANGILDER, J. W. Understandng and abstractng 11 USENIX Assocaton 1th Internatonal Conference on Autonomc Computng (ICAC 13) 313

12 total data center power. In Proc. Workshop on Energy Effcent Desgn (WEED) (29). [34] QURESHI, A., WEBER, R., BALAKRISHNAN, H., GUTTAG, J., AND MAGGS, B. Cuttng the electrcty bll for Internet-scale systems. In Proc. ACM SIGCOMM (29). [35] RAO, L., LIU, X., XIE, L., AND LIU, W. Mnmzng electrcty cost: Optmzaton of dstrbuted Internet data centers n a mult-electrcty-market envronment. In Proc. IEEE INFOCOM (21). [36] REISS, C., TUMANOV, A., GANGER, G. R., KATZ, R. H., AND KOZUCH, M. A. Heterogenety and dynamcty of clouds at scale: Google trace analyss. In Proc. ACM SoCC (212). [37] TELEGEOGRAPHY RESEARCH. Global Internet geography executve summary. bpastudo.csudh.edu/fac/lpress/471/ hout/telegeographygg_execsumm.pdf, 28. [38] URDANETA, G., PIERRE, G., AND VAN STEEN, M. Wkpeda workload analyss for decentralzed hostng. Elsever Computer Networks 53, 11 (July 29), [39] XU, H., FENG, C., AND LI, B. Temperature aware workload management n geo-dstrbuted datacenters. Tech. rep., Unversty of Toronto, ~henryxu/share/geodc-preprnt.pdf, 213. [4] ZHOU, R., WANG, Z., MCREYNOLDS, A., BASH, C., CHRISTIAN, T., AND SHIH, R. Optmzaton and control of coolng mcrogrds for data centers. In Proc. IEEE ITherm (212) th Internatonal Conference on Autonomc Computng 12 (ICAC 13) USENIX Assocaton