How To Improve Power Demand Response Of A Data Center Wth A Real Time Power Demand Control Program

Transcription

1 Demand Response of Data Centers: A Real-tme Prcng Game between Utltes n Smart Grd Nguyen H. Tran, Shaole Ren, Zhu Han, Sung Man Jang, Seung Il Moon and Choong Seon Hong Department of Computer Engneerng, Kyung Hee Unversty, Korea; emal: {nguyenth, smjang, moons85, cshong}@khu.ac.kr School of Computng and Informaton Scences, Florda Internatonal Unversty, USA; emal: sren@cs.fu.edu Electrcal and Computer Engneerng Department, Unversty of Houston, USA; emal: zhan2@uh.edu Abstract We study the demand response DR) of geodstrbuted data centers DCs) usng a dynamc prcng scheme. Our proposed prcng scheme s constructed based on a formulated two-stage Stackelberg game where each utlty sets a real-tme prce to maxmze ts own proft n Stage I; and based on these prces, the DCs servce provder mnmzes ts cost va workload shftng and dynamc server allocaton n Stage II. Frst, we show that there exsts a unque Stackelberg equlbrum. Then, we propose an teratve and dstrbuted algorthm that converges to ths equlbrum, where the rght prces are set for the rght demand. Fnally, we verfy our proposal by traced-base smulaton and results show that our prcng scheme outperforms other baselne schemes sgnfcantly. I. INTRODUCTION A. Motvatons, Challenges and Contrbutons Data centers DCs) are well-known as large-scale consumers of electrcty and a study shows that many DC operators pad more than $1M [1] on ther annual electrcty blls, whch contnues to rse wth the flourshng of cloud-computng servces. Recent works have shown that DC operators can save more than 5% 45% [2] operaton cost by leveragng tme and locaton dverstes of electrcty prces. However, most of the exstng research s based on one mportant assumpton: the electrcty prce applyng on DC does not change wth demand, whch s not true snce DCs have enormous energy consumpton and have mpacts on power prces. Many DC operators are consderng how to run ther geo-dstrbuted DCs on smart grd, whch s desgned to coordnate the energy supply and demand more effectvely through ts advanced two-way communcatons. An mportant feature of smart grd s demand response DR). DR programs seek to provde ncentves to nduce dynamc demand management of customers electrcty load n response to power supply condtons. Due to ts huge and rapdly ncreasng energy consumpton, DCs should be encouraged sgnfcantly to partcpate n the DR programs. One of the DR programs s usng real-tme prcng schemes to reduce the peak-to-average PAR) load rato by encouragng customers to shft ther energy demand away from peak hours. The challenge of an effectve prcng scheme s how to charge the customers wth a rght prce not only at the rght tme but also on the rght amount of customers demand. A real-tme prcng scheme s consdered effectve f t can mtgate the large fluctuaton of energy consumpton between peak and offpeak hours to ncrease power grd s relablty and robustness. We consder the problem of usng real-tme prcng of utltes to enable the geo-dstrbuted DCs partcpaton nto the DR program. We show that there s an nteracton between geo-dstrbuted DCs and ther local utltes; and t s the frst challenge of ths DR problem. Specfcally, when partcpatng n the DR program, DCs operator wll dstrbute ts energy demand geographcally based on the electrc prces adjusted ntellgently by the local utltes. However, the utltes set ther prces based on the total demand ncludng the DCs demand, whch s only known when the prce s avalable. We clearly see that ths dependency makes t dffcult for both DCs and utltes to make ther decsons. The second challenge s an nteracton among local utltes feedng power to the geodstrbuted DCs. Specfcally, the DCs decsons depend on the electrc prces set by local utltes; therefore, f any local utlty changes ts prces, t can affect other DCs prcng decson. Snce n practce the utltes are non-cooperatve, how to desgn a prcng mechansm that can enable an equlbrum prce profle s the bottleneck of ths DR program. To tackle two above dscussed challenges, our contrbutons can be summarzed as follows: Frst, we transform the functonal space of the geo-dstrbuted DCs DR program nto a mathematcal space of a formulated two-stage Stackelberg game. In ths game, each utlty wll set a real-tme prce to maxmze ts own proft n Stage I; and gven these prces, the DCs operator wll mnmze ts cost va workload shftng and dynamc server allocaton n Stage II. Second, we use the backward nducton method to fnd a unque Stackelberg equlbrum of ths two-stage game. Based on ths result, we propose an teratve and dstrbuted algorthm to acheve the Stackelberg equlbrum. We also examne the algorthm s convergence where the rght prces are set for the rght demand. Fnally, we perform real-world trace-based smulaton to soldfy the analyss. The results show that our proposed prcng scheme can flatten the workload not only over tme but also over space. Due to space lmtatons, all proofs can be found n the techncal report avalable onlne [3]. B. Related Work There are many exstng research on DCs cost mnmzaton takes the electrcty prce for granted [2], [4], [5], whch does not follow any DR programs. For those work consderng DR of geo-dstrbuted DCs, based on the nteractons between DCs

2 and utltes, we smply dvde them nto two categores. 1) One-way nteracton: In realty one of the most popular DR programs of DCs s Concdent Peak Prcng CPP), whch s studed n [6]. However, current DCs do not respond actvely to the warnng sgnals due to the uncertanty of these warnngs [6], whch motvates researchers to devse more effectve DR approaches. The authors n [7] use a predctonbased method where the customers DCs) respond to the prces whch are chosen based on a supply functon. Hence, n ths work only customers respond to a predcted prce whle there s no acton from the power supplers to set the prces correspondng to the demand. 2) Two-way nteracton: Two recent papers [8], [9] n ths category are hghly related to our work. Both consder dynamc prcng mechansms that make utltes and DCs coupled. However, the system model of [9] assumes that all utltes cooperate to solve a socal optmzaton problem, whch s not relevant to current practce snce there s no nformaton exchange between utltes n realty. On the other hand, the prcng scheme of [8] s based on a heurstc approach, whch cannot maxmze the utltes proft as well as mnmze ther cost. II. SYSTEM MODEL AND PROBLEM FORMULATION We consder dscrete tme model t T = {1,..., T } of a bllng cycle e.g., typcally a month), where the length of a tme slot t matches an nterval at whch the DCs decsons and utltes real-tme prces can be updated such as one hour). Let I = {1,..., I} denote the set of stes where DCs are located. Each DC s assumed to be powered by a local utlty company and have S homogeneous servers. We observe that there exsts a specal mutual nteracton between DCs and utltes that can be modeled as a leaderfollower game,.e. two-stage Stackelberg game. Specfcally, the utltes are the leaders that smultaneously set the prces to maxmze ther proft n Stage I and DCs wll make ther decsons on workload shftng and dynamc server provsonng to mnmze ther cost n Stage II. We descrbe ths two-stage game formulaton n the reverse sequence, startng wth Stage- II optmzaton problem. A. DCs Cost Mnmzaton n Stage II We frst descrbe the workload model of a typcal DCs. We then elaborate the DCs cost focusng on the energy cost and delay cost model. Fnally, we formulate the Stage-II DC s cost mnmzaton. 1) Workload Model: Even though DCs can support a wde range of workloads, we generally dvde them nto two typcal types of workload: nteractve jobs and batch jobs. Whle the former s delay-senstve and non-flexble e.g. bankng servce, onlne game, etc.), the later s delay-tolerant and flexble to schedule e.g. scentfc applcaton, map reduce workload, etc.). We assume that each DC processes ts batch jobs locally.e. batch jobs cannot be re-drected to other DCs for load balancng) smlarly to [5]. For nteractve jobs, we denote the total arrval rate to the DCs front-end server,.e. all DCs are managed by a DC servce provder DSP)) at tme t by Λt) and ths front-end server s responsble for splttng the total ncomng workload Λt) nto separate workloads of geo-dspersed DCs, denoted by {λ t)} I. 2) DC s Cost and QoS Model: The DSP tres to not only mnmze ts energy and mgraton cost but also guarantee the QoS requrements of the nteractve jobs. Energy Cost: Snce batch jobs are flexble to schedule, we assume that the batch job processng consumes an amount of energy e b t) of each DC n tmeslot t. On the other hand, the energy consumpton 1 of delay-sensve jobs at DC s [1] e d t) = s t) ) P dle +P peak P dle )U t)+p UEt) 1)P peak where s t) s the server count, µ s the servce rate of a server, P peak and P dle are the server s peak and dle power, respectvely, U t) = λt) s t)µ s the average server utlzaton, and P UEt) s the power usage effectveness measurng the energy e.g. coolng) effcency of the DC. We can rewrte e d t) as follows e d t) = a λ t) + b t)s t), I, t T, 1) where a = P peak P dle )/µ and b t) = P dle +P UEt) 1)P peak. Therefore, denotng the total energy by e t) = e d t) + e b t), 2) and gven a prce p t) at tme t, the energy cost of DC s e t)p t). Mgraton Cost: Snce mgratng the workload from frontend server to geo-dstrbuted DCs can be very costly e.g. mgratng vrtual machnes or vdeo content requests over the Internet could be expensve due to reservng bandwdth from ISP), we model the mgraton cost to DC as ωd c λ ), 3) where d s the transmsson delay from front-end server to DC, ω s a weght factor and c λ ) s a functon assumed to be strctly ncreasng and convex. Snce d s proportonal to the dstance, t can be assumed to be a constant and we see that mgratng more requests from the front-end server to a farther DC s more costly. For analyss tractablty, we choose a quadratc functon c λ t)) = λ t) 2 snce t s wdely used to penalze the acton n control theory. QoS Constrant: We assume that each delay-senstve request mposes a maxmum delay D that the DSP has to guarantee when shftng ths request to DC. Therefore, the QoS constrant n terms of delay guarantee can be modeled as follows 1 s t)µ λ t) + d D,, 4) where 1/s t)µ λ t)) s the average delay tme of a request processed n DC wth arrval rate λ t) and servce rate s t)µ by queueng theory. 1 We alternatvely use ether power or energy snce the tme slot s the same.

3 3) Problem Formulaton: Our model focuses on two key controllng knobs of DCs cost mnmzaton: the workload shftng to DC λ t) and the number of actve servers provsoned s t) at ste,. Then, the Stage-II DC cost mnmzaton s gven by DC : mnmze T t=1 =1 I e t)p t) + ωd λ t) 2 5) subject to constrants 1), 2), 4), I λ t) = Λt), t, 6) =1 s t) S,, t, 7) λ t) s t)µ,, t, 8) varables s t), λ t),, t. 9) Whle constrants 1), 2) and 4) are the defntons of the objectve functon and QoS contrant, the remanng constrants are straght-forward. In 6), all of the ncomng workload must be served by some DCs. Moreover, 7) lmts the number of actve servers and 8) means that the total workload assgned to a DC must be less than ts capacty. B. Non-Cooperatve Prcng Game n Stage I In ths stage, we frst descrbe the utlty s revenue and cost models to form the ndvdual objectve of each utlty s proft maxmzaton. We next formulate the non-cooperatve prcng game between utltes. 1) Utlty Revenue s Model: The optmal energy consumpton of DCs at tme t that can be obtaned from solvng DC depends on prces p t),, of all utltes. Denote the correspondng optmal power demand by e pt)), where pt) := {p t)} I. We further assume that due to the grd regulatons at each regon, the lower and upper bound of the real-tme prce should be mposed and denoted by p l and pu,, t, respectvely. Furthermore, besdes the power demand of DCs, each utlty has ts own background load e.g. resdental demand). Snce there are consderable works focusng on the resdental DR programs, we assume that the background load of utlty, denoted by B p t)), also responds to the prce and can be modeled by the followng functon B l, p t) p l ; B pt)) = α βp t), p l p t) p u ; 1) B u, p t) p u, where B l and Bu are the mnmum of maxmum background demands of ste due to the physcal constrants of consumers.e. maxmum and mnmum power of electrc devces or vehcles). Based on the total power requested by DCs and background s demands, the revenue of utlty at tme t s gven by rev pt)) = e pt)) + B p t)) ) p t). 11) 2) Utlty Cost s Model: On the other hand, every utlty ncurs a cost when t serves the customers load. When load ncreases, the utlty s cost also ncreases snce normally blackouts happen due to the overload, whch s a dsaster to any utltes. Hence, we can model the utlty s cost based on a wdely-used electrc load ndex ELI) as follows ) 2 e pt)) + B p t)) cost pt)) = γeli = γ C t), C t) where C t) s utlty capacty at tme t, and γ reflects the weght of the cost. ELI s an mportant economc ndcator where a hgh value of ELI notfes the utlty to spend more for stablty nvestment [9]. 3) Stage-I Prcng Game Formulaton: In realty, the geodstrbuted utltes usually have no communcaton exchange to optmze the socal performance. Instead, each utlty at tme slot t has ts own goal to maxmze ts proft, whch s defned as the dfference between revenue and cost as follows u p t), p t)) = rev pt)) cost pt)), 12) where p t) denotes the prce vector of other utltes except. Ths notaton comes from an observaton that there s a game between utltes because the proft of each utlty not only depends on ts energy prce but also on the others. Hence, the Stage-I utlty proft maxmzaton game s defned as follows Players: the utltes n the set I; Strategy: p l p t) p u, I, t T ; Payoff functon: T t=1 u p t), p t)), I. III. TWO-STAGE STACKELBERG GAME: EQUILIBRIA AND ALGORITHM In ths secton, we frst apply the backward nducton method to solve the Stackelberg game. Then, we propose an teratve algorthm to reach an equlbrum of ths game. A. Backward Inducton Method 1) Optmal Solutons at Stage II: We realze that the stage-ii DCs cost mnmzaton can be decomposed nto ndependent problems at each tme slot t. Henceforth, we only consder a specfc tme perod and drop the tme dependence notaton for ease of presentaton. In ths stage, DCs cooperate wth each other to mnmze the total cost by determnng the workload allocaton λ and the number of actve servers s at each DC. It s easy to see that the DCs cost mnmzaton s a convex optmzaton problem. Frst, we observe that constrant 4) must be actve because otherwse the DSP can decrease ts energy cost by reducng s t). Hence, we have 4) s equvalent to [ 1 s λ ) = λ + µ D )] S 1, 13) where [.] y x s the projecton onto the nterval [x, y] and D := D d. In practce most DCs can have enough number of servers to serve all requests at the same tme due to the lluson of nfnte capacty of DCs [4]. Therefore, we adopt s λ ) =

4 1 µ λ + D ) 1 n the sequel. By substtutng ths s λ ) nto the objectve of DC, we have an equvalent problem DC as follows DC : mn. λ I I =1 f λ ) 14) s.t. λ = Λ, =1 15) λ,, 16) ) where f λ ) := ωd λ 2 +p a + b µ λ +p e b + b D 1 µ ). It can be seen that DC s a strctly convex problem, whch has a unque soluton. Snce DSP lkes to have λ >,, n order to utlze all DCs resources, we characterze the unque soluton of DC and a necessary condton to acheve ths soluton wth the optmal λ >,, as the followng result. Theorem 1. Gven a prce vector p, we have the unque solutons of Stage-II DC problem as follows only f λ = ν p A 2ωd >, 17) s = 1 µ λ + D 1 ),, 18) ω > ωth 1 := ˆd max {p A } I ) p A /d /2Λ, 19) =1 where ˆd := I =1 1/d, A := a + b µ and ν = 2ωΛ + ) I =1 p A /d. 1ˆd We can consder condton 19) as a lower bound) gudelne for DSP to choose an approprate weght factor ω to ensure all DCs have postve requests. 2) Nash Equlbrum at Stage I: We contnue to characterze the Nash equlbrum of the Stage-I game based on the Stage-II solutons. In the non-cooperatve game, one of the most mportant questons s whether there exsts a unque Nash equlbrum. In ths Stage-I game, gven all other utltes strateges p, a natural strategy of utlty s the best response strategy as follows BR p ) = arg where u p, p ) = max p l p pu u p, p ),. 2) ) e p) + B p ) ) e 2 γc p)+bp) C and e p) = a λ + b s ) + eb. Wth λ and s obtaned from Theorem 1, e p) s equal to A 2 p 1 1) + 2ωd ˆdd A 2ω ˆdd j A j p j d j p + A Λ ˆdd + b µ D + e b. When all utltes play best response strateges, a Nash equlbrum p e s a profle that satsfy p e = BR p e ),,.e. every utlty s strategy s ts best response to others strateges. Then we have ths result. Theorem 2. Exstence and Unqueness) There exst a Nash equlbrum of the Stage-I game. Furthermore, f { A ω ωth 2 j := max A j/d j A 2 ˆd1 } 1/d ˆd)) 2β ˆdd, 21) then startng from any ntal pont, the best response strateges converge to a unque Nash equlbrum p e of the Stage-I game. B. Dstrbuted Algorthm We frst descrbe the detaled operatons of the proposed algorthm and provde ts convergence performance. Next, we dscuss about the practcal mplementaton ssue of the algorthm. Algorthm 1 Demand Response of Data Center wth Real-tme Prcng 1: ntalze: Set k =, p ) = p u,, and ω satsfes 21); 2: repeat 3: Utlty broadcasts ts p k) to all customers, ; 4: The front-end server collects p k) from all DCs, updates e p)k) and send t back to DC, ; 5: Each DC reports ts e p)k) to the local utlty; 6: Utlty receves the demand responses from the local DC e p)k) and background users B p) k), then solves p k+1) = BR p k)), ; 7: untl p k+1) p k) < ɛ. 1) Proposed Algorthm s Operatons and Convergence: We contnue proposng a dstrbuted algorthm, shown n Algorthm 1 Alg. 1), that can acheve the Nash equlbrum. We assume that Alg. 1 operates at the begnnng of each prcng update perod.e. one hour) and the algorthm runs for many teratons communcaton rounds wth a parameter k) untl t converges to a prce settng equlbrum. Here, based on the total ncomng workload, the front-end server of the DSP frst collects all prces from ts local DCs and calculate the optmal energy consumpton lne 4). After that the frontend server wll feedback these energy consumpton data to ts local DCs, whch then forwards ts own nformaton to the local utlty lne 5). Each utlty solves ts own proft maxmzaton problem to fnd an optmal prce, then broadcasts ths prce to ts local DCs and background customers lne 6). The process repeats untl the game reach the Nash equlbrum as the prces converge lne 7). At ths state the prce settng s fnalzed and appled to the whole tme slot t. We can see that Alg. 1 converge to a unque Nash equlbrum of Stage-I game accordng to the best response strateges by Theorem 2. 2) Practcal Issues and Implementaton Dscusson: Frst, we assume the DSP deploys a front-end server to dstrbute the ncomng workload to DCs. Ths can be done by usng varous practcal solutons such as ncorporatng the authortatve DNS servers whch s used by Akama) or HTTP ngress proxes whch s used by Google and Yahoo) nto the front-end servers. Furthermore, n realty there s only a sub-set of

5 Workload ebt) MW) FIU Google Fg. 1. The request rate from FIU top) and batch job power from Google bottom). Prces $/MW) Hour Councl Bluffs, IA Lenor, NC Baselne 1 Baselne 2 Alg Hour Fg. 2. Optmal prces at sx locatons. TABLE I AVERAGE OPTIMAL PRICES COMPARISONS WITH γ EFFECT Stes FIU Baselne Alg. 1 Alg. 1 Alg. 1 1 γ = 1 γ = 4 γ = DCs to whch a workload type can be routed to due to the avalablty resource constrant of each DC. Ths ssue can be easly addressed by ncorporatng more constrant nto our model such as [1], and n practce we can mplement t by classfyng the workload types at front-end server before routng. Second, we assume that DCs communcate wth ts front-end server by choosng one of the egress lnks of ts Internet Servce Provder ISP). Specfcally, the total tme of one teraton conssts of the transmsson tme and computatonal tme. Whle the transmsson tme from utltes to DCs and vce versa) s from 1 to 1 ms over a broadband speed of 1 Mbps, t s from 5 to to 1 ms for a one-way communcaton between DCs and the front-end servers over a current ISP s path. The computatonal tme depends on the processng power of the front-end server and smart meters on calculatng 2), whch s low-complexty problem and can be n the tme-scale of mcrosecond [11]. Based on our smulaton results, the equlbrum can be reached n less than 5 teratons, whch means that the total tme of Alg. 1 can be approxmately one second for each one-hour tme slot. IV. TRACE-BASED SIMULATIONS In ths secton, we conduct trace-based smulatons to valdate our analyss and evaluate the performance of Alg. 1. Frst, we present the smulaton setups. Next, we descrbe the baselne prcng methods for comparson. Fnally, we show the results and analyze the performance comparson. A. Setups DCs cost $) Utltes proft $) γ γ Baselne 1 Alg. 1 Fg. 3. Effect of γ to average DCs cost and utltes proft. 1) DCs: We consder sx geo-dstrbuted DCs where ther PUEs are set to. The homogeneous servers of sx DCs have peak power of 2 W and dle power of 1 W, and the servce rate of each server s chosen unformly between 1.1 and 1.2. We set ω to 1 to satsfy 19) wth d s proportonal to the dstance from front-end server and D s chosen unformly between 1 and 3 ms,. There are two realstc traces that we use for the smulaton. The frst trace s the ncomng workload at the front-end server, whch s scaled respectvely to servce rates and shown n Fg. 1. Ths data s profled from January 1 to June 3, 212, at the Florda Internatonal Unversty FIU) [5]. The second trace s the power demand of delay-tolerant batch jobs e b t) of Google by recent study [12]. The workload seres and batch job power demand spans over 3 days and each pont of seres s a one-hour perod. 2) Utltes: Snce lackng the publc nformaton of local utltes, we assume that at each tme slot all utltes have the capactes C t) unformly dstrbuted n the range of 25 and 3 MW, whch s a standard measure for a medum-sze utlty. The lower and upper bounds of the real-tme prce, p l and pu, are set to 1 and 3 $/MWh), respectvely. The utlty cost parameter γ s set to 1 unless otherwse stated. Regardng to the resdental power demand Bp), α and β parameters are chosen unformly n the range of [25, 3] and [.25,.3], respectvely.

6 PAR Baselne 1 Baselne Alg PAR Baselne 1 Baselne Alg PAR Baselne 1 Baselne Alg Councl Bluffs, IA Lenor, NC a).5 Councl Bluffs, IA Lenor, NC b) Fg. 4. PAR at sx locatons wth: a) γ = 1, b) γ = 4, c) γ = 8..5 Councl Bluffs, IA Lenor, NC c) B. Baselne Prcng Schemes for Comparson We consder two baselne prcng schemes for the smulaton comparson as follows. 1) Baselne 1: The frst baselne s based on the proposed dynamc prcng scheme of [8]. At each utlty, ths prcng scheme can be brefly descrbed as follows p t + 1) = δp D t) P S t)) + p t), 22) where P D and P S are the power demand and supply of utlty at tme t. We set δ to.5 n all smulaton scenaros. 2) Baselne 2: The second baselne s based on the Google s contract wth ther local utltes. Accordng to the emprcal study n [13], there are sx Google s DCs powered by ther local utltes at the followng locatons: ; Councl Bluffs, IA; ; Lenor, NC; and. In these locatons, Google s DCs are nfered to have longterm contracts wth ther local utltes as the followng fxed rates [37, 42.73, 36.41, 4.68, 44.44, 39.97] $/MWh, respectvely. We use ths baselne manly for PAR comparsons snce t s not far to compare statc prces versus dymamc prces n terms of cost or proft. C. Results We frst show the optmal prces by Alg. 1 to compare wth other baselne schemes. Then we compare the total DCs cost and utltes proft. Fnally, we compare the PAR performance of three schemes. 1) Optmal solutons: We frst provde a sample-path optmal prces of three schemes at sx locatons n Fg. 2 correspondng to two workload traces. Snce Baselne 1 and Alg. 1 employ dynamc prcng mechansms, we can observe that the utltes prces of these two schemes vary accordng to the workload pattern. We also observe the effect of mgraton cost to the optmal prces n ths fgure. Snce the nearest DCs to the front-end server are stes 2 and 3, Fg. 2 shows that all dynamc prcng schemes set hgh prces at these stes compared wth the other stes. Furthermore, we also nvestgate the effect γ n Stage I snce our smulaton shows that the average prces of Alg. 1 are not affected by ω. Table I shows that f we ncrease γ, then the Alg 1 s optmal prces also ncrease snce the hgher the weght utltes ELI cost factor s, the more conservatve utltes are n terms of relablty by rasng the prces. We also see that Baselne 1 always overprces Alg. 1. 2) Total DCs cost and utltes proft: We also evaluate the effect of parameter γ to average DCs cost and utltes proft n Fg. 3. Frst, we can see that Baselne 1 wth hgher prces has hgher DCs cost and utltes proft than those of Alg. 1. Therefore, Alg. 1 can gve more ncentves to encourage the DCs to jon the DR program. Second, we can see that when γ ncreases, the utltes proft of both schemes decrease due the cost n 12). Wth Alg. 1, we see that small γ s favorable because t can provde low DCs cost and hgh utltes proft. 3) PAR: The fnal factor that we examne s PAR, whch s one of the most mportant metrcs to measure the effectveness of desgns for smart grd snce the fluctuaton of energy consumpton between peak and off-peak hours ndcate power grd s relablty and robustness. Reducng PAR s the ultmate goal of any DR program desgns, so s our proposed Alg. 1. Fg 4 compares the PAR of three schemes wth dfferent γ. The most mportant observaton s that the PAR s performance of Alg. 1 outperforms those of other schemes over tme and space sgnfcantly. V. CONCLUSION AND FUTURE WORK We study the DR of geo-dstrbuted DCs usng smart grd. We frst formulate ths DR program nto a two-stage Stackelberg game to model the nteractons between utltes and DCs. Specfcally, n ths game the role of each utlty s settng a prce to maxmze ts proft, whle the DCs mnmze ts cost. We then characterze the exstence of a Stackelberg equlbrum of ths game where all utltes agree on a stable prce settng wthout devaton ntenton. We next develop an teratve and dstrbuted algorthm to reach one equlbrum pont. We valdate and our proposal s effectveness wth the smulaton results based on realstc traces. ACKNOWLEDGMENT Ths research was funded by the MSIP Mnstry of Scence, ICT and Future Plannng), Korea n the ICT R&D Program 214. We sncerely thank Cheng Wang for provdng the useful traced data.

7 REFERENCES [1] A. Quresh, R. Weber, H. Balakrshnan, J. Guttag, and B. Maggs, Cuttng the electrc bll for nternet-scale systems, n Proc. ACM SIGCOMM 29, Barcelona, Span, 29, pp [2] L. Rao, X. Lu, L. Xe, and W. Lu, Mnmzng Electrcty Cost: Optmzaton of Dstrbuted Internet Data Centers n a Mult-Electrcty- Market Envronment, n Proc. IEEE INFOCOM, San Dego, CA, USA, Mar. 21, pp [3] N. H. Tran, S. Ren, Z. Han, S. man Jang, S. I. Moon, and C. S. Hong, Demand Response of Data Centers:A Real-tme Prcng Game between Utltes n Smart Grd, Tech. Rep., 214. [Onlne]. Avalable: [4] Z. Lu, M. Ln, A. Werman, S. H. Low, and L. L. Andrew, Greenng geographcal load balancng, n Proc. ACM SIGMETRICS, San Jose, Calforna, USA, Jun. 211, pp [5] S. Ren and Y. He, COCA: onlne dstrbuted resource management for cost mnmzaton and carbon neutralty n data centers, n Proc. SC13 Int. Conf. Hgh Perform. Comput. Networkng, Storage Anal., Denver, Colorado, USA, Nov. 213, pp. 39:1 39:12. [6] Z. Lu, A. Werman, Y. Chen, B. Razon, and N. Chen, Data center demand response: Avodng the concdent peak va workload shftng and local generaton, Perform. Eval., vol. 7, no. 1, pp , Oct [7] P. Wang, L. Rao, X. Lu, and Y. Q, D-Pro: Dynamc Data Center Operatons Wth Demand-Responsve Electrcty Prces n Smart Grd, IEEE Trans. Smart Grd, vol. 3, no. 4, pp , Dec [8] Y. L, D. Chu, C. Lu, and L. Phan, Towards dynamc prcng-based collaboratve optmzatons for green data centers, n IEEE 29th Int. Conf. Data Eng. Work., Brsbane, Australa, 213, pp [9] H. Wang, J. Huang, X. Ln, and H. Mohsenan-Rad, Explorng smart grd and data center nteractons for electrc power load balancng, ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 3, pp , Jan [1] P. X. Gao, A. R. Curts, B. Wong, and S. Keshav, It s not easy beng green, n Proc. ACM SIGCOMM, Helsnk, Fnland, Aug. 212, pp [11] S. Boyd and L. Vandenberghe, Convex Optmzaton. Cambrdge Unversty Press, Mar. 24. [12] C. Wang, B. Urgaonkar, and Q. Wang, Data center cost optmzaton va workload modulaton under real-world electrcty prcng, Arxv Prepr. arxv , pp. 1 14, 213. [Onlne]. Avalable: http: //arxv.org/abs/ [13] H. Xu and B. L, Reducng Electrcty Demand Charge for Data Centers wth Partal Executon, arxv Prepr. arxv , 213. [Onlne]. Avalable: