- and Energy-Aware Load Dsrbuon Across Daa Ceners Ken Le, Rcardo Banchn, Margare Maronos, and Thu D. Nguyen Rugers Unversy Prnceon Unversy Inroducon Today, many large organzaons operae mulple daa ceners. The reasons for hs nclude naural busness dsrbuon, he need for hgh avalably and dsaser olerance, he sheer sze of her compuaonal nfrasrucure, and/or he desre o provde unform access mes o he nfrasrucure from wdely dsrbued clen ses. Regardless of he reason, hese organzaons consume sgnfcan amouns of energy and hs energy consumpon has boh a fnancal and envronmenal cos. Ineresngly, he geographcal dsrbuon of he daa ceners ofen exposes many opporunes for opmzng energy consumpon and coss by nellgenly dsrbung he compuaonal workload. We are neresed n hree such opporunes. Frs, we seek o explo daa ceners ha pay dfferen and perhaps varable elecrcy prces. In fac, many power ules now allow consumers o choose hourly prcng, e.g. []. Second, we seek o explo daa ceners ha are locaed n dfferen me zones, whch adds an exra componen o prce varably. For example, one daa cener may be under peakdemand prces whle ohers are under off-peak-demand prces. Thrd, we seek o explo daa ceners locaed near ses ha produce renewable (hereafer called green ) elecrcy o reduce brown energy consumpon ha s mosly produced by carbon-nensve means, such as coal-fred power plans. To make our nvesgaon of hese degrees of freedom more concree, n hs paper we consder mul-daacener Inerne servces, such as Google or Tunes. These servces place her daa ceners behnd a se of fron-end devces. The fron-ends are responsble for nspecng each clen reques and forwardng o one of he daa ceners ha can serve, accordng o a reques dsrbuon polcy. Despe her wde-area dsrbuon of requess, servces mus srve no o volae her servcelevel agreemens (SLAs). Ths paper proposes and evaluaes a framework for opmzaon-based reques dsrbuon. The framework enables servces o manage her energy consumpon and coss, whle respecng her SLAs. I also allows servces o ake full advanage of he degrees of freedom menoned above. Based on he framework, we propose wo reques dsrbuon polces. For comparson, we also propose a greedy heursc desgned wh he same goals and consrans as he oher polces. Operaonally, an opmzaon-based polcy defnes he fracon of he clens requess ha should be dreced o each daa cener. The fron-ends perodcally (e.g., once per hour) solve he opmzaon problem defned by he polcy. Afer fracons are compued, he fron-ends abde by hem unl hey are recompued. The heursc polcy operaes que dfferenly. Durng each hour, frs explos he daa ceners wh he bes power effcency, and hen sars explong he daa ceners wh he cheapes elecrcy. Our evaluaon uses a day-long race from a commercal servce. Our resuls show ha he opmzaonbased polces can accrue subsanal cos reducons by nellgenly leveragng me zones and hourly elecrcy prces. The resuls also show ha we can explo green energy o acheve sgnfcan reducons n brown energy consumpon for small ncreases n cos. Relaed work. The vas majory of he prevous work on daa cener energy managemen has focused on a sngle daa cener. We are no aware of any prevous work ha addresses load dsrbuon across daa ceners wh respec o her energy consumpon or energy coss. Moreover, we are no aware of oher works on leveragng me zones, varable elecrcy prces, or green energy sources. The excepon here s [], whch leverages elecrcy prce dversy o shu down enre daa ceners when her elecrcy coss are relavely hgh. Fnally, we know of no prevous work on opmzaon-based reques dsrbuon n Inerne servces, besdes our own [8]. However, our prevous work dd no address energy ssues, me zones, or heurscs a all. 2 Reques Dsrbuon Polces We assume ha a fron-end s chosen o frs handle a clen reques va round-robn DNS or some oher hghlevel polcy. The fron-ends execue one of our polces and forward each reques o a daa cener ha can serve. Typcally, a reques can only be served by 2 or 3 mrror daa ceners; furher replcang conen would ncrease he sae-conssency raffc whou a meanngful benef
Overall = ( f () LT () ()) + ( B (offered, )) () Polcy EPrce: () = c () and B (offered, ) = b (offered, ) Polcy GreenDC: () = c green () and B (offered, ) = b green (offered, ), f green energy consumed so far GE () = c () and B (offered, ) = b (offered, ), oherwse (2). f ().e., each fracon canno be negave. 2. f() =.e., he fracons for each reques ype need o add up o. 3. (f () LR()) LC.e., he offered load o a daa cener should no overload. 4. (f() LT () CDF(L, offered )) / (3) LT () P.e., he SLA mus be sasfed. Symbol Meanng f () % requess o be forwarded o cener Overall Toal energy cos ($) (), c () Avg. cos ($) of a reques a cener c green () Avg. cos ($) of a reques a cener usng green energy B (offered, ), Base energy coss ($) of cener b (offered, ), under offered load b green (offered, ) GE Amoun of green energy ha green cener can consume LC Load capacy (reqs/sec) of cener LR() Expeced peak servce rae (reqs/sec) LT() Expeced oal servce load (#reqs) offered LR() mes f () (reqs/sec) CDF (L, offered ) Expeced % requess ha complee whn L me, gven offered load Table : Framework parameers. () represens me. n avalably or performance. The reply s sen o he orgnal fron-end, whch n urn forwards o he clen. 2. Prncples and Gudelnes For our polces o be praccal, s no enough o mnmze energy coss; we mus also guaranee hgh performance and avalably. Our polces respec hese requremens by havng he fron-ends: () preven daa cener overloads; and (2) monor he response me of he daa ceners, and adjus he reques dsrbuon o correc any performance or avalably problems. We assume ha he servce has a sngle SLA wh s cusomers, whch s enforced on a daly bass, he accounng perod. The SLA s specfed as (L, P ), meanng ha a leas P % of he requess mus complee n less han L me, as observed by he fron-end devces. The SLA guaranee provded by our polces and framework can be combned wh Inerne QoS approaches o acheve end-o-end guaranees []. Noe he SLA defnon mples ha he servce does no need o selec a fron-end devce and daa cener ha are closes o each clen for he lowes response me possble; all needs s o have respeced he SLA a he end of each accounng perod. We assume ha each daa cener reconfgures self by leavng only as many servers acve as necessary o servce he expeced load for he nex hour (plus an addonal 2% slack for unexpeced ncreases n load); oher servers can be urned off, as n [4, 5, 6, 9]. 2.2 Opmzaon-Based Dsrbuon Our framework comprses he parameers lsed n Table. Usng hese parameers, we can formulae opmzaon problems defnng he behavor of our reques dsrbuon polces. The opmzaon seeks o fnd he fracon f () of requess ha should be sen o each mrror daa cener, durng epoch. (An epoch s defned as a perod of fxed fracons. There can be many epochs durng a sngle accounng perod.) The nex subsecon descrbes wo specfc opmzaon problems (polces). Secon 2.2.2 descrbes he nsanaon of he parameers. Secon 2.2.3 dscusses how o solve he problems. 2.2. Problem Formulaons Polcy EPrce: Leveragng me zones and varable elecrcy prces. The f () fracons should mnmze he overall energy cos, Overall. Equaon defnes Overall wh wo addve componens. The frs represens he energy cos of processng he clen requess ha are offered o he servce. The second represens he base energy cos,.e. he cos of he energy ha s spen when he acve servers are dle. In he EPrce polcy, he per-reques and he base energy cos B have rval defnons (Equaon 2). Overall should be mnmzed under he consrans ha follow he equaons. Polcy GreenDC: Leveragng daa ceners powered by green energy. The formulaon above does no dsngush daa ceners based on her energy source. However, we expec ha daa ceners wll ncreasngly ofen be locaed near sources of green energy, such as wnd and solar farms. In hs scenaro, he same servce could have some daa ceners ha are powered by brown energy (brown daa ceners), and ohers ha are powered by green energy (green daa ceners). Because he supply of green energy may no be enough o power a daa cener hroughou he enre perod, green daa ceners 2
mus also be conneced o he regular elecrcal grd. To formalze hs scenaro, we can redefne and B for he green daa ceners as n he GreenDC polcy (Equaon 2), where GE s he amoun of green energy ha green cener can consume durng he accounng perod. The defnons of and B for brown daa ceners say he same as before. Oher opons. We have no ye explored servces wh sesson sae,.e. sof sae ha only lass a user s sesson wh he servce. In such servces, he dsrbuon s consraned snce all requess of a sesson mus be sen o he same daa cener. Neverheless, s easy o exend our work o handle sessons by () esmang he average number of requess per sesson; and (2) compung fracons ha gude he dsrbuon of he frs reques of a sesson. I s also farly easy o handle () requess ha nvolve wres o perssen sae and (2) mulple reques ypes, nsead of averagng across all ypes lke we do now. We wll explore hese ssues n our fuure work. 2.2.2 Insanang Parameers To nsanae he parameers of our formulaons exacly, he fron-ends would have o communcae and coordnae her decsons. To avod hese overheads, we explore a smpler approach n whch he opmzaon problem s solved ndependenly by each of he fron-ends. If he fron-ends guaranee ha he consrans are sasfed from her ndependen pons of vew, he consrans wll be sasfed globally. In hs approach, LT () and LR() (and consequenly offered ) are defned for each fron-end. In addon, he load capacy of each daa cener s dvded by he number of fron-ends. To nsanae CDF, each fron-end collecs he recen hsory of response mes of cener when he fron-end drecs offered load o. For hs purpose, each fron-end has a able of hese <offered load, percenage> enres for each daa cener ha s flled over me. Smlarly, we creae a able of <offered load, base energy cos> enres o nsanae B. 2.2.3 Solvng he Opmzaon Problem The soluon for an enre accounng perod provdes he bes energy cos. However, such a soluon s only possble when we can predc fuure load nenses, elecrcy prces, and daa cener response me dsrbuons (CDF ). Elecrcy prce predcons are rval when he prce s consan or when here are wo prces. When prces vary hourly, we can use he day-ahead predcon ha s provded by he uly for each day []. Typcally, hese day-ahead prces are reasonably good predcons of acual prces. For predcng load nenses, we consder Auo-Regressve Movng Average (ARMA) modelng [3]. We do no aemp o predc CDF. Insead, Characersc Opmzaon CA-Heursc (EPrce & GreenDC) Accounng perod day day Epoch lengh 4 hours hour Load predcons Per fron-end Per fron-end for enre day for nex hour Energy prce predcons Enre day Nex hour Recompuaon decson Epoch boundary Epoch boundary Communcaon wh DCs Yes Yes Table 2: Man characerscs of dsrbuon approaches. we assume he curren CDF ables as predcons. We canno use fas Lnear Programmng (LP) solvers, because solvng for an enre day a once nvolves a few non-lnear funcons (e.g., B and CDF ). Insead of LP, we use Smulaed Annealng [7] and dvde he day no sx 4-hour epochs,.e. =..6. We wll consder runnng an LP solver every hour n our fuure work. Because he assumpons/predcons ha we make/use when compung a soluon may become nvald/naccurae over me, we mus check for devaons. If here s any sgnfcan devaon a an epoch boundary, we recompue he soluon. We mus also recompue f a daa cener becomes unavalable (or, n our second formulaon, he green energy expres a a daa cener). In pracce, recompuaons occur a he granulary of mulple hours. Afer a recompuaon and every hour, he fron-ends nform he daa ceners abou her predced loads for he nex hour, so ha hey can reconfgure. The Opmzaon column of Table 2 summarzes our approach. 2.3 Heurscs-Based Reques Dsrbuon We also propose a heursc polcy (CA-Heursc) ha s sll cos-aware, bu s smpler and less compuaonally nensve han he opmzaon-based approach. The heursc s greedy and uses -hour epochs. A each epoch boundary, each fron-end compues R = P E (he number of requess ha mus have lower laency han L), where E s he number of requess he fron-end expecs n he nex epoch. E can be predced usng ARMA for each fron-end. Each fron-end also orders he daa ceners ha have CDF (L, LC ) P accordng o he rao ()/CDF (L, LC ), from lowes o hghes rao. The remanng daa ceners are ordered by he same rao. A fnal ls, called ManOrder, s creaed by concaenang he wo lss of daa ceners. Requess are forwarded o he frs daa cener n Man- Order unl s capacy s me. A ha pon, new requess are forwarded o he nex daa cener on he ls and so on. Afer he fron-end has served R requess n less han L me, can dsregard ManOrder and sar forwardng requess o he cheapes daa cener (lowes ()) unl s capacy s me. A ha pon, he nex 3
Daa Cener Brown energy Green energy Capacy (cens/kwh) (cens/kwh) (reqs/s) DC (Wes US). 5. (solar) 25 DC 2 (Eas US).7 25 DC 3 (Europe) 9.7 8. (wnd) 25 Table 3: Defaul smulaon parameers. Capaces have been scaled down o mach our reques race. cheapes daa cener can be exercsed and so on. If he predcon of he number of requess o be receved n an epoch conssenly underesmaes he offered load, servng R requess whn L me may no be enough o sasfy he SLA. To preven hs suaon, whenever he predcon s naccurae, he heursc adjuss he R value for he nex epoch o compensae. A each epoch boundary, he fron-ends nform he ceners abou her predced loads for he nex epoch. The las column of Table 2 overvews our heursc. 3 Evaluaon 3. Mehodology We mplemened a smulaor of a large Inerne servce. For smplcy, we smulae a sngle fron-end locaed on he Eas Coas of he US. The fron-end dsrbues requess o 3 daa ceners, each of hem locaed on he Wes Coas, on he Eas Coas, and n Cenral Europe. Reques race. Our day-long race s a represenave fracon of he requess receved by Ask.com. Fgure shows he 9h percenle of he acual and ARMApredced reques raes durng each hour. The fgure shows ha he ARMA predcons are very accurae. Our race does no nclude response mes. To generae realsc daa cener response mes, we nsalled a smple servce on 3 PlaneLab machnes n he rgh me zones. The requess were made from a machne a Rugers,.e. our fron-end. We assume ha he average raw processng me of each reques s 2 ms. To mmc he effec of load nensy and nework congeson, we ncrease he response mes based on he load offered o each cener (5% ncrease n me for each 25% ncrease n load). Elecrcy prces, sources, and me zones. We smulae schemes wh one elecrcy prce, wo prces (on/off peak), and hourly prces. For he on/off-peak scheme (On/Off), off-peak hours are from 9pm o 7am. For he hourly scheme (Dynamc), Fgure 2 shows he dayahead and acual brown elecrcy prces we use []. The day-ahead prces predc rends farly accuraely, bu no absolue prces. To mmc dfferen brown elecrcy prces for each daa cener, we smply shf our defaul prces 3 hours earler or 6 hours laer. To make all prcng schemes comparable, he prces for he consan and on/off-peak schemes are compued based on he real prces n Fgure 2. When we consder green daa ceners, her elecrcy prce s always assumed consan. We also assume ha he amoun of green energy avalable daly a each green se s enough o process 25% of he requess n he race. The consan brown and green prces are lsed n Table 3. Oher parameers. We assume ha a reques consumes 6 J of dynamc energy o process by 2 machnes, ncludng coolng, converson, and delvery overheads. Ths s equvalen o consumng 5 W of dynamc power per machne durng reques processng. By defaul, we sudy machnes ha are fully energy-proporonal [2],.e. hey consume no base energy. In addon, we sudy he mpac of hs assumpon. The SLA requres 9% of he requess o complee n 7 ms (processng me plus 5 ms) or less. The SLA was sasfed a he end of he accounng perod (one day) n all our smulaons. Table 3 lss he daa cener capaces. -unaware dsrbuon. As he smples bass for comparson, we use a cos-unaware polcy (CU- Heursc) ha s smlar o CA-Heursc. I orders daa ceners accordng o performance,.e. CDF (L, LC ), from hghes o lowes. Requess are forwarded o he frs daa cener on he ls unl s capacy s me. A ha pon, new requess are forwarded o he nex daa cener on he ls and so on. Daa cener reconfguraon happens as n CA-Heursc. 3.2 Resuls Effec of cos-awareness and prcng scheme. Fgure 3 depcs he energy cos of he EPrce, CA-Heursc, and CU-Heursc polces under he hree (brown) elecrcy prcng schemes. The fgure shows many mporan resuls: () As expeced, boh cos-aware polces reduce coss compared o CU-Heursc, even under consan prcng; (2) he On/Off and Dynamc schemes enable sgnfcan cos reducons compared o consan prcng, especally under EPrce; and (3) EPrce always acheves lower cos han CA-Heursc. In fac, combnng cos-awareness and dynamc prcng enables EPrce o reduce cos by 25%. In general, EPrce behaves beer han CA-Heursc because ofen uses he cheapes bu wors-performng daa cener (Europe), nsead of he expensve bu bes-performng daa cener (US Eas Coas). The reason s ha EPrce predcs ha can compensae for he poor performance of he European daa cener durng fuure perods of low load. Effec of me zones. Fgure 3 assumes ha each daa cener s n a dfferen me zone. When hs s no he case, servng a reques coss he same a any daa cener. For hs reason, all polces acheve he same cos, regardless of prcng scheme. Ths cos s slghly hgher han ha of CU-Heursc n Fgure 3, suggesng ha mulple me zones are crcal o enable cos savngs. 4
35 2 3 8 6 Reqs/s 25 2 5 Cens/KWH 4 2 8 6 Acual Predced 5 4 8 2 6 2 Hour Fgure : Acual and predced load nenses. 4 2 Acual Day-Ahead 4 8 2 6 2 Hour Fgure 2: Acual and day-ahead brown elecrcy prces..4.2.8.6.4.2 EPrce CA-Heursc CU-Heursc.97.98.98..93.92.9.75.78 and Brown Energy.8.6.4.2.8.6.4.2.3.23.35 GreenDC CA-Heursc CU-Heursc All-Brown...85.87.65.4.2.8.6.4.2 EPrce CA-Heursc CU-Heursc....94.96.96.87.89.76 Dynamc On/Off Consan Fgure 3: Prcng and cos-awareness. Effec of green daa ceners. Fgure 4 depcs he cos and brown energy consumpon of he GreenDC polcy, CA-Heursc, and CU-Heursc, under dynamc prcng. The resuls are normalzed agans he defaul resuls for EPrce wh dynamc prcng ( All-Brown ),.e. he lefmos bar n Fgure 3. Fgure 4 shows ha GreenDC can decrease brown energy consumpon by 35% by leveragng he green daa ceners a only a 3% cos ncrease. The heursc polces save subsanally less brown energy a much hgher coss han GreenDC. Agan, he reason s ha he heursc polces ofen use he Eas Coas daa cener, nsead of he wnd-based European daa cener. Effec of base energy. The resuls above all assume ha servers do no consume any power when dle. Fgure 5 quanfes he effec of he base energy by comparng he defaul resuls for EPrce, CA-Heursc, and CU-Heursc o hose when a server consumes 75W and 5W when dle. We assume ha no daa cener consumes green energy. Ths fgure shows ha ncreasng he base energy reduces he cos savngs achevable by our opmzaon approach. The gans are smalles (bu sll non-rval) for Base = 5W. Ths resul shows ha he benefs of our approach wll ncrease wh me, as servers become more energy-proporonal. 4 Conclusons Brown Energy Fgure 4: Green daa ceners. Was 75 Was 5 Was Fgure 5: Base energy. In hs paper, we proposed a framework for opmzaonbased reques dsrbuon n mul-daa-cener Inerne servces. We also proposed wo polces for managng hese servces energy consumpon and cos, whle respecng her SLAs. The polces ake advanage of me zones, varable elecrcy prces, and green energy. Fnally, we proposed a smple heursc for achevng he same goals. Our evaluaon showed posve resuls. References [] Ameren. Real-me Prces. hps://www2.ameren.com/- RealEnergy/realmeprces.aspx. [2] L. A. Barroso and U. Holzle. The Case for Energy-Proporonal Compung. IEEE Compuer, 4(2), December 27. [3] G. Box and G. Jenkns. Tme Seres Analyss, Forecasng and Conrol. Holden-Day, Incorporaed, 99. [4] J. Chase e al. Managng Energy and Server Resources n Hosng Ceners. In Proceedngs of SOSP, Ocober 2. [5] G. Chen e al. Energy-Aware Server Provsonng and Load Dspachng for Connecon-Inensve Inerne Servces. In Proceedngs of NSDI, Aprl 28. [6] Y. Chen e al. Managng Server Energy and Operaonal s n Hosng Ceners. In Proceedngs of SIGMETRICS, June 25. [7] S. Krkparck, C. D. Gela, and M. P. Vecch. Opmzaon by smulaed annealng. Scence, 22(4598), May 983. [8] K. Le, R. Banchn, and T. D. Nguyen. A -Effecve Dsrbued Fle Servce wh QoS Guaranees. In Proceedngs of Mddleware, November 27. [9] E. Pnhero e al. Load Balancng and Unbalancng for Power and Performance n Cluser-Based Sysems. In Proceedngs of COLP, Sepember 2. [] A. Quresh. Pluggng Ino Energy Marke Dversy. In Proceedngs of HoNes, Ocober 28. [] W. Zhao, D. Olshefsk, and H. Schulzrnne. Inerne Qualy of Servce: An Overvew. Techncal Repor CUCS-3-, Deparmen of Compuer Scence, Columba Unversy, February 2. 5