Energy and Performance Management of Green Data Centers: A Profit Maximization Approach

Transcription

1 Energy and Performance Managemen of Green Daa Ceners: A Profi Maximizaion Approach Mahdi Ghamkhari, Suden Member, IEEE, and Hamed Mohsenian-Rad, Member, IEEE Absrac While a large body of work has recenly focused on reducing daa cener s energy expenses, here exiss no prior work on invesigaing he rade-off beween minimizing daa cener s energy expendiure and maximizing heir revenue for various Inerne and cloud compuing services ha hey may offer. In his paper, we seek o ackle his shorcoming by proposing a sysemaic approach o maximize green daa cener s profi, i.e., revenue minus cos. In his regard, we explicily ake ino accoun pracical service-level agreemens (SLAs) ha currenly exis beween daa ceners and heir cusomers. Our model also incorporaes various oher facors such as availabiliy of local renewable power generaion a daa ceners and he sochasic naure of daa ceners workload. Furhermore, we propose a novel opimizaion-based profi maximizaion sraegy for daa ceners for wo differen cases, wihou and wih behind-hemeer renewable generaors. We show ha he formulaed opimizaion problems in boh cases are convex programs; herefore, hey are racable and appropriae for pracical implemenaion. Using various experimenal daa and via compuer simulaions, we assess he performance of he proposed opimizaion-based profi maximizaion sraegy and show ha i significanly ouperforms wo comparable energy and performance managemen algorihms ha are recenly proposed in he lieraure. Keywords: Green daa ceners, behind-he-meer renewable power generaion, energy and performance managemen, servicelevel agreemens, profi maximizaion, convex opimizaion. I. INTRODUCTION The growing demand for Inerne services and cloud compuing has significanly increased he elecric power usage associaed wih large daa ceners - such as hose owned and operaed by Google, Microsof, and Amazon - over he pas few years. Each daa cener includes hundreds of housands of compuer servers, cooling equipmen, and subsaion power ransformers. For example, consider Microsof s daa cener in Quincy, WA. I has 43,6 square meers of space and uses 4.8 kilomeers of chiller piping, 965 kilomeers of elecric wire, 9,9 square meers of drywall, and.5 meric ons of backup baeries. The peak power consumpion of his faciliy is 48 megawas, which is enough o power 4, homes []. As anoher example, he Naional Securiy Agency is currenly building a massive daa cener a For Williams in Uah which is expeced o consume over 7 megawas elecriciy []. Due o he increasing cos of elecriciy associaed wih daa ceners, here has been a growing ineres owards developing echniques and algorihms o minimize daa ceners energy expendiure. One hread of research focuses on reducing he amoun of energy consumed by compuer servers [3]. Anoher Manuscrip received April, ; revised Augus 5, and November 5, ; acceped December 9,. This work was suppored in par by he Naional Science Foundaion hrough gran ECCS The auhors are wih he EE Deparmen, Universiy of California a Riverside, Riverside, CA, USA, s: {ghamkhari, hamed}@ee.ucr.edu. hread of research is dynamic cluser server configuraion o reduce he oal power consumpion by consolidaing workload only on a subse of servers and urning off he res, during low workload hours [4], [5]. A similar approach is dynamic CPU clock frequency scaling [6], [7]. In his approach, a higher frequency, imposing higher energy consumpion, is chosen only a peak workload hours. Finally, some recen sudies aimed o uilize price-diversiy in deregulaed elecriciy markes as well as locaional-diversiy in renewable power generaion. The idea is o consanly monior he price of elecriciy and he amoun of renewable power generaed a differen regions and forward he workload owards daa ceners ha are locaed in regions wih he lowes elecriciy price [8], [9] or highes renewable power available [], []. While a large body of work has addressed minimizing daa ceners cos, e.g., in [4] [], o he bes of our knowledge, no prior work has addressed he rade-off beween minimizing daa cener s energy expendiure and maximizing heir revenue for various Inerne and cloud compuing services ha hey offer. Such rade-off is due o he fac ha minimizing daa cener s energy cos is achieved essenially by urning off some servers, scaling down CPU clocks, or migraing some workload, which can all poenially lead o degrading he qualiy-of-services offered by daa cener and consequenly is income, considering he sochasic naure of workload. Therefore, in his paper, we seek o ackle his shorcoming by proposing a sysemaic approach o maximize green daa cener s profi, i.e., revenue minus cos. In his regard, we explicily ake ino accoun pracical service-level agremens (SLAs) ha currenly exis beween daa ceners and heir cusomers. In summary, our conribuions are as follows: We develop a mahemaical model o capure he radeoff beween minimizing a daa cener s energy cos versus maximizing he revenue i receives for offering Inerne services. We ake ino accoun compuer server s power consumpion profiles, daa cener s power usage effeciveness, price of elecriciy, availabiliy of renewable generaion, oal workload in erms of he rae a which service requess are received a each ime of day, pracical service-level agreemens and heir parameers for service deadline, service paymen, and service violaion penaly. We propose a novel opimizaion-based profi maximizaion sraegy for daa ceners for wo differen cases, wihou and wih behind-he-meer renewable generaors. The laer is he scenario applicable o green daa ceners. We show ha he formulaed opimizaion problems in boh cases are convex programs; herefore, hey are racable and appropriae for pracical implemenaion. We use experimenal daa, e.g., for workload, price of

2 elecriciy, renewable power generaion, and SLA parameers, o assess he accuracy of he proposed mahemaical model for profi and also he performance of he proposed opimizaion-based profi maximizaion sraegy via compuer simulaions. We show ha our proposed opimizaion-based designs significanly ouperform wo comparable energy and performance managemen algorihms ha have recenly been proposed in he lieraure. The res of his paper is organized as follows. The sysem model and noaions are defined in Secion II. The proposed opimizaion-based profi maximizaion sraegy is presened in Secion III. Simulaion resuls are presened in Secion IV. Conclusions and fuure work are discussed in Secion V. II. SYSTEM MODEL Consider an Inerne or cloud compuing daa cener wih M max compuer servers as shown in Fig.. Nex, we explain he sysem model in erms of power consumpion, price of elecriciy, incoming workload, and qualiy-of-service. A. Power Consumpion The oal amoun of power consumpion in a daa cener is obained by adding he oal power consumpion a he compuer servers o he oal power consumpion a he faciliy, e.g., for cooling, lighing, ec. For a daa cener, power usage effeciveness (PUE), denoed by E usage, is defined as he raio of he daa cener s oal power consumpion o he daa cener s power consumpion a he compuer servers []. The PUE is considered as a measure for daa cener s energy efficiency. Currenly, he ypical value for mos enerprise daa ceners is. or more. However, recen sudies have suggesed ha many daa ceners can soon reach a PUE of.7. A few sae-of-he ar faciliies have reached a PUE of. []. Le P idle denoe he average idle power draw of a single server and P peak denoe he average peak power when a server is handling a service reques. The raio P peak /P idle denoes he power elasiciy of servers. Higher elasiciy means less power consumpion when he server is idle, no handling any service reques. Le M M max denoe he number of servers ha are on a daa cener. The oal elecric power consumpion associaed wih he daa cener can be obained as [3] [6]: P = M[P idle + (E usage )P peak + (P peak P idle )U], where U is he CPU uilizaion of servers. From (), he power consumpion a daa cener increases as we urn on more compuer servers or run servers a higher uilizaion. B. Elecriciy Price The elecriciy pricing models ha are deployed for each region usually depend of wheher he elecriciy marke is regulaed or deregulaed in ha region. The elecriciy prices ofen have fla raes and do no change during he day when he elecriciy marke is regulaed. On he oher hand, he prices may significanly vary during he day when he elecriciy marke is deregulaed as he prices would reflec () he flucuaions in he wholesale elecriciy marke. Some of he non-fla pricing ariffs in deregulaed elecriciy markes include: Day-ahead pricing (DAP), ime-of-use pricing (TOUP), criical-peak pricing (CPP), and real-ime pricing (RTP). Our proposed energy and performance managemen design in his paper is applicable o no only fla rae bu also non-fla rae pricing ariffs. In our sysem model, he insananeous price of elecriciy is denoed by ω. In Secion III, we will use pricing informaion o obain daa cener s cos of elecriciy. C. Renewable Power Generaion In order o reduce cos of elecriciy, a daa cener may be equipped wih behind-he-meer renewable generaors, e.g., a wind urbine, in addiion o being conneced o he power grid. Le G denoe he renewable power generaed by renewable generaors. The amoun of power exchange wih he power grid is obained as P G. If local renewable power generaion is lower han local power consumpion, i.e., P > G, hen P G is posiive and he power flow is in he direcion from he power grid o he daa cener. If P = G hen he daa cener operaes as a zero-ne energy faciliy [7]. If P < G, hen P G is negaive and he power flow is in he direcion from he daa cener o he power grid. However, in his case, wheher he grid compensaes he daa cener s injeced power depends on he marke model being used. Alhough, in some areas, he uiliy pays for he power injeced ino he grid, currenly, here is no specific marke model for behind-hemeer generaion in mos regions in he U.S. Therefore, in our sysem model, while we allow a daa cener o injec is excessive renewable generaion ino he power grid, we assume ha i does no receive compensaion for he injeced power. D. Qualiy-of-Service Because of he limied compuaion capaciy of daa ceners and given he sochasic naure of mos pracical workload, daa ceners canno process he incoming service requess, immediaely afer hey arrive. Therefore, all arriving service requess are firs placed in a queue unil hey can be handled by any available compuer. In order o saisfy qualiy-ofservice requiremens, he waiing ime/queuing delay for each incoming service reques should be limied wihin a cerain range which is deermined by he Service Level Agreemen (SLA). The exac SLA depends on he ype of service offered which may range from cloud-based compuaional asks o video sreaming and HTML web services. Examples of wo ypical SLAs for an Inerne or cloud compuing daa cener are shown in Fig. [8]. In his figure, each SLA is idenified by hree non-negaive parameers D, δ, and γ. Parameer D indicaes he maximum waiing ime ha a service reques can olerae. Parameer δ indicaes he service money ha he daa cener receives when i handles a single service reques before deadline D. Finally, parameer γ indicaes he penaly ha he daa cener has o pay o is cusomers every ime i canno handle a service reques before deadline D. For he Gold SLA in Fig., we have D = 3 ms, δ = 7 5 dollars, and γ = dollars. For he Silver SLA, we have D = ms, δ = 5 5 dollars, and γ =.6 5 dollars.

3 Users Power Grid P G Service Requess Queue P Incoming Service Requess Daa Cener G Fig.. In he sudied sysem model, service requess are firs placed in a queue before hey are handled by one of he available compuer servers. Service Paymen / Penaly (Dollars 5 ) Fig Silver SLA Gold SLA 3 4 Queueing Delay (ms) Two sample service-level agreemens (SLAs) in daa ceners. E. Service Rae Le µ denoe he rae a which service requess are removed from he queue and handled by a server. The service rae depends on he number of servers ha are swiched on. Le S denoe he ime i akes for a server o finish handling a service reques. Each server can handle = /S service requess per second. Therefore, he oal service rae is obained as µ = M M = µ. () As we increase he number of swiched on servers and accordingly he service rae, more service requess can be handled before he SLA-deadline D, which in urn increases he paymens ha he daa cener receives as explained in Secion II-D. On he oher hand, i will also increase he daa cener s power consumpion and accordingly he daa cener s energy expendiure as explained in Secions II-A and II-B. Therefore, here is a rade-off when i comes o selecing he daa cener s service rae, as we will discuss in deail nex. III. PROBLEM FORMULATION The rae a which service requess arrive a a daa cener can vary over ime. To improve daa cener s performance, he number of swiched on servers M should be adjused according o he rae of receiving service requess. More servers should be urned on when service requess are received a higher raes. By monioring service reques rae and adjusing he number of swiched on servers accordingly, only a proporional-o-demand porion of servers will be on. This resuls in reducing daa cener s power consumpion. Because of he ear-and-wear cos of swiching servers on and off, and also due o he delay in changing he saus of a compuer, M canno be change rapidly. I is raher desired o be updaed only every few minues. Therefore, we divide running ime of daa cener ino a sequence of ime slos Λ, Λ,, Λ N, each one wih lengh T, e.g. T = 5 minues. The number of swiched on servers are updaed only a he beginning of each ime slo. For he res of his secion, we focus on mahemaically modeling he energy cos and profi of he daa cener of ineres a each ime slo Λ Λ, Λ,, Λ N as a funcion of service rae µ and consequenly as a funcion of M, based on (). A. Revenue Modeling Le q(µ) denoe he probabiliy ha he waiing ime for a service reques exceeds he SLA-deadline D. Obaining an analyical model for q(µ) requires a queueing heoreic analysis ha we will provide in Secion III-C. Nex, assume ha λ denoes he average rae of receiving service requess wihin ime slo Λ of lengh T. The oal revenue colleced by he daa cener a he ime slo of ineres can be calculaed as Revenue = ( q(µ))δλt q(µ)γλt, (3) where ( q(µ))δλt denoes he oal paymen received by he daa cener wihin inerval T, for he service requess ha are handled before he SLA-deadline, while q(µ)γλt denoes he oal penaly paid by he daa cener wihin inerval T for he service requess ha are no handled before he SLA-deadline. B. Cos Modeling Wihin ime inerval T, each urned on server handles T ( q(µ))λ (4) M service requess. This makes each server busy for T ( q(µ))λ/m seconds. By dividing he oal CPU busy ime by T, he CPU uilizaion for each server is obained as U = ( q(µ))λ M. (5)

4 Replacing () and (5) in (), he power consumpion associaed wih he daa cener a he ime slo of ineres is obained as aµ + bλ( q(µ)) P =, (6) where a = P idle + (E usage )P peak, (7) b = P peak P idle. (8) Muliplying (6) by he elecriciy price ω, he oal energy cos a he ime inerval of ineres is obained as [ ] aµ + bλ( q(µ)) Cos = T ω. (9) C. Probabiliy Model of q(µ) Consider a new service reques ha arrives wihin ime slo Λ. Le Q denoe he number of service requess waiing in he service queue righ before he arrival of he new service reques. All he Q requess mus be removed from he queue, before he new reques can be handled by any available server. Since he daa cener s service rae is µ, i akes Q/µ seconds unil all exising requess are removed from he queue. Hence, he new service reques can be handled afer Q/µ seconds since is arrival. According o he SLA, if Q/µ D, hen he service reques is handled before he deadline D. If Q/µ > D, he service reques is no handled before he deadline D and i is dropped. Therefore, we can model he SLA-deadline by a finie-size queue wih he lengh µd. A service reques can be handled before he SLA-deadline, if and only if i eners he aforemenioned finie size queue. We assume ha he service reques rae has an arbirary and general probabiliy disribuion funcion. On he oher hand, since he service rae µ is fixed over each ime inerval of lengh T, q(µ) can be modeled as he loss probabiliy of a G/D/ queue. Therefore, following he queuing heoreic analysis in [9], we can obain where q(µ) = α(µ) e min n m n(µ), () α(µ) = λ πσ e (µ λ) σ and for each n we have µ (r µ)e (r λ) σ dr () (Dµ + n(µ λ)) m n (µ) =. () nc λ () + n C λ (l)(n l) Here, σ = C λ () and C λ denoes he auo-covariance of he service reques rae s probabiliy disribuion funcion. I is known ha he model in () is mos accurae when he service reques rae can be modelled as a Gaussian Process, bu i also works well for any general probabiliy disribuion [9], as we will confirm hrough simulaions in Secion IV. Our model is Here, since our focus is on energy and performance managemen of daa ceners, he cos model only indicaes he cos of elecriciy. However, oher cos iems can also be included in he model in a similar fashion. l= paricularly more accurae han he exising models ha are based on Poisson workload arrival disribuions, e.g., in []. Before we end his secion, we shall clarify our assumpion on discarding service requess ha are no handled before heir deadlines. The benefis of his approach are addressed in deailed in []. Based on he experimenal cusomer sudies in [], [3], he auhors in [] argue ha for mos daa cener applicaion raffic, a nework flow is useful, and conribues o applicaion hroughpu and operaor revenue, if and only if i is compleed wihin is deadline. For example, services such as Web search, reail, adverisemen, social neworking and recommendaion sysems, while are very differen, share a common underlying heme ha hey need o serve users in a imely fashion. Consequenly, when he ime expires for a service reques, responses, irrespecive of heir compleeness, are shipped ou. However, he compleeness of he responses direcly governs heir qualiy, and in urn, operaor s revenue based on he conraced SLA, as we explained in Secion III- A. Some oher sudies ha similarly assume dropping service requess if hey canno mee he deadline include [4] [7]. D. Profi Maximizaion wihou Local Renewable Generaion A each ime slo Λ, daa cener s profi is obained as P rofi = Revenue Cos, (3) where revenue is as in (3) and cos is as in (9). We seek o choose he daa cener s service rae µ o maximize profi. This can be expressed as he following opimizaion problem: Maximize T λ [( q(µ))δ q(µ)γ] λ µ M max ( ) aµ + bλ( q(µ)) T ω, (4) where q(µ) is as in () and M max denoes he oal number of servers available in he daa cener. We noe ha he service rae µ is lower bounded by λ. This is necessary o assure sabilizing he service reques queue [], [9], []. We also noe ha Problem (4) needs o be solved a he beginning of every ime slo Λ {Λ,..., Λ N }, i.e., once every T minues. E. Profi Maximizaion wih Local Renewable Generaion When a daa cener is supplied by boh he power grid and also a local behind-he-meer renewable generaor, hen he opimum choice of service rae for maximizing profi is obained by solving he following opimizaion problem Maximize T λ [( q(µ))δ q(µ)γ] λ µ M max [ + aµ + bλ( q(µ)) T ω G], (5) where [x] + = max(x, ). Noe ha, (aµ+bλ( q(µ)))/ G indicaes he amoun of power o be purchased from he grid. As discussed in Secion II-C, if his erm is negaive, he daa cener s elecriciy cos will be zero, given he assumpion ha he grid does no provide compensaion for he injeced power.

5 F. Soluion and Convexiy Properies.44 In his Secion, we characerize opimizaion problems (4) and (5), and show ha hey can be solved using efficien opimizaion echniques, such as he inerior poin mehod (IPM) [8]. Firs, consider he following useful heorem. Theorem : The probabiliy funcion q(µ) in () is nonincreasing and convex if he service rae is limied o 6/π µ λ + σ, λ π π σ. (6) Profi / Time Slo (Dollars) Simulaion Opimum 3.7 Analyical Opimum Simulaion Analyical The proof of Theorem is presened in Appendix A. Noe ha, he inerval in (6) can be approximaely expressed as [λ +.77σ, λ σ]. (7) Nex, we noe ha handling a single service reques increases power consumpion of a single server from P idle o P peak for / seconds. This increases he energy cos by ω(p peak P idle )/ = ωb/ and also increases he revenue by δ. Thus, running he daa cener is profiable only if δ ωb/ >. (8) From his, ogeher wih Theorem, we can now provide he following key resuls on racabiliy of Problems (4) and (5). Theorem : Assume ha condiion (8) holds and he service rae µ is limied in he range indicaed in (6). (a) Problem (4) is convex in is curren form. (b) Problem (5) is equivalen o he following convex opimizaion problem: Maximize T λ [( q(µ))δ + q(µ)γ] λ µ M max ( ) aµ + bλ( q(µ)) T ω G Subjec o aµ + bλ( q(µ)) G. (9) The proof of Theorem is given in Appendix B. Noe ha, wo opimizaion problems are called equivalen if hey have equal opimal soluions such ha solving one can readily solve he oher one [8, pp. 3-35]. From Theorem, Problems (4) and (5) are boh racable and can be solved efficienly using convex programming echniques, such as he IPM [8]. An ineresing exension for he design problem in (9) is he case when an SLA requires ha he probabiliy of dropping a packe is upper bounded by a consan L. This requiremen can be incorporaed in our design by adding he following consrain o opimizaion problems (4), (5) and (9): q(µ) L. () However, we can show ha he above consrain is convex. Noe ha, from Theorem, q(µ) is a convex funcion. Therefore, consrain forms a convex se [8, Secion 4..] and he convexiy of problems (4), (9) sill holds. As a resul, he proposed convex opimizaion framework is sill valid Service Rae µ (Reques/Sec) Fig. 3. Comparing analyical and simulaion resuls in calculaing daa cener profi as a funcion of µ over a sample T = 5 minues ime slo. A. Simulaion Seing IV. PERFORMANCE EVALUATION Consider a daa cener wih a maximum of M max = 5, servers. The exac number of swiched on servers M is updaed periodically a he beginning of each ime slo of lengh T = 5 minues by solving Problems (4) and (5) for he cases wihou and wih behind-he-meer renewable power generaion respecively. For each swiched on server, we have P peak = was and P idle = was []. We assume ha E usage =. []. The elecriciy price informaion is based on he hourly real-ime pricing ariffs currenly praciced in Illinois Zone I, spanning from June, o July 9, [9]. We also assume ha =. and consider he SLA o be according o he Gold SLA curve of Fig.. To simulae he oal workload, we use he publicly available World Cup 98 web his daa, spanning from June, 998 o July 9, 988, as he rend for rae of incoming service requess [3]. B. Simulaion Resuls for a Single Time Slo To gain insighs abou he achievable profi, in his secion, we focus on a single ime slo of lengh T = 5 minues a : PM on June 9 and invesigae he soluion of he profi maximizaion problem in (4). In Fig. 3, he analyical profi curve is compared wih he profi curve ha is calculaed using an even-based simulaor. We can see ha he analyical curve is a close approximaion of he simulaion one. Based on he analyical curve, he opimum µ is 74.7 requess/second. This is only 3.7 requess/second greaer han he rue opimum service rae obained from he simulaion curve. Tha is, he opimaliy loss due o analyical modeling error is only.4%. C. Simulaion Resuls for an Enire Day Fig. 4 shows a comparison beween he proposed design wih he one in [] and [] by simulaion over 4 hours operaion of he daa cener based on he June 9 daa. The ime-of-day price of elecriciy [9] as well as he workload [3] are shown in Fig. 4(e) and Fig. 4(d), respecively. We can

6 Normilized Profi Gain (a). Design in [] Design in [] Service Rae µ (Reques/Sec) Opimum Design Design in [] Design in [] (b) Servers Uilizaion, U (c) Workload (Reques/Sec) 5 Elecriciy Price (Cen/kwh) 5 (d) (e) Fig. 4. Performance comparison beween he proposed design and he designs in [] and [] over 4 hours running ime of a daa cener: (a) Normalized profi gain. (b) The choice of service rae µ. (c) Servers uilizaion. (d) Toal workload [3]. (e) Hourly price of elecriciy [9]. see ha he normalized profi gain of he proposed design in Fig. 4(a) is very close o opimal. The normalized profi gain is calculaed as (P rofi P rofi Base ) divided by (P rofi Max P rofi Base ). Here, P rofi Base is he profi obained when we simply se µ = λ [3] and P rofi Max is he maximum of he profi curve obained by simulaion. Furhermore, we can see ha our proposed design can ouperform he wo designs in [] and []. The reason is wo-fold. Firs, our design explicily akes ino accoun a mahemaical model for daa cener s profi as a funcion of service rae. Second, we use an accurae G/D/ queuing model while he designs in [] and [] are based on less accurae M/M/ queueing models which canno capure he workload disribuion well. Finally, Fig. 4(b) shows he opimum choice of service rae, obained from various designs, and Fig. 4(c) shows he server uilizaion U for he case of each design. We can see ha he proposed design can achieve close o opimal service raes as well as close o opimal server uilizaion levels. The designs in [] and [] resul in under and over uilized servers, respecively. Nex, we compare he normalized daily profi gain achieved by our design wih hose obained by he designs in [] and [] over 3 days operaion of he daa cener. The resuls are shown in Fig. 5. We can see ha he proposed design works beer in all days, wih an average opimaliy of 96.3%. I is ineresing o also poin ou ha in 84% of he oal = 88 ime slos being simulaed, he opimum analyical service rae, i.e., he maximizer of he analyical profi curve, drops wihin inerval (6), which is he range in which Problem (4) is a convex program. Finally, he normalized profi gain of he hree aforemenioned designs, when S = / changes, are compared in Fig. 6 for June 9 daa. We can see ha he proposed design is no sensiive o he service ime parameer and can ouperform [] and [] in all cases. The loss probabiliy of dropping a service reques for differen design approaches over 4 hours running ime of he daa cener are shown in Fig. 7. Here, he choice of simulaion parameers are he same as hose in Fig 4. The resuls in Fig. 7 show ha he loss probabiliy for our proposed design is closer o he opimum loss probabiliy, compared o he cases of he designs in [] and []. In order o undersand he cause for his observaion, we noe ha based on he resuls in Fig. 4-(c) he designs in [] and [] under-uilize and overuilize he servers, respecively. For an under-uilized design, he loss probabiliy is lower han he opimum value. For an over uilized design, he loss probabiliy is higher han he opimum value. Clearly, we can draw a similar conclusion based on he loss probabiliy resuls shown in Fig. 7. D. Impac of Behind-he-Meer Renewable Generaion Nex, assume ha he daa cener is equipped wih one local.5 Megawas wind urbine. The power oupu for he urbine is assumed o be as in Fig. 8(a), based on he June wind speed daa available in [3]. In his case, he opimal service rae is obained by solving Problem (5). The corresponding addiional profi (in percenage) due o local renewable generaion is shown in Fig. 8(b). We can see ha local renewable generaion can significanly increase he daa cener s daily profi by offseing par of is energy cos.

7 Normilized Profi Gain Design in [] Design in [] Loss Probabiliy Opimum Design Design in [] Design in [] Days Fig. 5. Daily normalized profi gain across 3 days. Fig. 7. Probabiliy of dropping a service reques due o full queues for various design approaches over 4 hours operaion of daa cener. Normilized Profi Gain Design in [] Design in [] 5 5 Service Time S=/ Wind Power (Megawas) Addiional Profi wih Renewable Generaors (%) (b) (a) Fig. 6. Normalized profi gain as a funcion of S = /. Fig. 8. Addiional profi gained in a sample 4 hours operaion of a daa cener wih local renewable generaion. Wind daa is obained from [3]. V. CONCLUSIONS AND FUTURE WORK In his paper, we proposed a novel analyical model o calculae profi in large daa ceners wihou and wih behindhe-meer renewable power generaion. Our model akes ino accoun several facors including he pracical service-level agreemens ha currenly exis beween daa ceners and heir cusomers, price of elecriciy, and he amoun of renewable power available. We hen used he derived profi model o develop an opimizaion-based profi maximizaion sraegy for daa ceners. We showed, under cerain pracical assumpions, he formulaed opimizaion problems are convex. Finally, using various experimenal daa and via compuer simulaions, we assess he accuracy of he proposed mahemaical model for profi and also he performance of he proposed opimizaion-based profi maximizaion sraegy. The resuls in his paper can be exended in several direcions. Firs, he considered cos model can be generalized o include cos elemens oher han energy cos. Second, given renewable energy forecasing models and day-ahead pricing ariffs, he proposed shor-erm energy and performance managemen can be furher exended o daily or monhly planning. Finally, he obained mahemaical model in his paper can be adjused o also include poenial profi if a daa cener paricipaes in ancillary services marke in he smar grid. REFERENCES [] R. Kaz, Tech ians building boom, IEEE Specrum, vol. 46, no., pp. 4 54, Feb. 9. [] H. Mohsenian-Rad and A. Leon-Garcia, Coordinaion of cloud compuing and smar power grids, in Proc. of he IEEE Smar Grid Communicaions Conference, Gaihersburg, MD, oc. [3] M. K. S. Gurumurhi, A. Sivasubramaniam and H. Franke, Reducing disk power consumpion in servers wih drpm, Compuer, vol. 36, no., pp , Dec. 3. [4] J. Heo, D. Henriksson, X. Liu, and T. Abdelzaher, Inegraing adapive componens: An emerging challenge in performance-adapive sysems and a server farm case-sudy, in Proc. of he IEEE Inernaional Real- Time Sysems Symposium, Tucson, AZ, Dec. 7. [5] E. Pinheiro, R. Bianchini, E. V. Carrera, and T. Heah, Dynamic cluser reconfiguraion for power and performance, in Compilers and Operaing Sysems for Low Power, M. K. L. Benini and J. Ramanujam, Eds. Kluwer Academic Publishers, 3. [6] X. Fan, W. D. Weber, and L. A. Barroso, Power provisioning for a warehouse-sized compuer, in Proc. of he ACM Inernaional Symposium on Compuer Archiecure, San Diego, CA, June 7. [7] J. S. Chase, D. Anderson, P. Thakar, A. Vahda, and R. Doyle, Managing energy and server resources in hosing ceners, in Proc. of ACM Symp. on Operaing sysems principles, Banff, Canada, Oc..

8 [8] L. Rao, X. Liu, L. Xie, and W. Liu, Minimizing elecriciy cos: Opimizaion of disribued inerne daa ceners in a muli-elecriciymarke environmen, in Proc. of IEEE INFOCOM, Orlando, FL,. [9] J. Li, Z. Li, K. Ren, and X. Liu, Towards opimal elecric demand managemen for inerne daa ceners, IEEE Transacions on Smar Grid, vol. 3, no., pp. 83 9, Mar.. [] M. Ghamkhari and H. Mohsenian-Rad, Opimal inegraion of renewable energy resources in daa ceners wih behind-he-meer renewable generaors, in Proc. of he IEEE Inernaional Conference on Communicaions, Oawa, Canada, June. [] Z. Liu, M. Lin, A. Wierman, S. Low, and L. L. H. Andrew, Geographical load balancing wih renewables, in Proc. of he ACM GreenMerics Workshop, San Jose, CA, Apr.. [] Unied Saes Environmenal Proecion Agency, EPA repor on server and daa cener energy efficiency, Final Repor o Congress, Aug. 7. [3] A. Qureshi, R. Weber, H. Balakrishnan, J. Guag, and B. Maggs, Cuing he elecric bill for inerne-scale sysems, in Proc. of he ACM SIGCOMM, Barcelona, Spain, 9. [4] P. R. S. Rivoire and C. Kozyrakis, A comparison of high-level fullsysem power models, in Proceedings of he 8 conference on Power aware compuing and sysems, ser. HoPower 8, 8. [5] T. W. S. Pelley, D. Meisner and J. VanGilder, Undersanding and absracing oal daa cener power, in Proc. of he ACM Workshop on Energy Efficien Design, ser. HoPower 8, June 9. [6] M. Ghamkhari and H. Mohsenian-Rad, Daa ceners o offer ancillary services, in Proc. of he IEEE Conference on Smar Grid Communicaions (SmarGridComm), Ocober. [7] P. Torcellini, S.. Pless, and M. Deru, Zero energy buildings : A criical look a he definiion preprin, Energy, p. 5, 6. [8] D. Kusic, J. O. Kephar, J. E. Hanson, N. Kandasamy, and G. Jiang, Power and performance managemen of virualized compuing environmens via lookahead conrol, in Inernaional Conference on Auonomic Compuing, june 8, pp. 3. [9] H. S. Kim and N. B. Shroff, Loss probabiliy calculaions and asympoic analysis for finie buffer muliplexers, IEEE/ACM Transacions on Neworking, vol. 9, no. 6, pp , dec. [] L. Rao, X. Liu, L. Xie, and W. Liu, Coordinaed energy cos managemen of disribued inerne daa ceners in smar grid, IEEE Transacions on Smar Grid, vol. 3, no., pp. 5 58, march. [] C. Wilson, H. Ballani, T. Karagiannis, and A. Rowsron, Beer never han lae: Meeing deadlines in daacener neworks, in Proc. of he ACM SIGCOMM, Aug.. [] S. Gurumurhi, A. Sivasubramaniam, M. Kandemir, and H. Franke, Conrolled experimens on he web: survey and pracical guide, Daa Mining and Knowledge Discovery, vol. 8, no., pp. 4 8, Feb. 9. [3] G. DeCandia, D. Hasorun, M. Jampani, G. Kakulapai, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels, Dynamo: Amazons highly available key-value sore, in Proc. of he ACM SIGOPS, Oc. 7. [4] C. Y. Hong, M. Caesar, and P. B. Godfrey, Finishing flows quickly wih preempive scheduling, in Proc. of he ACM SIGCOMM, Aug.. [5] J. Rubio-Loyola, A. Galis, A. Asorga, J. Serra, L. Lefevre, A. Fischer, A. Paler, and H. Meer, Scalable service deploymen on sofware-defined neworks, IEEE Communicaions Magazine, vol., no. 49, pp , Dec.. [6] D. Zas, T. Das, and R. H. Kaz, DeTail: Reducing he flow compleion ime ail in daacener neworks, Technical repor, EECS Deparmen, Universiy of California, Berkeley, Mar.. [7] T. Wood, P. Shenoy, A. Venkaaramani, and M. Yousif, Sandpiper: Black-box and gray-box resource managemen for virual machines, Compuer Neworks, vol. 53, no. 7, p , 9. [8] S. Boyd and L. Vandenberghe, Convex Opimizaion. New York, NY, USA: Cambridge Universiy Press, 4. [9] hps:// [3] Hp://ia.ee.lbl.gov/hml/conrib/WorldCup.hml. [3] X. Zheng and Y. Cai, Reducing elecriciy and nework cos for online service providers in geographically locaed inerne daa ceners, in IEEE/ACM Inernaional Conference on Green Compuing and Communicaions, Aug., pp [3] hp:// [33] M. Abramowiz and I. A. Segun, Handbook of mahemaical funcions wih formulas, graphs, and mahemaical ables, Aug. 7. A. Proof of Theorem APPENDIX Firs, we show ha q(µ) in () is non-increasing. We define (µ λ)/σ. () From (), and afer reordering he erms, we can show ha α(µ) = σ λ e e u du. () π Once we ake he derivaive wih respec o µ, we have α (µ) = λ ( + )e e u du. (3) π From [33, Formula 7..3], he following bounds always hold: + ( + 4) < e e u du + ( + 8/π). (4) From he lower bound in (4) and he rivial inequaliy , (5) we have α (µ). Tha is, α(µ) is non-increasing. On he oher hand, for each n, we have m n(µ) [9]. As a resul, exp( min n m n (µ)) is non-increasing. Therefore, q(µ) in () is non-increasing, i.e., q (µ). Nex, we prove ha q(µ) is convex over inerval (6). From (), and since e x is non-increasing, we have q(µ) = max n α(µ) e mn(µ). (6) Therefore, from [8, Secion 3..3], q(µ) is proven o be a convex funcion if we can show ha for each n, funcion q n (µ) α(µ) e mn(µ) (7) is convex. Tha is, for each n, he second derivaive ( q n (µ) =e mn(µ) α (µ) + α(µ) m n (µ) 4 ) (8) α (µ)m n(µ) α(µ) m n(µ). Nex, we show (8) hrough he following five seps: Sep : We show ha α(µ). Firs, we noe ha (r µ)e (r λ) σ, r [µ, ]. (9) Therefore, he inegral in () is non-negaive. From his, ogeher wih he fac ha /λ πσ and exp (µ λ) σ are boh non-negaive erms, we can readily conclude ha α(µ). Sep : We show ha α (µ) over inerval (6). Afer aking he second derivaive of α respec o µ, we have α (µ) = λ ( + ) ( 3 + 3)e e u du. πσ (3)

9 Using simple algebra, we can show ha + for all p ( + 8/π) v u u B. Proof of Theorem 3 8 π /π q + 9 (3) Par (a): We can rewrie he objecive funcion in (4) as T λ(δ + ωb/) (T ωa/)µ T λq(µ)(δ ωb/ + γ). (4). (3) 6 π δ ωb/ + γ δ ωb/ >, Noe ha, from (), condiion (3) holds since (6) holds. Togeher, from (3), (3), and he upper bound in (4), we can direcly conclude ha α (µ) over inerval (6). Sep 3: We show ha α (µ) p +4. α(µ) σ From () and (3), we have R u e e du α (µ). = u R α(µ) σ e e du From Theorem, q(µ) is convex. Therefore, we need o show ha he coefficien of q(µ) in (4) is non-negaive. We have (33) (43) where he sric inequaliy is due o (8). Par (b): From Theorem, T λ [( q(µ))δ q(µ)γ] and (aµ + bλ( q(µ)))/ G are boh non-decreasing. Clearly, hey are also coninuous funcions. Hence, for each µ, where aµ + bλ( q(µ)) < G, (44) here exiss an >, such ha we have a(µ + ) + bλ( q(µ + )) = G, (34) From he lower bound in (4) and afer reordering, we have Z u e e du. (35) +4+ Togeher, from (35) and he lower bound in (4), we have R u e e du + +4 =. (36) R u e e du (45) and he objecive value of Problem (5) a service rae µ + becomes no less han he objecive value a service rae µ. Therefore, a opimaliy of Problem (5), we have aµ + bλ( q(µ)) G. This leads o he formulaion of equivalen Problem (9). Nex, we show ha Problem (9) is convex. From Par (a), he objecive funcion of Problem (9) is concave as i is he same as he objecive funcion in Problem (4). Moreover, since q(µ) is convex, he expression aµ+bλ( q(µ)) is a concave funcion in µ and consequenly, he consrain in Problem (9) forms a convex se [8]. By replacing (36) in (34) and afer reordering we obain (33). Sep 4: We show ha, over inerval (6), we have mn (µ) p +4. (37) mn (µ) σ From () and afer aking he derivaives over µ, we have: mn (µ) D+n = =, (38) mn (µ) Dµ + n(µ λ) (µ λ) σ where he las equaliy is due o (). Noe ha, for each n, we always have mn (µ), mn (µ), and mn (µ) [9]. Nex, from he lower bound in (6), we have σ. From his, and by applying simple algebra, we can show ha p +4. (39) σ σ Replacing (39) in (38), we can direcly conclude (37). Sep 5: From Seps 3 and 4, over inerval (6), we have mn (µ) α (µ) mn (µ) α(µ) (4) m (µ) α (µ)mn (µ) α(µ) n. Furhermore, from Seps and, over inerval (6), we have mn (µ). (4) 4 From (4) and (4) and since he exponenial funcion is nonnegaive, we can conclude (8) and he proof is complee. α (µ) + α(µ) Mahdi Ghamkhari (S 3) received his B.Sc. degree from Sharif Universiy of Technology, Tehran, Iran, in, and his M.Sc. degree from Texas Tech Universiy, Lubbock, TX, USA, in, boh in Elecrical Engineering. Currenly, he is a PhD suden in he Deparmen of Elecrical Engineering a he Universiy of California, Riverside, CA, USA. His research ineress include, Daa Cener and Cloud Compuing, Convex and Sochasic Opimizaion, Queuing Theory and applicaion of Game heory in deregulaed Power Marke. Mr. Mahdi Ghamkhari has been a recipien of several graduae suden awards boh a Texas Tech Universiy and a he Universiy of California a Riverside. Hamed Mohsenian-Rad (S 4-M 9) is currenly an Assisan Professor of Elecrical Engineering a he Universiy of California a Riverside. He received his Ph.D. degree in Elecrical and Compuer Engineering from The Universiy of Briish Columbia (UBC) - Vancouver in 8. Dr. Mohsenian-Rad is he recipien of he NSF CAREER Award and he Bes Paper Award from he IEEE Inernaional Conference on Smar Grid Communicaions. He is an Associae Edior of he IEEE Communicaions Surveys and Tuorials, an Associae Edior of he IEEE Communicaion Leers, a Gues Edior of he ACM Transacions on Embedded Compuing Sysems - Special Issue on Smar Grid, and a Gues Edior of he KICS/IEEE Journal of Communicaions and Neworks - Special Issue of Smar Grid. His research ineress include he design, opimizaion, and game-heoreic analysis of power sysems and smar power grids.