Greening Multi-Tenant Data Center Demand Response

(215 1 3 Greeig Multi-Teat Data Ceter Demad Respose Niagju Che a, Xiaoqi Re a, Shaolei Re b, Adam Wierma a a Computig ad Mathematical Scieces Departmet, Califoria Istitute of Techology b Uiversity of Califoria, Riverside Abstract Data ceters have emerged as promisig resources for demad respose, particularly for emergecy demad respose (EDR, which saves the power grid from icurrig blackouts durig emergecy situatios. However, curretly, data ceters typically participate i EDR by turig o backup (diesel geerators, which is both expesive ad evirometally ufriedly. I this paper, we focus o greeig demad respose i multi-teat data ceters, i.e., colocatio data ceters, by desigig a pricig mechaism through which the data ceter operator ca efficietly extract load reductios from teats durig emergecy periods for EDR. I particular, we propose a pricig mechaism for both madatory ad volutary EDR programs, ColoEDR, that is based o parameterized supply fuctio biddig ad provides provably ear-optimal efficiecy guaratees, both whe teats are pricetakig ad whe they are price-aticipatig. I additio to aalytic results, we exted the literature o supply fuctio mechaism desig, ad evaluate ColoEDR usig trace-based simulatio studies. These validate the efficiecy aalysis ad coclude that the pricig mechaism is both beeficial to the eviromet ad to the data ceter operator (by decreasig the eed for backup diesel geeratio, while also aidig teats (by providig paymets for load reductios. Keywords: demad respose, mechaism desig, multi-teat data ceter, supply fuctio biddig 1. Itroductio Data ceters have emerged as a promisig demad respose opportuity. However, data ceter demad respose today is ot evirometally friedly sice data ceters typically participate by turig o backup (diesel geerators. I this paper, we focus o desigig a pricig mechaism for multi-teat data ceters, which is a crucial class of data ceters for demad respose. Our pricig mechaism allows the data ceter operator to obtai load sheddig amog teats efficietly, reducig the eed for use of backup (diesel geeratio ad thus greeig data ceter demad respose. Data ceter demad respose. Power-hugry data ceters have bee quickly expadig i both umber ad scale to support the explodig IT demad, cosumig 91 billio kilowatt-hour (kwh electricity i 213 i the U.S. aloe [1]. While traditioally viewed purely as a egative, the massive eergy usage of data ceters has recetly begu to be recogized as a opportuity. I particular, because the eergy usage of data ceters teds to be flexible, they are promisig cadidates for demad respose, which is a crucial tool for improvig grid reliability ad icorporatig reewable eergy ito the power grid. From the grid operator s perspective, a data ceter s flexible power demad serves as a valuable eergy buffer, helpig balace grid power s supply ad demad at rutime [2]. This work is supported i part by the U.S. NSF CNS-131982, EPAS-137794, CNS-1423137, ad CNS-1453491. Email addresses: cche@caltech.edu (Niagju Che, xre@caltech.edu (Xiaoqi Re, sre@ece.ucr.edu (Shaolei Re, adamw@caltech.edu (Adam Wierma 1

/ (215 1 3 2 To this poit, data ceters are a promisig, but still largely uder-utilized opportuity for demad respose. However, this is quickly chagig as data ceters play a icreasig role i emergecy demad respose (EDR programs. EDR is the most widely-adopted demad respose program i the U.S., represetig 87% of demad reductio capabilities across all reliability regios [3]. Specifically, durig emergecy evets (e.g., extreme weather or atural disasters, EDR coordiates may large eergy cosumers, icludig data ceters, to shed their power loads, servig as the last protectio agaist cascadig blackouts that could potetially result i ecoomic losses of billios of dollars [4, 5]. The U.S. EPA has idetified data ceters as critical resources for EDR [6], which was attested to by the followig example: o July 22, 211, hudreds of data ceters participated i EDR by cuttig their electricity usage before a large-scale blackout would have occurred [5]. While data ceters are icreasigly cotributig to EDR, they typically participate by turig o their o-site backup diesel geerators, which is either cost effective or evirometally friedly. For example, i Califoria (a major data ceter market, a stadby diesel geerator ofte produces 5-6 times more itroge oxides (a smogformig pollutat compared to a typical power plat for each kwh of electricity, ad diesel particulate represets the state s most sigificat toxic air pollutio problem [7]. I additio, relyig o diesel geeratio for EDR presets emergig challeges which, if left uaddressed, may forfeit data ceter s EDR capability. First, as EDR becomes more frequet [4, 8], the curret fiacial compesatio offered by power grid to data ceters (for committed eergy reductio durig EDR may ot be eough to cover the growig cost of diesel geeratio. Secod, data ceter operators are aggressively cuttig the huge capital ivestmet i their power ifrastructure (e.g., 1-25$/watt [9, 1], by dow-sizig the capacity of diesel geerator ad uiterrupted power supply (UPS systems [11]. Such uder-provisioig of diesel geeratio may compromise EDR capability. Therefore, to retai ad ecourage data ceter participatio i EDR without cotamiatig the eviromet, it is critical ad urget that data ceters seek alterative ways to shed load. Cosequetly, modulatig server eergy for gree EDR (as well as other demad respose programs such as regulatio service [12] has received a icreasig amout of attetio i recet years, e.g., [13, 14, 15, 16, 17, 12, 2]. These studies leverage various widely-available IT computig kobs (e.g., server turig o/off ad workload migratio i data ceters ad provide algorithms to optimize them for participatio i demad respose markets. Importatly, these are ot simply theoretical studies. For example, a field study by Lawrece Berkeley Natioal Laboratory (LNBL has illustrated that data ceters ca reduce eergy cosumptio by 1-25% i respose to demad respose sigals, without oticeably impactig ormal operatio [18]. Demad respose i collocatio data ceters. While existig studies o data ceter demad respose show promisig progress, they are primarily focused o ower-operated data ceters (e.g., Google whose operators have full cotrol over both servers ad facilities. Ufortuately, such compaies may actually be the least likely to participate i demad respose programs, because may of their workloads are extremely delay sesitive ad their data ceters have bee optimized for miimum delay. I this paper, we focus o aother type of data ceters multi-teat colocatio data ceters (e.g., Equiix. These have bee ivestigated much less frequetly, but are actually better targets for demad respose tha oweroperated data ceters. I a colocatio data ceter (simply called colocatio or colo, multiple teats deploy ad keep full cotrol of their ow physical servers i a shared space, while the colo operator oly provides facility support (e.g., high-availability power ad coolig. Colos are less studied tha ower-operated data ceters, but they are actually more commo i practice. Colos offer data ceter solutios to may idustry sectors, ad serve as physical home to may private clouds, medium-scale public clouds (e.g., VMware [19], ad cotet delivery providers (e.g., Akamai. Further, a recet study shows that colos cosume early 4% of data ceter eergy i the U.S., while Google-type data ceters collectively accout for less tha 8%, with the remaiig goig to eterprise i-house data ceters [1]. I additio to cosumig a sigificat amout of eergy (more tha Google-type data ceters, colos are ofte located i places more useful for demad respose. While may mega-scale ower-operated data ceters are built i rural areas, colos are mostly located i metropolita areas (e.g., Los Ageles, New York [2], which are the very places where EDR is most eeded. For all these reasos, colos are key participats i EDR programs. Further, teats workloads i colos are highly heterogeous, ad may teats ru o-missio-critical workloads (e.g., lab computig [21] that have very high schedulig flexibilities, differet delay sesitivities, peak load periods, etc., which is ideal for demad respose participatio. Thus, teats load sheddig potetials, if appropriately exploited, ca altogether form a gree alterative to diesel geeratio for colo EDR. Noetheless, teats 2

/ (215 1 3 3 maage their ow servers idepedetly ad may ot have icetive to cooperate with the operator for EDR, thus raisig the research questio: how ca a colo operator efficietly icetivize its teats load sheddig for EDR? 1 Cotributios of this paper. I this paper, we focus o greeig colocatio demad respose by extractig load reductio from teats istead of relyig o backup diesel geeratio. We study both madatory EDR, a type of EDR program i which participats sig cotracts ad are obliged to reduce loads whe requested [8], ad volutary EDR, where participats volutarily reduce loads for fiacial compesatio upo grid request [4]. I both cases, we propose a ew pricig mechaism with which colo operators ca extract load sheddig from teats. I particular, our proposed approach, called ColoEDR, ca effectively provide icetives for teats to reduce eergy cosumptio durig EDR evets, complemetig (ad eve substitutig for the high-cost ad evirometally-ufriedly diesel geeratio. ColoEDR works as follows. After a EDR sigal arrives at the colo operator, teats bid usig a parameterized supply fuctio, ad the the colo operator aouces a market clearig price which, whe plugged ito the bids, specifies how much eergy teats will reduce ad how much they will be paid. Participatio by the teats is easy, sice they are asked to bid oly oe parameter, which ca be viewed as a proxy of how much flexibility i eergy reductio they have at that momet. This participatio ca be automated ad so ca be easily icorporated ito curret practice [22], ad mimics the way geeratio resources participate i electricity markets more broadly. For example, colo operators, like Verizo Terremark, already commuicate with their teats i preparatio for a EDR evet. The mai techical cotributio of the paper is the aalysis of the efficiecy of the supply fuctio mechaism proposed i ColoEDR. I particular, while there is a large literature studyig supply fuctio biddig [23, 24, 25, 26, 27], our settig here is ovel ad differet. For madatory EDR, the colo operator ca either satisfy the EDR request usig flexibility from the teats (as i prior supply fudig literature or through its backup diesel geerator. Thus, the diesel geerator is a outside optio that allows for elasticity i the amout of respose extracted from the teats. Further, the colo operator ca combie ad balace betwee its two optios (i.e., teat load sheddig ad backup geerator i order to miimize costs. For volutary EDR, the amout of respose extracted from the teats is also a elastic decisio by the colo operator, sice there is o obligatio for the colo to reduce eergy. Thus, for both madatory ad volutary EDR, the elastic amout of respose from teats creates a multi-stage game ad adds a cosiderable complexity as compared to the stadard settig without such elasticity, e.g., [23]. Despite the added complexity, our aalysis precisely characterizes the equilibrium outcome, both whe teats are price-takig ad whe they are price-aticipatig. I both cases, our results highlight that ColoEDR suffers little performace loss compared to the socially optimal outcome, both from the operator s ad the teats perspectives. However, our aalysis does highlight oe possible drawback of ColoEDR. I the worst case, it is possible that ColoEDR may result i usig sigificatly more o-site diesel geeratio tha would the socially optimal. However, this bad evet occurs oly i cases where oe teat has a overwhelmigly fractio of the servers ad has a uit cost (for eergy reductio just below that of o-site diesel geeratio. Such a exploitatio of market power is ulikely to be possible i practical multi-teat colocatio data ceters where multiple teats with comparable sizes house their servers. I additio to our theoretical aalysis, we ivestigate a case study of (madatory EDR i 6 usig trace-based experimets. The results further validate the desig of ColoEDR, ad show that it achieves the madatory eergy reductio for EDR while beefitig teats through fiacial icetives ad decreasig the operator s cost. Moreover, our simulatio study shows that the efficiecy loss i practical settigs is eve lower tha what is suggested by the aalytic bouds. This is especially true for the amout of o-site geeratio, which the aalytic results suggest ca (i the worst-case be sigificatly larger tha socially optimal but i realistic settigs is very close to the social optimal. 2. Modellig Multi-Teat Data Ceter EDR Our focus is the desig of a mechaism for a colo operator to extract teat load reductios i respose to a EDR sigal. Thus, we eed to begi by describig a model for a colo operator. Recall that the colo operator is resposible for o-it facility support (e.g., high-availability power, coolig. We capture the o-it eergy cosumptio usig Power Usage Effectiveess (PUE γ, which is the ratio of the total data 1 Teats receive UPS-protected power from the colo operator ad share coolig systems. I other words, teats total eergy cosumptio is ot directly provided by the grid ad icludes o-metered coolig eergy, which makes teats ieligible for direct participatio i EDR [4]. 3

/ (215 1 3 4 ceter eergy cosumptio to the IT eergy cosumptio. Typically, γ rages from 1.1 to 2., depedig o factors such as outside temperature. Whe the operator receives a EDR sigal from the LSE (Load Servig Etity, it has two optios for satisfyig the load reductio. First, without ivolvig the teats, the colo operator ca use its o-site backup diesel geerator. 2 We deote the amout of eergy reductio by diesel geeratio by y ad the cost per kwh of diesel geeratio (e.g., for fuels by. Alteratively, the colo operator could try to extract IT eergy reductios from the teats. We cosider a settig where there are N teats, i N = {1, 2,, N}. Whe sheddig eergy cosumptio, a teat i will icur some costs ad we deote the cost from sheddig s i by a fuctio c i (s i. These costs could be due to wear-ad-tear, performace degradatio, workload shiftig, etc. For the purposes of our model, we do ot specify which techique reduces the IT eergy, oly its cost. For details o how oe might model such costs, see [28, 29, 3, 31]. A stadard, atural assumptio o the costs is the followig. Assumptio 1. For each, the cost fuctio c (s is cotiuous, with c (s = if s. Over the domai s, the cost fuctio c is covex ad strictly icreasig. Ituitively, covexity follows from the covetioal assumptio that the uit cost icreases as teats reduce more eergy (e.g., utilizatio becomes higher whe servers are off, leadig to a faster icrease i respose time of teats workloads. 3. Pricig Teat Load Sheddig i Madatory EDR EDR is the last lie of protectio agaist cascadig power failures, ad represets 87% of demad reductio capabilities across all the U.S. reliability regios [3]. I geeral, there are two types of EDR programs: madatory ad volutary (also called ecoomic [4, 8]. We focus o madatory EDR first, ad retur to volutary EDR i Sectio 5. For madatory EDR, participats typically sig cotracts with a load servig etity (LSE i advace (e.g., 3 years ahead i Pesylvaia-New Jersey-Marylad Itercoectio (PJM [4] ad receive fiacial rebates for their committed eergy reductio eve if o EDR sigals are triggered durig the participatio year, whereas o-compliace (i.e., failure to cut load as required durig EDR icurs a heavy pealty [4]. If a LSE aticipates that a emergecy will occur, participats are otified, usually at least 1 miutes i advace, ad obliged to fulfill their cotracted amouts of eergy reductio for the legth of the evet, which may spa a few miutes to a few hours. I madatory EDR, the colo operator ca reduce load i respose to a EDR sigal either through teats or by turig o a o-site geerator. Sice the madatory EDR target is fixed, the operator must balace betwee payig teats for reductio ad usig o-site geeratio i order to miimize cost. Note that teats load reductio ca also reduce the usage of diesel geerator, mitigatig evirometal impacts. Noetheless, the challege is that the operator does ot kow the teat cost fuctios, ad so caot determie the cost-miimizig price. Cosequetly, the operator has two optios to determie the price: (i predict the teat supply fuctio ad compute prices based o the predictios, or (ii allow teats to supply some iformatio about their cost fuctios through bids. Clearly, there is a tradeoff here betwee the accuracy of predictios ad the maipulatio possible i the bids. Both of these approaches have bee studied i the literature [32, 16, 23, 24, 33], though ot i the cotext of colo demad respose. I geeral, the broad coclusio is that approach (i is appropriate whe predictios are accurate ad oe bidder has market power (e.g., is sigificatly larger tha other bidders. While market power is a cosiderable issue for the participatio of ower-operated data ceters i demad respose programs due to their large size compared to other participats, it is ot a issue withi a specific colo that houses multiple teats (typically of comparable sizes, ad so we adopt approach (ii i this paper. Specifically, we desig a mechaism, amed ColoEDR, where teats bid usig parameterized supply fuctios ad the, give the bids, the operator decides how much load to shed via teats ad how much to shed via osite geeratio. I the followig, we describe the mechaism ad the cotrast our approach with other potetial alteratives. 2 Other alteratives, e.g., battery [11], usually oly last for < 5 miutes. So, diesel geeratio is the typical method [6]. 4

/ (215 1 3 5 Note that, throughout this paper, we focus o oe EDR evet, thus we omit the time idex. I the case of multiple cosecutive EDR evets, ColoEDR will be executed oce at the begiig of each evet, as is stadard i the literature [16, 34]. 3.1. A overview of ColoEDR The operatio of ColoEDR is summarized below, ad the discussed i detail i the text that follows. 1. The colo operator receives a EDR reductio target δ ad broadcasts the supply fuctio S (, p specified by(1 to teats; 2. Participatig teats respod by placig their bids b ; 3. The colo operator decides the amout of o-site geeratio y ad market clearig price p to miimize its cost, usig equatios (2 to set the market clearig price p ad (3 to set y i order to miimize the cost of EDR; 4. EDR is exercised. N, teat sheds S (b, p, ad receives ps (b, p reward. Give the overview above, we ow discuss each step i more detail. Step 1. Upo receivig a EDR otificatio of a eergy reductio target δ, the colo operator broadcasts a parameterized supply fuctio S (b, p to teats (by, e.g., sigallig to the teats server cotrol iterfaces, which are widely i use today [22]. The form of S (b, p is the followig parameterized family 3 : S (b, p = δ b p. (1 where p is a offered reward for each kwh of eergy reductio ad b is the biddig values that ca be chose by teat. This form is ispired by [23], where it is show that by restrictig the supply fuctio to this parameterized family, the mechaism ca guide the firms i the market to reach a equilibrium with desirable properties. 4 Note that, to be cosistet with the supply fuctio literature, we exchageably use price ad reward rate wherever applicable. Step 2. Next, accordig to the supply fuctio, each participatig teat submits its bid b to the colo operator. This bid specifies that, at each price p, it is willig to reduce S (b, p uits of eergy. The bid is chose by teats idividually to maximize their ow utility ad ca be iterpreted as, e.g., the amout of IT service reveue that teat is willig to forgo. Note that b ca be chose to esure that teat will ot be required to reduce more eergy tha its capacity. To see this, ote that sice the operator is cost-miimizig, p(b, y always holds, i.e., the market clearig price is lower tha the uit cost of diesel geeratio. Hece, if K is the capacity of reductio for teat, as log as b (δ K, the S (b, p = δ b p δ b K. A importat ote about the teat bids is that the supply fuctio is likely of a differet form tha the true cost fuctio c, ad so it is ulikely for the teats to reveal their cost fuctios truthfully. This is ecessary i order to provide a simple form for teat bids. Biddig their true cost fuctios is too complex ad itrusive. However, a cosequece of this is that oe must carefully aalyze the emerget equilibrium to uderstad the efficiecy of the pricig mechaism. We study both the cases of price-takig ad price-aticipatig equilibrium i 4. Step 3. After teats have submitted their bids, the colo operator decides the amout of eergy y to produce via o-site geeratio ad the clearig price p. Give y, the market clearig price has to satisfy Σ S (p(b, b + y = δ, thus b p(b, y = (N 1δ + y. (2 3 The supply fuctio allows teats to have egative supply, i.e., teats cosume more eergy itetioally, which is either profit maximizig or practical. We show i 4 that eergy reductio of each teat is always oegative i both equilibrium ad social optimal outcomes. 4 [23] studies the case where firms bid to supply a ielastic demad, which is equivalet to fixig the diesel geeratio y = i our case. Allowig the operator to choose y i a cost-miimizig maer leads to sigificatly differet results, as will be show i 4.1 ad 4.2. 5

/ (215 1 3 6 To determie the amout of local geeratio y, the operator miimizes the cost of the two load-reductio optios, i.e., y = arg mi(δ y p(b, y + y. (3 y δ Step 4. Fially, EDR is exercised ad teats receive fiacial compesatio from the colo operator via the realized price i (2, shed load S (p, b, ad o-site geeratio produces (3. 3.2. Discussio To the best of our kowledge, this paper represets the first attempt to desig a supply fuctio biddig mechaism for colocatio demad respose. Although alterative mechaisms may be applicable, there are compellig advatages to the supply fuctio approach. First, biddig for the teats is simple they oly eed to commuicate oe umber, ad it is already commo practice for operators to commuicate with teats before EDR evets [22], so the overhead is margial. Secod, the colo operator collects just eough iformatio (i.e., how much eergy reductio each teat will cotribute to EDR, while teats private iformatio (i.e., how much performace pealty/cost each for eergy reductio is masked by the form of the supply fuctio ad hece ot solicited. Third, ColoEDR guaratees that the colo operator will ot icur a higher cost tha the case where oly backup geerators are used. Further, ColoEDR pays a uiform price to all participatig teats ad hece esures fairess. The most atural alterative biddig mechaism to supply fuctio biddig is a Vickrey-Clarke-Groves (VCG- based mechaism, as is suggested i [35]. While VCG-based mechaisms have the beefit of icetive compatibility, these mechaisms violate all the four properties discussed above. Uder such approaches, teats must submit very complex bids describig their precise cost fuctios, the true private cost of teats is disclosed, paymet made to teats may be ubouded, ad prices to differet teats are differetiated ad thus raises ufairess issues. Due to these shortcomigs, VCG-based mechaisms are typically ot adopted i complex resource allocatio settigs such as power markets, where supply-fuctio based desigs are commo [23]. I fact, early all geeratio markets use variatios of supply fuctio biddig. 4. Efficiecy Aalysis of ColoEDR for Madatory EDR Give the ColoEDR mechaism described above, our task ow is to characterize its efficiecy. There are two potetial causes of iefficiecy i the mechaism: the cost miimizig behavior of the operator ad the strategic behavior (biddig of the teats. I particular, sice the forms of the teat s cost fuctios are likely more complex tha the supply fuctio bids, teats caot bid their true cost fuctio eve if they wated to. This meas that evaluatig the equilibrium outcome is crucial to uderstadig the efficiecy of the mechaism. Further, the equilibrium outcome that emerges depeds highly o the behavior of the teats whether they are price-takig, i.e., they passively accept the offered market price p as give whe decidig their ow bids; or priceaticipatig, i.e., they aticipate how the price p will be impacted by their ow bids. We ivestigate both models, i 4.1 ad 4.2, respectively. I both cases, the goal of our aalysis is to assess the efficiecy of ColoEDR. To this ed, we adopt a otio of a (socially optimal outcome, ad focus o the followig social cost miimizatio (SCM problem. SCM : mi y + c i (s i (4a i N s.t. y + γ s i = δ i N (4b s i, i N, y. (4c where s i ad c i are teat i s eergy reductio ad correspodig cost, respectively. The objective i SCM ca be iterpreted as the teats cost plus the colo operator s cost. Note that the iteral paymet trasfer betwee the colo operator ad teats cacels, ad does ot impact the social cost. Also, ote that paymet from the LSE to the colo operator is ot icluded i the social cost objective, sice it is idepedet of how the operator obtais the amout δ of load reductio. Additioally, we do ot iclude the optio of igorig the EDR 6

/ (215 1 3 7 sigal ad takig the pealty, sice the o-compliace pealties are typically extreme [4]. Fially, the Lagrage multiplier of (4b ca be iterpreted as the social optimal price p, i.e., give this price as reward for eergy reductio, each teat will idividually reduce their eergy by s that correspods to the social cost miimizatio solutio i (4. Before movig to the aalysis, i order to simplify otatio, we suppress the PUE γ by, without loss of geerality, settig γ = 1. To obtai results for γ 1, simply take the results assumig γ = 1 ad modified them i the followig way: let y, δ ad be the diesel geeratio, EDR target ad diesel price that appear i the results for γ = 1, replace them by y = y/γ, δ = δ/γ, ad = γ where y, δ, are the respective quatities whe γ 1. 4.1. Price-Takig Teats Whe teats are price-takig, they maximize their et utility, which is the differece betwee the paymet they receive ad the cost of eergy reductio, give the assumptio that they cosider their actio does ot impact the price. A price-takig teat will try to maximize the followig payoff P (b, p: P (b, p = ps (b, p c (S (b, p (5a = pδ b c ( δ b p. (5b Here, the price-takig assumptio implies that the variable p is cosidered to be as is. The price-takig assumptio ormally holds whe the market cosists of may players of similar sizes who have little power to impact the market clearig price. The other market model, whe teats are price-aticipatig, is aalyzed i Sectio 4.2.The market equilibrium for price-takig teats is thus defied as follows: Defiitio 1. A triple (b, p, y is a (price-takig market equilibrium if each teat maximizes its payoff defied i (5, market is cleared by settig price p accordig to (2, ad the amout of o-site geeratio is decided by (3, i.e., P (b ; p P ( b ; p b, = 1,..., N. (6 i N b i p = (N 1δ + y. (7 y = arg mi(δ y p(b, y + y. (8 y δ 4.1.1. Market Equilibrium Characterizatio The key to our aalysis is the observatio that the equilibrium ca be characterized by a optimizatio problem. Oce we have this optimizatio, we ca use it to characterize the efficiecy of the equilibrium outcome. This approach parallels that used i [23]; however, the optimizatio obtaied has a differet structure due to local diesel geeratio. Note that, though we use a optimizatio to characterize the equilibrium, the game is ot a potetial game sice the objective (9a below is ot a potetial fuctio. Our first result highlights that, give ay choice for o-site geeratio, a uique market equilibrium exists for the teats, ad ca be characterized via a simple optimizatio. Propositio 1. Uder Assumptio 1, whe teats are price-takig, for ay o-site geeratio level y < δ, there exists a market equilibrium, i.e., a vector b t = (b t 1,..., bt N ad a scalar p > that satisfies (2, ad the resultig allocatio s = S (b, p is the optimal solutio of the followig mi c i (s i (9a s i N s.t. s i = (δ y, (9b i N s i, i N. (9c This result is a key tool for uderstadig the overall market outcome. Ituitively, the operator ruig ColoEDR is more likely (tha the social optimal to use o-site geeratio, sice this reduces the price paid to teats. The followig propositio quatifies this statemet. 7

/ (215 1 3 8 Propositio 2. Uder Assumptio 1, it is optimal for price-takig teats to use o-site geeratio if ad oly if < (Σ b (N 1δ.5 (1 However, whe the operator is profit maximizig, it will tur o o-site geeratio if ad oly if < N (Σ b N 1 (N 1δ. (11 This propositio is a importat buildig block because the most iterestig case to cosider is whe it is optimal to use some o-site geeratio ad some teat load sheddig, i.e., δ > y >. Otherwise the EDR requiremet should be etirely fulfilled by teats, ad the aalysis reduces to the case of a ielastic demad, as studied i [23]. Thus, subsequetly, we make the followig assumptio, which esures that o-site geeratio is valuable. Assumptio 2. The uit cost of o-site geeratio is cheap eough that the optimal o-site geeratio is o-zero, i.e., satisfies (1. Note that, whe Assumptio 2 holds, by first-order optimality coditio of (3 we have (Σi N b i Nδ y = (N 1δ, (12 ad so the market clearig price for the teats give o-site geeratio is i N b i p = (N 1δ + y = (Σi N b i. (13 Nδ Usig these allows us to prove a complete characterizatio of the market equilibrium uder price-takig teats. This theorem is the key to our aalysis of market efficiecy. Theorem 3. Whe Assumptios 1 ad 2 hold there is a uique market equilibrium, i.e., a vector b t = (b t 1,..., bt N, y t > ad a scalar p t > that satisfies (6-(8, ad the resultig allocatio (s t, y t where s t = S (b t, p t is the optimal solutio of the followig problem mi s,y s.t. c (s + (y + (N 1δ2 (14a 2Nδ s = δ y, (14b s,, y. (14c 4.1.2. Boudig Efficiecy Loss We ow use Theorem 3 to boud the efficiecy loss due to strategic behavior i the market. Deote the socially optimal o-site geeratio by y, the optimal price that leads to the optimal allocatio s i, i N by p, ad let y t ad p t be the allocatio uder the price-takig assumptio. Our first result highlights that, due to the cost-miimizig behavior of the operator, the equilibrium outcome uses more o-site geeratio ad pays a lower price to the teats tha the social optimal. 5 We adopt the covetio that = ad x = + whe x >. Therefore, whe N = 1, uless the bid is, the coditio is always satisfied. 8

/ (215 1 3 9 Propositio 4. Suppose that Assumptios 1 ad 2 hold. Whe teats are price-takig, the operator ruig ColoEDR uses more o-site geeratio ad pays a lower price for power reductio to its teats tha the social optimal. Specifically, y t y ad N 1 N p p t p. Now, we move to more detailed comparisos. There are three compoets of market efficiecy that we cosider: social welfare, operator cost, ad teat cost. First, let us cosider the social cost. Theorem 5. Suppose that Assumptios 1 ad 2 hold. Let (s t, y t be the allocatio whe teats are price-takig, ad (s, y be the optimal allocatio. The the welfare loss is bouded by: c (s t + y t c (s + y + δ/2n. Importatly, this theorem highlights that the market equilibrium is quite efficiet, especially if the umber of teats is large (the efficiecy loss decays to zero as O(1/N. However, the market could maitai good overall social welfare at the expese of either the operator or the teats. The followig results show this is ot true. Let cost o (p, y be the operator s cost, i.e., The, we have the followig results. cost o (p, y = p(δ y + y. (15 Theorem 6. Suppose that Assumptios 1 ad 2 are satisfied. The cost of colo operator with price-takig teats is smaller tha the cost i the socially optimal case. Further, we have cost o (p, y δ/n cost o (p t, y t cost o (p, y. 4.2. Price-Aticipatig Teats I cotrast to the price-takig model, price-aticipatig teats realize that they ca chage the market price by their bids, i.e., that p is set accordig to (13, ad adjust their bids accordigly. The price-aticipatig model is suitable whe the market cosists of a few domiat players, who have sigificat power to impact the market price through their bids, i.e., the oligopoly settig. Clearly, this additioal strategic behavior ca lead to larger efficiecy loss. However, i this sectio, we show that the extra loss is surprisigly small, especially whe a large umber of teats participate i ColoEDR. Give bids from the other teats, each price-aticipatig teat optimizes the followig cost over biddig value b Q (b, b = p(bs (b, p c (S (b, p where we use b to deote the vector of bids of teats other tha ; i.e., b = (b 1,..., b 1, b +1,..., b N. Thus, substitutig (1 ad (13, we have (Σ b δ Q (b ; b = b c N δ b Nδ Σm b m. (16 Note that the payoff fuctio Q is similar to the payoff fuctio P i the price-takig case, except that the teats aticipate that the colo operator will set the price p accordig to p = p(b, y from (13. Defiitio 2. A triple (b, p, y is a (price-aticipatig market equilibrium if each teat maximizes its payoff defied i (16, the market is cleared by settig the price p accordig to (2 ad the amout of o-site geeratio is decided by (3, i.e., Q (b ; b Q ( b ; b b, = 1,..., N (17 b p = (N 1δ + y. (18 y = arg mi(δ y p(b, y + y. (19 y δ 9

/ (215 1 3 1 Note that our aalysis i this sectio requires oe additioal techical assumptio about the teat cost fuctios. Assumptio 3. For all teats, the margial cost of eergy reductio at is greater tha 2N, i.e., + c ( 2N,. This assumptio is quite mild, especially if the umber of teats N is large. Ituitively, it says that the uit cost of o-site geeratio is competitive with the cost of teats reducig their server eergy. 4.2.1. Market Equilibrium Characterizatio Our aalysis of market equilibria proceeds alog parallel lies to the price-takig case. We agai show that there exists a uique equilibrium ad, furthermore, that the teats ad operator behave i equilibrium as if they were solvig a optimizatio problem of the same form as the aggregate cost miimizatio (4, but with modified cost fuctios. Theorem 7. Suppose that Assumptio 1-3 are satisfied, the there exists a uique equilibrium of the game defied by (Q 1,..., Q satisfyig (17-(19. For such a equilibrium, the vector s a defied by s a = S (p(b a, b a is the uique optimal solutio to the followig optimizatio: mi ĉ (s + (y + (N 1δ2 2Nδ (2a s.t. s = δ y (2b where, for s, ĉ (s = 1 2 ad for s <, ĉ (s =. y, s, = 1,..., N, (2c ( c (s + s + 1 s ( + c (z 2 + 2 + c (z z dz, (21 2N 2 z 2N z Nδ Although the form of ĉ (s looks complicated, there is a simple liear approximatio that gives useful ituitio. Lemma 8. Suppose that Assumptio 1-3 are satisfied. For all modified cost ĉ, 1,..., N, for ay s δ, c (s ĉ (s c (s + s 2N, Furthermore, whe the left or right derivatives of ĉ( is defied, it ca be bouded by c (s ĉ(s + ĉ(s + c (s + 2N. The form of Lemma 8 shows that the differece betwee the modified cost fuctio i (21 ad the true cost dimiishes as N icreases, ad this is the key observatio that uderlies our subsequet results upper boudig the efficiecy loss of ColoEDR. 4.2.2. Boudig Efficiecy Loss We ow use Theorem 7 to boud the efficiecy loss due to strategic behavior. Note that, by comparig to both the socially optimal ad the price-takig outcomes, we ca uderstad the impact of both strategic behavior by the operator ad the teats. Our first result focuses o comparig the price-aticipatig ad price-takig equilibrium outcomes. It highlights that price-aticipatig behavior leads to teats receivig higher price while sheddig less load. 1

/ (215 1 3 11 Theorem 9. Suppose Assumptio 1-3 hold. Let (p t, y t be the equilibrium price ad o-site geeratio whe teats are price-takig, ad (p a, y a be those whe teats are price-aticipatig, the we have, y t y a y t + δ/2 ad p t p a p t + /2N. Next, combiig Theorem 9 ad Propositio 4 yields the followig compariso betwee the price-aticipatig ad socially optimal outcomes. Corollary 1. Suppose Assumptio 1-3 hold. Whe teats are price-aticipatig, a operator ruig ColoEDR uses more o-site geeratio ad pays lower market price tha i the socially optimal case, i.e., y a y ad N 1 N p p a p. Now, we move to more detailed comparisos. There are three compoets of market efficiecy that we cosider: social cost, operator cost, ad teat cost. First, let us cosider the social cost. Theorem 11. Suppose that Assumptio 1-3 hold. Let (s a, y a be the allocatio whe teats are price-aticipatig, ad (s, y be the optimal allocatio. The welfare loss is bouded by: c (s a + y a c (s + y + δ/n. Similarly to the price-takig case, the efficiecy loss i the price-aticipatig case decays to zero as O(1/N, oly with a larger costat. Also, as i the case of price-takig teats, we agai see that either the teats or the operator suffers sigificat efficiecy loss. Theorem 12. Suppose that Assumptio 1-3 hold. The cost of colo operator for price-aticipatig teats is smaller tha the cost i the socially optimal case. Further, we have cost o (p, y δ N cost o(p a, y a cost o (p, y, cost o (p a, y a δ N cost o(p t, y t cost o (p a, y a Fially, let us ed by cosiderig the amout of o-site geeratio used i equilibrium. Here, i the worst-case, the o-site geeratio at equilibrium for price-aticipatig teats ca be arbitrarily worse tha the socially optimal, i.e., the socially optimal ca use o o-site geeratio while the equilibrium outcome uses oly o-site geeratio. Theorem 13. Suppose that Assumptio 1-3 hold. For ay ε >, N 1, there exist cost fuctios c 1,..., c N, such that the o-site geeratio i the market equilibrium compared to the optimal is give by y a y δ ε. This is a particularly disappoitig result sice a key goal of the mechaism is to obtai load sheddig from the teats. However, the proof emphasizes that this is ulikely to occur i practice. I particular, the worst-case sceario is that there exists a domiat (moopoly teat, which is ulikely i a multi-teat colo, that has a cost fuctio asymptotically liear with uit cost roughly matchig the o-site geeratio price. We cofirm this i a case study i Sectio 6. 4.3. Discussio The mai results for the price-takig ad price-aticipatig aalyses are summarized i Table 1. Note that simplified bouds are preseted i the table, to ease iterpretatio, ad the iterested reader should refer to the theorems i 4.1 ad 4.2 for the actual bouds. Also, ote that the bechmark for social cost we cosider is a ideal, but ot achievable, mechaism. Teats Price Ratio Colo Savig Welfare Loss Price-takig [ N 1 N, 1] [, δ/n] [, δ/2n] Price-aticipatig, 1] [, δ/n] [, δ/n] [ N 1 N Table 1. Performace guaratee of ColoEDR compared to the social optimal allocatio. 11

/ (215 1 3 12 To summarize the results i Table 1 briefly, ote first that ColoEDR always beefits the operator, sice the price paid to teats to reduce eergy is always less tha the socially optimal price, ad the total cost icurred by operator for eergy reductio is also less tha that of the social optimal. Secodly, ColoEDR also gives the teats approximately the social optimal paymet, sice the operator s additioal beefit is bouded above by δ/n, ad the welfare loss is bouded above by δ/n. This aturally meas that the loss i paymet for teats compared to the social optimal is at most 2δ/N, which approaches as N grows. Third, regardless of teats beig price-takig or price-aticipatig, ColoEDR is approximately socially cost-miimizig as the umber of teats grows. However, while ColoEDR is good i terms of operator, teat, ad social cost, it may ot use the most evirometally friedly form of load reductio: i the worst case, the upper boud o the extra o-site geeratio that ColoEDR uses is ot decreasig with N. However, the aalysis highlights that this worst-case occurs whe there exists a domiat teat with uit cost of eergy reductio that is cosistetly just below the cost of diesel over a large rage of eergy reductio. As our case study i 6 shows, this is ulikely to occur i practice. So, ColoEDR ca be expected to use a evirometally friedly mix i most realistic situatios. 5. Pricig Teat Load Sheddig i Volutary EDR We ow tur from madatory EDR to volutary EDR ad show how the aalysis ad desig of ColoEDR ca be exteded. Uder volutary EDR, a colo operator is offered a certai compesatio rate for load reductio ad ca cut ay amout of eergy at will without ay obligatio. Volutary EDR ofte supplemets madatory EDR, ad both are widely adopted i practice [4, 8]. Sice the colo operator ca freely decide o the amout of eergy to cut based o the compesatio rate [4], the amout of eergy reductio from teats is fully elastic, differig from madatory EDR where the total eergy reductio (icludig diesel geeratio if ecessary eeds to satisfy a costrait δ. I the followig, we formulate the problem ad geeralize ColoEDR for the volutary EDR settig. Furthermore, we illustrate that the efficiecy aalysis, though more complicated, parallels that of madatory EDR. 5.1. Problem Formulatio Durig a volutary EDR evet, the LSE offers a reward of u for each uit of eergy reductio (or diesel geeratio if applicable. I our settig, the colo operator aims at maximizig its profit through extractig loads from teats usig parameterized supply fuctio biddig, as cosidered for madatory EDR. A key differece with the case of madatory EDR is that, sice the reductio is flexible, diesel geeratio eed ot be cosidered. I particular, if the reward offered the LSE for reductio is larger tha the cost of diesel, the the operator ca cotribute its whole diesel capacity ad, if the reward is smaller tha the cost of diesel, o diesel eed be used. I the madatory EDR settig, operator eeds to use diesel geeratio whe teats eergy reductio (i.e., teats bids are high is ot eough, i order to meet the reductio requiremet δ; i the volutary EDR case, there is o madatory eergy reductio target ad thus, the optimizatio of diesel geeratio by the operator is separable from the optimizatio of teat reductio. This yields a situatio where the et profit (from teat reductio received by the colo operator is: u d p d (22 where p is the uit price the colo operator pays to the teats to solicit d uits of reductio i aggregate from N teats, where teat i has reductio capacity D i. A overview of ColoEDR. It is straightforward to adapt ColoEDR to this settig. We outlie its operatio i four steps below, which parallel the steps i the case of madatory EDR. 1. The colo operator receives the volutary EDR reductio price u ad broadcasts the supply fuctio S (b, p to teats accordig to where D i is the capacity of teat i for reductio determied exogeously. S i (b i, p = D i b i p, (23 12

/ (215 1 3 13 2. Participatig teats respod by placig their bids b i order to maximize their ow payoff; 3. The colo operator decides the total amout of reductio from teats d ad market clearig price p to maximize its utility. Give the bids b = (b 1,..., b, if the operator decides to offer d amout of eergy reductio to the utility, the the market clearig price p eeds to satisfy i=1 S i (b i, p = d ad hece, p will be i=1 b i p = i=1 D i d. (24 Hece, to maximize the operator s profit, the operator will chooose d such that ( u d = arg max d i=1 D i (u pd = i=1 b i i=1 D i d 4. Volutary EDR is exercised. N, teat sheds S (b, p, ad receives ps (b, p reward. d. (25 Discussio. A key differece i the operatio of ColoEDR for madatory EDR ad volutary EDR is i the form of the supply fuctio (23. I particular, for volutary EDR, we allow heterogeeity i the supply fuctio for teats i terms of their capacity D. Recall, however, that i the case of madatory EDR the required reductio capacity δ was used. This differece stems from the fact that the reductio target is flexible for volutary EDR ad also creates sigificat challeges both i terms of efficiecy, sice it allows the chace of market power to emerge because of capacity differeces, ad for aalysis, sice it adds cosiderable complexity. 5.2. Efficiecy Aalysis of ColoEDR for Volutary EDR Give the adaptatio of ColoEDR to the volutary EDR settig, it is atural to ask how the efficiecy of the mechaism chages whe the operator has full flexibility i decidig the amout of respose to a volutary EDR sigal. Ituitively, the icreased flexibility leads to the possibility of more iefficiecy, but how large is this effect? We agai quatify efficiecy through a compariso with the (socially optimal outcome. Assume that each teat has a cost c i ( associated with eergy reductio that is covex, icreasig, ad c i (x =, x (Assumptio 1. The, the allocatio that maximizes social utility (the sum of operator s ad teats utility solves the followig problem max d,s subject to ud c i (s i (26a i=1 s i = d i=1 (26b s i D i. (26c Fially, ote that our aalysis makes the followig atural assumptios o the uit price u ad the margial cost of each teat. Note that these are aalogous to Assumptio 2 ad Assumptio 3, respectively. Assumptio 4. The market clearig price p is lower tha the price offered by the utility for ay d >, i.e., u Assumptio 5. The margial cost of each teats satisfies + c (z z ν u z= 2,. i=1 b i i=1 D i. Before movig to the mai results, let us first defie some otatios. Lettig ν = D i=1 D i, we have ν = 1. Here ν behaves like market share of teat i the volutary EDR market. I the madatory EDR case, ν = 1/N for all. Furthermore, defie ν = max ν, as the domiat share i load reductio amog the teats, ad D = max D. 5.3. Market Equilibrium Characterizatio As i the case of madatory EDR, we cosider both price-takig ad price-aticipatig teats. 13

/ (215 1 3 14 5.3.1. Price-takig Teats Recall that a price-takig teat cosiders the price as is without accoutig for the impact of its biddig decisio o the market clearig price. Hece, give the other teats biddig decisios, each price-takig teat optimizes the followig payoff over biddig value b, P (b, b = ps (b, p c (S (b, p = pd b c (D b p So, i a price-takig equilibrium (b, d, p, P (b ; b P ( b ; b holds for each teat over all b. Also, the market clearig price p must satisfy (24 ad the total reductio d must satisfy (25. Usig techiques similar to the proof of Theorem 3, we ca completely characterize the price-takig equilibrium of ColoEDR i volutary EDR as follows: Theorem 14. There exists a uique equilibrium of the game defied by (P 1,..., P N for ColoEDR i volutary EDR. For such a equilibrium, the vector s t defied by s t = S (p(b t, b t is the uique optimal solutio to the followig optimizatio: max ud ud2 2 c (s D s.t. s = d (27a (27b d, s D, = 1,..., N, (27c 5.3.2. Price-aticipatig Teats Recall that a price-aticipatig teat actively seek to chage market price through its bid to maximize payoff. Hece, give the other teats biddig decisios, each price-aticipatig teat optimizes the followig payoff over biddig value b, the payoff fuctio Q (b, b ca be derive i a similar maer as (16: Q (b, b = p(bs (b, p c (S (b, p = ν Σm b m u D i b c (D b i=1 D i, Σ m b m u So, i a price-aticipatig equilibrium (b, d, p, we must have Q (b ; b Q ( b ; b for all over all b. Also, the market clearig price p must satisfy (24 ad the total reductio d must satisfy (25. Usig techiques similar to the proof of Theorem 7, we ca completely characterize the price-aticipatig equilibrium of ColoEDR i volutary EDR as follows. Theorem 15. There exists a uique equilibrium of the game defied by (Q 1,..., Q N for ColoEDR i volutary EDR. For such a equilibrium, the vector s a defied by s a = S (p(b a, b a is the uique optimal solutio to the followig optimizatio: i=1 max ud ud2 2 ĉ (s D s.t. s = d (28a (28b d, s D, = 1,..., N, (28c 14

/ (215 1 3 15 Teats Price Ratio Colo Extra Profit Welfare Loss Price-takig [1 d Σ D, 1] [, ud 2 /Σ D ] [, ud 2 /2Σ D ] Price-aticipatig [1 d Σ D, 1] [, ud 2 /Σ D ] [, u(σ D ν + d 2 /Σ D /2] Table 2. Performace guaratee of ColoEDR compared to the social optimal allocatio. where, for s, ĉ (s = 1 2 ad for s <, ĉ (s =. ( ν u s 2 + c (s + 1 s (ν 2 u 2 2 + c (z + 2 + c (z zu dz, (29 z z Σ i D i Like i the case of madatory EDR, the above characterizatio ca be approximated usig a modified cost fuctio whe ν is small, i.e., whe there are a large umber of teats ad all teats have similar market shares. Lemma 16. For s D, the modified cost i (29 ca be upper ad lower bouded by, c (s ĉ (s c (s + s ν u 2, Furthermore, where the left or right derivatives are defied, we have c (s ĉ (s + ĉ (s + c (s + ν u 2. (3a 5.4. Boudig Efficiecy Loss We ow use the characterizatio results i Theorem 14 ad Theorem 15 to aalyze the social efficiecy of ColoEDR i the volutary EDR settig for both price-takig ad price-aticipatig teats. Theorem 17. For price takig teats, the welfare loss of ColoEDR i volutary EDR is bouded by ud t c (s t ud c (s ud 2 2 D. Moreover, the boud is tight. Theorem 18. For price aticipatig teats, the welfare loss of ColoEDR i volutary EDR is bouded by ud a c (s a ud ( c (s u 2 Σ D ν + d 2 Σ D. Theorem 17 highlights that the price-takig market equilibrium is efficiet whe the optimal eergy reductio d is small. This is due to the profit maximizig behavior of the operator: whe the social optimal d is large, the operator has greater opportuity to raise his profit by lowerig the market price. Comparig Theorem 18 with Theorem 17, we ca see that the additioal welfare loss due to price-aticipatig behavior of teats is a fuctio of ν, the market share of the teats. It is easy to see that the additioal loss of social welfare is miimized whe ν = 1/N for all, i.e., whe the reductio capacity of each teat is equal. Additioally, we ca obtai tight bouds o the market clearig price, eergy reductio quatity, ad operator s profit i a similar fashio as our aalysis doe for the madatory EDR case usig Theorem 14 ad Theorem 15. Due to space costraits, we summarize the results i Table 2 ad Table 3. Table 2 shows that as the optimal reductio d icreases, there is more opportuity for the operator to profitably reduce market price ad icrease his ow profit. Table 3 shows further that, whe teats are price-aticipatig, they will drive the market clearig price up, provide less eergy reductio ad reduce the operator s profit. However, all these additioal losses ca be bouded by liear fuctios of ν, the domiat share of the eergy reductio capacity. Hece, the loss due to price-aticipatig behavior of teats is miimized D 1 = D 2 = = D N. 15

/ (215 1 3 16 Price Markup Load Reductio Operator s cost [, uν/2] [ D/2, ] [, ud] Table 3. Performace guaratee of ColoEDR whe teats are price-aticipatig compared to them beig price-takig. 6. Case Study Our goal i this sectio is to ivestigate ColoEDR i a realistic sceario. Give the theoretical results i the prior sectios, we kow that ColoEDR is efficiet for both the operator ad teats whe the umber of teats is large, but that it may use excessive o-site geeratio (i the worst case. Thus, two importat issues to address i the case study are: How efficiet is the pricig mechaism i small markets, i.e., whe N is small? What is the impact of the pricig mechaism o o-site geeratio i realistic scearios? Additioally, the case study allows us to better uderstad whe it is feasible to obtai load sheddig from teats, i.e., how flexible must teats be i order to actively participate i a load sheddig program? Due to space costraits, we discuss oly madatory EDR i this sectio. The results i the case of volutary EDR are parallel ad hece omitted for brevity. 6.1. Simulatio Settigs We use trace-based simulatios i our case study. Our simulator takes the teats workload trace ad a trace of madatory EDR sigals from Pesylvaia-New Jersey-Marylad Itercoectio (PJM [4] as its iputs. It the executes ColoEDR (by emulatig the biddig process ad teats eergy reductio for EDR at each timestamp of the EDR sigal, ad outputs the resultig equilibrium. The settigs we use for modelig the colocatio data ceter ad the teat costs follow. Colocatio data ceter setup. We cosider a colocatio data ceter located i Ashbur, VA, which is a major data ceter market served by PJM Itercoectio. By default, there are three participatig teats iterested i EDR, though we vary the umber of participatig teats durig the experimets. Each participatig teat has 2, servers, ad each server has a idle ad peak power of 15W ad 25W, respectively. The default PUE of the colo is set to 1.5 (typical for colo, ad hece, wheever a teat reduces 1kWh eergy, the correspodig eergy reductio at the colo level amouts to 1.5kWh. Thus, the maximum possible power reductio is 2.25MW (i.e., 1.5MW IT plus.75mw o-it. We assume that the colo operator couts the extra eergy reductio at the colo level as part of the teats cotributios, ad rewards the teats accordigly. The colo has a o-site diesel geerator, which has cost $.3/kWh estimated based o typical fuel efficiecy [36]. For settig the eergy reductio target received by the colo, we follow the EDR sigals issued by PJM Itercoectio from 5:am to 11:am o Jauary 7, 214, whe may states i the easter U.S. experieced a extremely cold weather ad faced a electricity productio shortage [37]. Fig. 1(b shows the total eergy reductio target set by PJM durig that day for all participatig colos. I our simulatio, we keep the shape of the eergy reductio target but scale dow the reductio amout based o real power cosumptio i our cosidered colo. Teat workloads characteristics. We choose three represetative types of workloads for participatig teats: teat 1 is ruig high delay-sesitive workloads (e.g., user-facig web service, teat 2 is ruig low delaysesitive workloads (e.g., eterprise s iteral services, ad teat 3 is ruig delay-tolerat workload (e.g., backed processig. The workload traces for the three participatig teats were collected from server utilizatio log of MSR [38], Wiki [39], ad Florida Iteratioal Uiversity, respectively. Fig. 1(a illustrates a sapshot of the traces of server cluster utilizatio over 24 hours, where the workloads are ormalized with respect to each teat s maximum service capacity. For our evaluatio based o PJM EDR sigals, we oly use the traces from Hour 5 11 (i.e., 5:am 11:am. The illustrated results use a average utilizatio for each teat of 3%, cosistet with reported values from real systems [9]. Our results are ot particularly sesitive to this choice. There are various power maagemet techiques, e.g., load migratio/schedulig, that ca be used for reducig teats server eergy. Here, as a cocrete example, we cosider that teats dyamically cosolidate workloads ad tur o/off servers for eergy savig subject to SLA [4]. This power-savig techique has bee widely studied [4, 41] ad also recetly applied i real systems (e.g., Facebook s AutoScale [42]. 16

/ (215 1 3 17 Workload.6.4.2 MSN MSR Wiki Uiversity 4 8 12 16 2 24 Hour (a EDR (MWh 25 2 15 1 5 4 5 6 7 8 9 1 11 12 Hour (b Figure 1. (a Workload traces. (b Eergy reductio for PJM s EDR o Jauary 7, 214 [37]. Social cost ($ 3 2 1 4 6 8 1 12 Hour (a Social cost Eergy reductio (kwh 1 5 T1 T2 T3 Diesel 4 6 8 1 12 Hour (b Eergy reductio Net utility ($ 2 15 1 5 T1 T2 T3 4 6 8 1 12 Hour (c Teats et profits Cost ($ 3 2 1 Teats Diesel 4 6 8 1 12 Hour (d Operator s total cost Price ($/kwh.4.3.2.1 Diesel price Utilizatio (% 1 5 Utilizatio boud Utilizatio (% 1 5 Utilizatio boud Utilizatio (% 1 5 Utilizatio boud 4 6 8 1 12 Hour 4 6 8 1 12 Hour 4 6 8 1 12 Hour 4 6 8 1 12 Hour (e Market clearig price (f Teat 1 s utilizatio (g Teat 2 s utilizatio (h Teat 3 s utilizatio Figure 2. Performace compariso uder default settigs. Throughout this ad later plots, the bars i each cluster are the price-takig, priceaticipatig, socially optimal, ad diesel oly (if applicable outcomes. Whe teats save eergy for EDR by turig off some servers, their applicatio performace might be affected. We adopt a simple yet commo model based o a M/G/1/Processor-Sharig queueig model, as follows. For a teat with M servers each with a service rate of µ, deote the workload arrival rate by λ. Whe m servers are shut dow, we model the total delay cost as c(m = λ β T delay(m = βt 1 νm 1 M m, where ν = λ µm deotes the ormalized workload arrival (i.e., utilizatio without turig off servers, T is the duratio of a EDR evet, ad β is a cost parameter ($/time uit/job. I our simulatios, we set the cost parameter for teat 1, teat 2 ad teat 3 as.1,.3,.6, respectively, which are already higher tha those cosidered i the prior cotext of turig off servers for eergy savig [4]. Note that we have experimeted with a variety of other models as well ad the results do ot qualitatively chage. We use a stadard model for eergy usage [9] ad take the eergy reductio s as liear i the umber of servers shut dow, i.e., s = θ m, where θ is a costat decided by server s idle power ad T. The, it yields the followig cost fuctio for a teat s eergy reductio c(s = c( s θ c(, where c( is defied i the above paragraph. Note that we have experimeted with a variety of other forms, ad our results are ot sesitive to the details of this cost fuctio. Fially, ote that teats typically have a delay performace requiremet. Based o the above queueig model, this ca be traslated as a utilizatio upper boud. Such a traslatio is also commo i real systems (e.g., default policy for auto-scalig virtual machies [43]. I our simulatio, we capture the performace costrait by settig utilizatio upper bouds for teat 1, teat 2, ad teat 3 as.5,.6, ad.8, respectively. Efficiecy bechmarks. Throughout our experimets, we cosider the price-takig, price-aticipatig, ad social optimal outcomes. Additioally, we cosider oe other bechmark, diesel oly, which is meat to capture commo practice today ad to highlight that ay teat respose extracted grees data ceter demad respose. Uder diesel oly, the full EDR respose is provided by the o-site diesel geerator. Throughout, our results are preseted i grouped bar plots with the bars represetig (from left to right the price-takig, price-aticipatig, social optimal, 17

/ (215 1 3 18 Social cost ($ 3 2 1 Price ($/kwh.4.3.2.1 Diesel price Net utility ($ 25 T1 T2 T3 2 15 1 5 Cost ($ Teats Diesel 4 2 3 6 9 12 15 18 # of teats 3 6 9 12 15 18 # of teats 3 6 9 12 15 18 # of teats 3 6 9 12 15 18 # of teats (a Social cost (b Market price (c Teats et profits (d Operator s total cost Figure 3. Impact of umber of teats. ad diesel oly (if applicable outcomes. While other mechaisms (e.g., direct pricig [16], auctio [35] have bee itroduced i recet papers, we do ot compare ColoEDR with them here because ColoEDR is already typically idistiguishable from the social optimal cost. 6.2. Performace Evaluatio We ow discuss our mai results, show i Fig. 2. Social cost. We first compare i Fig. 2(a the social costs (4 icurred by differet algorithms. Note that ColoEDR is close to the social cost optimal uder both price-takig ad price-aticipatig cases eve though there are oly three participatig teats. Further, the resultig social costs i both the price-takig ad price-aticipatig scearios are sigificatly lower tha that of the diesel oly outcome. This shows a great potetial of teats IT power reductio for EDR, which is cosistet with the prior literature o ower-operated data ceter demad respose [15, 16, 2]. Eergy reductio cotributios. Fig. 2(b plots EDR eergy reductio cotributios from teats ad the diesel geerator. As expected from aalytic results, both price-takig ad price-aticipatig teats ted to cotribute less to EDR (compared to the social optimal because of their self-iterested decisios. I other words, give self-iterested teats, the colo operator eeds more diesel geeratio tha the social optimal. Noetheless, the differece is fairly small, much smaller tha predicted by the worst-case aalytic results. This highlights that worst-case results were too pessimistic i this case. Of course, oe must remember that all teat reductio extracted is i-place of diesel geeratio, ad so serves to make the demad respose more evirometally friedly. Beefits for teats ad colocatio operator. We show i Fig. 2(c ad Fig. 2(d that both the teats ad the colo operator ca beefit from ColoEDR. Specifically, Fig. 2(c presets et profit (i.e., paymet made by colo operator mius performace cost received by teats, showig that all participatig teats receive positive et rewards. While price-aticipatig teats ca receive higher et rewards tha whe they are price-takig, the extra reward gaied is quite small. Similarly, Fig. 2(d shows cost savig for the colo operator, compared to the diesel oly case. Market clearig price. Fig. 2(e shows the market clearig price. Naturally, whe usig ColoEDR to icetivize teats for EDR while miimizig the total cost, the colo operator will ot pay the teats at a price higher tha its diesel price (show via the red horizotal lie. We also ote that the price uder the price-aticipatig case is higher tha that uder the price-takig case, because the price-aticipatig teats are more strategic. However, the price differece betwee price-aticipatig ad price-takig cases is quite small, which agai cofirms our aalytic results. Teat server utilizatio. Teats server utilizatios are show i Figs. 2(f, 2(g ad 2(h, respectively. These illustrate that, while teats reduce eergy for EDR, their server utilizatios still stay withi their respective limits (show via the red horizotal lies, satisfyig performace costraits. This is because teats typically provisio their servers based o the maximum possible workloads (plus a certai margi, while i practice their workloads are usually quite low, resultig i a slackess that allows for savig eergy while still meetig their performace requiremets. 6.3. Sesitivity Aalysis To complete our case study, we ivestigate the sesitivity of the coclusios discussed above to the settigs used. For each study, we oly show results that are most sigificatly differet tha those i Fig. 2. Impact of the umber of teats. First, we vary the umber of participatig teats ad show the results i Fig. 3. To make results comparable, we fix the EDR eergy reductio requiremet as well as total umber of servers: 18

/ (215 1 3 19 Social cost ($ 6 4 2.1.2.3.4.5.6 Diesel price ($/kwh Eergy reductio (kwh 1 5 T1 T2 T3 Diesel.1.2.3.4.5.6 Diesel price ($/kwh Price ($/kwh.6.4.2 Diesel price.1.2.3.4.5.6 Diesel price ($/kwh Net utility ($ 4 T1 T2 T3 3 2 1.1.2.3.4.5.6 Diesel price ($/kwh (a Social cost (b Eergy reductio (c Market price (d Teats et profits Figure 4. Impact of diesel price. Eergy reductio (kwh 2 T1 T2 T3 Diesel 15 1 5 2 4 6 8 1 12 EDR requiremet (% Price ($/kwh.4.3.2.1 Diesel price 2 4 6 8 1 12 EDR requiremet (% (a Eergy reductio (b Market price Figure 5. Impact of EDR eergy reductio target. teat 1, teat 2 ad teat 3 are each equally split ito multiple smaller teats, each havig fewer servers with the same workload arrival rate scaled dow accordigly. We the aggregate replicas of the same teat together for a easy viewig i the figures, e.g., teat 1 i the figures represet the whole group of teats that are obtaied by splittig the origial teat 1. Oe iterestig observatio is that as more teats participate i EDR, the market becomes more competitive. Hece, each idividual teat ca oly gai less et reward, but both the price ad the aggregate et reward become higher (see Figs. 3(b ad 3(c. Motivated by this, oe might suggest a possible trick: a teat may gai more utility by splittig its servers ad pretedig to be multiple teats. I practice, however, each teat has oly oe accout (for billig, etc. which requires cotracts ad base fees, ad thus pretedig as multiple teats is ot viable i a colo. Impact of the price of diesel. Fig. 4 illustrates how our result chages as the diesel price varies. Ituitively, as show i Fig. 4(a, the social cost (which icludes diesel cost as a key compoet icreases with the diesel price. We see from Figs. 4(b ad 4(c that, whe diesel price is very low (e.g., $.1/kWh, the colo operator is willig to use more diesel ad offers a lower price to teats. As a result, teats cotribute less to EDR. As the diesel price icreases (e.g., from $.2/kWh to $.3/kWh, the colo operator icreases the market price (but still below the diesel price to ecourage teats to cut more eergy for EDR. Noetheless, teats eergy reductio cotributio caot icrease arbitrarily due to their performace costraits. Specifically, after the diesel price exceeds $.4/kWh, teats will ot cotribute more to EDR (i.e., almost all their IT eergy reductio capabilities have bee exploited, eve though the colo operator icreases the reward. I this case, teats simply receive higher et rewards without further cotributig to EDR, as show i Fig. 4(d. Impact of EDR requiremet. Fig. 5 varies the EDR eergy reductio target, with the maximum reductio ragig from 2% to 12% of the colo s peak IT power cosumptio. As the EDR eergy reductio target icreases, teats eergy reductio for EDR also icreases; after a certai threshold, diesel geeratio becomes the mai approach to EDR, while the icrease i teat s cotributio is dimiishig (eve though the colo operator icreases the market price, because of teats performace requiremets that limit their eergy reductio capabilities. Impact of teats workloads. I Fig. 6(a-6(b, we vary the teats workload itesity (measured i terms of the average server utilizatio whe all servers are active from 1% to 5%, while still keepig the maximum utilizatio bouds to 5%, 6% ad 8% as the performace requiremets for the three teats, respectively. While it is straightforward that whe teats have more workloads, they ted to cotribute less to EDR, because they eed to keep more servers active to deliver a good performace. Noetheless, eve whe their average utilizatio without turig off servers is as high as 5% (which is quite high i real systems, cosiderig that the average utilizatio 19

/ (215 1 3 2 Social cost ($ 3 2 1 1 2 3 4 5 Utilizatio (% Eergy reductio (kwh 1 5 T1 T2 T3 Diesel 1 2 3 4 5 Utilizatio (% Social cost ($ 3 2 1 5 1 15 2 Over predictio rate (% Price ($/kwh.4.3.2.1 Diesel price 5 1 15 2 Over predictio rate (% (a Social cost (b Eergy reductio (c Social cost (d Market price Figure 6. Impact of teats workloads ad the workload predictio errors. is oly aroud 1-3% [9], teats ca still cotribute more tha 2% of EDR eergy reductio uder ColoEDR, showig agai the potetial of IT power maagemet for EDR. Impact of workload predictio error. I practice, teats may ot perfectly estimate their ow workload arrival rates. To cope with possible traffic spikes, teats ca either keep more servers active as a backup or deliberately overestimate the workload arrival rate by a certai overestimatio factor. We choose the later approach i our simulatio. Fig. 6(c-6(d shows the result uder workload predictio errors. We see that both the social cost ad market price are fairly robust agaist teats workload over-predictios. For example, the social cost icreases by less tha 1%, eve whe teats overestimate their workloads by 2% (which is already sufficietly high i practice, as show i [41]. Other results (e.g., teats et reward, colo operator s total cost are also oly miimally affected, thereby demostratig the robustess of ColoEDR agaist teats workload over-predictios. 7. Related Work Our work cotributes both to the growig literature o data ceter demad respose, ad to the literature studyig supply fuctio equilibria. We discuss each i tur below. Recetly, data ceter demad respose has received a growig amout of attetio. A variety of approaches have bee cosidered, such as optimizig a grid operator s pricig strategies for data ceters [16] ad tuig computig (e.g., server cotrol ad schedulig ad/or o-computig kobs (e.g., coolig system i data ceters for various types of demad respose programs [15, 44, 13, 14]. Field tests by LBNL also verify the practical feasibility of data ceter demad respose usig a combiatio of existig power maagemet techiques (e.g., load migratio [18]. These studies, however, have all focused o large ower-operated data ceters. I cotrast, to the best of our kowledge, colocatio demad respose has bee ivestigated by oly a few previous works. The first is [34], which proposes a simple mechaism, called icode, to icetivize teats load reductio. But, icode is purely for volutary EDR ad does ot iclude ay eergy reductio target (eeded for madatory EDR. More importatly, icode is desiged without cosiderig strategic behavior by teats, ad ca be compromised by price-aticipatig teats [34]. More relevat to the curret work is [35], which proposes a VCG-type auctio mechaism where colo participates i EDR programs. While the mechaism is approximately truthful, it asks participatig teats to reveal their private cost iformatio through complex biddig fuctios. Further, the colo operator may be forced to make arbitrarily high paymets to teats. I cotrast, our proposed solutio provides a simple biddig space, protects teats private valuatio, ad esures that the colo operator does ot icur a higher cost for EDR tha the case teat cotributios. Thus, ulike [35], ColoEDR beefits both colo operator ad teats, givig both parties icetives to cooperate for EDR. Further, ColoEDR is applicable to both madatory ad volutary EDR, both of which are importat EDR programs [8]. Fially, it is importat to ote that our approach builds o, ad adds to, the supply fuctio mechaism literature. Supply fuctio biddig (c.f. the semial work by [45] is frequetly used i electricity markets due to its simple biddig laguage ad the avoidace of the ubouded paymets typical i VCG-like mechaisms. Supply fuctio biddig mechaisms have bee extesively studied, e.g., [24, 25, 26, 27, 33, 46]. The literature primarily focuses o existece ad computatio of supply fuctio equilibrium, sometimes additioally provig bouds o efficiecy loss. Our work is most related to [23], which cosiders a ielastic demad δ that must be satisfied via extractig load sheddig from cosumers ad proves efficiet bouds o supply fuctio equilibrium. I cotrast, our work assumes that the operator has a outside optio (diesel that ca be used to satisfy the ielastic demad. This leads 2

/ (215 1 3 21 to a multistage game betwee the teats ad the profit-maximizig operator, a dyamic which has ot bee studied previously i the supply fuctio literature. 8. Coclusio I this paper, we focused o greeig colocatio demad respose by desigig a pricig mechaism that ca extract load reductios from teats durig EDR evets. Our mechaism, ColoEDR, ca be used i both madatory ad volutary EDR programs ad is easy to put i place give systems available i colos today. The mai techical cotributio of the work is the aalysis of the ColoEDR mechaism, which is a supply fuctio mechaism for a elastic demad, a settig for which efficiecy results have ot previously bee attaied i the supply fuctio literature. Our results highlight that ColoEDR provides provably ear-optimal efficiecy guaratees, both whe teats are pricetakig ad whe they are price-aticipatig. We also evaluate ColoEDR usig trace-based simulatio studies ad validate that ColoEDR is ot oly grees multi-teat EDR by reducig diesel geeratio, it also beefits the colo operator by reducig costs ad the teats by providig paymets for reductios. Refereces [1] NRDC, Scalig up eergy efficiecy across the data ceter idustry: Evaluatig key drivers ad barriers, Issue Paper. [2] A. Wierma, Z. Liu, I. Liu, H. Mohseia-Rad, Opportuities ad challeges for data ceter demad respose, i: IGCC, 214. [3] K. Maaga, Demad repsose: A market overview (214, http://eaxiscosultig.com. [4] PJM, Emergecy demad respose (load maagemet performace report 212/213. [5] A. Misra, Respodig before electric emergecies (http://www.afcom.com/digital-library/pub-type/commuique/ respodig-before-electric-emergecies/. [6] EerNOC, Esurig U.S. grid security ad reliability: U.S. EPA s proposed emergecy backup geerator rule (213. [7] Sata Babara Couty, Air Pollutio Cotrol District (http://www.ourair.org/do-you-really-eed-a-diesel-geerator/. [8] PJM, Retail electricity cosumer opportuities for demad respose i PJM s wholesale markets (http://www.pjm.com. [9] L. A. Barroso, J. Clidaras, U. Hoelzle, The Dataceter as a Computer: A Itroductio to the Desig of Warehouse-Scale Machies, Morga & Claypool, 213. [1] H. Lim, A. Kasal, J. Liu, Power budgetig for virtualized data ceters, i: USENIX ATC, 211. [11] D. Wag, S. Govida, A. Sivasubramaiam, A. Kasal, J. Liu, B. Khessib, Uderprovisioig backup power ifrastructure for dataceters, i: ASPLOS, 214. [12] S. Li, M. Brocaelli, W. Zhag, X. Wag, Data ceter power cotrol for frequecy regulatio, i: PES, 213. [13] B. Aksali, T. S. Rosig, Providig regulatio services ad maagig data ceter peak power budgets, i: DATE, 214. [14] H. Che, A. K. Cosku, M. C. Caramais, Real-time power cotrol of data ceters for providig regulatio service, i: CDC, 213. [15] D. Aikema, R. Simmods, H. Zareipour, Data cetres i the acillary services market, i: IGCC, 212. [16] Z. Liu, I. Liu, S. Low, A. Wierma, Pricig data ceter demad respose, i: SIGMETRICS, 214. [17] M. Ghamkhari, H. Mohseia-Rad, Data ceters to offer acillary services, i: SmartGridCom, 212. [18] G. Ghatikar, V. Gati, N. E. Matso, M. A. Piette, Demad respose opportuities ad eablig techologies for data ceters: Fidigs from field studies (212. [19] Data Ceter Kowledge, Iside SuperNAP 8: Switch s Tier IV data fortress, Feb. 11, 214. [2] DataceterMap, Colocatio USA (http://www.datacetermap.com/usa/. [21] J. Verge, Symatec Sigs Multi-Megawatt Lease at Sata Clara Data Ceter (215, http://www.dataceterkowledge.com/. [22] Equiix, Customer portal (http://www.equiix.com/services/support/customer-portal/. [23] R. Johari, J. N. Tsitsiklis, Parameterized supply fuctio biddig: Equilibrium ad efficiecy, Operatios research 59 (5 (211 179 189. [24] C. J. Day, B. F. Hobbs, J.-S. Pag, Oligopolistic competitio i power etworks: a cojectured supply fuctio approach, Power Systems, IEEE Trasactios o 17 (3 (22 597 67. [25] R. Baldick, R. Grat, E. Kah, Theory ad applicatio of liear supply fuctio equilibrium i electricity markets, Joural of Regulatory Ecoomics 25 (2 (24 143 167. [26] R. J. Gree, D. M. Newbery, Competitio i the british electricity spot market, J. of Political Ecoomy (1992 929 953. [27] R. Gree, Icreasig competitio i the british electricity spot market, J. of Idustrial Ecoomics (1996 25 216. [28] S. Og, P. Deholm, E. Doris, The impacts of commercial electric utility rate structure elemets o the ecoomics of photovoltaic systems, Natioal Reewable Eergy Laboratory, 21. [29] X. Fa, W.-D. Weber, L. A. Barroso, Power provisioig for a warehouse-sized computer, i: ACM SIGARCH Computer Architecture News, Vol. 35, ACM, 27, pp. 13 23. [3] L. L. Adrew, M. Li, A. Wierma, Optimality, fairess, ad robustess i speed scalig desigs, SIGMETRICS Perform. Eval. Rev. 38 (1 (21 37 48. doi:1.1145/181199.181144. URL http://doi.acm.org/1.1145/181199.181144 [31] A. Wierma, L. L. Adrew, A. Tag, Power-aware speed scalig i processor sharig systems, i: INFOCOM 29, IEEE, IEEE, 29, pp. 27 215. [32] A.-H. Mohseia-Rad, A. Leo-Garcia, Optimal residetial load cotrol with price predictio i real-time electricity pricig eviromets, Smart Grid, IEEE Trasactios o 1 (2 (21 12 133. 21

/ (215 1 3 22 [33] E. J. Aderso, X. Hu, Fidig supply fuctio equilibria with asymmetric firms, Oper. Res. 56 (3 (28 697 711. [34] S. Re, M. A. Islam, Colocatio demad respose: Why do I tur off my servers?, i: ICAC, 214. [35] L. Zhag, S. Re, C. Wu, Z. Li, A truthful icetive mechaism for emergecy demad respose i colocatio data ceters, i: INFOCOM, 215. [36] Wikipedia (http://e.wikipedia.org/wiki/diesel_geerator. [37] PJM, Demad respose activity o Jauary 7-8, 214 (http://www.pjm.com. [38] E. Thereska, A. Doelly, D. Narayaa, Sierra: a power-proportioal, distributed storage system, Tech. Rep. MSR-TR-29-153. [39] G. Urdaeta, G. Pierre, M. Va Stee, Wikipedia workload aalysis for decetralized hostig, Computer Networks. [4] M. Li, A. Wierma, L. L. H. Adrew, E. Thereska, Dyamic right-sizig for power-proportioal data ceters, i: IEEE Ifocom, 211. [41] A. Gadhi, M. Harchol-Balter, R. Raghuatha, M. A. Kozuch, Autoscale: Dyamic, robust capacity maagemet for multi-tier data ceters, ACM Tras. Comput. Syst. 3 (4 (212 14:1 14:26. [42] Q. Wu, Makig facebook s software ifrastructure more eergy efficiet with autoscale (214. [43] Microsoft Azure, How to use the autoscalig applicatio block (http://azure.microsoft.com. [44] H. Wag, J. Huag, X. Li, H. Mohseia-Rad, Explorig smart grid ad data ceter iteractios for electric power load balacig, SIG- METRICS Perform. Eval. Rev. 41 (3 (214 89 94. [45] P. D. Klemperer, M. A. Meyer, Supply fuctio equilibria i oligopoly uder ucertaity, Ecoometrica: Joural of the Ecoometric Society (1989 1243 1277. [46] X. Vives, Strategic supply fuctio competitio with private iformatio, Ecoometrica 79 (6 (211 1919 1966. [47] N. Che, X. Re, S. Re, A. Wierma, Greeig multi-teat data ceter demad respose, arxiv preprit arxiv:154.738. 22

/ (215 1 3 23 Due to space costraits, we oly iclude proofs of the results for the madatory EDR sceario. The aalysis for volutary EDR is aalogous to the madatory case, though more complex. Full proofs for all the results ca be foud i the techical report [47]. Appedix A. Price takig teats Appedix A.1. Proof of Propositio 1 Whe teats are price takers, they maximize the payout P (b, p = ps (b, p c (s over the bid b. Note that b [, pδ] as o teat will bid beyod pδ otherwise the payout P <. Hece b = (b 1,..., b is a equilibrium if ad oly if the followig coditio is satisfied c (s p, b < pδ, (A.1a + c (s p, < b pδ. (A.1b At least oe feasible solutio to (9 exists because it is miimizig a cotiuous fuctio over a compact set. Furthermore, (9b - (9c satisfy stadard costrait qualificatio, hece for the Lagragia L(s, µ = c (s + µ((δ y s, there exists optimal primal dual pair (s, µ, such that (9b ad (9c are satisfied, ad c (s µ, s >, (A.2a + c (s µ, s. (A.2b Give the optimal (s, µ, let p = µ, ad b = p(δ s, the (9b implies p satisfies (2, ad (A.2a-(A.2b implies (A.1a - (A.1b, hece a equilibrium exists. Coversely, if (b, p is a equilibrium ad p satisfies (2, the resultig allocatio s is optimal to (9. To see this, if s < δ y for all, (A.1a-(A.1b is equivalet to (A.2a-(A.2b if we set µ = p, hece (s, µ is primal dual optimal pair for (9. If s = (δ y, the s m =, m. I this case, we set µ = mi{p, + c (s / }, ad we ca check that (s, µ is the primal dual optimal solutio for (9. Appedix A.2. Proof of Theorem 3 By Propositio 1, whe teats are price-takig, for ay y, the there is always a equilibrium, ad the resultig s is always the optimal allocatio to provide (δ y eergy reductio. Hece we oly eed to verify that the o-site geeratio level y is the solutio to (14a-(14c. Similar to the proof of Propositio 1, by Assumptio 2, the first order optimality coditio for the y i (14a-(14c is Nδ (y + (N 1δ = p. By Propositio 1, p satisfies the relatio (2, substitute the left-had-side ito (2 ad solve for y, we have y = Σ b Nδ (N 1δ. This is exactly the o-site geeratio y that miimizes cost o (b, y give i (12. Hece the dataceter will always pick y that is optimal for (14a-(14c, together with Propositio 1, a equilibrium exists, ad the resultig allocatio (s, y is optimal for (14a-(14c. Appedix A.3. Proof of Propositio 4 Sice y, it suffices to prove that wheever the optimal o-site geeratio is o-zero, y >, y t y. From (4, the Lagragia of SCM is L(s, y, µ, λ = c (s + y + µ ((δ y s λ y. By costrait qualificatio ad the KKT coditios, assumig y >, the λ =, µ =, hece the market clearig price i the optimal allocatio should be p =. 23

/ (215 1 3 24 Next, cosider the market price for price takig teats. From (13, p t i N b t i (Σ i N b t i = (N 1δ + y t =. (A.3 Nδ The secod equality yields i N b t i = ((N 1δ+yt 2 Nδ. Substitute this back to (A.3, p t = i N b t i (N 1δ + y t = (N 1δ + yt. (A.4 Nδ Ad ote that y t [, δ] ad p =, thus (A.4 yields N 1 N p p t p. Fially, from (14, the Lagragia of the price-takig characterizatio optimizatio is, L(s, y, µ t, λ t = c (s + 2Nδ (y + (N 1δ2 + µ t ((δ y s λ t y. By examiig the KKT coditio ad usig a similar argumet to the proof of Propositio 1, we have p t = µ t, also, p t p + c (s s. Thus,, s t s. Sice y = δ s, y t y. c (s t s t Appedix A.4. Proof of Propositio 2 From the proof of Propositio 4, we see that whe y >, λ =, ad µ =. Furthermore, we have s < δ, but s = δ b µ. Hece (Nδ Σ b < δ. Coversely, if (1 holds, the (N 1δ < b. But by Propositio 1 ad (2, we have b = (p (N 1δ + y. By combiig the two equatios above: (N 1δ < p ((N 1δ + y. However, from the proof i Propositio 1, we have p, hece we must have y >. O the other had, whe the data ceter operator is profit maximizig, the cost to the operator cost o (b, y = + y is a covex fuctio i y over the domai y. By first order coditio, the cost is miimized whe (Σ b (δ y (N 1δ+y N y δσ b = (N 1δ, the y = y if ad oly if y [, δ]. However, Σ b = Σ p(δ s = p((n 1δ + y (Nδ, where the last iequality is because y δ, ad p, sice operator always has the optio to use o-site geeratio to get uit cost of eergy reductio at. Hece we always have y δ. So, if y >, by (A.5, (11 must hold, coversely, if (11 holds, the by (A.5, y >, so operator will use y = y. Appedix A.5. Proof of Theorem 5 Note that (s, y is a feasible solutio to (14. By Theorem 3, we have c (s t + 2Nδ (yt +(N 1δ 2 c (s + 2Nδ (y + (N 1δ 2. Rearragig, we have c (s t + y t c (s + y 2Nδ (yt y ( 2δ (y t + y = 2Nδ [ (yt y 2 + 2(δ y (y t y ] 2Nδ [ (yt y (δ y 2 + (δ y 2 ] = 2Nδ (δ y 2 δ 2N. Appedix A.6. Proof of Theorem 6 From Propositio 4, we have N 1 N pt p =, ad y t δ, which yields cost o(p, y cost o (p t, y t = p (δ y + y ( p t (δ y t + y t = ( p t (δ y t Substitutig the above bouds for p t ad y t gives cost o(p, y cost o (p t, y t δ N. 24 (A.5

/ (215 1 3 25 Appedix B. price-aticipatig teats Appedix B.1. Proof of Theorem 7 The proof proceeds i a umber of steps. We first show that the payoff fuctio Q is a cocave ad cotiuous fuctio for each firm. We the establish ecessary ad sufficiet coditios for b to be a equilibrium; these coditios look similar to the optimality coditios (A.1a-(A.1b i the proof of Propositio 1, but for a modified cost fuctio defied accordig to (21. We the show the correspodece betwee these coditios ad the optimality coditios for the problem (2a-(2c. This correspodece establishes existece of a equilibrium, ad uiqueess of the resultig allocatio. Step 1: If b is a equilibrium, ad Assumptio 2 is satisfied, at least oe coordiate of b is positive. By Assumptio 2, < < Σ b (N 1δ, hece at least oe coordiate of b must be positive. Step 2: The fuctio Q ( b ; b is cocave ad cotiuous i b, for b. From (16 ad by pluggig p(b ito s i (1, we have (Σ m b m + b δ Q ( b ; b = b c N δ b Nδ Σm b m + b. Whe Σ m b m + b >, the fuctio b / Σ m b m + b is a strictly cocave fuctio of b (for b. Sice c is assumed to be covex ad odecreasig (ad hece cotiuous, it follows that Q ( b, b is cocave ad cotiuous i b, for b. It is easy to show that for s to be positive, we eed b b where b = 1 2 Step 3:I a equilibrium, b b,. ( δ N + Teat would ever bid more tha b give b. If b > b, the S (p(b, b = δ Q (b ; b becomes egative; o the other had, Q (b ; b =. δ N ( δ N + 4Σ m b m We specify the followig coditio whe margial cost of productio is ot less tha the price: b b Nδ +Σ m b m <. so the payoff., c (s p(b, s >. (B.1 This coditio is satisfied whe teats are price-takig, i the ext step, we show that (B.1 also holds i a equilibrium outcome whe teats are price-aticipatig. Step 4: The vector b is a equilibrium if ad oly if (B.1 is satisfied, at least oe compoet of b is positive, ad for each, b [, b ], ad the followig coditios hold: if < b b, if b < b, ( 1 + c (s 2 ( c (s 1 2 + 2N + 2N + 1 ( + c (s 2 ( c (s + 1 2 2 + + c (s 2N 2 + c (s 2N 2s Nδ 2s Nδ p(b, (B.2a p(b. (B.2b By Step 2, Q (b ; b is cocave ad cotiuous for b. By Step 3, b [, b ]. b must maximize Q (b ; b over b b, ad satisfy the followig first order optimality coditios: + Q (b ; b, b if < b b ; Q (b ; b, b if b < b ; 25

/ (215 1 3 26 1 Recallig the expressio for p(b give i (13, ad ote that by (13 ad (1, we have : Σm = 1 b m p(b Nδ, ad b Σm = (δ s b m Nδ. Expadig the first order optimality coditios with (13 ad simplify with the two equatios ito the above, we have ( 1 2p(b N 1 + c (s 1 1 1 p(b 1 2p(b N 1 + + c (s 1 p(b N 2p(b ( 1 1 2p(b N δ s. δ δ s δ To show (B.1 holds, we divide ito two cases, whe N 2, by rearragig (B.3a, we have. (B.3a (B.3b c (s 1 p(b 2N p(b 2N p(b δ s δ 1. This is because by Assumptio 2, 2N p(b > whe N 2. Also, we have 2N p(b δ s δ 2N p(b. Hece (B.1 holds for N 2. Whe N = 1, we ca simplify (B.3a further to 1 2p(b 1 + c (s 1 2p(b, p(b 1 2 ( + c (s c (s. The last iequality is because c (s, otherwise p(b >, but profit maximizig operator will ot pay for price more tha, cotradictio. Hece (B.1 must hold for all N. After multiplyig through (B.3a-(B.3b by p(b ad rearragig, we have two quadratic iequalities i terms of p(b. Solvig the iequalities lead to two sets of coditios of p(b that satisfy the first order optimality coditios, they are: if b < b, if < b b, ( 1 c (s 2 ( + c (s 1 2 + 2N + 2N ± 1 ( c (s 2 ( + c (s ± 1 2 2 + 4 c (s 2N 2 + 4 + c (s 2N s 2Nδ p(b s 2Nδ p(b (B.4a (B.4b However, oly the coditios with plus sig satisfies (B.1, the coditios with mius sig violates (B.1 because sice s >, p(b 2N + c ( < c (s. Hece we discard the coditios with mius sig ad ote that (B.4b-(B.4a correspods to (B.2a-(B.2b. Coversely, suppose that b has at least oe strictly positive compoet, that b b, ad that b satisfies (B.1 ad (B.2a-(B.2b. The we may simply reverse the argumet: by Step 2, Q (b ; b is cocave ad cotiuous i b, ad i this case the coditios (B.2a-(B.2b imply that b maximizes Q (b ; b over b b. Sice we have already show that choosig b > b is ever optimal for firm, we coclude that b is a equilibrium, ad it is easy to check that i this case coditio (B.1 is satisfied. Step 5: If Assumptio 2 holds, the the fuctio ĉ (s defied i (21 is cotiuous, ad strictly covex ad strictly icreasig over s, with ĉ(s = for s. ĉ (s is cotiuous o s > by cotiuity of c ad o s < by defiitio. We oly eed to show that ĉ ( =, this is because whe s =, c (s =, s 2N =, ad itegratig from to s is. Hece ĉ (s = for s. 26

For s, we simply compute the directioal derivatives of ĉ : + ĉ (s = 1 ( 2 2N + + c (s ĉ (s = 1 ( 2 2N + c (s / (215 1 3 27 ( + 1 2 + 1 2 2N + c (s ( 2N + c (s Sice c is strictly icreasig ad covex, for s < s, we will have + ĉ(s < ĉ( s + ĉ( s. This guaratees that ĉ is strictly icreasig ad strictly covex over s. 2 + 2 + c (s 2 + 2 + c (s Step 6: There exists a uique vector s, y ad at least oe scalar ρ > such that: ( 1 + c (s 2 ( c (s + 2N 1 + + 1 2 2N 2 (y + (N 1δ = ρ; Nδ (B.5c s = (δ y. + 1 ( + c (s 2 ( + c (s 2 + + c (s 2N 2 + + c (s 2N The vector s ad y is the the uique optimal solutio to (2a-(2c. s Nδ, s Nδ. 2s Nδ ρ, if s ; (B.5a 2s Nδ ρ, if s > ; (B.5b (B.5d By Step 5, sice ĉ is cotiuous ad strictly over the covex, compact feasible regio for each, we kow that (2a-(2c have a uique optimal solutio s, y. As i the proof of Propositio 1, form the Lagragia L(s, y; ρ = ĉ (s + 2Nδ (y + (N 1δ2 + ρ((δ y s. By assumptio 2, y >, ad by the fact that ĉ (s = for s, s. there exists a Lagrage multiplier ρ such that (s, y, ρ satisfy the statioarity coditios which correspods to (B.5a-(B.5c whe we expad the defiitio of ĉ (s, together with the costrait (B.5d. The fact that ρ > follows by (B.5c as y >. Step 7: If s, y ad ρ > satisfy (B.5a-(B.5d, the the triple (b, ρ, y defied by b = (δ s ρ is a equilibrium as defied i (17 ad (18. First observe that with this defiitio, together with (B.5d ad the fact that s, we have b for all. Furthermore, we ca show b b, sice s, b ρδ, but by (B.5c-(B.5d, we have ρ = Nδ (y + (N 1δ = Nδ (Nδ s (B.6 Substitute the defiitio s = δ b ρ ito (B.6, we have ρ = Σ b Σ b ρ = Nδ ρ Nδ. (Σ Substitutig (B.7 ito b ρδ, we have b m b m +b δ N, Solvig this iequality we have b b. 27 (B.7

/ (215 1 3 28 Fially, at least oe compoet of b is strictly positive, sice otherwise we have s 1 = s 2 = δ for some 1 2, i which case Σ s > δ, which cotradicts (B.5d. (or s = δ, y =, cotradictig our assumptio that y >. By Step 4, to check that b is a equilibrium, we must oly check the statioarity coditios (B.2a-(B.2b. We simply ote that uder the idetificatio b = ρ(δ s, usig (B.7 ad (B.5c, we have y = Σ b Nδ Σ b (N 1δ; ρ = (N 1δ + y = p(b. Substitute p(b ito (B.5a will correspod to (B.2a, ad (B.5b implies (B.2b ad (B.1 because c (s Thus (b, ρ, y is a equilibrium. + c (s. Step 8: If (b, p(b, y is a equilibrium, the there exists a scalar ρ such that the vector b defied by s = S (p(b, b satisfies (B.5a-(B.5d. We simply reverse the argumet of Step 7. Sice b is a equilibrium bids, by (18 ad s = S (p(b, b, we have s = (δ y, i.e., (B.5d is satisfied. By Step 4, b satisfies (B.2a-(B.2b. Sice y > by Assumptio 2, s < δ for all, let { ρ = max p(b, 1 ( c (s + + 1 ( + c (s 2 2N 2 2N 2 + + c (s } 2s. Nδ I this case ρ > ad b b for all, so (B.2b implies (B.5b by defiitio of ρ, ad (B.5a holds by (B.2a ad the fact that c (s + c (s (by covexity. Step 9: There exists a equilibrium b, ad for ay equilibrium that price is greater tha margial cost, the vector s defied by s = S (p(b, b is the uique optimal solutio of (B.5a-(B.5d. The coclusio is ow straightforward. Existece follows from Steps 6 ad 7. Uiqueess of the resultig productio vector s, ad the fact that s is a optimal solutio to (2a-(2c, follows by Steps 6 ad 8. Appedix B.2. Proof of Lemma 8 We exploit the structure of the modified cost ĉ to prove the result. Note that, for all, s, if we defie G (s = s ( + c (z z 2N 2 + + c (z 2z z Nδ dz, the G (s s ( + c (z z 2 dz = c (s s 2N 2N. First iequality is because z, last equality is because by covexity ad Assumptio 3, we have + c (z 2N. ( Hece we have ĉ (s = 1 2 c (s + s 2N + 1 2 G (s c (s. O the other had, otice that s δ, we have: z + c ( G (s = s s ( + c (z 2 + + c (z z 2N z ( + c (z + z 2N 2δ Nδ dz 2 dz = c (s + s 2N. ( Hece we have ĉ (s = 1 2 c (s + s 2N + 1 2 G (s c (s + s 2N. The bouds for the left ad right derivatives ca be obtaied from takig the left (or right derivatives at the bouds of G (s. 28

Appedix B.3. Proof of Theorem 9 / (215 1 3 29 Firstly we will prove oe side of the iequality p t p a, y t y a. Recall that by the examigig the Lagragias of the optimizatios i Propositio 4 i ad Theorem 7, we have p t c (s t /, p t + c (s t /, p a ĉ (s a /, p a + ĉ (s a /, at the domai where the left or right derivative is defied, ad p t = Nδ (yt + (N 1δ, p a = Nδ (ya + (N 1δ. If y t > y a, the p t > p a. Also, because the total eergy reductio δ is costat, we have s t < s a. Hece there exist s r > such that s a r > s t r for some r {1,..., N}. Therefore, by strict covexity of c (Assumptio 1: However, by Lemma 8 we have ĉ r (s r s r c r (s r s r p t + c r (s t r s r < c r (s a r s r. (B.8. Hece, we have p a ĉ r (s a r s r c r (s a r s r. (B.9 Combiig (B.8 ad (B.9, we have p t < p a, cotradictio. Hece we have y t y a, ad p t p a. Next we show the other side of the iequality p a p t + 2N, ya y t + δ 2, by the previous part, we have s a s t. Let = arg max m (s t m s a m, clearly s t s a, otherwise s t < s a, cotradictio. If s t = s a, the m, s t m = s a m, ad y t = y a, the p t = p a. If s t > s a, the by strict covexity of c (assumptio 1, ad the fact that s a, s t >, we have + ĉ (s a c (s t s p t. Also, by Lemma 8, we have + ĉ (s + c (s the two previous iequalities about p t ad p a, we have p a < p t + 2N + 2N, this gives us pa + ĉ (s a + c (s a. Hece we have Nδ (ya + (N 1δ < Nδ (yt + (N 1δ + 2N ya < y t + δ 2. s < + 2N. Combiig Appedix B.4. Proof of Theorem 11 As (s, y is a feasible solutio to (2, by Theorem 7, we have ĉ (s a + 2Nδ (ya + (N 1δ 2 ĉ (s + 2Nδ (y + (N 1δ 2. (B.1 Rearragig, we have ĉ (s a + y a ( ĉ (s + y ( N (y a y (1 ya +y 2δ. By Corollary 1 ad the fact that y δ, y a δ, both terms i the brackets are positive, hece right-had-side expressio is maximized whe y + ad y a = δ, hece ĉ (s a + y a ĉ (s + y δ 2N. (B.11 However, by Lemma 8, we have ĉ (s c (s + 2N ( s c (s + δ 2N ; ad ĉ (s a c (s a. Substitutig the above relatios ito (B.11 ad rearragig, we have the desired result. Appedix B.5. Proof of Theorem 12 First, we compare the cost by operator betwee the price-takig ad price aticipatig cases, by defiitio (15 ad rearragig, we have cost o (p a, y a cost o (p t, y t = (p a p t ( δ y t + ( p a (y a y t. By the fact that p a = Nδ (ya + (N 1δ (show i Theorem 9 ad the fact that y a δ, we have ( N 1 p a. (B.12 N 29

/ (215 1 3 3 By the upper boud of p a i (B.12 ad the upper bouds of p t, y t i Theorem 9, we have cost o (p a, y a cost o (p t, y t. (B.13 Similarly, usig the lower boud of p a i (B.12 ad the upper bouds of p a, y a i Theorem 9, we have ( ( cost o (p a, y a cost o (p t, y t (δ + 1 ( δ = δ 2N N 2 N. Secod, we compare the cost by the operator to the social optimal. Sice the eergy reductio goal δ is the same, by Propositio 4 ad Corollary 1, we have p t p ad p a p. Hece we have cost o (p t, y t cost o (p a, y a cost o (p, y. Furthermore, cost o (p, y cost o (p t, y t = δ (p t (δ y t + y t ( δ y =( p t (δ y t t = (δ y t δ Nδ N. (B.14 Lastly by (B.13 ad (B.14, we have cost(p, y cost(p a, y a cost(p, y cost(p t, y t δ N. Appedix B.6. Proof of Theorem 13 Give ay ε >, let ε = 1 2ε. Cosider the followig set of cost fuctio: 2N s 1, if s 1 < ε ; c 1 (s 1 = (1 3ε 2Nδ s 1 + C 1, ε s 1 δ ε ; 2s 1 + C 2, s 1 > δ ε where C 1, C 2 are costats that make c 1 cotiuous 6, the c 1 is piece-wise liear ad covex. Also, m 1, c m (s m = 2s m. It is easy to see that s 1 = δ ε ad y = ε is the optimal allocatio. Let s a 1 = ε, y a = δ ε, ad m 1, s a m =, we claim that (s a, y a is the uique optimal solutio to (2a-(2c. To see this, let ρ = (1 ε/(nδ, the, Nδ (ya + (N 1δ = ρ; ĉ 1 (s a 1 s 1 ρ; s a = δ y a ; + ĉ 1 (s a 1 s 1 ρ; (B.15a + ĉ m ( s m ρ, m 1. (B.15b where the secod iequality is because if we let H be the term uder square root for + ĉ (s, the ( + c (s H = ( 2N 2 s N δ + ( 2 (δ + s (δ s N 2 δ 2 + c (s ( 2N s N δ. Note that + ĉ (s = 1 2 ( + c (s + 2N + 1 2 H. Hece we have + ĉ 1 (s a 1 s 1 + c 1 (s a 1 s 1 + s 1 2Nδ = ρ. These coditios correspod to (B.5a-(B.5d, so we coclude that (s a, y a is the uique optimal solutio to (2a-(2c. Hece y a y = δ 2ε = δ ε. 6 C 1 = ε ( (2N 1δ 3ε 2Nδ, ad C 2 = Nδ (Nδ2 + δε 3ε 3