Data Center Demand Response: Avoiding the Coincident Peak via Workload Shifting and Local Generation

Transcription

1 (213) 1 28 Data Cente Demand Response: Avoiding the Coincident Peak via Wokload Shifting and Local Geneation Zhenhua Liu 1, Adam Wieman 1, Yuan Chen 2, Benjamin Razon 1, Niangjun Chen 1 1 Califonia Institute of Technology 2 HP Labs 1 {zhenhua,adamw,ben,ncchen}@caltech.edu 2 [email protected] Abstact Demand esponse is a cucial aspect of the futue smat gid. It has the potential to povide significant peak demand eduction and to ease the incopoation of enewable enegy into the gid. Data centes paticipation in demand esponse is becoming inceasingly impotant given thei high and inceasing enegy consumption and thei flexibility in demand management compaed to conventional industial facilities. In this pape, we study two demand esponse schemes to educe a data cente s peak loads and enegy expenditue: wokload shifting and the use of local powe geneation. We conduct a detailed chaacteization study of coincident peak data ove two decades fom Fot Collins Utilities, Coloado and then develop two algoithms fo data centes by combining wokload scheduling and local powe geneation to avoid the coincident peak and educe the enegy expenditue. The fist algoithm optimizes the expected cost and the second one povides a good wost-case guaantee fo any coincident peak patten, wokload demand and enewable geneation pediction eo distibutions. We evaluate these algoithms via numeical simulations based on eal wold taces fom poduction systems. The esults show that using wokload shifting in combination with local geneation can povide significant cost savings (up to 4% unde the Fot Collins Utilities chaging scheme) compaed to eithe alone. c 213 Published by Elsevie Ltd. 1. Intoduction Demand esponse (DR) pogams seek to povide incentives to induce dynamic demand management of customes electicity load in esponse to powe supply conditions, fo example, educing thei powe consumption in esponse to a peak load waning signal o equest fom the utility. The National Institute of Standads and Technology (NIST) and the Depatment of Enegy (DoE) have both identified demand esponse as one of the pioity aeas fo the futue smat gid [1, 2]. In paticula, the National Assessment of Demand Response Potential epot has identified that demand esponse has the potential to educe up to 2% of the total peak electicity demand acoss the county [3]. Futhe, demand esponse has the potential to significantly ease the adoption of enewable enegy into the gid. Data centes epesent a paticulaly pomising industy fo the adoption of demand esponse pogams. Fist, data cente enegy consumption is lage and inceasing apidly. In 211, data centes consumed appoximately 1.5% of all electicity woldwide, which was about 56% highe than the peceding five yeas [4, 5, 6, 7]. Second, data centes ae highly automated and monitoed, and so thee is the potential fo a high-degee of esponsiveness. Fo example, 1

2 / (213) today s data centes ae well instumented with a ich set of sensos and actuatos. The powe load and state of IT equipment (e.g., seve, stoage and netwoking devices) and cooling facility (e.g., CRAC units) can be continuously monitoed and panoamically adjusted. Thid, many wokloads in data centes ae delay toleant, and can be scheduled to finish anytime befoe thei deadlines. This enables significant flexibility fo managing powe demand. Finally, local powe geneation, e.g., both taditional backup geneatos such as diesel o natual gas poweed geneatos and newe enewable powe installations such as sola PV aays, can help educe the need fom the gid by supplying the demand at citical times. In paticula, local powe geneation combined with wokload management has a significant potential to shed the peak load and educe enegy costs. Despite wide ecognition of the potential fo demand esponse in data centes, the cuent eality is that industy data centes seemingly pefom little, if any, demand esponse [4, 5]. One of the most common demand esponse pogams available is Coincident Peak Picing (CPP), which is equied fo medium and lage industial consumes, including data centes, in many egions. These pogams wok by chaging a vey high pice fo usage duing the coincident peak, often ove 2 times highe than the base ate, whee the coincident peak is the when the most electicity is equested by the utility fom its wholesale electic supplie. It is common fo the coincident peak chages to account fo 23% o moe of a custome s electic bill accoding to Fot Collins Utilities [8]. Hence, fom the pespective of a consume, it is citical to contol and educe usage duing the peak. Although it is impossible to accuately pedict exactly when the peak will occu, many utilities identify potential peak s and send waning signals to customes, which helps customes manage thei loads and make decisions about thei enegy usage. Fo example, Fot Collins Utilities sends coincident peak wanings fo 3-22 s each month with aveage 14.5 in summe months and 1 in winte ones. Depending on the utility, wanings may come between 5 minutes and 24 s ahead of time. Coincident peak picing is not a new phenomenon. In fact, it has been used fo lage industial consumes fo decades. Howeve, it is ae fo lage industial consumes to have the esponsiveness that data centes can povide. Unfotunately, data centes today eithe do not espond to coincident peak wanings o simply espond by tuning on thei backup powe geneatos [9]. Using backup powe geneation seems appealing since it can be automated easily, it does not impact opeations, and it povides demand esponse fo the utility company. Howeve, the taditional backup geneatos at data centes can be vey dity in some cases even not meeting Envionmental Potection Agency (EPA) emissions standads [4]. So, fom an envionmental pespective this fom of esponse is fa fom ideal. Futhe, unning a backup geneato can be expensive. Altenatively, poviding demand esponse via shifting wokload can be moe cost effective. One of the challenges with wokload shifting is that we need to ensue that the Sevice Level Ageements (SLAs), e.g., completion deadlines, emain satisfied even with uncetainties in coincident peak and waning pattens, wokload demand, and enewable geneation Summay of contibutions Ou main contibutions ae the following. Fist, we pesent a detailed chaacteization study of coincident peak picing and povide insight about its popeties. Section 2 discusses the chaacteization of 26 yeas coincident peak picing data fom Fot Collins Utilities in Coloado. The data highlights a numbe of impotant obsevations about coincident peak picing (CPP). Fo example, the data set shows that both the coincident peak occuence and waning occuence have stong diunal pattens that diffe consideably duing diffeent days of the week and seasons. Futhe, 2

3 / (213) the data highlights that coincident peak wanings ae highly eliable only twice did the coincident peak not occu duing a waning. Finally, the data on coincident peak wanings highlights that the fequency of wanings tends to decease though the month, and that thee tend to be less than seven days pe month on which wanings occu. Second, we develop two algoithms fo avoiding the coincident peak and educing the enegy expenditue using wokload shifting and local powe geneation. Though thee has been consideable ecent wok studying wokload planning in data centes, e.g., [1, 11, 12, 13, 14, 15, 16, 17, 18, 19, 2], the uncetainty of the occuence of the coincident peak pesents significant new algoithmic challenges beyond what has been addessed peviously. In paticula, small eos in the pediction of wokload o enewable geneation have only a small effect on the esulting costs of wokload planning; howeve, eos in the pediction of the coincident peak have a theshold effect if you ae wong you pay a lage additional cost. This lack of continuity is well known to make the development of online algoithms consideably moe challenging. Given the challenges associated with the combination of uncetainty about the coincident peak and waning s, wokload demand, and enewable geneation, we conside two design goals when developing algoithms: good pefomance in the aveage case and in the wost case. We develop an algoithm fo each goal. Fo the aveage case, we pesent a stochastic optimization based algoithm given the estimates of the likelihood of a coincident peak o waning duing each of the day, and pedictions of wokload demand and enewable geneation. The algoithm povides povable obustness guaantees in tems of the vaiance of the pediction eos. Fo the wost case scenaio, we popose a obust optimization based algoithm that is computationally feasible and simple, and guaantees that the cost is within a small constant of the optimal cost of an offline algoithm fo any coincident peak and waning pattens, wokload demand, and enewable geneation pediction eo distibutions with bounded vaiance. Note that a distinguishing featue of ou analysis is that we povide povable bounds on the impact of pediction eos. In pio wok on data cente capacity povisioning pediction eos have almost always been studied via simulation, if at all. The thid main contibution of ou wok is a detailed study and compaison of the potential cost savings of algoithms via numeical simulations based on eal wold taces fom poduction systems. The expeimental esults in Section 5 highlight a numbe of impotant obsevations. Most impotantly, the esults highlight that ou poposed algoithms povide significant cost and emission eductions compaed to industy pactice and povide close to the minimal costs unde eal wokloads. Futhe, ou expeimental esults highlight that both local geneation and wokload shifting ae impotant fo ensuing minimal enegy costs and emissions. Specifically, combining wokload shifting with local geneation can povide 35-4% eductions of enegy costs, and 1-15% eductions of emissions. We also illustate that ou algoithms ae obust to pediction eos Related wok While the design of wokload planning algoithms fo data centes has eceived consideable attention in ecent yeas, e.g., [1, 11, 12, 13, 14, 15, 16, 17, 18, 19, 2] and the efeences theein; demand esponse fo data centes is a elatively new topic. Some of the initial wok in the aea comes fom Ugaonka et al. [21], which poposes an appoach fo poviding demand esponse by using enegy stoage to shift peak demand away fom high peak peiods. This technique complements othe demand esponse schemes such as wokload shifting. Conceptually, using local stoage is simila to the use of local powe geneation studied in the cuent pape. In this pape, we conside both the wokload shifting and local powe geneation. The integation of enegy stoage to ou famewok is a topic of ou 3

4 / (213) futue wok. Anothe ecent appoach fo data cente demand esponse is Iwin et al. [22], which studies a distibuted stoage solution fo demand esponse whee compatible stoage systems ae used to optimize I/O thoughput, data availability, and enegy-efficiency as powe vaies. Pehaps the most in depth study of data cente demand esponse to this point is the ecent epot eleased by Laweence Bekeley National Laboatoies (LBNL) [5]. This epot summaizes a field study of fou data centes and evaluates the potential of diffeent appoaches fo poviding demand esponse. Such appoaches include adjusting the tempeatue set point, shutting down o idling IT equipment and stoage, load migation, and adjusting building popeties such as lighting and ventilation. The esults show that data centes can povide 1-12% enegy usage savings at the building level with minimal o no impact to data cente opeations. This epot highlights the potential of demand esponse and shows that it is feasible fo a data cente to espond to signals fom utilities, but stops shot of poviding algoithms to optimize cost in demand esponse pogams, which is the focus of the cuent pape. 2. Coincident peak picing Most typically, the demand esponse pogams available fo data centes today ae some fom of coincident peak picing. In this section, we give an oveview of coincident peak picing pogams and then do a detailed chaacteization of the coincident peak picing pogam un by Fot Collins Utilities in Coloado, whee HP has a data cente chaged by this company An oveview of coincident peak picing In a coincident peak picing pogam, a custome s monthly electicity bill is made up of fou components: (i) a fixed connection/mete chage, (ii) a usage chage, (iii) a peak demand chage fo usage duing the custome s peak, and (iv) a coincident peak demand chage fo usage duing the coincident peak (CP), which is the duing which the utility company s usage is the highest. Each of these is descibed in detail below. Connection/Mete chage. The connection and mete chages ae fixed chages that cove the maintenance and constuction of electic lines as well as sevices like mete eading and billing. Fo medium and lage industial consumes such as data centes, these chages make up a vey small faction of the total powe costs. Usage chage. The usage chage in CPP pogams woks similaly to the way it does fo esidential consumes. The utility specifies the electicity pice $p(t)/h fo each. This pice is typically fixed thoughout each season, but can also be time-vaying. Usually p(t) is on the ode of seveal cents pe h. Peak demand chage. CPP pogams also include a peak demand chage in ode to incentivize customes to consume powe in a unifom manne, which educes costs fo the utility due to smalle capacity povisioning. The peak demand chage is typically computed by detemining the of the month duing which the custome s electicity use is highest. This usage is then chaged at a ate of $p p /h, which is much highe than p(t). It is typically on the ode of seveal dollas pe h. Coincident peak chage. The defining featue of CPP pogams is the coincident peak chage. This chage is simila to the peak chage, but focuses on the peak fo the utility as a whole fom its wholesale electicity povide (the coincident peak) athe than the peak fo an individual consume. In paticula, at the end of each month the peak usage fo the utility, t cp, is detemined and then all consumes ae chaged $p cp /h fo thei usage 4

5 / (213) CP occuence winte months summe months CP occuence waning occuence winte months summe months waning occuence (a) Time of the day (b) Days of the week (c) Time of the day (d) Days of the week Figue 1. Occuence of coincident peak and wanings. (a) Empiical fequency of CP occuences on the time of day, (b) Empiical fequency of CP occuences ove the week, (c) Empiical fequency of waning occuences on the time of day, and (d) Empiical fequency of waning occuences ove the week. duing this. This ate is again at the scale of seveal dollas pe h, and can be significantly lage than the peak demand chaging ate p p. Note that customes cannot know when the coincident peak will occu since it depends on the behavio of all of the utility s customes. As a esult, to aid customes the utility sends wanings that paticula s may be the coincident peak. Depending on the utility, these wanings can be anywhee fom 5 minutes to 24 s ahead of time, though they ae most often in the 5-1 minute time-fame. These wanings can last multiple s and can occu anywhee fom two to tens of times duing a month. In pactice, these wanings ae extemely eliable the coincident peak almost neve occus outside of a waning. This is impotant since wanings ae the only signal the utility has fo achieving esponsiveness fom customes A case study: Fot Collins Utilities Coincident Peak Picing (CPP) Pogam In ode to povide a moe detailed undestanding of CPP pogams, we have obtained data fom the Fot Collins Utilities on the CPP pogam they un fo medium and lage industial and commecial customes. The data we have obtained coves the opeation of the pogam fom Januay 1986 to June 212. It includes the date and of the coincident peak each month as well as the date,, and length of each waning peiod. In the following we focus ou study on thee aspects: the ates, the occuence of the coincident peak, and the occuence of the wanings. Rates. We begin by summaizing the pices fo each component of the CPP pogam. The ates fo 211 and 212 ae summaized in the Table 1. It is woth making a few obsevations. Fist, note that all the pices ae fixed and announced at the beginning of the yea, which eliminates any uncetainty about pices with espect to data cente planning. Futhe, the pices ae constant within each season; howeve the utility began to diffeentiate between summe months and winte months in 212. Second, because the coincident peak pice and the peak pice ae both so much highe than the usage pice, the costs associated with the coincident peak and the peak ae impotant components of the enegy costs of a data cente. In paticula, p p p is 194 and 148, and p cp p is 514 and 219, in 211 and winte 212, espectively. Hence, it is vey citical to educe both the data cente peak demand and the coincident peak demand in ode to lowe the total cost. A final obsevation is that the coincident peak pice is highe than the peak demand pice, 2.6 times and 1.4 times highe in 211 and winte 212, espectively. This means that the eduction of powe demand duing the coincident peak is moe impotant, futhe highlighting the impotance of avoiding coincident peaks. Coincident peak. Undestanding popeties of the coincident peaks is paticulaly impotant when consideing data cente demand esponse. Figue 1 summaizes the coincident peak data we have obtained fom Fot Collins Utilities fom Januay 1986 to June 212. Figue 1(a) depicts the numbe of coincident peak occuences duing each 5

6 / (213) Chaging ates Fixed $/month Additional mete $/month CP summe $/h CP winte $/h Peak $/h Enegy summe $/h Enegy winte $/h Table 1. Summay of the chaging ates of Fot Collins Utilities duing 211 and 212 [8]. of the day. Fom the figue, we can see that the coincident peak has a stong diunal patten: the coincident peak nealy always happens between 2pm and 1pm. Additionally, the figue highlights that the coincident peak has diffeent seasonal pattens in winte and summe: the coincident peak occus late in the day duing winte months than duing summe months. Futhe, the time ange that most coincident peaks occu is naowe duing winte months. The numbe of coincident peak occuences on a weekly basis is shown in Figue 1(b). The data shows that the coincident peak has a stong weekly patten: the coincident peak almost neve happens on the weekend, and the likelihood of occuence deceases thoughout the weekdays. Wanings. To facilitate customes managing thei demand, Fot Collins Utilities identify potential peak s and send waning signals to customes. These wanings ae the key tool though which utilities achieve esponsiveness fom customes, i.e., demand esponse. On aveage, wanings fom Fot Collins Utilities cove 12 s fo each month. Figues 1(c), 1(d), and 2 summaize the data on wanings announced by Fot Collins Utilities between Januay 21 and June 212. We limit ou discussion to this peiod because the algoithm fo announcing wanings was consistent duing this inteval. Duing this peiod, wanings wee announced 5-1 minutes befoe the waning peiod began. Note that wanings ae only useful if they do in fact align with the coincident peak. Within ou data set, all but two coincident peak fell duing a waning peiod. Futhe, upon discussion with the manage of the CPP pogam, these two mistakes ae attibuted to human eo athe than an unpedicted coincident peak. Figue 1(c) shows the numbe of wanings on the time of the day. Given that the wanings ae well coelated with the coincident peak shown in Figue 1(a), it is impotant to undestand thei fequency and timing. Unsupisingly, the announcement of wanings has stong diunal patten simila to that of the coincident peak: most wanings happen between 2pm and 1pm. The seasonal patten is also simila to that of the coincident peak: winte months have wanings late in the day than summe months, and the time ange in which most wanings occu is naowe duing winte months. Additionally, summe months have significantly moe wanings than winte month do (14.5 wanings pe month in summe compaed to 1 in winte). The numbe of wanings ove the week is shown in Figue 1(d). Simila to that of the coincident peak shown in Figue 1(b), the wanings have a stong weekly patten: few wanings happen duing the weekends, and the numbe of wanings deceases thoughout the weekdays. Some othe inteesting phenomena ae shown in Figue 2. In paticula, the fequency of wanings deceases duing the month, the length of consecutive wanings tends to be 2-4 s. the numbe of wanings in a month vaies fom 3 to 22, and the numbe of days with wanings duing a month tends to be less than seven. 6

7 / (213) waning occuence daily occuence 7 day aveage day of the month (a) Fequency of wanings duing a month occuence lasting s numbe of wanings in a month (b) Length of consecutive wanings (c) Numbe of wanings pe month occuence occuence days with wanings in a month (d) Numbe of days with wanings pe month Figue 2. Oveview of waning occuences showing (a) daily fequency, (b) length, and (c)-(d) monthly fequency. (h) (a) in June in Fot Collins, Coloado nomalized CPU usage (b) Inteactive wokload fom a photo shaing web sevice nomalized CPU demand (c) Facebook Hadoop wokload PUE (d) Google data centes PUE Figue 3. One week taces fo (a), (b) non-flexible wokload demand, (c) flexible wokload demand, and (d) cooling efficiency. 3. Modeling The coe of ou appoach fo developing data cente demand esponse algoithms is an enegy expenditue model fo a data cente paticipating in a CPP pogam. We intoduce ou model fo data cente enegy costs in this section. It builds on the model used by Liu et al. in [23], which is in tun elated to the models used in [24, 25, 26, 27, 12, 28, 29, 3]. The key change we make to [23] is to incopoate chages fom CPP, wokload demand and enewable geneation pediction eos into the objective function of the optimization. This is a simple modeling change, but one that ceates significant algoithmic challenges (see Section 4 fo moe details). Ou cost model is made up of models chaacteizing the powe supply and powe demands of a data cente. On the powe supply side, we model a powe mico-gid consisting of the public gid, local backup powe geneation, and/o a enewable enegy supply. On the powe demand side, we conside both non-flexible inteactive wokloads and flexible batch-style wokloads in the data centes. Futhe, we conside a cooling model that allows fo a mixtue of diffeent cooling methods, e.g., fee outside ai cooling and taditional mechanical chille cooling. Thoughout, we conside a discete-time model whose time slot matches the time scale at which the capacity povisioning and scheduling decisions can be updated. Thee is a (possibly long) planning hoizon that we ae inteested in, {1, 2,..., T}. In pactice, T could be a day and a time slot length could be 1. 7

8 / (213) Powe Supply Model The electicity cost fom the gid includes thee non-constant components as descibed in Section 2, denote by p(t) the usage pice, p p the (custome) peak pice, and p cp the coincident peak pice. We assume all pices ae positive without loss of geneality. Most data centes ae equipped with local powe geneatos as backup, e.g., diesel o natual gas poweed geneatos. These geneatos ae pimaily intended to povide powe in the event of a powe failue; howeve they can be valuable fo data cente demand esponse, e.g., shedding peak load by poweing the data cente with local geneation. Typically, the costs of opeating these geneatos ae dominated by the cost of fuel, e.g., diesel o natual gas. Note that the effective output of such geneatos can often be adjusted. In many cases the backup geneation is povided by multiple geneatos which can be opeated independently [31], and in othe cases the geneatos themselves can be adjusted continuously, e.g., in the case of a GE gas engine [32]. To model such local geneatos, we assume that the geneato has the capacity to powe the whole data cente, which is quite common in industy [31], i.e., the total capacity of local geneatos C g = C, whee C is the total data cente powe capacity. We denote the cost in dolla of geneating 1h powe using backup geneato by p g. Finally, we denote the geneation povided by the local geneato at time t by g(t). In addition to local backup geneatos, data centes today inceasingly have some fom of local enewable enegy available such as PV [33]. The effective output of this type of geneation is not contollable and is often closely tied to extenal conditions (e.g., wind speed and sola iadiance). Figue 3(a) shows the powe geneated fom a 1 PV installation in June in Fot Collins, Coloado. The fluctuation and vaiability pesent a significant challenge fo data cente management. In this pape, we conside both data centes with and without local enewable geneation. To model this, we use (t) to denote the actual enewable enegy available to the data cente at time t and use ˆ(t) fo the pedicted geneation. We denote (t) = (1 + ˆɛ )ˆ(t), whee ˆɛ is the pediction eo. We assume unbiased pediction E [ˆɛ ] = and denote the vaiance V [ˆɛ ] by σ 2, which can be obtained fom histoic data. These ae standad assumptions in statistics. Let ˆξ denote the distibution of ˆɛ. In the model, we ignoe all fixed costs associated with local geneation, e,g., capital expenditue and enewable opeational and maintenance cost Powe Demand Model The powe demand model is deived fom models of the wokload and the cooling demands of the data cente. Wokload model. Most data centes suppot a ange of IT wokloads, including both non-flexible inteactive applications that un 24x7 (such as Intenet sevices, online gaming) and delay toleant, flexible batch-style applications (e.g., scientific applications, financial analysis, and image pocessing). Flexible wokloads can be scheduled to un anytime as long as the jobs finish befoe thei deadlines. These deadlines ae much moe flexible (seveal s to multiple days) than that of inteactive wokload. The pevalence of flexible wokloads povides oppotunities fo poviding demand esponse via wokload shifting/shaping. We assume that thee ae I inteactive wokloads. Fo inteactive wokload i, the aival ate at time t is λ i (t). Then based on the sevice ate and the taget pefomance metics (e.g., aveage delay, o 95th pecentile delay) specified in SLAs, we can obtain the IT capacity equied to allocate to each inteactive wokload i at time t, denoted by a i (t). Hee a i (t) can be deived fom eithe analytic pefomance models, e.g., [34], o system measuements as a function of 8

9 / (213) λ i (t) because pefomance metics geneally impove as the capacity allocated to the wokload inceases, hence thee is a shap theshold. Inteactive wokloads ae typically chaacteized by highly vaiable diunal pattens. Figue 3(b) shows an example fom a 7-day nomalized CPU usage tace fo a popula photo shaing and stoage web sevice which has moe than 85 million egisteed uses in 22 counties. Flexible batch jobs ae moe difficult to chaacteize since they typically coespond to intenal wokloads and ae thus hade to attain accuate taces fo. Figue 3(c) shows an example fom a 7-day nomalized CPU demand tace geneated using aival and job infomation about Facebook Hadoop wokload [35, 36]. We assume thee ae J classes of batch jobs. Class j jobs have total demand B j, maximum paallelization MP j, stating time S j and deadline E j. Let b j (t) denote the amount of capacity allocated to class j jobs at time t. We have b j (t) MP j, t and t [S j,e j ] b j (t) = B j. Given the above models fo inteactive and batch jobs, the total IT demand at time t is given by d IT (t) = I a i (t) + i=1 J b j (t). (1) j=1 The total IT capacity in units of h is D, so d IT (t) D, t. Since ou focus is on enegy costs, we intepet d IT (t), a i (t), and b j (t) as being the enegy necessay to seve the demand, and thus in units of h. Cooling model. In addition to the powe demands of the wokload itself, the cooling facilities of data centes can contibute a significant potion of the enegy costs. Cooling powe demand depends fundamentally on the IT powe demand, and so is deived fom IT powe demand though cooling models, e.g., [37, 38]. Hee, we assume the cooling powe associated with IT demand d IT, c(d IT ), is a convex function of d IT. One simple but widely used model is Powe Usage Effectiveness (PUE) as follows: c(d(t)) = (PUE(t) 1) d(t). Note that PUE(t) is the PUE at time t, and vaies ove time depending on envionmental conditions, e.g., the outside ai tempeatue. Figue 3(d) shows one week fom a tace of the aveage PUE of Google data centes. Moe complex models of the cooling cost have also been deived in the liteatue, e.g., [23, 37, 38]. Total powe demand. The total powe demand is denoted by d(t) = d IT (t) + c(d IT (t)). (2) We use ˆd(t)to denote the pedicted demand. We denote d(t) = (1 + ˆɛ d ) ˆd(t), whee ˆɛ d is used to stand fo the pediction eo. Again, we assume E [ˆɛ d ] = and denote V [ˆɛ d ] by σ 2 d, which can be obtained fom histoic data. Let ˆξ d denote the distibution of ˆɛ d Total data cente costs Using the above models fo the powe supply and powe demand at a data cente, we can now model the opeational enegy cost of a data cente, which ou data cente demand esponse algoithms seek to minimize. In paticula, they take the powe supply cost paametes, including the gid powe picing and fuel cost, as well as the wokload demand and SLAs infomation, as input and seek to povide an nea-optimal wokload schedule and a local powe geneation plan given uncetainties about wokload demand and enewable geneation. This planning poblem can be fomulated 9

10 as the following constained convex optimization poblem given t cp. / (213) min b,g T T p(t)e(t) + p p max t e(t) + p cp e(t cp ) + p g g(t) t=1 s.t. e(t) (d(t) (t) g(t)) + C, t b j (t) = B j, j t [S j,e j ] t=1 (3a) (3b) (3c) b j (t) MP j, j, t (3d) d IT (t) D, t (3e) g(t) C g. t (3f) In the above optimization, the objective (3a) captues the opeational enegy cost of a data cente, including the electicity chage by the utility and the fuel cost of using local powe geneation. The fist thee tems descibe gid powe usage chage, peak demand chage, and coincident peak chage, espectively. The fuel cost of the local powe geneato is specified in the last tem. Futhe, the fist constaint (3b) defines e(t) to be the powe consumption fom the gid at time t, which depends on the IT demand d IT (t) defined in (1) and theefoe futhe depends on batch job scheduling b j (t), the cooling demand, the availability of enewable enegy, and the use of the local backup geneato. Constaint (3c) equies all jobs to be completed. Constaint (3d) limits the paallelism of the batch jobs. Constaint (3e) limits the demand seved in each time slot by the IT capacity of the data cente. The final constaint (3f) limits the capacity of the local geneation. 4. Algoithms We now pesent ou algoithms fo wokload and geneation planning in data centes that paticipate in CPP pogams. In paticula, ou stating point is the data cente optimization poblem intoduced in (3a) in the pevious section, and ou goal is to design algoithms fo optimally combining local geneation and wokload shifting in ode to minimize the opeational enegy cost. Moe specifically, the algoithmic poblem we appoach is as follows. We assume that the planning hoizon being consideed is one day and that the wokload, pices, cooling efficiency, and enewable availability can be pedicted with easonable accuacy in this hoizon, but that the planne does not know when the coincident peak and the coesponding wanings will occu. The algoithmic goal is thus to geneate a plan that minimizes cost despite this unknown infomation and pediction eos. Since the costs associated with the coincident peak can be a lage faction of the data cente electicity bill, this lack of infomation is a significant challenge fo planning. As we have aleady discussed, designing fo this uncetainty about the coincident peak is fundamentally diffeent than designing fo pediction eos on factos such as wokload demand o enewable geneation since inaccuacies in the pediction of the coincident peak and the coesponding wanings have a discontinuous theshold effect on the ealized cost. As a esult, even small pediction eos can esult in significantly inceased costs. Such effects ae well-known to make the design of online algoithms difficult. We conside two appoaches fo handling uncetainty about the coincident peak. The fist appoach we follow is to estimate when the coincident peak and the coesponding wanings will occu. Using the estimated likelihood 1

11 / (213) of a waning and/o coincident peak duing each, we can fomulate a convex optimization poblem to minimize the expected cost in the planning hoizon. The second appoach we follow is to fomulate a obust optimization that seeks to minimize the wost case cost given advesaial placement of wanings and the coincident peak. Note that thoughout this pape we estict ou attention to algoithms that do non-adaptive wokload shifting, i.e., algoithms that plan wokload shifting once at the beginning of the hoizon and then do not adjust the plan duing the hoizon in ode to make them moe easily adoptable. Howeve, we do allow local geneation to be tuned on adaptively when wanings ae eceived. This estiction is motivated by industy pactice today adaptive wokload shifting fo demand esponse is nealy non-existent, but data centes that actively paticipate in demand esponse pogams do adjust local geneation when wanings ae eceived. This estiction can easily be elaxed in what follows. 1 Howeve, the fact that ou analytic esults povide guaantees fo non-adaptive wokload planning means they ae stonge. Futhe, ou numeical expeiments studying the impovements fom adaptive wokload planning (omitted due to space estictions) highlight that the benefit of such adaptivity is not lage. This can be seen aleady in ou esults since the gap between the costs of ou non-adaptive algoithms and the cost of the offline optimal is small Expected cost optimization The stating point fo ou algoithms is the data cente optimization in (3a). In this section, ou goal is to plan wokload allocation and local geneation in ode to minimize the expected cost of the data cente given estimates fom histoical data about when the wanings and the coincident peak will occu. In paticula, ou appoach uses histoical data about when wanings will occu in ode to estimate the likelihood that time slot t will be a waning. We denote the estimate at time t by ŵ(t), and the full estimato by Ŵ. Since the data cente has local backup geneation, it can povide demand esponse even without using adaptive wokload shifting by tuning on the backup geneato when wanings ae eceived fom the utility. Today, those data centes that actively paticipate in demand esponse pogams typically use this appoach. The eason is that the cost of local geneation is typically significantly less than the coincident peak pice, and the numbe of wanings pe month is small enough to ensue that it is cost efficient to always tun on geneation wheneve wanings ae given. Of couse, thee ae dawbacks to using local geneation, since it is typically povided by diesel geneatos, which often have vey high emissions and costs [39, 4]. Thus, it is impotant to do wokload shifting in a manne that minimizes the use of local geneation, if possible. Befoe stating the algoithm fomally, let biefly discuss its stuctue. Using the estimates of waning occuences, wokload demand and enewable geneation, we fist solve a stochastic optimization (given in Algoithm 1 below) to obtain a wokload schedule b(t) and local geneato usage plan g 1 (t). Then, in untime, when the pediction eo is hamful, i.e., when min{e(t), ɛ d ˆd(t) ɛ ˆ(t)} >, (4) use the backup geneato to emove this effect, i.e., use geneation g ɛ (t) = max{, min{(e(t), ɛ d ˆd(t) ɛ ˆ(t)}}. 2 Additionally, if a waning occus, tun on the local geneato to educe the demand fom the gid to zeo, which we denote 1 If it is elaxed, eplanning afte wanings occu can be beneficial. Inteestingly, such eplanning could have simila and only slightly bette pefomance in the wost case. We omit the esults due to space constaints. 11

12 / (213) by g 2 (t) = e(t) g ɛ (t) when t is a waning peiod in ode to ensue that the coincident peak payment is zeo. (Recall that the coincident peak happens within a waning peiod with nea cetainty.) The total local geneation used is thus g(t) = g 1 (t) + g ɛ (t) + g 2 (t), t. Moe fomally, to wite the objective function used fo the fist step of planning we fist need to estimate g 2 (t), which can be done as follows: e(t) g ɛ (t) if t is a waning g 2 (t) = othewise This is feasible since in pactice the geneato has the capacity to powe the whole data cente [31], i.e., C g = C. We can now fomally define the planning algoithm fo expected cost minimization. Define ê(t) ( ˆd(t) ˆ(t) g 1 (t) ) + as the pedicted powe demand fom utility at time t, and σ max{σ d, σ } as a uppe bound of nomalized vaiance of the powe demand fom utility. Algoithm 1. Estimate ŵ(t) fo all t in the planning peiod. Then, solve the following convex optimization: min b,g 1 T ( ) T (1 ŵ(t))p(t) + ŵ(t)pg ê(t) + pp max t ê(t) + p g g 1 (t) t=1 s.t. ê(t) ( ˆd(t) ˆ(t) g 1 (t) ) + C, t (5b) b j (t) = B j, j (5c) t [S j,e j ] b j (t) MP j, j, t (5d) ˆd(t) D, t (5e) g 1 (t) C g. t (5f) t=1 (5a) Duing opeation, if the pediction eo has negative effect satisfying (4), use backup geneation to emove the eo. 2 If a waning is eceived, use the local geneato to educe the powe usage fom the gid to zeo until the waning peiod ends. Of couse thee ae many appoaches fo estimating ŵ(t) in pactice. In this pape, we do this using the histoical data summaized in Section 2. Since ou data is ich, and the occuence of the wanings is faily stationay, this estimato is accuate enough to achieve good pefomance, as we show in Section 5. Of couse, in pactice pedictions could likely be impoved by incopoating infomation such as weathe pedictions. It is clea that the pefomance of Algoithm 1 is highly dependent on the accuacy of pedictions, thus it is impotant to chaacteize this dependence. To accomplish this, denote the objective function in (3a) by f (b, g). Then [ the expected cost of Algoithm 1 is Eξ ˆd,Ŵ f (b s, g s ) ]. We compae this cost to the expected cost of oacle-like offline [ algoithm that knows wokload demand and enewable geneation pefectly, which we denote by Eξ ˆd,Ŵ f (b, g ) ]. To chaacteize the pefomance of the algoithm we use the competitive atio, which is defined as the atio of the 2 Note that, in pactice, one would not want to use geneation to coect fo all pediction eos, such a coection would only be done if the pediction eo was exteme. Howeve, fo analytic simplification, we assume that all pediction eos ae eased in this manne and evaluate the esulting cost. Ou simulations esults in Section 5 use the geneato only to coect fo exteme pediction eos. 12

13 / (213) cost of a given algoithm to the cost of the offline optimal algoithm. The following theoem (poven in Appendix A) shows that the cost of the online algoithm is not too much lage than optimal as long as pedictions ae accuate. Theoem 1. Given that the standad deviation of pediction eos fo the wokload and enewable geneation ae bounded by σ and the distibution of coincident peak wanings is known pecisely, Algoithm 1 has a competitive atio of 1 + Bσ, whee B = p gσ t ( ˆd s (t)+ˆ(t)) 2E εd [ f (e,g )] + p gσ t ( ˆd (t)+ˆ(t)) 2E εd [ f (e,g )]. That is, E [ ξˆ d,ŵ f (b s, g s ) ] [ /Eξ ˆd,Ŵ f (b, g ) ] 1 + Bσ. It is woth noting that it is ae fo the impact of pediction eo on a data cente planning algoithm to be quantified analytically, nealy all pio wok eithe does not study the impact of pediction eos, o studies thei impact via simulation only. Additionally, it is impotant to point out that Theoem 1 does not make any distibutional assumption on the pediction eos othe than bounded vaiance. The key obsevation povided by Theoem 1 is that the competitive atio is a linea function of pediction standad deviation, which implies when pediction eos decease to, this competitive atio also deceases to 1. Thus, the algoithm is faily obust to pediction eos. Ou tace-based simulations in Section 5 cooboate this conclusion Robust optimization While pefoming well fo expected cost is a natual goal, the algoithm we have discussed above depends on the accuacy of estimatos of the occuence of the coincident peak o waning peiods. In this section, we focus on poviding algoithms that maintain wost-case guaantees egadless of pediction accuacy, i.e., that minimize the wost case cost. To chaacteize the pefomance of the algoithm we again use the competitive atio. In ou setting, we conside the cost only duing one planning peiod. Thus, the diffeence in infomation between the offline algoithm and ou algoithm is knowledge of when the wanings will occu, exact wokload demand and enewable geneation. We do assume that the online algoithm has an uppe bound on the numbe of wanings that may occu. In ode to minimize the wost case cost, the natual appoach is to incease the penalty on the peak peiod. This follows because, if an advesay seeks to maximize the cost of an algoithm, it should place wanings on the peiods whee the algoithm uses the most enegy. This obsevation leads us to the following algoithm: Algoithm 2. Conside an uppe bound on the numbe of waning peiods W. Solve the following convex optimization T p(t)ê(t) + ( p p + W ( p g min t p(t) )) T max t ê(t) + p g g 1 (t) (6a) min b,g 1 t=1 s.t. ê(t) ( ˆd(t) ˆ(t) g 1 (t) ) + C, t (6b) b j (t) = B j, j (6c) t [S j,e j ] b j (t) MP j, j, t (6d) ˆd(t) D, t (6e) g 1 (t) C g. t (6f) t=1 Duing opeation, if the pediction eo has negative effect satisfying (4), use backup geneation to emove the eo. 2 If a waning is eceived, use the local geneato to educe the powe usage fom the gid to zeo until the waning peiod ends. 13

14 / (213) This algoithm epesents a seemingly easy change to the oiginal data cente optimization in (3a); howeve the subtle diffeences ae enough to ensue that it povide a vey stong wost case cost guaantee. In paticula, it povides the minimal competitive atio achievable. Theoem 2. Given that the standad deviation of pediction eos fo the wokload and enewable geneation ae bounded by σ, Algoithm 2 has a competitive atio of W ( p g min t p(t) ) W ( p g min t p(t) ) 1 + Bσ + Tmin t p(t)/pmr 1 + Bσ +, + p p p p W(p g min t p(t)) whee B = p gσ t ( ˆd w (t)+ˆ(t)) 2E εd [ f (e,g )] + p gσ t ( ˆd (t)+ˆ(t)) 2E εd [ f (e,g )]. Futhe, if W = W then thee is a lowe bound 1 + Tmin t p(t)/pmr +p p on the competitive atio achievable unde any online algoithm, even one with exact pedictions of wokloads and enewable geneation. The key contast between Theoem 2 and Theoem 1 is that Theoem 1 assumes that the distibution of coincident peak wanings is known pecisely, while Theoem 2 povides a bound even when the coincident peak wanings ae advesaial. As such, it is not supising that the competitive atio is lage in Theoem 2. Howeve, note that the competitive atio of Algoithm 1 in the context of Theoem 2 can be easily shown to be unbounded, and so one should not think of Theoem 1 as a stonge bound than Theoem 2. Inteestingly, the fom of Theoem 2 paallels Theoem 1, except with an additional tem in competitive atio. Thus, again the competitive atio gows linealy with the vaiance of the pediction eo. Additionally, note that when σ =, the competitive atio matches the lowe bound, which highlights that the additional tem in Theoem 2 is tight. Futhe, since the additional tem is defined in tems of the elative pices of local geneation and the peak, it is easy to undestand its impact in pactice. In pactice, p g is less than $.3/h [4] and the numbe of waning s is oughly between 3 and 22, with an aveage of 12 waning s pe month. So, this tem is typically less than 1, which highlights that the wost-case bound on Algoithm 2 nealy matches the bound on Algoithm 1 in the case whee the coincident peak waning distibution is known. Note that, if thee is no local geneato, then we can deive a simila esult to Theoem 2, whee W ( p g min t p(t) ) is eplaced by p cp. The compaison of these esults highlights the cost savings povided by using a local backup geneato. Since the data cente does not know the exact numbe of wanings fo a paticula month, whethe o not using local geneation is beneficial depends on the pedicted bound on the numbe of wanings pe month. If it is smalle than (25 in winte and 36 in summe fo 212 in the utility scheme shown in Table 1 with high p cp p g min t p(t) local geneation cost), it should use local geneation. This highlights that if a utility wishes to incentivize the data cente to use local geneation to elieve its pessue, then it should not send too many wanings Implementation consideations Ove the past decade thee has been significant effot to addess data cente enegy challenges via wokload management. Most of these effots focus on impoving the enegy efficiency and achieving enegy popotionality of data centes via wokload consolidation and dynamic capacity povisioning, e.g., [1, 11, 12, 13, 14, 15, 16, 17, 18, 19, 2]. Recently, such wok has begun to exploe topics such as shifting (tempoal) o migating (spatial) wokloads to bette use enewable enegy souces [41, 25, 28, 42, 43, 29, 44, 45]. 14

15 / (213) The algoithms pesented in this section ae both optimization-based appoaches fo tempoal wokload management and, as such, build on this liteatue. In paticula, optimization based appoaches have eceived significant attention in ecent yeas, and have been shown to tansition easily to lage scale implementations, e.g., [23, 1, 5]. In this pape, we evaluate the algoithms pesented above via both wost-case analysis and tace-based simulations. Howeve, fo completeness we comment biefly hee on the impotant consideations fo implementation of these designs. Fo moe details, the eade should consult [23, 1, 5]. Implementation consideations typically fall into two categoies: (i) obtaining accuate pedictions of wokload, enewable geneation, costs, etc.; (ii) implementing the plan geneated by the algoithm. Each of these challenges has been well studied by pio liteatue, and we only povide a bief desciption of each in the following. Pedictions. Ou algoithms exploit the statistical popeties of the coincident peak as well as pedictions of IT demand, cooling costs, enewable geneation, etc. Histoical data about the coincident peak is geneally available, fo lage industial consumes, fom the utilities opeating demand esponse pogams. In pactice, coincident peak pedictions can also be impoved using factos such as the weathe. Othe paametes needed by ou algoithm ae also faily pedictable. Fo example, in a data cente with a enewable supply such as a sola PV system, ou planning algoithms need the pedicted enewable geneation as input. This can be done in many ways, e.g., [46, 23, 44] and a ballpak appoximation is often sufficient fo planning puposes. Similaly, IT demands typically exhibit clea shot-tem and long-tem pattens. To pedict the esouce demand fo inteactive applications, we can fist pefom a peiodicity analysis of the histoical wokload taces to eveal the length of a patten o a sequence of pattens that appea peiodically via Fast Fouie Tansfom (FFT). An auto-egessive model can then be ceated and used to pedict the futue demand of inteactive wokloads. Fo example, this appoach was followed by [23]. The total esouce demand (e.g., CPU s) of batch jobs can be obtained fom uses o fom histoical data o though offline benchmaking [47]. Like supply pediction, a ballpak appoximation is typically good enough. Finally, thee ae many appoaches fo deiving cooling powe fom IT demand, fo example the models in [37, 23]. Execution. Given the pedictions fo the coincident peak, IT demand, cooling costs, enewable geneation, etc., ou poposed algoithms poceed by solving an optimization poblem to detemine a plan. Since the optimization poblems used ae convex and in simple fom, they can be solved efficiently. Given the esulting plan, the emaining wok is to implement the actual wokload placement and consolidation on physical seves. This can be done using packing algoithms, e.g., simple techniques such as Best Fit Deceasing (BFD) o moe advanced algoithms such as [48]. Finally, the execution of the plan can be done by a untime wokload geneato, which schedules flexible wokload and allocates CPU esouces accoding to the plan. This can be easily implemented in vitualized envionments. Fo example, a KVM o Xen hypeviso enables the ceation of vitual machines hosting batch jobs; the adjustment of the esouce allocation (e.g., CPU shaes o numbe of vitual CPUs) at each vitual machined; and the migation and consolidation of vitual machines. An example using this appoach is [23]. Futhe, [5] povides moe concete details of implementing the plan in the field. These suggest that the benefits fom ou algoithms ae attainable in eal system, and we will focus on numeical simulations in the following section. 15

16 / (213) Case study To this point we have intoduced two algoithms fo managing wokload shifting and local geneation in a data cente paticipating in a CPP pogam. We have also povided analytic guaantees on these algoithms. Howeve, to get a bette pictue of the cost savings such algoithms can povide in pactical settings, it is impotant to evaluate the algoithms using eal data, which is the goal of this section. We use numeical simulations fed by eal taces fo wokloads, cooling efficiency, electicity picing, coincident peak, etc., in ode to contast the enegy costs and emissions unde ou algoithms with those unde cuent pactice Expeimental setup Wokload and cost settings. To define the wokload fo the data cente we use taces fom eal data centes fo inteactive IT wokload, batch jobs, and cooling data. The inteactive wokload tace is fom a popula web sevice application with moe than 85 million egisteed uses in 22 counties (see Figue 3(b)). The tace contains aveage CPU utilization and memoy usage as ecoded evey 5 minutes. The peak-to-mean atio of the inteactive wokload is about 4. The batch job infomation comes fom a Facebook Hadoop tace (see Figue 3(c)). The total demand atio between the inteactive wokload and batch jobs is 1:1.6. This atio can vay widely acoss data centes, and ou pevious wok studied its impacts [23]. The deadlines fo the batch jobs ae set so that the lifespan is 4 times the time necessay to complete the jobs when they ae un at thei maximum paallelization. The maximum paallelization is set to the total IT capacity divided by the mean job submission ate. The time vaying cooling efficiency tace is deived fom Google data cente data and the PUE (see Figue 3(d)) is between 1.1 and 1.5. The pediction eo of wokload and cooling powe demand has a standad deviation of 1% fom ou simple pediction algoithm. The total IT capacity is set to 35 seves (7). Seve idle powe is 1W and peak powe is 2W. The enegy elated costs ae detemined fom the Fot Collins Utilities data descibed in Section 2. The pices ae chosen to be the 211 ates in Table 1. The local powe geneation of the data cente is set as follows. In diffeent settings the data cente may have both a local diesel geneato and a local PV installation 3. When a diesel geneato is pesent, we assume it has the capacity to powe the full data cente, which is set to be 1. The cost of geneation is set at $.3/h [4] fo consevative estimates. The emissions ae set to be 3.288kg CO 2 equivalent pe h [39]. The emission of gid powe is set to be.586kg CO 2 equivalent pe h [4]. The PV capacity is set to be 7 and the pediction eo of has standad deviation 15% fom ou pediction algoithm. Compaison baselines. In ou expeiments, ou goal is to evaluate the pefomance of the algoithms pesented in Section 4. We conside a planning peiod that is 24-s stating at midnight. The planne detemines wokload shifting and local geneation usage at an ly level, i.e., the amount of capacity allocated to each batch job and the amount of powe geneated by the local diesel geneato at each time slot. The length of each time slot is one. In this context, we compae the enegy costs and emissions of the algoithms pesented in Section 4 with two baselines, which ae meant to model industy standad pactice today. In ou study, Algoithm 1 is temed Pediction (Ped), which utilizes pedictions about the coincident peak wanings to minimize the expected cost. Similaly, Algoithm 2 optimizes the wost-case cost, and ae temed Robust. The baseline algoithms ae Night, Best 3 we have moe esults about othe combinations, but omit due to space constaint. 16

17 / (213) powe consumption 15 1 idle powe non flexible wokload flexible wokload cooling powe 15 1 powe consumption (a) Pediction: one week plan (b) Pediction: one day plan (c) Robust: one week plan 15 1 idle powe non flexible wokload flexible wokload cooling powe 15 1 powe consumption 15 1 idle powe non flexible wokload flexible wokload cooling powe (d) Robust: one day plan (e) Night: one week plan (f) Night: one day plan 15 1 powe consumption 15 1 idle powe non flexible wokload flexible wokload cooling powe 15 1 powe consumption (g) Best Effot (BE): one week plan (h) Best Effot (BE): one day plan (i) Optimal: one week plan idle powe non flexible wokload flexible wokload cooling powe (j) Optimal: one day plan annual expenditue ($) x 1 4 usage chaging geneation cost peak chaging CP chaging Ped. Robust Night BE Offline (k) Enegy costs annual CO2e emission (kg) x 16 gid geneato Ped. Robust Night BE Offline (l) Emissions Figue 4. Compaison of enegy costs and emissions fo a data cente with a local PV installation and a local diesel geneato. (a)-(j) show the plans computed by ou algoithms and the baselines. Effot (BE), and Optimal. Night and Best Effot ae meant to mimic typical industy heuistics, while Optimal is the offline optimal plan given knowledge of when the coincident peak will occu, exact wokload demand and enewable geneation. Best Effot finishes jobs in a fist-come-fist-seve manne as fast as possible. Night ties to un jobs duing night if possible and othewise un these jobs with a constant ate to finish them befoe thei deadlines Expeimental esults In ou expeimental esults, we seek to exploe the following issues: (i) How much cost and emission savings can ou algoithms achieve? How close to optimal ae ou algoithms on eal wokloads? (ii) What ae the elative benefits of local geneation and wokload shifting and a mixtue of both with espect to cost and emission eductions? (iii) 17

18 / (213) powe consumption 15 1 idle powe non flexible wokload flexible wokload cooling powe 15 1 powe consumption (a) Pediction: one week plan (b) Pediction: one day plan (c) Robust: one week plan idle powe non flexible wokload flexible wokload cooling powe (d) Robust: one day plan annual expenditue ($) x 1 4 usage chaging geneation cost peak chaging CP chaging Ped. Robust Night BE Offline (e) Enegy costs annual CO2e emission (kg) x 16 gid geneato Ped. Robust Night BE Offline (f) Emissions Figue 5. Compaison of enegy costs and emissions fo a data cente without local geneation o. (a)-(d) show the plans computed by ou algoithms. What is the impact of eos in pedictions of the coincident peak and the coesponding wanings? Cost savings and emissions eductions We stat with the key question fo the pape: how much cost and emission savings do ou algoithms povide? Figue 4 shows ou main expeimental esults compaing ou algoithms with baselines. The weekly powe pofile fo the fist week of June 211 is shown in the fist plot fo each algoithm, including powe consumption, and, and coincident peak wanings. The detailed daily powe beakdown fo the fist Monday in June 211 is shown in the second plot fo each algoithm, including idle powe, powe consumed by seving flexible wokload and non-flexible wokload, cooling powe, local geneation and wanings. Futhe, the last two plots includes a cost compaison and an emissions compaison fo ove one yea of opeation, including usage costs, peak costs, CP costs, local geneation costs, and emissions fom both the gid powe and local geneation used. As shown in the figue, ou algoithms povide 4% savings compaed to Night and Best Effot. Specifically, Pediction eshapes the flexible wokload to pevent using the time slots that ae likely to be waning peiods o the coincident peak as shown in Figues 4 (a) and (b), while Robust ties to make the gid powe usage as flat as possible as shown in Figues 4 (c) and (d). Both algoithms ty to fully utilize. In contast, Night and Best Effot do not conside the wanings, the coincident peak, o enewable geneation. Theefoe, they have significantly highe coincident peak chages and local geneation costs (Night has highe cost hee because it wastes even moe enewable geneation). Since the waning and coincident peak pedictions ae quite accuate, Pediction woks bette than Robust and simila to Optimal Local geneation vesus wokload shifting A second impotant goal of this pape is to undestand the elative benefits of local geneation planning and wokload shifting fo data centes paticipating in CPP pogams. Though ou algoithms have focused on the case 18

19 / (213) powe consumption 15 1 idle powe non flexible wokload flexible wokload cooling powe 15 1 powe consumption (a) Pediction: one week plan (b) Pediction: one day plan (c) Robust: one week plan idle powe non flexible wokload flexible wokload cooling powe (d) Robust: one day plan annual expenditue ($) x 1 4 usage chaging geneation cost peak chaging CP chaging Ped. Robust Night BE Offline (e) Enegy costs annual CO2e emission (kg) x 16 gid geneato Ped. Robust Night BE Offline (f) Emissions Figue 6. Compaison of enegy costs and emissions fo a data cente with a local PV installation, but without local geneation. (a)-(d) show the plans computed by ou algoithms. of local geneation, they can be easily adjusted to the case whee thee is no local geneato. In fact, simila analytic esults hold fo that case but wee omitted due to space constaints. Instead, we use simulation esults to exploe this case. In paticula, to evaluate the elative benefits of local geneation and wokload shifting in pactice, we can contast Figues 4 7. These simulation esults highlight that local geneation is cucial, in ode to povide esponses to waning signals fom the utility; but at the same time, even when local geneation is pesent, wokload shifting can povide significant cost savings, and can lead to a significant eduction in the amount of local geneation needed (and thus emissions). Moe specifically, compaed with the case of no local geneation, the use of local geneation can help educe the coincident peak costs; howeve one must be caeful when using local geneation to coect fo pediction eo since this added cost is not woth it unless the pediction eo is exteme. The aggegate effect is pehaps smalle than expected, and can be seen by compaing Figue 5(e) with 7(e) and Figue 6(e) with 4(k). As discussed in Section 4, the benefit of local geneation depends on the numbe of wanings, the local geneation cost, and the pediction eo. With fewe wanings and cheape local geneation, local geneatos can help educe costs moe. Howeve, this benefit comes with highe emissions (5-1% in the expeiments) since local geneatos ae usually not envionmentally fiendly. This can be seen fom the emission compaison between Figues 5(f) and 7(f), and Figues 6(f) and 4(l). Impotantly, enewable geneation can help educe both enegy costs and emissions significantly, especially when combined with wokload management. This can be seen fom cost and emission compaisons acoss Figues 5 and 6, and Figues 7 and Sensitivity to pediction eos The final issue that we seek to undestand using ou expeiments is the impact of pediction eos. We have aleady povided an analytic chaacteization of the impact of pediction eos on wokload and enewable geneation 19

20 / (213) powe consumption 15 1 idle powe non flexible wokload flexible wokload cooling powe 15 1 powe consumption (a) Pediction: one week plan (b) Pediction: one day plan (c) Robust: one week plan idle powe non flexible wokload flexible wokload cooling powe (d) Robust: one day plan annual expenditue ($) x 1 4 usage chaging geneation cost peak chaging CP chaging Ped. Robust Night BE Offline (e) Enegy costs annual CO2e emission (kg) x 16 gid geneato Ped. Robust Night BE Offline (f) Emissions Figue 7. Compaison of enegy costs and emissions fo a data cente with a local diesel geneato, but without local. (a)-(d) show the plans computed by ou algoithms. in Section 4 and so (due to limited space) we only biefly comment on numeical esults cooboating ou analysis hee Figue 8(a) shows the gowth of the competitive atio as a function of the standad deviation of the pediction eo. Recall that all esults in Figues 4 7 incopoate pediction eos as well. Moe impotantly, we focus this section on coincident peak and waning pediction eos. Figue 8 studies this issue. In this figue, the pedictions used by Pediction ae manipulated to ceate inaccuacies. In paticula, the pedictions calculated via the histoical data ae shifted ealie/late by up to 6 s, and the coesponding enegy costs and emissions ae shown. Of couse, the costs and emissions of Robust ae unaffected by the change in the pedictions; howeve the costs and emissions of Pediction change damatically. In paticula, Pediction becomes wose than Robust if the shift (and the eo) in the pediction distibution is lage than 3.5 s. 6. Concluding Remaks Ou goal in this pape is to povide algoithms to plan fo wokload shifting and local geneation usage at a data cente paticipating in a CPP demand esponse pogam with uncetainties in coincident peak and wanings, wokload demand and enewable geneation. To this end, we have obtained and chaacteized a 26-yea data set fom the CPP pogam un by Fot Collins Utilities, Coloado. This chaacteization povides impotant new insights about CPP pogams that can be useful fo data cente demand esponse algoithms. Using these insights, we have pesented two appoaches fo designing algoithms fo wokload management and local geneation planning at a data cente paticipating in a CPP pogam. In paticula, we have pesented a stochastic optimization based algoithm that seeks to minimize the expected enegy expenditue using pedictions about when the coincident peak and coesponding wanings will occu, wokload demand and enewable geneation, and anothe obust optimization based algoithm designed to povide minimal wost case guaantees on enegy expenditue given all uncetainties. Finally, we have 2

21 / (213) Competitive atio Pediction Robust standad deviation (a) Wokload/Renewable eo annual expenditue ($) 1.15 x Pediction Robust CP distibution shift (s) (b) Spead out CPs annual expenditue ($) 1.4 x Pediction Robust waning distibution shift (s) (c) Spead out wanings Figue 8. Sensitivity analysis of Pediction and Robust algoithms with espect to (a) wokload and enewable geneation pediction eo and (b) & (c) coincident peak and waning pediction eos. In all cases, the data cente consideed has a local diesel geneato, but no local PV installation. evaluated these algoithms using detailed, eal wold tace-based numeical simulation expeiments. These expeiments highlight that the use of both wokload shifting and local geneation ae cucial in ode fo a data cente to minimize its enegy costs and emissions. Thee ae a numbe of futue eseach diections that build on the wok in this pape. In paticula, an inteesting diection is to adapt the algoithms pesented hee in ode to incopoate enegy stoage at the data cente. Moe geneally, Intenet-scale systems ae typically povided by a geogaphically distibuted data centes, and so it would be inteesting to undestand how the geogaphical load balancing pefomed by such systems inteacts with coincident peak picing. This moving bits, not watts scheme can significantly educe local powe netwok pessue without adding futhe load to the (possibly aleady) congested tansmission netwok. Additionally, CPP pogams ae just one example of demand esponse pogams. Though CPP pogams ae cuently the most common fom of demand esponse pogam, a numbe of new pogams ae emeging. It is impotant to undestand how each of these pogams, e.g., [49], inteact with data cente planning. Refeences [1] National Institute of Standads and Technology, NIST famewok and oadmap fo smat gid inteopeability standads. NIST Special Publication 118, 21. [2] Depatment of Enegy, The smat gid: An intoduction. [3] Fedeal Enegy Regulatoy Commission, National assessment of demand esponse potential. 29. [4] NY Times, Powe, Pollution and the Intenet. [5] G. Ghatika, V. Ganti, N. Matson, and M. Piette, Demand esponse oppotunities and enabling technologies fo data centes: Findings fom field studies, 212. [6] Repot to congess on seve and data cente enegy efficiency. 27. [7] J. Koomey, Gowth in data cente electicity use 25 to 21, Oakland, CA: Analytics Pess. August, vol. 1, p. 21, 211. [8] [9] htm. [1] A. Gandhi, Y. Chen, D. Gmach, M. Alitt, and M. Mawah, Minimizing data cente sla violations and powe consumption via hybid esouce povisioning, in Poc. of IGCC, 211. [11] Y. Chen, D. Gmach, C. Hyse, Z. Wang, C. Bash, C. Hoove, and S. Singhal, Integated management of application pefomance, powe and cooling in data centes, in Poc. of NOMS, 21. [12] M. Lin, A. Wieman, L. L. H. Andew, and E. Theeska, Dynamic ight-sizing fo powe-popotional data centes, in Poc. of INFOCOM, 211. [13] S. Govindan, J. Choi, B. Ugaonka, A. Sivasubamaniam, and A. Baldini, Statistical pofiling-based techniques fo effective powe povisioning in data centes, in Poc. of EuoSys, 29. [14] J. Choi, S. Govindan, B. Ugaonka, and A. Sivasubamaniam, Pofiling, pediction, and capping of powe consumption in consolidated envionments, in MASCOTS,

22 / (213) [15] J. Heo, P. Jayachandan, I. Shin, D. Wang, T. Abdelzahe, and X. Liu, Optitune: On pefomance composition and seve fam enegy minimization application, Paallel and Distibuted Systems, IEEE Tansactions on, vol. 22, no. 11, pp , 211. [16] A. Vema, G. Dasgupta, T. Nayak, P. De, and R. Kothai, Seve wokload analysis fo powe minimization using consolidation, in USENIX ATC, 29. [17] D. Meisne, C. Sadle, L. Baoso, W. Webe, and T. Wenisch, Powe management of online data-intensive sevices, in Poc. of ISCA, 211. [18] Q. Zhang, M. Zhani, Q. Zhu, S. Zhang, R. Boutaba, and J. Hellestein, Dynamic enegy-awae capacity povisioning fo cloud computing envionments, in ICAC, 212. [19] H. Xu and B. Li, Cost efficient datacente selection fo cloud sevices, 212. [2] Y. Yao, L. Huang, A. Shama, L. Golubchik, and M. Neely, Data centes powe eduction: A two time scale appoach fo delay toleant wokloads, in Poc. of INFOCOM, pp , 212. [21] R. Ugaonka, B. Ugaonka, M. Neely, and A. Sivasubamaniam, Optimal powe cost management using stoed enegy in data centes, in Poc. of the ACM Sigmetics, 211. [22] D. Iwin, N. Shama, and P. Shenoy, Towads continuous policy-diven demand esponse in data centes, Compute Communication Review, vol. 41, no. 4, 211. [23] Z. Liu, Y. Chen, C. Bash, A. Wieman, D. Gmach, Z. Wang, M. Mawah, and C. Hyse, Renewable and cooling awae wokload management fo sustainable data centes, in Poc. of ACM Sigmetics, 212. [24] K. Le, O. Bilgi, R. Bianchini, M. Matonosi, and T. D. Nguyen, Capping the bown enegy consumption of intenet sevices at low cost, in Poc. IGCC, 21. [25] Z. Liu, M. Lin, A. Wieman, S. H. Low, and L. L. H. Andew, Geening geogaphical load balancing, in Poc. ACM Sigmetics, 211. [26] L. Rao, X. Liu, L. Xie, and W. Liu, Minimizing electicity cost: Optimization of distibuted intenet data centes in a multi-electicity-maket envionment, in Poc. of INFOCOM, 21. [27] P. Wendell, J. W. Jiang, M. J. Feedman, and J. Rexfod, Dona: decentalized seve selection fo cloud sevices, in Poc. of ACM Sigcomm, 21. [28] Z. Liu, M. Lin, A. Wieman, S. H. Low, and L. L. H. Andew, Geogaphical load balancing with enewables, in Poc. ACM GeenMetics, 211. [29] M. Lin, Z. Liu, A. Wieman, and L. Andew, Online algoithms fo geogaphical load balancing, in Poc. of IGCC, 212. [3] D. Meisne, J. Wu, and T. Wenisch, Bighouse: A simulation infastuctue fo data cente systems, in Poc. of ISPASS, pp , 212. [31] L. Baoso and U. Hölzle, The datacente as a compute: An intoduction to the design of waehouse-scale machines, Synthesis Lectues on Compute Achitectue, vol. 4, no. 1, pp. 1 18, 29. [32] [33] [34] B. Ugaonka, G. Pacifici, P. Shenoy, M. Speitze, and A. Tantawi, An analytical model fo multi-tie intenet sevices and its applications, in Poc. of ACM Sigmetics, 25. [35] Y. Chen, S. Alspaugh, and R. Katz, Inteactive analytical pocessing in big data systems: a coss-industy study of mapeduce wokloads, Poc. of VLDB, 212. [36] M. Zahaia, D. Bothaku, J. Sama, K. Elmeleegy, S. Shenke, and I. Stoica, Job scheduling fo multi-use mapeduce clustes, UCB/EECS , 29. [37] T. Been, E. Walsh, J. Punch, C. Bash, and A. Shah, Fom chip to cooling towe data cente modeling: Influence of seve inlet tempeatue and tempeatue ise acoss cabinet, Jounal of Electonic Packaging, vol. 133, no. 1, 211. [38] C. Patel, R. Shama, C. Bash, and A. Beitelmal, Enegy flow in the infomation technology stack, in Poc. of IMECE, 26. [39] EPA, US Emission Standads fo Nonoad Diesel Engines. [4] C. Ren, D. Wang, B. Ugaonka, and A. Sivasubamaniam, Cabon-awae enegy capacity planning fo datacentes, in MASCOTS, pp , IEEE, 212. [41] A. Queshi, R. Webe, H. Balakishnan, J. Guttag, and B. Maggs, Cutting the electic bill fo intenet-scale systems, in Poc. of ACM Sigcomm, 29. [42] C. Stewat and K. Shen, Some joules ae moe pecious than othes: Managing enewable enegy in the datacente, in Poc. of HotPowe, 29. [43] K. Le, R. Bianchini, M. Matonosi, and T. Nguyen, Cost-and enegy-awae load distibution acoss data centes, Poceedings of HotPowe, 29. [44] Í. Goii, K. Le, T. Nguyen, J. Guitat, J. Toes, and R. Bianchini, Geenhadoop: leveaging geen enegy in data-pocessing famewoks, in Poc. of EuoSys, 212. [45] N. Deng, C. Stewat, J. Kelley, D. Gmach, and M. Alitt, Adaptive geen hosting, in Poceedings of ICAC, 212. [46] N. Shama, P. Shama, D. Iwin, and P. Shenoy, Pedicting sola geneation fom weathe foecasts using machine leaning, in Poc. of SmatGidComm, 211. [47] Y. Becea, D. Caea, and E. Ayguade, Batch job pofiling and adaptive pofile enfocement fo vitualized envionments, in Poc. of ICPDNP, 29. [48] J. Choi, S. Govindan, B. Ugaonka, and A. Sivasubamaniam, Powe consumption pediction and powe-awae packing in consolidated envionments, IEEE Tansactions on Computes, vol. 59, no. 12, 21. [49] D. Aikema, R. Simmonds, and H. Zaeipou, Data centes in the ancillay sevices maket, 22

23 / (213) Appendix A. Poofs In this appendix we include poofs fo bounds on the competitive atio of ou both algoithms in Section 4. Because the poof of Theoem 1 uses simplified vesions of many pats of the poof of Theoem 2, we stat with the poof of Theoem 2 and then descibe how to specialize the appoach to Theoem 1. To pove Theoem 2, we stat with some notation and simple obsevations. Fist, in this context, the offline optimal is defined as follows: (b, g ) agmin b,g f (e, g), whee f (e, g) Σ t p(t)e(t) + p p max t e(t) + p cp e(t cp ) + p g Σ t g(t). Hee b stands fo the wokload management, and g denotes the local backup geneato usage, e(t) = (d(t) (t) g(t)) + is the gid powe usage, we assume the offline optimal have pefect knowledge of d(t), (t), and when coincidental peak occus. In contast, the plan deived fom Algoithm 2, denoted by (ê w 1, gw 1 ), minimizes f w (ê, g) Σ t p(t)ê(t) + ( p p + W ( p g min t p(t) )) max t ê(t) + p g Σ t g(t) using pediction of wokload ˆd(t) and pediction of enewable geneation ˆ(t) without any knowledge of coincidental peak (CP) o wanings except W. Hee ê(t) = ( ˆd(t) ˆ(t) g(t)) +. In addition, Algoithm 2 uses minimal local geneation to emove hamful pediction eo when (4) occus, i.e., g w ε (t) = max{, min{e w (t), ε d ˆd w (t) ε ˆ(t)}}. Also, Algoithm 2 uses local geneation wheneve wanings ae eceived, i.e., g w 2 (t) = I {t W}e w 1 (t), t, whee I {t W} is the indicato function, which equals to 1 if t is a time when waning is eceived and othewise and e w 1 (t) = (d w 1 (t) (t) gw 1 (t) gw ε (t)) +. Theefoe the eal gid powe usage at time t is e w (t) ê w 1 (t) gw 2 (t), and local powe geneation is g w (t) = g w 1 (t) + gw ε (t) + g w 2 (t), t. Note hee (êw 1, gw 1 ) is the day-ahead plan, while (ew, g w ) is the eal gid powe consumption and local geneation afte using local geneation to compensate fo undeestimation and duing waning peiods. Poof of Theoem 2. Note that f and f w ae optimizations using diffeent data ( f uses pefect knowledge of d(t) and (t), while f w uses pediction ˆd(t) and ˆ(t)), to bidge this gap, we fist obseve the following: f (e, g ) f (ê, g + g ε) p g Σ t g ε(t) (A.1) whee ê is the optimize of f using pediction ˆd(t) and ˆ(t), and g ε is defined in a simila way to g w ε, g ε(t) = max{, min{ê (t), ε d ˆd(t) ε ˆ(t)}} which emoves all the hamful pediction eos. The ight hand side of the inequality is essentially evaluating the same objective using pediction, but is given g ε of local powe fo fee. As g ε emoves all hamful effects of pediction, using pediction will not incease the objective. 23

24 The key step is to bound E ˆ ξ d, ˆ ξ [ f w (ê w 1, gw 1 )] in tems of E ˆ ξ d, ˆ ξ [ f (ê, g + g ε)] / (213) E ˆ ξ d, ˆ ξ [ f (ê, g + g ε)] = E ˆ ξ d, ˆ ξ [ f w (ê, g + g ε))] W ( p g min t p(t) ) E ˆ ξ d, ˆ ξ [max t ê (t)] + p cp E ˆ ξ d, ˆ ξ [ê (t cp )] E ˆ ξ d, ˆ ξ [ f w (ê, g + g ε))] W ( p g min t p(t) ) E ˆ ξ d, ˆ ξ [max t ê (t)] E ˆ ξ d, ˆ ξ [ f w (ê w 1, gw 1 )] W ( p g min t p(t) ) E ˆ ξ d, ˆ ξ [max t ê (t)] E ˆ ξ d, ˆ ξ [ f w (ê w 1, gw 1 + gw ε )] p g Σ t E ˆ ξ d, ˆ ξ [g w ε (t)] W ( p g min t p(t) ) E ˆ ξ d, ˆ ξ [max t ê (t)] E ˆ ξ d, ˆ ξ [ f (e w, g w )] p g Σ t E ˆ ξ d, ˆ ξ [g w ε (t)] W ( p g min t p(t) ) E ˆ ξ d, ˆ ξ [max t ê (t)] (A.2) Hee the fist inequality holds because p cp E ˆ ξ d, ˆ ξ [ê (t cp )]. The second inequality is fom the optimality of (ê w 1, gw 1 ) in minimizing f w (e, g). Howeve, the last inequality is moe involved. We show the last step of (A.2) by fist witing out the day-ahead plan ê w 1 (t) = ( ˆd w 1 (t) ˆ(t) gw 1 (t)) +, and the actual powe demand e w (t) = ( d w 1 (t) (t) gw 1 (t) gw ε (t) g w 2 (t)) +. Futhemoe, denote e w 2 (t) as the electicity demand of Algoithm 2 without using local geneation to espond to CP waning. Then e w (t) = e w 2 (t) gw 2 (t), and gw 2 (t) = e w 2 (t)i {t W}, so we have e w 2 (t) = ( d w 1 (t) gw 1 (t) gw ε (t) ) + ( ˆd w 1 (t) ˆ(t) gw 1 (t)) + = ê w 1 (t) Hence e w (t) = e w 2 (t) gw 2 (t) êw 1 (t) gw 2 (t). Next, we bound f (e w, g w ) as follows. f (e w, g w ) = f (e w, g w 1 + gw ε + g w 2 ) = Σ t p(t)e w (t) + p p max t e w (t) + p cp e w (t cp ) + p g Σ t g w (t) = Σ t W p(t)e w 2 (t) + p pmax t W e w 2 (t) + p ( g Σt (g w 1 (t) + gw ε (t)) + Σ t W e w 2 (t)) Σ t p(t)ê w 1 (t) + p pmax t ê w 1 (t) + p gσ t (g w 1 (t) + gw ε (t)) + Σ t W (p g p(t))ê w 1 (t) Σ t p(t)ê w 1 (t) + p pmax t ê w 1 (t) + p gσ t (g w 1 (t) + gw ε (t)) + W(p g min t p(t))max t ê w 1 (t) = f w (ê w 1, gw 1 + gw ε ) (A.3) The second equality is because g w 2 (t) = I {t W}e w 2 (t), t. The fist inequality is fom max t We w 2 (t) max te w 2 (t) and e w 2 (t) êw 1 (t). The second inequality holds because Σ t W(p g p(t))ê w 1 (t) Σ t W(p g min t p(t))ê w 1 (t) = ( p g min t p(t) ) Σ t W ê w 1 (t) ( p g min t p(t) ) Σ t W max t ê w 1 (t) W(p g min t p(t))max t ê w 1 (t). Finally, we can combine (A.1) and (A.2) to obtain E ˆ ξ d, ˆ ξ [ f (e, g )] E ˆ ξ d, ˆ ξ [ f (ê, g + g ε)] p g Σ t E ˆ ξ d, ˆ ξ [g ε(t)] Eξ ˆd f (e w, g w ) p g Σ t Eξ ˆd [g w ε (t) + g ε(t)] W ( p g min t p(t) ) E ˆ ( ˆd Eξ ˆd [ f (e w, g w w (t) + ˆd (t) )] p g σσ t + ˆ(t) 2 ) ξ d, ˆ ξ [max t ê (t)] W ( p g min t p(t) ) E ˆ ξ d, ˆ ξ max t ê (t), (A.4) 24

25 / (213) whee (A.4) deives fom the following E ˆ ξ d, ˆ ξ [g w ε (t) + g ε(t)] = E ˆ ξ d, ˆ ξ [max{, min{e w (t), ε d ˆd w (t) ε ˆ(t)}} + max{, min{e (t), ε d ˆd (t) ε ˆ(t)}}] Eξ ˆd [(ε d ˆd w (t) ε ˆ(t)) + ] + Eξ ˆd [(ε d ˆd (t) ε ˆ(t)) + ] ( = E[ε w (t) + ] + E[ε (t) + ] let ε w (t) = ε d ˆd w (t) ε ˆ(t), ε (t) = ε d ˆd (t) ε ) ˆ(t) 1 2 σ ε w (t) σ ε (t) = 1 ( ) ˆd 2 w (t) 2 σ 2 d + ˆ(t)2 σ 2 + ˆd (t) 2 σ 2 d + ˆ(t)2 σ 2 1 ( ( ˆd w (t) + ˆ(t)) max(σ d, σ ) + ( ˆd (t) + ˆ(t)) max(σ d, σ ) ) 2 ( ˆd w (t) + ˆd ) (t) = + ˆ(t) σ 2 (A.5) The second last equality holds because ε d and ε ae independent, and the last inequality holds because ˆd(t) and ˆ(t) ae nonnegative. The key is the second inequality, as the cases fo ε w (t) and ε (t) ae the same, we just need to show this inequality holds fo any ε(t) has zeo mean and fixed vaiance σ 2 ε(t). Note that ε(t) = ε(t)+ ε(t), hence E[ε(t)] = E[ε(t) + ] = E[ε(t) ]. It follows that σ 2 ε(t) = E[ε(t)2 ] = E[(ε(t) + ) 2 ] + E[(ε(t) ) 2 ] 2E[ε(t) + ε(t) ] = E[(ε(t) + ) 2 ] + E[(ε(t) ) 2 ] E[ε(t)+ ] 2 P(ε(t) ) + E[ε(t) ] 2 P(ε(t) < ) ( ) = E[ε(t) + ] 2 1 P(ε(t) ) P(ε(t) ) ( ) = E[ε(t) + ] 2 1 (P(ε(t) ))(1 P(ε(t) )) 4E[ε(t) + ] 2 Reaanging, we have E[ε(t) + ] 1 2 σ ε(t). The last inequality attains equality when P(ε(t) + ) = P(ε(t) < ) = 1/2. The thid equality follows because ε(t) + and ε(t) cannot be simultaneously non-zeo. The fist inequality 25

26 / (213) follows because = E[(ε(t) + ) 2 ]P(ε(t) ) ˆ x 2 df ε(t) (x) ˆ ) 2 (ˆ x 1dF ε(t) (x) = E[ε(t) + ] 2 1dF ε(t) (x) E[(ε(t) + ) 2 ] E[ε(t)+ ] 2 P(ε(t) ) The fist inequality follows fom Cauchy-Schwaz inequality, and the inequality attains equality when the distibution of ε(t) + is a point mass. By simila agument we can show that E[ε(t) ] 2 E[ε(t) ] 2, and equality is attained when the distibution of ε(t) is a point mass. P(ε(t)<) Using the obsevation above and the pevious obsevation that P(ε(t) + ) = P(ε(t) < ) = 1/2, we can see that E[ε(t) + ] = 1 2 σ ε(t) when the distibution of ε(t) is two equal point masses located at σ ε(t) and σ ε(t) espectively. P(ε(t) = x) 1 2 x σ ε(t) σ ε(t) Figue A.9. Illustation of pdf of ε(t) that attains E[ε(t) + ] = 1 2 σ ε(t) fo E[ε(t)] = and Va(ε(t)) = σ ε(t). Finally, combining the above, we can compute the competitive atio as follows Eξ ˆd [ f (e w, g w )] Eξ ˆd [ f (e, g )] W ( p g min t p(t) ) E ˆ ξ d, ˆ ξ [max t e (t)] + p g σσ t ( ˆd w (t)+ ˆd (t) 2 + ˆ(t)) Σ t p(t)eξ ˆd [e (t)] + p p Eξ ˆd [max t e (t)] + p cp Eξ ˆd [e (t cp )] + p g Σ t g (t) W ( p g min t p(t) ) Σ t p(t)eξ ˆd [e (t)]/eξ ˆd [max t e + Bσ, (t)] + p p W ( p g min t p(t) ) 1 + min t p(t)σ t Eξ ˆd [e (t)]/eξ ˆd [max t e + Bσ (t)] + p p W ( p g min t p(t) ) = 1 + Tmin t p(t)/pmr + Bσ + p p W ( p g min t p(t) ) Bσ p p ) B = p ˆdw(t)+ ˆd (t) gσ t ( 2 + ˆ(t)) Eξ ˆd [ f (e, g )] (A.6) 26

27 / (213) It emains to show that no online algoithm can have competitive atio smalle than (1 + W(p g min t p(t)) p p ) even with pefect infomation of wokload and enewable geneation. To pove this, we use the instance summaized in Figue A.1. demand D/(T-1) D/T Local geneation OPT ALG t w1 t w2 t w3 (t cp ) T time Figue A.1. Instance fo lowe bounding the competitive atio fo setting with local geneation. In this instance, PUE is the same acoss all time slots and small. Thee is no local enewable supply o inteactive wokload. The total flexible wokload demand is D. The (discete) time hoizon is [1,T], whee t wi, i = 1,..., W ae the time slots with wanings (thee wanings ae shown in the figue) and the total numbe of wanings is W with bound W W known to the online algoithm. The final coincident peak is t cp and it is among the wanings (t w3 in the figue). The usage-based electicity pice p(t) = p, t and is much smalle than p p and p cp. Also, in this instance, p p T 1 p g (using local geneation is moe expensive than demand shifting and paying (slightly) inceased peak demand chaging) and p g p cp, which ae common in pactice. In this setting, the offline optimal solution plans accoding to the geen cuve: it does not use the coincident peak time slot but speads the demand evenly acoss the othe T 1 time slots. The cost of the offline optimal solution is D theefoe f (e, g ) = pd + p p T 1. In contast, any online algoithm can at best plan accoding to the ed cuve: speading the wokload evenly among all T time slots and using local geneation when wanings ae eceived. To see this, note that thee is no benefit to speading the wokload unevenly since that inceases local geneation usage fo the wost-case instance and possibly the peak chaging, while not saving any usage based cost. The cost of the best online non-adaptive solution is theefoe f (e ALG, g ALG D ) = pd + p p T + W ( p g p ) D T. The best competitive atio is theefoe: f (e ALG, g ALG ) f (e, g ) = pd + p p D + W ( p T g p ) D T D pd + p p D T 1 = 1 + p p + W ( p T(T 1) g p ) D T D pd + p p T 1 = 1 + W(p g p) pp T pt + p p As T, taking the usage cost pt as the same o smalle ode of magnitude as the peak cost p p, this becomes 1 + W(p g p) pt + p p The above matches the bound in equation (A.6) when W = W, which completes the poof. 27 T 1 T 1

28 / (213) Poof Sketch of Theoem 1. The poof of Theoem 1 is simila in stuctue to that of Theoem 2, only simple. Thus, we outline only the main steps and highlight the similaities with the poof of Theoem 2. In paticula, the following povides the majo steps needed to bidge the expected cost of Algoithm 1 and the cost of the offline algoithm with exact IT demand and enewable geneation knowledge: Eξ ˆd,Ŵ EŴ = EŴ EŴ EŴ [ f (e, g ) ] E ˆ ξ d, ˆ ξ f (ê, g + g ɛ) p g E ˆ ξ d, ˆ ξ [ f s (ê, g + g ɛ) ] p g E ˆ ξ d, ˆ ξ [ f s (e s, g s 1 )] 1 2 σp g E ˆ ξ d, ˆ ξ [ f (e s, g s ) ] 1 2 σp g T g ε(t) t=1 T t=1 E ˆ ξ d, ˆ ξ [ g ε (t) ] (A.7a) (A.7b) T ( ˆd (t) + ˆ(t) ) (A.7c) t=1 T ( ˆd (t) + ˆ(t) ) 1 T 2 σp ( g ˆd s (t) + ˆ(t) ) (A.7d) t=1 It is easy to see that the theoem follows fom this geneal appoach, but of couse each step equies some effot to justify. Howeve, the justification of each step paallels calculations fom the poof of Theoem 2. In paticula, (A.7a) is paallel to (A.1), (A.7b) is because f ( ) and f s ( ) ae equivalent when taking expectation, (A.7c) is paallel to (A.5), and (A.7d) is paallel to (A.2). Since the veification of these is simple than in the case of Theoem 2, we omit the details. t=1 28