A Agent-Based Homeostatic Control for Green Energy in the Smart Grid

Transcription

1 A Agent-Based Homeostatc Control for Green Energy n the Smart Grd SARVAPALI D. RAMCHURN, PERUKRISHNEN VYTELINGUM, ALEX ROGERS, and NICHOLAS R. JENNINGS Intellgence, Agents, Multmeda Group School of Electroncs and Computer Scence Unversty of Southampton Southampton, SO17 1BJ, UK {sdr,pv,acr,nrj}@ecs.soton.ac.uk Wth dwndlng non-renewable energy reserves and the adverse effects of clmate change, the development of the smart electrcty grd s seen as key to solvng global energy securty ssues and to reducng carbon emssons. In ths respect, there s a growng need to ntegrate renewable (or green) energy sources n the grd. However, the ntermttency of these energy sources requres that demand must also be made more responsve to changes n supply, and a number of smart grd technologes are beng developed, such as hghcapacty batteres and smart meters for the home, to enable consumers to be more responsve to condtons on the grd n real-tme. Tradtonal solutons based on these technologes, however, tend to gnore the fact that ndvdual consumers wll behave n such a way that best satsfes ther own preferences to use or store energy (as opposed to that of the suppler or the grd operator). Hence, n practce, t s unclear how these solutons wll cope wth large numbers of consumers usng ther devces n ths way. Aganst ths background, n ths paper, we develop novel control mechansms based on the use of autonomous agents to better ncorporate consumer preferences n managng demand. These agents, resdng on consumers smart meters, can both communcate wth the grd and optmse ther owner s energy consumpton to satsfy ther preferences. More specfcally, we provde a novel control mechansm that models and controls a system comprsng of a green energy suppler operatng wthn the grd and a number of ndvdual homes (each possbly ownng a storage devce). Ths control mechansm s based on the concept of homeostass whereby control sgnals are sent to ndvdual components of a system, based on ther contnuous feedback, n order to change ther state so that the system may reach a stable equlbrum. Thus, we defne a new carbon-based prcng mechansm for ths green energy suppler that takes advantage of carbon-ntensty sgnals avalable on the nternet n order to provde real-tme prcng. The prcng scheme s desgned n such a way that t can be readly mplemented usng exstng communcaton technologes and s easly understandable by consumers. Buldng upon ths, we develop new control sgnals that the suppler can use to ncentvse agents to shft demand (usng ther storage devce) to tmes when green energy s avalable. Moreover, we show how these sgnals can be adapted accordng to changes n supply and to varous degrees of penetraton of storage n the system. We emprcally evaluate our system and show that, when all homes are equpped wth storage devces, the suppler can sgnfcantly reduce ts relance on other carbon-emttng power sources to cater for ts own shortfalls. By so dong, the suppler reduces the carbon emsson of the system by up to 25% whle the consumer reduces ts costs by up to 14.5%. Fnally, we demonstrate that our homeostatc control mechansm s not senstve to small predcton errors and the suppler s ncentvsed to accurately predct ts green producton to mnmse costs. Categores and Subject Descrptors: I.2.11 [Computng Methodologes]: Artfcal Intellgence Dstrbuted Artfcal Intellgence General Terms: Agents, Mult-Agent Systems Addtonal Key Words and Phrases: Computatonal Sustanablty, Electrcty, Mult-Agent Systems, Agent-Based Control, Agents. 1. INTRODUCTION Achevng energy securty and reducng carbon emssons have been recognsed as two of the most mportant challenges of ths century gven the rapd depleton of worldwde ol and gas reserves and the potentally devastatng effects of clmate change [DECC 2009a; US Department Of Energy 2003]. To ths end, the creaton of a smart electrcty grd has been advocated as a key component n ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

2 A:2 Ramchurn, Vytelngum, Rogers, and Jennngs delverng effcent low-carbon energy. In partcular, the vson of a smart grd ncludes technologes that reduce losses n the transmsson of electrcty, that ntegrate ntermttent renewable energy generators (such as wnd turbnes and solar panels), and that attempt to reduce or shft demand by provdng real-tme nformaton about consumpton and prces usng smart meters n homes. In ths paper we focus n partcular on the pressng problem of ntegratng renewable energy generaton nto the smart grd. Ths s an mportant challenge because the ntegraton of renewable energy generators requres a sgnfcant shft from the tradtonal prncple of supply follows demand whereby generators always have to keep up wth demand n the grd. To date, ths has been acceptable only because the output from non-renewable energy sources (such as coal or gas fred power statons) can be turned up or down at wll. In contrast, renewable energy generators (.e., green supplers) are senstve to seasons and weather condtons and are therefore varable and ntermttent n ther output. Ths means that they cannot be powered up at wll to meet the demand, nor s t possble to accurately predct exactly how much energy they wll generate. Whle, from a techncal pont of vew, ntegratng renewable energy requres the development of technologes to ensure power flows n a controlled manner from renewable energy generators to the grd, from an economc pont of vew, renewable energy supplers need to devse new strateges to trade effectvely n the electrcty markets n whch they operate n order to ensure that they can make a proft whle stll satsfyng the demand from ther consumers gven the ntermttency n ther supply. When the suppler s not able to satsfy the demand of ts customers from ts own renewable generaton capacty, t must typcally buy addtonal electrcty at short notce (usually ncurrng a hgh cost whch reduces ts potental profts) from the wholesale electrcty market wthn whch t operates [Krschen and Strbac 2004]. Conversely, when the suppler produces more than ts customers demand, t must sell ths extra electrcty at a lower prce than what t would typcally obtan from ts customers. 1 It s therefore crucal for the suppler to predct ts producton accurately n order to bd effectvely n markets, but more mportantly, the suppler should be able to nfluence the demand from ts customers to mtgate the mpact of low producton. Now, whle the ssue of predctng renewable energy producton and bddng n electrcty markets has been actvely researched n the power systems doman [Krschen and Strbac 2004; Mllgan et al. 2009; Morales et al. 2010], there s relatvely lttle work studyng how to nfluence the demand of resdental consumers (see Secton 2 for more detals). Typcal approaches to demand management nvolve the suppler sendng a predcted real-tme prce, va the consumers smart meter, to ncentvse them to ncrease or decrease demand [Faruqu and George 2005]. These mechansms assume that by feedng more nformaton to consumers, the latter wll be able to understand the condtons on the grd (e.g., generator outages or a hgh prce for electrcty at peak tme) and react to such condtons by swtchng devces on or off, or changng ther thermostat settngs on ther heaters (or ar condtoners), and n future, use electrcty stored n ther home or electrc car battery. However, such an assumpton s challenged by the fact that the sgnals from the grd, asde from beng rather complex for the average user [Schweppe et al. 1989], requre the user to perform complex calculatons n order to optmse her consumpton and storage of electrcty (.e., to know what s the best tme to store or use electrcty). Gven ths, n ths paper, we advocate the use of autonomous software agents as a decentralsed control paradgm whereby each owner would have her own agent, runnng on her smart meter, advsng on the best consumpton pattern she could adopt to acheve her goals (e.g., mnmsng carbon emssons and/or costs). A key advantage of ths approach s that consumers do not need to be experts at energy tradng or schedulng n order to maxmse ther benefts, therefore ther adopton allows for a larger mpact n mplementng demand management strateges based on complex sgnals from the grd. 1 In general, a suppler acts as the mddle-man between the generators from whch t buys electrcty and the customers to whch t sells electrcty. However, we consder the case of the generator and the suppler beng the same actor as s often the case n the UK or US electrcty market, partcularly wth regards to green focused supplers such as Ecotrcty (UK), New England GreenStart, or Green Mountan Energy (US). ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

3 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A:3 In more detal, agents can be endowed wth the ablty to learn ther owner s preferences from ther consumpton hstory and can run optmsaton algorthms that solve complex schedulng tasks (e.g., heatng cycles of bolers or chargng profles of ther electrc car battery) wth very lttle drect nput, f at all, from ther owner. In so dong, these agents can mnmse ther owner s cost, gven prce sgnals from the grd, by effcently schedulng the on/off cycles of deferrable devces (such as washng machnes or bolers) or the storage profles of batteres (.e., storng when prces are low and usng the stored electrcty when prces are hgh). However, even f a real-tme prce sgnal s provded by the grd or suppler, agents have no ndcaton as to how much they should change ther usage (by deferrng loads) or store electrcty n order to meet the capacty of the renewable energy suppler. Ths s mportant because, f, for example, the agents are ncentvsed to consume more, based on a low predcted prce, the aggregate mght end up consumng much more than can be provded by the suppler. Conversely, gven a hgh prce, the collecton of agents mght consume much less than what the suppler produces leavng the suppler wth unused green energy. Hence, n general, relyng solely on the prce sgnal may result n a msmatch between the producton capacty of the suppler and the demand from the consumers that mght requre the suppler to buy extra capacty at a hgh prce or sell ts extra producton at a loss on the wholesale electrcty market. The suppler may, n turn, pass on such costs to ts customers and also ncrease the consumers carbon emssons as t uses carbon-emttng generators (from whch t buys on the wholesale market) to supply ther extra demands. Moreover, provdng only a prce sgnal gnores the fact that some agents may also be amng to reduce ther owner s carbon emssons (as there s a growng populaton of such green users [Kockar et al. 2009]) snce such a sgnal gnores the carbon ntensty (.e., the carbon emssons per unt of energy) of the energy beng suppled. Gven these ssues wth realtme prcng n ths doman, t s crucal that better control mechansms are developed to mtgate the occurrence of such msmatches n order to make the ntegraton of renewable energy nto the smart grd more proftable and more relable. In partcular, gven the reacton of the agents to such sgnals, the controller (.e., suppler) should be able to provde feedback to the agents n order for them to adjust ther consumpton profle whle stll allowng them to mnmse ther costs and ther carbon emssons. The challenge here s to generate such sgnals and stablse the system wthout the suppler beng able to fully observe the state of the ndvdual agents n terms of ther profle of electrcty usage. Ths challenge s further exacerbated by the fact that the agents may not wsh to reveal ther battery storage capacty (whch determnes by how much they can defer demand to follow supply) as they may not trust the provder to use ths nformaton to ther beneft. To meet ths challenge, n ths paper, we ntroduce the concept of agent-based homeostatc control to coordnate a green suppler and ts consumers, who may own storage capablty, wthn the smart grd. 2 In partcular, we model a system composed of a green energy suppler (e.g., a wnd and/or solar powered generator) provdng electrcty to a large number of consumers whose ams are to reduce ther ndvdual costs and carbon emssons. To capture the consumers preferences and goals, we develop agents that can optmse ther owner s battery storage profle accordng to sgnals from the grd. Our model ams to capture real-lfe settngs where, for example, a thrd-party wnd turbne/photovoltac array provdes energy to a small number of houses or buldngs, or where a regonal/natonal suppler serves customers across a wde area. We then provde a novel control mechansm to ensure that demand from the agents follows the producton from the green suppler. Our approach s nspred from the prncple of homeostass [Mareb and Hoehn 2007]. The latter descrbes a lvng organsm s adjustment of ts nternal nterdependent components to mantan a stable constant condton and was frst used n the context of the electrcty grd by Schweppe et al. [1980]. The analogy between our system and a lvng organsm s approprate gven that our system s based on the three underpnnng concepts of homeostass: sensng, sendng a control sg- 2 Our approach can be generalsed to consder loads such as washng machnes or bolers that can be deferred to acheve results smlar to usng storage, but ths s beyond the scope of ths paper. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

4 A:4 Ramchurn, Vytelngum, Rogers, and Jennngs nal, and feedback. Frst, our system s able to sense and predct 3 the effect of external condtons (.e., weather n our case) usng wdely avalable weather sensors and satellte magery. Second, the suppler can compute a control sgnal to be sent to the agents based on the mpact of weather condtons on ts producton of electrcty. Thrd, through the resultng response of the agents n the system (.e., the combned effect of ts load and storage profles), feedback s provded to the suppler whch decdes how to adapt ts control sgnal to ensure the system reaches the state t desres. Through several teratons of the above process, the system ams to converge to a more effcent soluton (see Secton 6). Whle the dea of homeostatc control n ths doman was poneered more than 30 years ago, t was not deemed practcal at the tme due to the lack of communcaton and dstrbuted computng technologes that would allow large populatons of consumers of dfferent types (of dfferent ncome and usage levels) to partcpate n such mechansms. However, wth the prevalence of broadband communcatons 4 and the development of agent-based technologes for a wde varety of applcatons ncludng the energy doman [Jennngs et al. 1996; Bussmann et al. 2004], t s now possble to revst ths proposton and nstantate homeostatc control mechansms wthn the smart grd. In more detal, ths paper advances the state of the art n the followng ways: (1) We mplement a novel real-tme carbon-based prcng mechansm that bulds on the carbon ntensty sgnals provded by the grd (e.g., the Natonal Grd n the UK or Ere Grd n Ireland already provde ths data n real-tme) through the nternet. 5 Usng ths mechansm, supplers, usng renewable energy sources or not, do not need to nstall addtonal hardware to compute and transmt a real-tme prce to consumers as t can easly be obtaned over an exstng broadband connecton. We show that by settng a prce plan to match the carbon ntensty of the grd, supplers can effectvely algn the carbon-based prces to the real-tme prces they effectvely trade at on the grd. Such a prcng scheme also means that, should demand be hgher than can be met by the green suppler, the greenest form of energy from the grd (where a mx of green and non-green supplers also exst) wll be bought by the suppler to meet excess demand. By so dong, carbon-senstve consumers are ncentvsed to subscrbe to the suppler snce they can mnmse both ther costs and carbon emssons at the same tme. In turn, ths helps the suppler reduce ts relance on other grd energy sources wth hgh carbon emssons (for whch the suppler may be taxed accordng to the market regme t operates wthn [Kockar et al. 2009; Hobbs et al. 2010]). It s also mportant to note that, our prcng scheme s not restrcted to just a green suppler as our prces (except the prce of energy strctly obtanable from the green suppler) are only dependent on the condtons on the grd (.e., grd carbon ntensty reflectng generaton costs). Thus, our prcng scheme could potentally be used by any suppler tryng to appeal to an envronmentally conscous consumer amng to reduce her contrbuton to the overall carbon emssons of the grd. (2) We develop a novel control sgnal whch ensures that agents are ncentvsed to store a gven quantty of electrcty accordng to changes n green energy generaton capacty caused by changng weather condtons. The sgnal provdes an estmate to the agent of how much t should am to change ts storage behavour n order to mnmse ts costs. (3) We develop algorthms, based on mathematcal programmng, for agents to optmse ther energy storage, gven the control sgnal they receve, n order to mnmse ther costs, and n so dong, ther carbon emssons. 3 Note that we do not develop novel sensng and predcton algorthms as ths s beyond the scope of ths work. Rather, we assume such predctons are avalable and focus on buldng the control mechansm that uses such predctons. However, as we show n Secton 6, our mechansm s robust to errors n predcton and hence, as better predcton technology s developed, the benefts accrued from usng our mechansm wll only mprove. Future work wll study such predcton algorthms n more detal. 4 Google PowerMeter ( and AlertMe ( systems already utlse home broadband lnks n order to montor energy consumpton over the web. 5 See and ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

5 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A:5 (4) We provde a novel adaptve mechansm for the green suppler to learn the best sgnal to send to the agents n order to maxmse ts effcency. In partcular, the mechansm works even when the suppler s unaware of the proporton of the consumers n the system that possess enough storage to cope wth the varablty n supply (whch cannot be accurately predcted by the suppler). (5) We emprcally evaluate our approach and show that our agent-based homeostatc control wth our novel adaptve mechansm (compared to a tradtonal real-tme prcng mechansm) results n up to 25% mprovement n effcency (whch maxmses the use of green energy by ts pool of consumers and hence reduces ther carbon emssons) for the suppler and up to 14.5% savngs for the agents. Furthermore, we show how errors n the suppler predctng ts green producton can result n hgher costs or carbon emssons. In partcular, we show that the suppler s ncentvsed to accurately predct ts green producton to mnmse ts costs and that our homeostatc control mechansm s not senstve to small predcton errors. The rest of ths paper s structured as follows. Secton 2 presents the general background of our work and outlnes the ratonale behnd our desgn choces. Secton 3 descrbes the model of the green suppler and the agents, each endowed wth ther own preferences and ther battery storage capabltes. Gven ths, n Secton 4, we present our novel carbon-based prcng mechansm. Then, Secton 5 detals our homeostatc control mechansm. Secton 6 evaluates our mechansm and Secton 7 concludes. 2. BACKGROUND AND RELATED WORK The ssue of ntegratng renewable energy supples nto the grd to reduce carbon emssons whle ensurng ther ntermttency does not have an adverse mpact (e.g., by requrng more spnnng reserves or causng brownouts) has been actvely researched for a number of years. The semnal dea of usng homeostatc control to account for ntermttency (for any energy source type) was ntally suggested by Schweppe et al. who foresaw that mprovements n communcaton technologes would pave the way for new control mechansms to coordnate energy supplers wth users n order to account for shortfalls (e.g., by askng them to reduce consumpton) or excesses (e.g., by lettng them consume more) n energy supples n real-tme [Schweppe et al. 1980]. Such mechansms would then allow demand to follow supply. Ther work, however, remaned prelmnary and, as such, untested under smulated settngs. In partcular, they dd not model the ndvdual behavour of agents (n terms of ther basc preferences and how they optmsed ther consumpton) and dd not consder the ssue of carbon emssons to be mportant at the tme. Moreover, they used the frequency of alternatng current (AC) as a vehcle for real-tme prcng. In the absence of other communcaton mechansms, ths choce was probably the only vable one. However, whle frequency s a good ndcator of generaton falng to meet demand, t does not correspond to the cost of electrcty n real-tme electrcty markets [Schweppe et al. 1989]. In contrast, our more up-to-date approach s able to explot recent communcaton and computatonal advances n order to model ndvdual agent behavours and develop prcng mechansms and control sgnals that be communcated n real-tme whle stll matchng the current market condtons. Snce Schweppe s work, most research n the power systems communty has focused on facltatng the ntegraton of ntermttent sources of energy by provdng more relable predctons and control of energy generaton from such sources [Mllgan et al. 2009; Morales et al. 2010] or by matchng renewable energy generators wth hgh-capacty electrcty storage provders (ether attached to wnd turbnes or performng arbtrage n an electrcty market) [Burgo et al. 2009; Lubosny and Balek 2007; Bathurst and Strbac 2003; Makarov et al. 2010; Korpaas et al. 2003]. Hence, most of ths work has focused on the use of large utlty-scale batteres controlled by one user as opposed to small domestc batteres (e.g., electrc cars or water heaters 6 ) where each battery s ndvdually owned and where the owner s only ntent on maxmsng her own preferences. Ths s manly due to the fact that coordnatng a large number of ndvdual users (ownng batteres) wth supplers 6 Note that the storage of hot water or ar s effectvely equvalent to the storage of electrcal energy n some settngs. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

6 A:6 Ramchurn, Vytelngum, Rogers, and Jennngs reles on robust communcatons that were prevously consdered to be too expensve [Krschen and Strbac 2004] and that such storage facltes may be too expensve to mplement on a large scale (.e., across 26M households n the UK). However, wth the advent of hgh-speed (moble) broadband and cheaper battery storage technology (e.g., wth electrc cars such as the Mn E and the Nssan Leaf, and deep-cycle battery technologes from Ceramatec for example), ths argument s no longer vald. Hence, several approaches and feld trals (e.g., the GrdWse Project, the Energy Demand Research Project [Smth 2010] or PowerMatcher 7 ) have emerged whch consder the use of demand-sde management technques (DSM) (based on real-tme prcng usng AC frequency or tme of use prcng) to control demand [Infeld et al. 2007; Hammerstrom et al. 2008]. However, such approaches do not address the use of storage devces to defer the consumpton of electrcty and agan gnore the ndvdual preferences of the users. 8 A number of agent-based approaches that do consder the ncentves of ndvdual actors on the grd have been developed over the last decade. For example, Ygge ntated work on abstractng electrcty markets as mult-commodty markets and showed how agents tradng energy for dfferent tmes of the day, could generate effcent allocatons [Ygge et al. 1999]. Moreover, Jennngs et al. developed coordnaton mechansms for dfferent actors on the grd to manage the allocaton of transmsson capacty [Jennngs et al. 1996] and, more recently, Testfatson et al. have developed agent-based electrcty market smulatons whch factored n transmsson constrants and dfferent types of buyers and sellers n the grd [Sun and Tesfatson 2007; L and Tesfatson 2009]. Kok and Venekamp [2010] take a smlar approach and mplement an agent-based archtecture to run the electrcty market where the ndvdual actors represent ether generators or devces n the home whereas Gerdng et al. [2011] and Chalkadaks et al. adopt a game theoretc approach to managng electrc vehcle chargng and organsng collectves of wnd energy generators respectvely. Moreover, L et al. [] recently proposed a mechansm that coordnates a populaton of agents n optmsng ther loads to avod gong over a gven lmt on demand. Ther agents are desgned to satsfy both the ndvdual s and the system s goals (n cappng demand). However, ther approach requres agents to communcate ther ndvdual plans (to optmse ther resources) to the system, leadng t to be scalable no more than ten thousand agents. In general, none of these works deal specfcally wth the varablty of supply when renewable energy sources are nvolved. Moreover, these approaches do not consder the daly consumpton profle of consumers and how an agent mght optmse ts consumpton to solely maxmse ts owner s beneft (n cost savngs). We beleve ths s crucal because agents have to be proftable to ther users, each wth her own specfc needs and lfestyle, n order to be commercally vable. Moreover, n our recent work, we showed how such agents can be modelled to ndvdually optmse the storage of electrcty gven a real-tme prce and how ths led to demand on the grd beng flattened (wthout explctly exchangng plans among agents) [Vytelngum et al. 2010; Ramchurn et al. 2011]. Whle we dd consder the preferences of the agents, we dd not address the ssue of external nfluences (.e., weather condtons n our case) on the supply of electrcty. Hence, ths paper brdges a major gap n ths area by modellng the agents n the system, each wth ts own preferences, and by provdng the control mechansm to ensure demand follows supply. Our work s also nspred from the recent upsurge n eco-frendly energy provders such as Ecotrcty, 9 Green Mountan Energy, 10 or New England GreenStart 11 that supply ther consumers from ther own green energy sources. Ther success stems from the fact that there s a large porton of consumers ntent on mnmsng ther ndvdual carbon emssons. However, such companes 7 See more detals at and 8 The GrdWse project trals allowed users to set ther preferences for the prce at whch devces should turn off. However, these preferences were assumed fxed throughout the day. However, t was recognsed that users preferences to use certan devces (e.g., to use heatng or hot water) vary across the day and, as a result, they were gven the capablty to overrde ther settngs at any tme ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

7 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A:7 cannot satsfy all ther demand from ther own renewable generators (e.g., Ecotrcty only generated around 41% of the energy they sold n ) and they have to buy the rest from other conventonal sources. Gven ths, the carbon-based prcng scheme we propose n ths paper departs from the fxed prcng scheme provded by these companes, to ncentvse users to mnmse ther carbon emssons n real-tme whle also mnmsng ther costs. Thus, our work sets the stage for new tarffs that can readly be appled once smart meters are rolled out. 12 Furthermore, n future energy markets where carbon emssons may be taxed [Kockar et al. 2009; Gajbhye and Soman 2010], our work on the carbon-based prcng scheme, also reduces the supplers operatonal rsks. The ntuton s that f supplers can pass on some of ther costs, due to carbon taxes, to ther consumers, the latter wll be ncentvsed to algn ther behavour to the needs of the suppler. Thus, f consumers are charged for ther carbon emssons, they wll optmse ther behavour to reduce ther emssons, and hence reduce the tax pad. Proposed approaches to mtgatng supplers carbon costs have focused on balancng the objectve of mnmsng costs and mnmsng carbon emssons as a complex optmsaton problem [Hobbs et al. 2010; Wong et al. 2010; Kockar et al. 2009; Mazer 2007; Harrs 2005]. Alternatve approaches consderng the use of carbon ntensty sgnals, as we do here, are practcally non-exstent snce, we beleve, such technologcal solutons nvolvng the use of nformaton and communcaton technologes, rather than meterng devces on the grd, are less popular among power systems engneers. Hence, ths paper embodes the fundamentals of computatonal sustanablty [Gomes 2009] by provdng computatonal solutons to real-tme electrcty prcng that can be rapdly and relably deployed n the smart grd to generate greater effcences. 3. GREEN SUPPLIER AND AGENT MODELS In ths work, we assume that t wll not be possble, at least n the near future, to be fully relant on green energy sources [DECC 2009b]. Rather, we focus on a more realstc scenaro whch s more representatve of the case today where a green suppler s operatng n a market where there are mostly brown (.e., carbon emttng) sources of electrcty (e.g., n markets run by Natonal Grd 13 or PJM 14 on the US east coast). Specfcally, Fgure 1 depcts the man elements of the system nvolvng the suppler and the consumers. Thus, the grd (consstng of electrcty markets and physcal networks) and weather (e.g., wnd speed or sunshne) act as external nfluences on them. Hence, n the followng sectons, we detal each element of our system and the nformaton flows between them by provdng a generc model of a green energy suppler (typcally wth wnd or solar power generaton capablty) and consumers (represented each by ts agent) whch may possess electrcty storage capacty (e.g., ether batteres or electrc vehcles) and whch are keen to mnmse ther carbon emssons (hence ther contract wth the green suppler). In our models, we assume that a day s dvded nto a set of half-hourly slots T = {1, 2,..., 48} such that each actor needs to decde ts behavour (typcally four hours before or a day ahead) for each slot. Smaller slots (e.g., 5 or 15 mnutes) could be consdered but half-hour slots are used here as ths s a common nterval used wthn electrcty meters and electrcty markets The Green Suppler We model the green suppler (e.g., wth wnd turbnes and/or solar power generators) as havng a pool of consumers that subscrbe to t for electrcty. 15 The suppler can predct, wth a reasonable 12 The UK government has planned to equp all 26M houses n the country wth a smart meter by 2020 whle n the US t s expected that about 60M houses wll be equpped by the same date. Smlar targets are n place n Europe wth Italy havng already completed ts smart meter deployment Our model consders only one green suppler operatng n the grd where other possbly non-green supplers also operate. If we were to consder a grd populated by mostly green supplers, then each green suppler would have to consder the possblty of not beng able to supply ts electrcty from the grd at tmes when none of the others are able to generate ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

8 A:8 Ramchurn, Vytelngum, Rogers, and Jennngs Suppler Prce Plan Prces Consumer demand va smart meter Carbon Intensty Consumers Homes wth dfferent or no batteres Electrcty Generaton Capacty Renewable energy Grd Generaton Transmsson Dstrbuton Markets Electrcty flow Informaton flow Fg. 1. Here we llustrate the electrcty and nformaton flows between the actors n the system we consder. Specfcally, our model specfes the prce plan used by the suppler, the heterogeneous storage devces used by the consumers as well as the decson makng nvolved n chargng or usng ther battery. We also assume that the grd provdes the carbon ntensty sgnals whch we use n our mechansm but do not consder how the electrcty s transmtted and delvered to consumers by the grd. level of accuracy (usually four hours ahead usng sensors and hstorcal data [Mllgan et al. 2009]), that t wll generate a gven amount of electrcty q R +, T. Now, n lne wth today s realty we model our suppler as beng able to satsfy only part of ts consumers demand, acqurng the remander from the wholesale electrcty market. Whle the avalablty of supply from the grd s guaranteed, ts prce s generally drven by the market demand and supply (see the relatonshp between prce and demand n Fgure 2(a)). In partcular, dependng on ts energy producton pattern, the suppler has a number of potental ways of prcng ts electrcty n real-tme based on what t expects the wholesale spot market prces to be and any exstng future and forward contracts that t holds. Thus, whle spot prces determne the suppler s real-tme tradng strategy, forward and futures contracts am to hedge the rsk of very hgh peak prces n the spot market. Typcally, buyng extra capacty on the spot market s more costly than n a forward contract and sellng extra capacty s usually done wth a very low or possbly negatve proft [Harrs 2005]. Hence, the green suppler needs to ensure that t mnmses the amount of extra capacty t needs to buy n spot markets by gettng ts consumers to follow ts producton pattern. Moreover, a key challenge here s to algn the goals of maxmsng proft whle meetng the goals of ts customers to mnmse ther carbon emssons. Hence, we address these ssues through a combnaton of prcng and control as we show n Sectons 4 and The Consumer Agents Here, we descrbe our model of the consumer agent, whch bulds upon and extends a recent model for homes equpped wth smart meters [Vytelngum et al. 2010]. Thus, we consder a set of consumers ndvdually represented by ther agent, where the set of agents s A and where each agent a Ahas a load profle l a T, (.e., ts owner s predcted consumpton of electrcty) such that l a s the amount of electrcty requred by agent a for tme nterval durng each day. Ths electrcty can ether be bought from the suppler or retreved from a storage devce. Thus, each agent a A may possess an electrcty storage devce wth capacty e a R +, effcency α a [0, 1], and runnng costs c a R +. Here, the cost c a represents mantenance costs or the loss n value of the devce suffcent electrcty. Ths would mean addng a hard constrant on the consumers demand and/or prce very expensvely any consumpton above the level provded by the grd. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

9 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A:9 through use over tme. We assume ths cost s drectly proportonal to the sze of the devce, such that f the sze of the battery s e a then the storage cost s c a e a. The storage effcency α a models the fact that f q kwh amount of energy s stored, then only α a q kwh may be subsequently used. The objectve of each agent a s to mnmse ts owner s cost and hence, t wll optmse ts storage profle, b a Iwhere ba b a b a +, where b a s the dschargng capacty of the storage (as lmted by the effcency and capacty of the battery), and b a +, the chargng capacty. Snce the dschargng effcency of the storage s regmented by α a, we separate out the chargng and dschargng profle such that for all T we have b a = ba+ b a, where b a+ s the chargng profle and b a, the dschargng profle. Then, we denote as l a = la + ba the total consumpton of an agent at tme and the aggregate consumpton of all agents as l A = a A la. In the next secton we descrbe how consumers are charged n real-tme for ther electrcty and elaborate on the technology to acheve ths. 4. THE CARBON-BASED PRICING MECHANISM To communcate electrcty prces, we develop a novel carbon-based prcng scheme that the suppler sends to ts customers. The scheme reles on the avalablty of a real-tme carbon ntensty sgnal (measured n gco 2 /kwh and representng the amount of CO 2 emtted for every unt of energy consumed) that s broadcast on the nternet 16 by natonal grd operators (as n the UK and Ireland). 17 The key feature of our prcng scheme s that the users are guaranteed to pay for the greenest form of energy at all tmes f they optmse ther consumpton pattern (to mnmse ether costs or carbon emssons). Whle such a scheme allows the suppler to pass on some of the costs for operatng n carbon-taxed electrcty markets [Hobbs et al. 2010; Kockar et al. 2009], t also follows from the smple fact that consumers usually subscrbe to green supplers n order to reduce ther carbon emssons and t would be smply nconsstent f any extra energy that the suppler cannot provde would be suppled by the worst carbon emtters for the same prce. The prcng scheme we develop explots the correlaton between prce and grd demand (shown n Fgure 2(a)) and between carbon ntensty and grd demand (shown n Fgure 2(b)) (based on real UK data). Usng ths data, we can nfer the correlaton between prce and carbon ntensty as shown n Fgure 2(c). Now, as wth other examples of prcng mechansms (e.g., fxed prcng, tme-of-use, or real-tme prcng), the retal prces must be calculated to reflect the suppler s retal margn proft and ts exposure to the rsk of peak prces n the wholesale market. 18 However ther complexty must be balanced aganst the need to provde a smple prcng scheme that would appeal to, and be understood by, the average consumer. Thus, we provde retal rates for dfferent bands of carbon emsson (n our case, Band 1 to Band 5) 19 rather than provdng the consumer wth a complex equaton-mappng between carbon and prce. More formally, these bands are computed as follows: 16 For example, the GrdCarbon Phone/Androd app uses nformaton provded by Natonal Grd n the UK to provde realtme updates on the generaton mx (.e., the percentage of energy generated by dfferent types of plant). Usng the latter and a functon mappng generaton source to carbon ntensty, the real-tme aggregate carbon ntensty s provded. For more detals see 17 The UK carbon ntensty can be computed from the generaton mx data provded by Elexon ( and s provded on a moble platform through the GrdCarbon app. Ireland s real-tme carbon ntensty can be found at The typcal carbon ntensty of the UK grd tends to be around 500gC0 2 /kwh (and range between 550 and 450 across a typcal day) and that of Ireland around 225 kgco 2 /kwh (and range between 280 and 200). 18 Whle we descrbe the prcng scheme here, the methodology to calculate the retal prces s beyond the scope of ths paper (and not crtcal n our work) as t s very dependent on the dfferent formats of futures and forward contracts that dfferent supplers would settle for. For the purpose of ths work, we set the band prces based on a retal margn and consderng spot prces of the wholesale market. The suppler would be requred to calculate the retal prces for dfferent carbon emssons on a month-ahead or even year-ahead bass, such that these rates would be fxed for long perods at a tme and perodcally revewed to match changng grd prces and aggregate consumers demand. 19 We choose fve bands only as a representatve set of bands that generally covers the trend we dentfy n the correlaton between carbon ntensty and demand. Whle more bands would result n a more accurate reflecton of the costs for the suppler, they would also be more complex for the consumers to understand we beleve. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

10 A:10 Ramchurn, Vytelngum, Rogers, and Jennngs (1) We splt the users aggregate consumpton n terms of the amount satsfed by the green suppler and the amount drawn from other grd sources such that l A = l A,green + l A,grd at tme T. Thus, l A,green R + s the load from the consumers that s completely satsfed by the green suppler from ts own renewable source such that l A,green q (remember q s the producton capacty of the suppler) whle l A,grd R + s suppled from other (possbly carbon emttng) sources on the grd. (2) l A,grd s prced by our carbon-based prcng scheme (based on fve dfferent bands) as p grd n where n {1, 2, 3, 4, 5} and p grd n < p grd m f n < m. We choose n based on the carbon ntensty sgnal ρ R + (kgco 2 /kwh) at tme, that s, p grd n s chosen f ρ mn n ρ ρ max n where ρ mn n and ρ max n are the upper and lower bounds of the band respectvely. For example, accordng to Fgure 2(c), p grd 1 =29 /MWh for 0 <ρ 275 and p grd 2 =41 /MWh for 275 <ρ 325. (3) l A,green (as a Band 0),.e., the amount consumed that s completely satsfed by the green suppler from ts own sources, s prced at ɛ less than the greenest band, Band 1 of the grd as p green = p grd 1 ɛ. Through the above prcng scheme the suppler ncentvses consumers to use the green energy (snce t always cheaper) t produces rather than grd energy. It s mportant to note that our carbon-based prcng scheme s not specfc to a green suppler and could be used by all supplers n general as the sgnals we used apply to the whole grd. Also, note that our approach s dfferent from typcal sldng scale or band prcng (where consumers smply pay dfferent rates for dfferent amounts consumed) n that, n our model, the band prce (p/kwh) appled to a consumer s usage s ndependent of the energy consumed by the consumer and, nstead, dependent on the real-tme carbon-ntensty of the grd (ncentvsng consumers to consume less when the grd has a hgh carbon ntensty). Buldng upon our prcng mechansm, we next develop an agent-based homeostatc control mechansm that provdes addtonal sgnals to the agents to ensure demand follows supply. 5. HOMEOSTATIC CONTROL Gven the varablty n weather condtons affectng the generaton capacty, the system, comprsng of the agents and the suppler, needs to contnuously adjust to these condtons to maxmse ts effcency. Ths needs to happen gven that both the suppler and the consumers am to maxmse ther ndvdual profts. More mportantly, they can only do so f they communcate to ensure that the aggregate demand from all the consumers s as close as possble to the real-tme supply from the producer, whle ensurng that any extra energy needed s bought at tmes when the carbon ntensty of the grd s the lowest. Only by dong so can all of the energy the suppler produces be used (whch ensures that both the suppler makes a proft and the users get cheap and greener electrcty) and that any extra energy needed s bought at the cheapest prce from the grd. Whle the sensng and predcton of short-term weather condtons can be easly mplemented usng exstng machne learnng technques [Alpaydn 2004], sgnfcant challenges le n determnng the approprate control sgnal to be sent to the agents to ensure that ther behavour n reacton to such a sgnal meets the above requrements. Moreover, the suppler may have to adapt the sgnals t sends based on the feedback (.e., consumpton levels at dfferent tmes of the day) t receves. In ths paper, we address these challenges and detal the dfferent elements of our soluton (graphcally represented n Fgure 3), n the followng subsectons The Sgnal to the Agents The nformaton that s typcally provded by the suppler to an agent s the prce of electrcty whether t s a fxed, real-tme or of our carbon-based type. Usng ths nformaton, an agent can then manage ts demand n order to both mnmse ts costs and also ts carbon emssons. Whle ths ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

11 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A:11 Wholesale UK Spot Prce (GBP/MWh) Market Demand and Supply Prce aganst Demand Best Ft Total Demand (MW) x 10 4 (a) The correlaton between UK spot prce and grd demand UK Grd Carbon Intensty aganst demand Carbon Intensty agant Demand Best Ft (lnear trend) Carbon Intensty (gco2/kwh) Total Demand (MW) x 10 4 (b) The correlaton between carbon ntensty and grd demand. Wholesale UK Spot Prce (GBP/MWh) Carbon based Prcng Scheme Carbon Intensty aganst Prce Carbon band Prces Carbon Intensty (gco2/kwh) (c) Prcng scheme obtaned from combnng the correlaton between prce and demand, and between carbon ntensty and demand. Fg. 2. Relatonshps between carbon ntensty and the prce of electrcty aganst demand on the balancng market (Source: NETA, UK August-September 2009). ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

12 A:12 Ramchurn, Vytelngum, Rogers, and Jennngs Grd Generator q Suppler Adaptve Mechansm γ t γ t+1 Carbon ntensty ρ Consumers Each agent a mnmses p green usng b a l a,green wth la,green + p grd l a,carbon γ t+1 l a l A Aggregate Demand l A = a A la + ba Fg. 3. Flow of nformaton n the system nvolvng the suppler and the consumers connected to the grd. Buldng on the system presented n Fgure 1, consumers receve two sgnals; a carbon ntensty sgnal ρ from the grd (whch they use to predct the next day s carbon ntensty) and (n addton to the prce plan shown n Fgure 1) γ t+1 (for the next day) computed by the suppler usng q (.e., ts producton for the current day) and l A (.e., the overall consumpton for the current day). The consumers then optmse ther electrcty usage gven the sgnal γ t+1 and usng ther battery. The suppler s nformed of the amount consumed by all ts customers (.e., l A ), hence closng the loop. Note that we do not show the payments the suppler receves based on splttng the consumpton of ts customers n terms of ther green and non-green components (.e., l A = l A,green + l A,grd ). approach s sutable for most supplers, t s neffcent for the green suppler wth ts own renewable energy generaton capacty. Specfcally, due to the ntermttent nature of ths generaton capacty, the suppler needs to sgnal to ts pool of consumers about ts generaton pattern so as to maxmse the use of ts green energy, thereby mnmsng the carbon emssons of ts consumers. Now, a prce sgnal would be an neffectve stmulus as t would provde no nformaton about how much the agents need to ncrease ther demand durng any partcular 30-mnute perod to maxmse the use of green energy (as dscussed n Secton 1). Thus, we propose two new sgnals (see Fgure 3) that address ths shortcomng. Frst, consumers are gven a grd carbon ntensty sgnal. Ths sgnal may be provded ether by the grd operators (over the nternet) or by the suppler (collectng the data from grd operators) through the meter (the band prces are agreed wth the suppler well before). The sgnal allows the consumer to compute ts cost of electrcty based on our prcng mechansm outlned earler on. Second, the suppler provdes the consumer wth a target γ R + T to ncrease (γ > 1) or decrease (γ < 1) ts consumpton wth respect to the prevous day s consumpton at tme such that demand follows supply n the day ahead. Ths target γ s based on the aggregate consumpton at tme nterval gven by l A and the producton from the suppler q, and s gven as γ = q. In Secton 5.3 we wll show how to adapt ths sgnal to consder the l A feedback that the suppler gets from the populaton of agents. For now, however, assumng that all consumers n the system react to ths sgnal (by storng electrcty when most ncentvsed to do so), we expect the aggregate demand to approach ths target, thus maxmsng green energy usage and mnmsng the carbon emssons of the consumers. Thus, f agents meet ths target exactly, they are expected to pay only p green for every unt of electrcty used. However, f an agent overuses (above the target), t ncreases the expected cost of electrcty as the suppler would need to acqure the addtonal demand at p grd prces. On the other hand, f t underuses (below the target), the agent msses out on cheaper and greener electrcty. Next, we present an optmsaton model for an agent that ams to mnmse ts costs gven ts load and storage avalablty (see Fgure 3). In partcular, we consder that the agent obtans part of ts energy demand from the green suppler descrbed n Secton 3.1 and s already contracted to pay p green for every kwh provded by ts suppler from ts own sources (wnd or sun). Then, the agent receves two sgnals from the suppler, that s p grd, computed based on the carbon-prcng scheme agreed beforehand and the real-tme carbon ntensty of the grd, and γ t+1 [0, 1]. We assume that ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

13 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A:13 the agent obtans these sgnals a day ahead (but ths could equally also be a few hours ahead as s commonly the case for wnd generators whch can predct reasonably accurately how much they are gong to be able to provde n such tme frames [Mllgan et al. 2009]). Thus, the agent would need to re-optmse ts behavour whenever t receves updates of γ The Agents Behavour Snce we assume that an agent s daly consumpton (load) profle s fxed (.e., prce does not affect the loads n a sngle tme slot but can affect stored or dscharged energy), an agent a can only try to mnmse ts costs by storng energy when prces are low and usng as much of that energy as possble when prces are hgh. Thus, we consder agent a havng to plan ts storage for the day ahead, that s t +1,ft s the current day. As opposed our prevous model [Vytelngum et al. 2010], 20 the agent does not need to predct the next day s prce for each tme slot. Rather, we assume the agent predcts the next day s carbon ntensty ρ for each half-hourly nterval of the day and usng the prce plan proposed n Secton 3.1, computes a prce p grd (snce p green s known to all agents a pror). Gven the uncertanty n ts own consumpton for the next day, the agent also needs to predct l a,t+1 whch s ts own expected total load at tme for the next day and here we assume that l a,t+1 = 1 t k t k la,t, that s, a movng average over k prevous (smlar) days consumpton (we choose k =3). 21 Now, the agent also has to decde how to adjust ts consumpton for ndvdual tme slots based on the sgnal γ sent by the suppler. Snce γ t+1 mples a factor change n the current day s overall consumpton l a,t, the best an agent can do s to try and bound the total amount of energy t consumes at by γ t+1 l a,t. By so dong, the agent can maxmse the probablty that t wll pay the best prce p green, as offered by the suppler (as explaned n Secton 3.1). Our soluton to ths optmsaton problem s formulated as a mathematcal program as gven n Fgure 4. The output s the storage profle, b a = b a+ b a at every tme-slot durng the day. The objectve of the agent s captured by the mnmsaton functon n Equaton (1). 22 Then Constrant 1 expresses the fact that the total load of the agent at any pont n tme s the sum of the amount t stores and the amount t consumes. Constrants 2 and 3 decouple the amount consumed by the agent n terms of the green and non-green components snce the green component (.e., l a,green ) s lmted by the suppler accordng to ts sgnal γ through γ l a,t (n Constrant 3). Constrants 4, 5, and 6 restrct the flow n the battery to ensure t respects the battery s effcency lmtatons (as lmted by α a ), that the amount (ds)charged n any tme slot respects the lmts b a and b a+, and that the energy that can be stored or used at any pont durng the day respects the state of the battery at that pont n the day. Fnally, Constrant 7 prevents the agent from smply storng electrcty to sell back to the grd. As can be seen, the mathematcal program only has lnear constrants and a lnear objectve functon. Hence, lnear programmng solvers (e.g., CPLEX or glpk) can be drectly 20 The model presented here s also dfferent from our prevous work n that the aggregate demand of the populaton of agents does not nfluence prces or carbon ntensty values on the grd. Ths s due to the fact that the populaton of agents ted to the green suppler s consdered to be too small to nfluence the grd s overall performance. 21 Consumpton on average tends to be smlar durng weekdays but weekday consumpton s typcally dfferent from week end consumpton. Moreover, load profles tend to change across seasons and for specal days (e.g., Chrstmas or for specal sportng events). Hence, agents would have to take these dfferences nto account n ther predcton f such a system were deployed. 22 The storage optmsaton model we present here tres to capture the key features of the storage devce and the load profle of the consumers. Whle other constrants could be added (e.g., a bound on the cost the consumer s wllng to pay, or the maxmum energy allowed to be charged n any tme slot to prevent the battery from overheatng), we beleve we have captured the man ones and have valdated ths wth our ndustral partners. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

14 A:14 Ramchurn, Vytelngum, Rogers, and Jennngs where l carbon mnmse I R +, subject to: Constrant 1: aggregate load l a,t+1 p green l a,green = l a,t+1 + b a+ + p grd l a,grd + c a e a (1) b a, I Constrant 2: separate usage nto green and non-green l a,t+1 = l a,green + l a,grd Constrant 3: restrct green part of consumpton 0 l a,green γ t+1 l a,t Constrant 4: storage effcency b a = α a b a+ I I Constrant 5: wthn chargng and dschargng capacty b a b a and ba+ b a + I Constrant 6: energy that can be stored or used at a tme-slot ( b a α a e a ( j=1 b a+ j b a ) ) j, I b a+ e a e a ( j=1 b a+ j b a ) j, I Constrant 7: no-resellng allowed l a b a 0, I Fg. 4. Mathematcal program used by an agent to fnd ts optmal storage strategy. appled to fnd the soluton. In our mplementaton, we found that the optmal soluton s obtaned wthn a few mllseconds. 23 Note that the last constrant can be removed n a system where consumers are allowed to sell power back to the grd and that e a can be fxed or unconstraned. In lne wth our model (see Secton 3), c a s the relatvely small dscounted runnng cost of usng storage, e a s the storage capacty, α a s the effcency of the agent s storage, b a + s ts maxmum chargng and ba ts maxmum dschargng rates and e a 0 s the storage at the begnnng of the day whch equals the storage at the end of the day (.e., chargng at the end of a day for the next day). Now, not all agents may possess storage capacty (or some may not have enough storage to cover ther whole consumpton) and ths s not known to the suppler a pror. The effect of ths can be qute dramatc (as we shall show n Secton 6). In partcular, wth lmted storage, agents are less able to react to the varablty n supply (communcated through the γ sgnals). Ths, n turn, can result n the suppler ncorrectly estmatng the outcome of aggregate consumers response to ts sgnals, leadng to greater demand from the grd at peak tmes and hence hgher prces. In the next subsecton, we present a novel adaptve mechansm that allows a suppler to determne the approprate γ to be sent to a populaton of agents that may have lmted storage. 23 We used IBM s ILOG CPLEX 12.2 to mplement the lnear programmng model. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

15 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A: The Adaptve Mechansm The am of the adaptve mechansm s to learn the optmal sgnal to send to the agents (see Fgure 3) (gven ther response to a prevously sent sgnal). Now, f the suppler were to have complete and perfect nformaton about the system (.e., the proporton of consumers wth demand-sde management capabltes that can use or store energy n reacton to a suppler s sgnal) and complete control of the system (.e., control the capabltes owned by ts consumers), t would be able to defne (and communcate) exactly the storage profle of each consumer to mnmse ts total cost. The same optmsaton model defned n Equaton 1 could be used by aggregatng the chargng and dschargng capabltes of all consumers and ther loads as follows: mn p green l a,green + p grd l a,grd + c a e a (2) a A I subject to Constrants 1 to 7 as n Fgure 4. In so dong, we can fnd an optmal soluton to the behavours of consumers that wll result n the most effcent use of green energy. Now, n a more realstc settng, the only feedback that the suppler obtans from ts consumers s ther total load on the grd, that s l + b (as shown n Fgure 3). Assumng that the suppler can accurately estmate the average load of each consumer l a, a smplstc approach to set the control sgnal γ t+1 would be: γ t+1 = qt (3) a A la Ths s smply the rato between the total green energy from the suppler q t and the total load from the agents at tme slot. The am here would be to tell the agents to adjust ther loads (usng the battery b a ) so that γ deally converges to 1 (.e., demand followng supply). However, ths approach assumes that all consumers have storage capablty, as the suppler s sgnallng to each one of ts consumers to ncrease ts demand by a porton of the green energy produced. Ideally, the suppler should do so only for those consumers that are able to react to such a sgnal (.e. those wth storage capablty). However, such nformaton s not readly avalable to the suppler as consumers do not have to and may not want to reveal such nformaton to ther suppler (because they may not trust t to use ths nformaton to ther beneft). Hence, the challenge s to adapt γ usng only the feedback from the agents. We present our soluton to ths problem n Fgure 5. γ t+1 = γ t + β S ( max qt l A,t 0,γ t β S qt l A,t ), f q t >la,t, otherwse where β S [0, 1] s the learnng rate Fg. 5. The adaptve mechansm gven a chosen learnng rate β S and nputs q t and la,t. In more detal, we propose a novel adaptve verson of γ based on current aggregate demand and the suppler s green energy producton. As can be seen, usng the adaptve sgnal, the homeostatc control mechansm tres to mnmse the dfference between green energy produced and current total demand based on a learnng rate β S whch determnes the rate at whch t wll change γ to reflect the feedback t receves. Ths rate s crtcal n determnng whether the mechansm converges to the optmal soluton (as we wll see n Secton 6). Thus, when the green energy produced s hgher ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

16 A:16 Ramchurn, Vytelngum, Rogers, and Jennngs than the total demand, t mples an opportunty to maxmse the use of green energy and reduce the carbon emssons of the consumers s lost and therefore γ s compensated postvely (.e., top condton n Fgure 5) to ncrease the total demand of all the consumers. However when the total demand s too hgh, the suppler sgnals ts consumers to reduce ther use of green energy (.e., bottom condton n Fgure 5) at tme (and ensurng that γ s never negatve). As we emprcally demonstrate n Secton 6, the daly total demand gradually changes such that the effcency of green energy usage converges to the optmal (defned by the optmal soluton of Equaton (2)). In the next secton, we evaluate our homeostatc control mechansm for dfferent proportons of the populaton wth storage capablty and demonstrate how our adaptve system remans robust (.e., keeps costs low and effcency of green energy usage hgh) even when the error wth whch the suppler predcts hs own supply of green energy (.e., q ) ncreases. 6. EVALUATION The am of ths evaluaton s to emprcally demonstrate the effectveness of the homeostatc control mechansm presented n the prevous secton n nducng demand to follow supply such that the carbon emssons of the system s reduced. We adopt an emprcal approach here, as opposed to an analytcal approach seekng equlbrum charactersatons, n order to evaluate the system under a wder varety of settngs than would be feasble wthn a theoretcal framework, 24 and predct equlbrum outcomes (for dfferently parametersed versons of our mechansm) wthout overly restrctve assumptons. As we wll see, despte the utlty-maxmsng behavour of consumers wth storage capablty, the system quckly converges to the optmal behavour. In ths context, we defne optmal behavour as the soluton to Equaton (2) as dscussed n Secton 5.3. In what follows, we frst detal our expermental setup and then go on to evaluate the effcency of our homeostatc control mechansm (aganst the optmal behavour) n terms of green energy use and costs of electrcty, n partcular for dfferent proportons of the populaton wth storage capablty. We focus on the proportons wth storage capacty to dentfy the crtcal amount of storage needed n the populaton to acheve the maxmum returns on nvestment n batteres for ndvdual users and for the populaton as a whole. We also demonstrate how the suppler s adaptve behavour sgnfcantly mproves the system. Fnally, we analyse the senstvty of our mechansm aganst predcton (of green energy producton) errors Expermental Setup We smulate a pool of 5000 consumers 26 subscrbed to a green suppler that s able to supply 50% of ts demand from ts own renewable sources and the remander bought from the grd. Our choce of the proporton of green energy was desgned to model one of the most popular green supplers n UK, namely Ecotrcty whch produced 41% of the electrcty t supples to ts consumers from ts own renewable sources n and expects to reach 50% wthn the year. Electrcty s prced usng the carbon-based prcng scheme detaled n Secton 4. Furthermore, we model the generaton capacty as values drawn from a unform dstrbuton (between 0 and 1) for each of the 30-mn tme slots. Each value s averaged usng a dscrete Gaussan flter (.e., usng normally dstrbuted weghts wth mean 0 and standard devaton of 1 to model the effect of 24 We acknowledge that a theoretcal analyss of our mechansm would allow us to ascertan the propertes of our control mechansm and, snce ths goes well beyond the scope of ths paper, we am to do such an analyss n future work. 25 Apart from the model presented by [Vytelngum et al. 2010], we are not aware of any other mcro-storage optmsaton models that have been devsed to respond to real-tme prces. Snce Vytelngum et al. s model dd not consder the control sgnal we use n our case, we benchmark aganst t as a system where homeostatc control s not used. 26 Larger populaton szes do not show any sgnfcant dfferences to the nsghts we provde here. Specfcally, the results wth dfferent populaton szes showed that the system becomes more effcent as the populaton sze ncreases. We also expermented wth real data from a populaton of about 1000 users n the south west of England and confrmed that smlar results were obtaned to the smulaton provded the optmsaton carred out by agents was carred out more frequently (on a half-hourly bass) to take nto account the spky nature of ther consumpton (as opposed to the smooth cycles smulated n our experments whch allow a daly optmsaton to work). However, due to UK data protecton ssues, such results cannot be reproduced here. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

17 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A:17 gradually changng wnd patterns by removng any hgh-frequency changes) over the vector of 48 tme-slots across the day. The values of the tme-slots are then lnearly scaled such that the sum total of these values equals 50% of the daly demand of the consumers. Furthermore, we set the lmt for the prces bands at ρ mn = 0, 275, 325, 375, 425, ρ max = 275, 325, 375, 425, and p grd = 29, 41, 51, 59, 71 assumng a 10% proft margn for the suppler. Fnally, the demand of each consumer s modelled on the real UK data profle of the average user on the Domestc Unconstraned Tarff (.e., the typcal UK average profle before any demand sde management technologes, such as storage, are used [Vytelngum et al. 2010]) and the smulaton s run 500 tmes, each run lastng 100 days. These values were chosen to ensure the statstcal sgnfcance of our analyss. 27 Next, we assume that only a proporton of the populaton has storage capablty (.e., they own storage devces capable of storng 4kWh wth chargng and dschargng effcency unformly dstrbuted between 0.3kW and 0.5kW to model current storage devces [Vytelngum et al. 2010]). The consumers wth storage capabltes are thus able to optmse ther storage profle to reduce ther costs (and mplctly reduce ther carbon emssons as a result of our carbon-based prcng scheme). Gven a control sgnal from the suppler, we emprcally demonstrate n ths secton how the system effcency mproves n terms of carbon emssons (.e., by mnmsng energy used from the grd and maxmsng green energy use) Effcency of Homeostatc Control Frst, we analyse the daly effect of homeostatc control on the system.we do so by comparng the green energy usage effcency (.e., the percentage of green energy that s drectly used by the consumers contracted to the suppler) and the cost of electrcty used (based on the carbon-based prcng scheme and the Band 0 prcng of the suppler) n a system wth and wthout homeostatc control. Furthermore, we also demonstrate the effectveness of our adaptve mechansm and how t allows the system to converge to an optmal soluton (gven n Subsecton 5.3). From Fgure 6, gven a populaton wth 50% havng storage capablty, we can see a daly convergence of our homeostatc control system wth an adaptve mechansm, for dfferent learnng rates β S, to the optmal soluton at 91.8%. As can be seen, the non-adaptve verson s not very effcent (.e., only 88.3%) as t overestmates the proporton of the populaton wth storage and understates γ that represents the amount of green energy avalable to consumers wth storage. When the suppler adopts the adaptve mechansm usng a hgher learnng rate (.e., β S =0.05), ths leads to a faster, though less smooth, convergence as opposed to a slower, but smoother convergence to the lower learnng rate (.e., β S =0.005). In general, ths demonstrates that the emergent behavour of ndvdually utlty-maxmsng agents s ndeed effcent such that the system maxmses ts use of green energy, avodng havng to acqure electrcty from the grd as the consumers change ther demand profles to follow green supply. Also, an effcent outcome means that the consumers have lower carbon emssons overall snce they maxmse ther use of renewable energy (and thus, mnmse ther use of grd energy). Because the objectves of reducng carbon emssons and mnmsng costs are algned through the carbon-based prcng scheme, the selfsh utlty-maxmsng behavour of a consumer allows t to be as green as t could be. Next, we analyse the system at an equlbrum state (.e., once the system has converged after a number of days), for dfferent proportons of the populaton that have storage capablty Effect of the Populaton Sze wth Storage Capablty Based on the prevous experment, here we set the learnng rate at β S =0.005 (to ensure convergence) for our homeostatc control mechansm and vary the proporton of the populaton wth storage capablty from 0 to 1. Fgure 7(a) shows the green energy usage effcency (as a percentage 27 Usng multple samples of the results, we valdated our conclusons usng a t-test wth α =0.05 and found that there was no sgnfcant dfference between the means of the samples. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

18 A:18 Ramchurn, Vytelngum, Rogers, and Jennngs % Green Producton and 50% of populaton wth storage 0.92 Effcency of green energy usage Optmal control Wth homeostatc control (non adaptve) Wth homeostatc control (adaptve, β S =0.005) Wth homeostatc control (adaptve, β S =0.05) Wthout homeostatc control Smulaton tme (days) Fg. 6. Effcency of green energy usage for the system wth and wthout the adaptve mechansm. The system wth no homeostatc control s only 79.5% effcent throughout. of the generaton capacty) for dfferent storage penetraton n the populaton (at the end of 100 smulaton days). Frst, we observe that when too many consumers adopt storage, the system wth no homeostatc control wastes more green energy (when the proporton s greater than 0.6 n partcular). Ths s because too many consumers are blndly optmsng ther demands based only on market prces, unaware of the green producton, wth effcency decreasng to 72%. Now, when gven a sgnal about green producton (as n the homeostatc control mechansms), the consumers are sgnfcantly greener as they are optmsng ther demand based on how much and when green energy s avalable to them. Wth no storage n the system (.e., no storage capablty), green energy usage effcency s the same wth and wthout homeostatc control as a sgnal s then redundant. As storage penetraton ncreases, the homeostatc control system becomes greener (havng to use less electrcty from the grd and more of the suppler s green energy). When storage penetraton s at ts hghest, we can see how the effect of homeostatc control peaks wth up to a 25% ncrease n green energy usage effcency (.e., a 25% decrease n carbon emssons) compared to wthout homeostatc control. In addton, Fgure 7(b) (log-scale) also compares the system, at equlbrum, wth and wthout homeostatc control aganst the optmal soluton (.e., we are comparng the relatve error between the optmal soluton and where the system converges to). We can see that our adaptve mechansm allows the system to converge to wthn 0.5% of the optmal soluton as opposed to up to 4.4% wthout an adaptve mechansm and up to 25% wthout homeostatc control. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

19 Agent-Based Homeostatc Control for Green Energy n the Smart Grd A: % Green Producton 0.95 Effcency of green energy usage Wthout homeostatc control Wth homeostatc control (non adaptve) Wth homeostatc control (adaptve) Proporton of populaton wth storage (a) % Green Producton Relatve devaton from optmal soluton (log scale) Wthout homeostatc control Wth homeostatc control (non adaptve) Wth homeostatc control (adaptve) Proporton of populaton wth storage (b) Fg. 7. Green energy usage effcency wth and wthout homeostatc control for dfferent proportons of the populaton wth storage. The relatve devaton from the optmal soluton (as computed n Equaton (2)) s gven n (b). ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.

20 A:20 Ramchurn, Vytelngum, Rogers, and Jennngs Next, we nvestgate the effect of dfferent penetratons of storage n the populaton on the cost of electrcty The Cost of Electrcty Here we use our carbon-based prcng scheme to evaluate the cost of electrcty gven that dfferent proportons of the populaton are able to utlse the full capacty of the green suppler. We consder cases where the cost of storage s added to the overall costs of the consumer or not. Thus, f cost of storage s fully subsdsed, we observe from Fgure 8 the decreasng trend whch results from more consumers beng able to change ther demand to use green and cheaper energy. Furthermore, the cost wthout homeostatc control s margnally hgher than wth homeostatc control. The ndvdual savngs of a consumer wth storage are maxmsed when the demand response of the populaton s maxmsed. At ths pont, t results n up to 14.5% lower costs because t can optmse ts demand for greener and cheaper-prced electrcty. Conversely, when there s no storage n the system, a sngle consumer decdng whether to adopt storage would be able to make 7.5% savngs even f she were the only one to have storage n the system. Indeed, ths ponts to a sgnfcant ncentve for consumers to adopt storage. Moreover, because the payoff of ownng storage (rather than not) s always hgher, t s lkely that all consumers wll eventually acqure storage devces and captalse on the suppler s control sgnal of green energy. 28 Daly average electrcty cost for household wth storage ( ) % 3.1% 50% Green Producton 14.5% 1.05 Wthout homeostatc control Wth homeostatc control (storage cost = 0p/kWh) Wth homeostatc control (storage cost = 6p/kWh) Proporton of populaton wth storage 9.8% Fg. 8. Average cost of electrcty for storage-capable consumers wth and wthout homeostatc control for dfferent proportons of the populaton wth storage. 28 Note that the settng we analyse here s dfferent from Vytelngum et al. s [2010] where they descrbe how eventually the system converges to an equlbrum where only 38% requre storage. In our context, we are lookng at a sngle suppler and ts consumers (wth changes n total demand profle small enough to have no effect on wholesale market prces), whle they consder the whole grd where changes n market demand affect wholesale market prces. ACM Transactons on Intellgent Systems and Technology, Vol. V, No. N, Artcle A, Publcaton date: January 2011.