Imiaive Learning for Online Planning in Microgrids Samy Aiahar 1(B), Vincen François-Lave 1, Sefan Lodeweyckx 2, Damien Erns 1, and Raphael Foneneau 1 1 Deparmen of Elecrical Engineering and Compuer Science, Universiy of Liège, Liège, Belgium saiahar@suden.ulg.ac.be, {v.francois,derns,raphael.foneneau}@ulg.ac.be hp://www.monefiore.ulg.ac.be 2 Enervalis, Greenville Campus, Greenville, Belgium sefan.lodeweyckx@enervalis.com hp://www.enervalis.com Absrac. This paper aims o design an algorihm dedicaed o operaional planning for microgrids in he challenging case where he scenarios of producion and consumpion are no known in advance. Using exper knowledge obained from solving a family of linear programs, we build a learning se for raining a decision-making agen. The empirical performances in erms of Levelized Energy Cos (LEC) of he obained agen are compared o he exper performances obained in he case where he scenarios are known in advance. Preliminary resuls are promising. Keywords: Machine learning Planning Imiaive learning Microgrids 1 Inroducion Nowadays, elecriciy is disribued among consumers by complex and large elecrical neworks, supplied by convenional power plans. However, due o he drop in he price of phoovolaic panels (PV) over he las years, a business case has appeared for decenralized energy producion. In such a conex, microgrids ha are small-scale localized saion wih elecriciy producion and consumpion have been developed. The exreme case of his decenralizaion process consiss in being fully off-grid (i.e. being disconneced from convenional elecrical neworks). Such a case requires o be able o provide elecriciy when needed wihin he microgrid. Since PV producion varies wih daily and seasonal flucuaions, sorage is required o balance producion and consumpion. In his paper, we focus on he case of fully off-grid microgrids. Due o he cos of baeries, sizing a baery sorage capaciy so ha i can deal wih seasonal flucuaions would be oo expensive in many pars of he world. To overcome his problem, we assume ha he microgrid is provided wih anoher sorage c Springer Inernaional Publishing Swizerland 2015 W.L. Woon e al. (Eds.): DARE 2015, LNAI 9518, pp. 1 15, 2015. DOI: 10.1007/978-3-319-27430-0 1
2 S. Aiahar e al. echnology, whose sorage capaciy is almos unlimied, such as for example as i is he case wih hydrogen-based sorage. However, his long-erm sorage capaciy is limied by he power i can exchange. Balancing he operaion of boh ypes of sorage sysems so as o avoid a bes power cus is challenging in he case where producion and consumpion are no known in advance. The conribuion presened in his paper aims o build a decision making agen for planning he operaion of boh sorage sysems. To do so, we propose he following mehodology. Firs, we consider a family of scenarios for which producion and consumpion are known in advance, which allows us o deermine he opimal planning for each of hem using he mehodology proposed in [4] which is based on linear programming. This family of soluions is used as an exper knowledge daabase, from which opimal decisions can be exraced ino a learning se. Such a se is used o rain a decision making agen using supervised learning, in paricular Exremely Randomized Trees [5]. Our supervised learning sraegy provides he agen wih some generalizaion capabiliies, which allows he agen o ake high performance decisions wihou knowing he scenarios in advance. I only uses recen observaions made wihin he microgrid. The ouline of his paper is he following. Secion 2 provides a formalizaion of he microgrid. Secion 3 describes he relaed work. Secion 4 inroduces a linear programming formalizaion of microgrid planning wih fully-known scenarios of producion and consumpion. Secion 5 describes our imiaive learning approach. Secion 6 repors and discusses empirical simulaions. Secion 7 concludes his paper. 2 Microgrids Microgrids are small srucures providing energy available wihin he sysem o loads. The availabiliy of he power srongly depends on he local weaher wih is own shor-erm and long-erm flucuaions. We consider he case where microgrids have boh generaors and sorage sysems and also loads. This secion formally defines such devices. I ends wih a definiion of Levelized Energy Cos wihin a fully off-grid microgrid. 2.1 Generaors Generaors conver any source of energy ino elecriciy. They are limied by he power hey can provide o he sysem. More formally, le us define G as he se of generaors, y g as he supply power limi of g G in Wp and η g he efficiency, i.e. he percenage of energy available afer generaion, of g G and p g he available power from he source of energy. The following inequaion describes he maximal power producion: p g η g y g. (1) In our work, we consider phoovolaic panels, for which he power limiaion is linearly dependen of he surface size of he PV panel which is expressed in m 2.
Imiaive Learning for Online Planning in Microgrids 3 Le x g be he surface size of g G. The oal power producion by he surface size is expressed as he following equaion, for which he consrain expressed above sill holds: φ g = x g p g η g. (2) 2.2 Sorage Sysems Sorage sysems exchange he energy wihin he sysem o mee loads demand and possibly o fill oher sorage sysems. They are limied eiher by capaciy or power exchange. We denoe hese limis by x σc and x σp respecively, where σ c Σ C, σ c Σ P. We denoe Σ = Σ P + Σ C as he se of boh kind of sorage sysems and η σ he efficiency of σ Σ. We consider also, σ Σ, he variables s σ for he sorage conen, a +,σ and a,σ for he decision variables corresponding discharge and he recharge amoun of he sorage sysem a each ime. Dynamics of sorage sysems are defined by he following equaions: s σ s,0 =0, σ Σ, (3) s σ = s σ ( 1) + a,σ 1 + a+,σ ( 1), 1 T 1, σ Σ. (4) 2.3 Loads Loads are expressed in kwh for each ime sep. Power cus occur when he demand is no me, and he lack is associaed wih a penaly cos. Formally, we define he ne demand d as he difference beween he consumpion, defined by c and he available energy from he generaors. The following equaion holds: d = c g G φ g, 0 T 1. (5) Finally, F is defined as he energy no supplied o loads, expressed in kwh, by he following equaion: F = d σ Σ η σ (a +,σ + a,σ ), 0 T 1. (6) We now inroduce in he model he possibiliy o have several levels of prioriy demand, defined by he cos associaed o power cus. Le Ψ be he se of prioriy demands, and F ψ he number of kwh no supplied for he demand prioriy group ψ Ψ. Hence, Eqs. (5) and (6) become: d = c ψ φ g, 0 T 1, (7) ψ Ψ g G ψ Ψ F ψ = d σ Σ η σ (a +,σ + a,σ ), 0 T 1. (8)
4 S. Aiahar e al. 2.4 Levelized Energy Cos Any planning of a microgrid given any scenario of producion and consumpion leads o a cos based on power cus and iniial invesmen. I also akes ino accoun economical aspecs (e.g. deflaion). This cos, called he Levelized Energy Cos (LEC) for a fully off-grid microgrid is formally defined below: where LEC = n I y M y y=1 n y=1 (1+r) + I y 0 ɛ y, (9) (1+r) y n = Considered horizon of he sysem in years; I y = Invesmens expendiures in he year y; M y = Operaional revenues performed on he microgrid in he year y (ake ino accoun he cos of power cus during he year y); ɛ y = Elecriciy consumpion in year y; r = Discoun rae which may refer o he ineres rae or discouned cash flow. 3 Relaed Work Differen seps of our conribuion have already been considered in various applicaions. We used linear programming for compuing exper sraegies. This kind of approach have already been discussed in [7 9] wih differen microgrid formulaions. We used supervised learning echniques wih soluions provided by linear programming. Cornélusse e al. [2] have considered his approach in he uni commimen problem. Our predicion model needs an addiional sep o ensure compliance of he policy learned by supervised learning algorihms wih consrains relaed o he sysem. This sep consiss o use quadraic programming o posprocess he soluion. Cornélusse e al. [1] have considered a similar approach. Online planning for microgrids has also been sudied wih ohers microgrid configuraions. For example, Debjyoi and Ambarnah [3] focuses on he specific case of online planning using auomaas, while Kuznesova e al. [6] focuses on he online planning using a model-based reinforcemen learning approach. 4 Opimal Sizing and Planning 4.1 Linear Program Objecive Funcion. Le k ψ, 0 T 1, ψ Ψ be he value of loss load of he prioriy demand ψ Ψ. The LEC is insaniaed in he following way: where y = /(24 365). LEC = T ψ Ψ kψ F ψ =1 n y=1 + I (1+r) y 0 ɛ y, (10) (1+r) y
Imiaive Learning for Online Planning in Microgrids 5 Consrains. Sorage sysems acions are limied by heir sizes. The following consrains are added: s σc x σc, σ c Σ C, 0 T 1, (11) a +,σp x σp, σ p Σ P, 0 T 1, (12) a,σp x σp, σ p Σ P, 0 T 1. (13) Figure1 shows he overall linear program. Min. T =1 ψ Ψ k ψ F ψ (1+r) + I y 0,y = /(365 24) (14a) n y=1 y (1+r) y (14c) S.., {0...T 1} : (14b) s σ = s σ 1 + a,σ 1 + a+,σ 1, σ Σ, s σc x σc, σ c Σ C, (14d) a +,σp x σp, σ p Σ P, (14e) a,σp x σp, σ p Σ P, (14f) ψ Ψ ψ Ψ F ψ d σ Σ η σ (a,σ + a +,σ ), (14g) F ψ 0, (14h) F ψ c ψ. (14i) Fig. 1. Overall linear program for opimizaion. 4.2 Microgrid Sequence When planning is performed given sequences of producion and consumpion of lengh T > 0, a sequence of sorage conens and a sequence of acions are generaed. Such a group of four sequences is called a microgrid sequence. In he following, we abusively denoe as an opimal microgrid sequence a se of sequences obained by solving linear programs. Figure 2 shows an illusraion of a sequence of decision, wih he wo kinds of sorage sysems. A microgrid sequence is formally defined below: (c 0...c T 1,φ 0...φ 0...T 1,s σ 0...s σ T 1,a σ 0...a σ T 1), (15) where {0...T 1},a σ = a +,σ + a,σ.
6 S. Aiahar e al. Fig. 2. Sequence scheme (discharging/recharging). The cos associaed o a microgrid sequence is defined below for any microgrid sequence s given any microgrid configuraion M, any sequence of producion φ 0...T 1 and any sequence of consumpion c 0...T 1 : LEC c 0...c T 1,φ 0...φ T 1 M (s) = T =1 ψ Ψ kψ ((c g G φg ) σ Σ ησ (a +,σ +a,σ )) + I (1+r) y 0 n ɛ y. y=1 (1+r) y (16) 5 Imiaive Learning Approach Opimal microgrid sequences are generaed as an exper knowledge daabase. The decision-making agen is buil using a subse of his daabase. Such an agen is evaluaed on a disinc subse. 5.1 Daa Given producion and consumpion sequences, we can generae microgrid sequences by solving linear programs. Formally, le (φ (k),c (k) ) {0...T 1},k {0...K} be a se of producion and consumpion scenarios, wih K N\0. To his se corresponds a se of microgrid sequences: (φ (k),c (k),s (k,σ),a (k,σ) ) k K, {0,...,T 1}. (17) 5.2 From Daa o Feaure Space For each ime {0,...,T 1}, producion and consumpion daa are known from0o.leφ (k) 0... = φ (k) 0...φ (k) and c (k) 0... = c (k) 0...c (k) be he sequences
Imiaive Learning for Online Planning in Microgrids 7 of producion and consumpion from 0 o. We define a microgrid vecor from he previous sequences: (φ (k) 0...,c(k) 0...,s(k,σ),a (k,σ) ) k K, {0...T 1}, σ Σ. (18) We now inroduce, {0...T 1}, he funcion e : R R R b R b where b =#Σ. Such a funcion builds an informaion vecor from sequences of producion and consumpion. Le v = e(φ (k) 0...,c(k) 0... ) be he informaion vecor, v l he l-h componen of v and L>0 he size of he vecor. Finally we define, from he definiion of microgrid vecor, a feaure space ha will be used wih supervised learning echniques as below: (v 1...v L,s (k,σ),a (k,σ) ) k K, {0...T 1}, σ Σ. (19) 5.3 Consrains Compliancy An addiional sep is o ensure ha he consrains relaed o he curren informaion of he sysem are no violaed wih he acions performed by he decision making agen. A quadraic program is designed o search for closes feasible acions. This program is defined in Fig. 3. We use consrains from Fig. 1 wih an exra one defined below which represens he limi of sorage sysem recharging regarding he overall available energy in he sysem. σ Σ a +,σ d σ Σ a,σ, (20) where d is defined below o ake ino accoun only he possible overproducion by he following equaion: d = max(0,d ). (21) We are going o illusrae he posprocessing par (see also Fig. 4). We will consider hree use cases below, wih a baery limied in capaciy by 11 kwh and a hydrogen ank wih a power exchange limi of 7 kwp. Underproducion wih boh sorage sysems empy. Iniial acions are boh discharging bu since his is no possible, he posprocessing par cancels he acions (Fig. 4 - op); Overproducion wih hydrogen ank empy and baery conaining 7 kwh. Iniial acions are boh charging. Bu he producion iself does no enirely mee he consumpion. Again, here is a projecion where only he baery is discharging (Fig. 4 - middle); Underproducion wih boh sorage sysems are no empy. The acions are boh charging bu he energy requesed does no mee enirely he consumpion. As a consequence, he levels of charging of he baery and of he hydrogen ank are decreased by he projecion (Fig. 4 - boom).
8 S. Aiahar e al. Min. (a +,σ S. : a +,σ ) 2 +(a,σ a,σ ) 2 F (22a) (22b) s σ + a,σ + a +,σ 0, σ Σ (22c) s σc x σc, σ c Σ C, (22d) a +,σp x σp, σ Σ P, (22e) a,σp x σp, σ Σ P, (22f) ψ Ψ ψ Ψ F ψ d σ Σ η σ a,σ + a +,σ η σ, (22g) F ψ 0, (22h) F ψ c ψ, (22i) d d, (22j) d 0, (22k) σ Σ a +,σ d σ Σ a,σ. (22l) Fig. 3. Quadraic program defined for any ime sep {0...T 1} (posprocessing par). 5.4 Evaluaion An evaluaion crierion consiss o compue he difference of cos observed beween conrol by he imiaive agen and he opimally conrol microgrid, for a given conex (i.e. profile of producion/consumpion and microgrid seings). More formally, le s consider s he opimal microgrid sequence and s he microgrid sequence generaed by he decision making agen. Then he cos difference is represened by he funcion below: Err c 0...cT 1,φ 0...φ T 1 M (s )=LEC c 0...cT 1,φ 0...φT 1 M (s ) LEC c 0...cT 1,φ 0...φ T 1 M (s ). (23) 6 Simulaions 6.1 Implemenaion Deails The programming language Pyhon 1 was used for all he simulaions, wih he library Gurobi 2 for opimizaion ools and sciki-learn 3 for machine learning ools. 1 www.pyhon.org. 2 hp://www.gurobi.com/. 3 www.sciki-learn.org.
Imiaive Learning for Online Planning in Microgrids 9 Fig. 4. Bar plos represening he projecion from acions o feasible ones when needed. 6.2 Microgrid Componens Devices below are considered for he microgrid configuraion. Phoovolaic panels accumulae energy from solar irradiance wih a raio of loss due o echnology and amospheric issues. According o Sec. 2, hey are defined in erms of m 2 and of Wp per m 2. Table 1 gives he values of he elemens describing he PV panels. Baeries are considered as shor-erm sorage sysems wih no consrain on power exchange bu wih limied capaciy. Table 2 gives he values of he elemens describing he baeries.
10 S. Aiahar e al. Hydrogen anks, wihou capaciy consrain, bu limied in power exchange. They are long-erm sorage sysems. Table 3 gives he values of he elemens describing he hydrogen anks. Table 1. Phoovolaic panels seings. Efficiency η PV 20 % Cos by m 2 200 e Wp/m 2 200 Lifeime 20 years Table 2. Baeries seings. Efficiency charging/discharging η B 90 % Cos per usable kwh 500 e Lifeime 20 years Table 3. Hydrogen ank seings. Efficiency charging/discharging η B 65 % Cos per kwp 14 e Lifeime 20 years 6.3 Available Daa Consumpion Profile. An arbirarily paern was designed as a represenaive model of a common residenial daily consumpion wih wo peaks of respecively 1200 and 1750 W. Figure 5 shows he daily graph of such a consumpion profile. Producion Profile. The producion scenarios are derived from he producion daa of a phoovolaic panel insallaion locaed in Belgium. These daa have been processed in a sraighforward way so as o have hisories of producion per m 2 of PV panels insalled. These will be used laer o define he producion scenarios by simply muliplying hem by he surface of he PV panels of he microgrid. Figure6 shows a ypical producion scenario for PV panels in Belgium.
Imiaive Learning for Online Planning in Microgrids 11 Fig. 5. Residenial consumpion profile. Fig. 6. Monhly producion profile of PV panels in Belgium. 6.4 Tes Proocol We spli he se of scenarios of producion and consumpion ino wo subses, a learning se for raining he agen and a es se o evaluae he performances. The learning se conains he wo firs years of producion and he es se conains he las year of producion. We also apply linear ransformaions as below o arificially creae more scenarios for boh learning and es ses. {(ic,jφ )}, {0...T 1}. (24) i {0.9,1,1.1} j {0.9,1,1.1} Table 4 deails he configuraion of our microgrid. The following informaion vecors have been considered, {0...T 1}:
12 S. Aiahar e al. Table 4. Microgrid configuraion. Phoovolaic panels area (in m 2 ) 42 Baery capaciy (in kwh) 13 Hydrogen nework available power (in kwp) 1 Fig. 7. Sample of ypical Belgium producion (1 year). 12 h of hisory, i.e. e(φ (k) 0...φ (k),c (k) 0...c (k) )= (φ (k) max( 12,0)...φ(k),c (k) max( 12,0)...c(k) ); 0...φ (k),c (k) 0...c (k) (φ (k) max( 24 30 3,0)...φ(k),c (k) max( 2160,0)...c(k) ); 3 monhs of hisory, i.e. e(φ (k) 12 h of hisory + summer equinox disance, i.e. e(φ (k) 0...φ (k),c (k) 0...c (k) )=(φ (k) where is he summer equinox daeime. )= max( 12,0)...φ(k),c (k) max( 12,0)...c(k), ), A fores of 250 rees have been buil wih he mehod of Exremely Randomized Trees proposed in [5]. Our imiaive learning agen and our opimal agen are also compared wih a so-called greedy agen ha behaves in he following way. Ifd 0, i.e. if underproducion occurs, sorage sysems are discharged in decreasing order of efficiency; Ifd 0, i.e. if overproducion occurs, sorage sysems are charged in decreasing order of efficiency. The main idea of his greedy agen is o keep as mos as possible energy ino he sysem.
Imiaive Learning for Online Planning in Microgrids 13 6.5 Resuls and Discussion Opimal Sequence of Acions. Figure 8 shows he evoluion of he sorage sysems conens given opimal sequences of acions, for a given scenario. The empirical mean LEC over he es se is 0.32e/kWh. The evoluion is ploed over 1 year. Fig. 8. Sorage sysem sae evoluion (opimal). As expeced, he baery ries o handle shor-erm flucuaions. On he oher hand, he hydrogen ank conen gradually increases during summer before gradually decreasing during winer. Greedy and Agen-Based Sequences of Acions. Table 5 shows he LEC for all he sequences generaed by he greedy algorihm and he conroller given several inpu spaces. Considering a hisory of producion and consumpion of only 12 h is more expensive in erms of LEC, compared o a hisory of producion and consumpion
14 S. Aiahar e al. Table 5. Overall mean LECs. Greedy conroller 0.6 12 h 0.44 3 monhs 0.43 12 h + summer equinox disance 0.42 of 3 monhs. I shows ha a decision making agen is more efficien wih longerm informaion. Addiionally, we also repor experimenal resuls for which he agen was also provided wih he disance (in ime) o summer equinox. This addiional informaion improves he performances. 7 Conclusion In his paper, we have proposed an imiaive learning-based agen for operaing boh long-erm and shor-erm sorage sysems in microgrids. The learning se was obained by solving a family of linear programs, each of hem being associaed wih a fixed producion and consumpion scenario. As having access o real daa is expensive, we plan o invesigae how o ransfer knowledge from one microgrid o anoher. In paricular, we will focus on ransfer learning sraegies [10]. Acknowledgmens. Raphael Foneneau is a Posdocoral Fellow of he F.R.S.-FNRS. The auhors also hank he Walloon Region who has funded his research in he conex of he BATWAL projec. The auhors also hank Berrand Cornelusse for valuable discussions. References 1. Cornélusse, B., Geurs, P., Wehenkel, L.: Tree based ensemble models regularizaion by convex opimizaion. In: 2nd NIPS Workshop on Opimizaion for Machine Learning, OPT 2009 (2009) 2. Cornélusse, B., Vignal, G., Defourny, B., Wehenkel, L.: Supervised learning of inra-daily recourse sraegies for generaion managemen under uncerainies. In: 2009 IEEE Buchares PowerTech, pp. 1 8. IEEE (2009) 3. Debjyoi, P., Ambarnah, B.: Conrol of sorage devices in a microgrid using hybrid conrol and machine learning. In: IET MFIIS 2013, Kolkaa, India, vol. 5, p. 34 (2013). ISBN: 978-93-82715-97-9 4. Francois-Lave, V., Gemine, Q., Erns, D., Foneneau, R.: Towards he minimizaion of he levelized energy coss of microgrids using boh long-erm and shor-erm sorage devices. To be published (2015) 5. Geurs, P., Erns, D., Wehenkel, L.: Exremely randomized rees. Mach. Learn. 63(1), 3 42 (2006) 6. Kuznesova, E., Li, Y.F., Ruiz, C., Zio, E., Aul, G., Bell, K.: Reinforcemen learning for microgrid energy managemen. Energy 59, 133 146 (2013)
Imiaive Learning for Online Planning in Microgrids 15 7. Moghaddam, A.A., Alireza, S., Niknam, T., Reza Alizadeh Pahlavani, M.: Muliobjecive operaion managemen of a renewable MG (micro-grid) wih back-up micro-urbine/fuel cell/baery hybrid power source. Energy 36(11), 6490 6507 (2011) 8. Morais, H., Kdr, P., Faria, P., Vale, Z.A., Khodr, H.: Opimal scheduling of a renewable micro-grid in an isolaed load area using mixed-ineger linear programming. Renew. Energy 35(1), 151 156 (2010) 9. Moevasel, M., Reza Seifi, A., Niknam, T.: Muli-objecive energy managemen of CHP (combined hea and power)-based micro-grid. Energy 51, 123 136 (2013) 10. Pan, S.J., Yang, Q.: A survey on ransfer learning. IEEE Trans. Knowl. Daa Eng. 22(10), 1345 1359 (2010)