Smart Grid Data Management Challenges

Smart Grid Data Management Challenges Marie-Luce PICARD EDF R&D marie-luce.picard@edf.fr 16 Novembre 2010

Outline 1. Smart grids : What? Where? What for? 2. A road map for Smart Grids functionalities and issues 3. Some experiments done at EDF R&D 4. Conclusion EDF R&D : Créer de la valeur et préparer l avenir 2

Smart Grids : What? Where? What for? EDF R&D : Créer de la valeur et préparer l avenir 3

Smart-Grids projects everywhere in the world... Key: red=electricity, green=gas, blue=water and triangle=trial or pilot where circle=project EDF R&D : Créer de la valeur et préparer l avenir 4

Smart-Grids projects everywhere in the world... EDF R&D : Créer de la valeur et préparer l avenir 5

Commission de Régulation de l Énergie (CRE) :... And in France www.smartgrids-cre.fr ERDF : linky.erdfdistribution.fr A law on smart-metering has been signed by the government by the end of August 2010. It implies : - On 01/01/2012 all new meters have to be smart - On 31/12/2014, 50% of installed meters have to be smart - On 31/12/2016, 95 % of installed meters have to be smart. EDF R&D : Créer de la valeur et préparer l avenir 6

Smart Grids : what? And what for? Environmental, economical, social and policy drivers lead to a deep change of the energy sector : Climate change, environmental concerns Increased pressure of operational and financial efficiency Increasing awareness of consumers, role of citizens Technological pressure (IT, smart devices) Source Wikipedia A smart grid delivers electricity from suppliers to consumers using digital technology with two-way communications to control appliances at consumers' homes to save energy, reduce cost and increase reliability and transparency. It overlays the electrical grid with an information and net metering system, and includes smart meters. Such a modernized eletricity network is being promoted by many governments as a way of addressing energy independence, global warming and emergency resilience issues.

A road map for Smart Grids functionalities and issues EDF R&D : Créer de la valeur et préparer l avenir 8

A road map for Smart Grids functionalities Future Decision support Integrated communication AMI / smart metering Today EDF R&D : Créer de la valeur et préparer l avenir 9

A road map for Smart Grids functionalities Future Decision support Integrated communication AMI / smart metering full integration of renewables and distributed generation increase customer participation SCADA, supervisory functions real time pricing, demand response self healing Individual load curves (hourly), available in a batch mode (the day after) Real time alarms Today EDF R&D : Créer de la valeur et préparer l avenir 10

A road map for Smart Grids functionalities and issues Future Decision support Integrated communication AMI / smart metering closed-loop system of systems transactive hierarchichal control privacy distributed intelligence (routers, database, CEP) real time processing for metering data and network data Entreprise service bus, SOA MDMS Standards : CIM, Zigbee Today Deployment but also complex IT and social projects EDF R&D : Créer de la valeur et préparer l avenir 11

Smart Grid Data Management challenges Data Management challenges : need for scalabality? Storage of large volume of time-series Historians, large scale relational data-bases, distributed NoSQL approaches Trade off between distributed intelligence and centralised processing Systems of systems, transactive hierachichal control Real-time processing that might be included into distributed architecture CEP tools Real time BI (low latency) Distributed data-mining Answer to scalabality, but also privacy Adaptive modelling, on-line machine learning The necessary functions remain the same, the key issue is The necessary functions remain the same, the key issue is manage manage the the complexity complexity to to support support the the necessary necessary business business capabilities at any scale as well as manage the separation capabilities at any scale as well as manage the separation of responsabilities to avoid dueling control systems of responsabilities to avoid dueling control systems (SDG&E) (SDG&E) EDF R&D : Créer de la valeur et préparer l avenir 12

Some experiments done at EDF R&D EDF R&D : Créer de la valeur et préparer l avenir 13

On-going research project : needs for storing large volume of time series Data : Individual up to 30 millions- and aggregated load curves (10 minutes intervals, at least 2 years lenght) Meteorological data (hourly measures of temperatures, georeferenced) Contractual information Needs : Curve selection Computation of aggregated curves Missing data processing, synchronisation, forecasting modeling...... EDF R&D : Créer de la valeur et préparer l avenir 14

On-going research project : needs for storing large volume of time series (2) Solutions considered : Data historians Relational data-bases NoSQL approaches (Hadoop HBASE with PigLatin and Hive scripts...) Local and cloud storage Work in progress EDF R&D : Créer de la valeur et préparer l avenir 15

Real-time computation of aggregated curves Computation of aggregated load curves : Grouped by logical criteria : i.e. customer segment Grouped by topological criteria : i.e. network node Issues Real-time processing (30 millions of curves, 700000 feeders) Poor data quality : missing data, delays... Bandwidth constraints Elements of answers : Broadcast multipath architecture On the fly estimation EDF R&D : Créer de la valeur et préparer l avenir 16

Real-time computation of aggregated curves (2) Use of summation sketches The method is duplication insensitive Implementation within the StreamBase CEP ; on the fly processing of 300000 curves (duplicated 6 times) Results :

Evolution of demand forecasting approaches Today : Short term (national) forecasts have a very good accuracy (MAPE : 1,5%) The current methods should work on stationary signals, and forecast the demand for well defined portfolio Tomorrow : Variable portfolio, non stationary signals (customers should leave and join the company, difficult periods to forecast, uncertainties, integration of renewables and new uses like electric vehicles) Massive individual data will be available : Streaming data : how to take them into account for short term (1 day ahead) or very short term (from 1 hour to 24 hours ahead) forecasts? Adaptivity Individual data : how could forecast approachs benefit from their use? Aggregation (clustering, sampling)

Evolution of demand forecasting approaches (2) Results : Work on adaptive GAM (Generalized Additive Models) for short term forecasting Example : Residus de prévision Résidus de prévision sur un an (fenêtre de taille 100, modélisation du niveau moyen et de l effet retard en ligne) RMSE GAM : 900 MW. RMSE GAM en ligne : 714 MW GAM GAM en ligne 1/9/2007 17/9/2007 4/10/2007 21/10/2007 16/11/2007 4/12/2007 20/12/2007 20/1/2008 6/2/2008 22/2/2008 10/3/2008 28/3/2008 16/4/2008 18/5/2008 3/6/2008 20/6/2008 7/7/2008 26/7/2008 12/8/2008 31/8/2008-2000 0 1000 3000

Conclusion EDF R&D : Créer de la valeur et préparer l avenir 20

Smart Grid data management challenges? Smart grids projects (or large experiments) do exist all around the world Deployment, IT and social projects Roadmap : long term vision will drive a very strong evolution of the energy sector Needs for scalability : Storage of large volumes of time series Centralised or distributed approaches Real-time processing (scalable and distributed CEP) Large-scale data-mining : Distributed data-mining could give answers to scalabality and privacy On-line models

Special thanks to Alexis Bondu Xavier Brossat Yousra Chabchoub Leeley Daio Pires Dos Santos Alzennyr Gomes Da Silva Yannig Goude Benoît Grossin Georges Hébrail Bruno Jacquin Sylvie Mallet Amandine Pierrot

Rererences MDMS : rapport du GTM Research www.gtmresearch.com/report/the-emergence-of-meter-data-management-mdm Smarter Energy @ IBM Research, Brian Gaucher, Manager Smarter Energy, IBM T.J. Watson Research Center Analytics and transactive control design for the Pacific Northwest Smart Grid Demonstration Project, P. Huang, J. Kalagnanam, R. Natarajan (IBM Research Watson), D. Hammerstrom and R. Melton, Battle Memorial Institue, Pacific Northwest Division, Richland. (http://www.ieeesmartgridcomm.org/techprogram.html) hadoop.apache.org Agrégation robuste de données en temps réel : application aux compteurs électriques communicants, Y. Chabchoub, B. Grossin, soumis à EGC 2011 Short term electricty load forecasting with adaptive GAM models, Y. Goude, A. Pierrot, ISF 2010 A range of methods for electrical consumption forecasting, X. Brossat, ISF 2010. http://www.csee.umbc.edu/~hillol/kargupta/pubs.html

MDMS : Metering Data Management Systems MDMS : software platform acquiring metering data from numerous sources and providing them, after integration, synchronisation and cleansing, to differents target users.