Data stream approaches for electric load curve analysis. Georges Hébrail, Marie-Luce Picard BILab : bilab.enst.fr ISF 2008, June 24th, Nice

Transcription

1 Data stream approaches for electric load curve analysis 1 Georges Hébrail, Marie-Luce Picard BILab : bilab.enst.fr ISF 2008, June 24th, Nice

2 Outline Deployment of a smart metering infrastructure What is a Data Stream? Data Stream Processing Approaches : Data Stream Management Systems Data Stream Mining Conclusion References 2

4 Deployment of a smart metering infrastructure Respecting the orientations given by the Commission de Régulation de l Energie smart metering is expected to be generalized in France (34 millions of meters until 2016) so that in the future all customers could possibly be observed precisely June 06 th, 2007 Thus these millions of automated electronic meters will be installed and connected directly to servers that will manage grid supervision, billing and customer services. Standard characteristics over the whole French territory 3 main objectives : 4 Customer point of view : better information (load curve ½ hour ) Market point of view : new offers, much data available (energy, quality, ) daily Distributor point of view : enhancing grid supervision, and reducing costs. ERDF pilot project ( ) : deployment planned until 2010

5 Automatic Meter Reading Reading (AMR) and Automatic Meter Management (AMM) deployments Vattenfall: meters before the end of 2008 (AMR then AMM) Fortum: objective of meters Ontario: 4.5 million, AMR, before 2010 E.ON: project in investigation, for 4,9 million meters EDP : pilot Fortum : deployment finalised in 2009 Vattenfall: AMR meters at the end of 2007 NESA : AMR meters installed Oxxio : agreement for the deployment of 1 million AMM Nuon: project of 2,7 million meters in 2010 SCE: 4,7 millions meters in 2012 PGE : 5,1 millions meters in 2011 SPAIN Royal decree for massive roll out of smart meters starting in 2010 ENEL : 27 millions AMM meters installed ACEA : 1,5 AMM meters deployed 5 mostly supported by the regulated distribution business in Europe

6 Impact Today lots of studies rely on panels and load-profiling Load-forecasting Macroscopic studies related to grid A management solution : or use customer datastreams knowledge approaches 6 Microscopic studies related to customer studies (offers, ) Profiling for different applications real data, real-time, on the fly computation Smart metering impact Highly reactive data acquisition build summaries Real data (<> profiling) (aggregated curves) Individual or local information add some dynamics on High temporal granularity panel constitution New possibilities : alarms (for example SCADA) allow adaptative clustering (aggregation/disintegration) Traditional approach Build a Data WareHouse : unscalable! Specific code development : expensive, not flexible

8 What is a data stream? Golab & Oszu (2003) : «A data stream is a real-time, continuous, ordered (implicitly by arrival time or explicitly by timestamp) sequence of items. It is impossible to control the order in which items arrive, nor is it feasible to locally store a stream in its entirety». Structured records (not audio or video data) Massive volumes of data ; records arrive at high rate Data push paradigm Infinite sequence of items : One item : structured information (tuple) Same structure for all items in a stream Windowing 8

9 Windowing (1) Queries can be applied either to the whole stream (from beginning to current time), either to a portion of the stream (called a window) Windowing politics : Definition of windows of interest on streams Fixed windows : June 2008 Sliding windows : last 2 days Landmark windows : from June 1 st, 2008 Window specification Refreshing rate Physical time : last 2 days, last 3 hours, last ½ hour Logical time : last 300 tuples Rate of producing results (every item, every 10 items, every 5 minutes) 9

10 Windowing (2) Examples of queries : Give me the number of calls every ½ hour, updated every 5 minutes Give me the total electrical consumption of Nice every 24 hours, updated every day Give me the max and the min price of the EDF SA stock last 24 hours, updated every week. 10

12 Data Stream Processing Approaches : Data Stream Management Systems Main characteristics 12 Generic tools Data and queries play an opposite role compared to standard DBMS approaches : Data is volatile and unpredictable Continuous queries SQL-like query language : Standard SQL on permanent relations Extended SQL on streams with windowing features (processed on the fly) Tools for capturing input streams and producing output streams Good performances : Optimization of computer resources Several input streams several queries Ability to face variations in arrival rates without any crash (includes loadshedding) Approximate or exact results unbounded memory (10 most sold stocks ) Too (many, fast ) : CPU and memory limit Sliding windows and refreshment rate, sampling and shedding, synopses

13 Data Stream Processing Approaches : Data Stream Management Systems - Examples Nom du système Origine Caractéristiques Services StreamBase Aurora-Medusa-Borealis Universités Brandeis, Brown, MIT Windows 2003 StreamBase Studio : développement graphique d applications, palette d opérateurs (Aminsight) Truviso TelegraphCQ Université de Berkeley Linux (Ubuntu). Fonctionne comme une extension de PostGreSQL Pas de studio de développement Visualisateur temps réel (environnement Flex). Aleri Société commerciale (services financiers aux sociétés bancaires) Version Windows 2003 annoncée. Choix de développement via un Studio, XML ou SQL. Possibilité de construire des cubes sur flux Coral8 Parmi les dirigeants, R. Motwani de Stanford (projet STREAM) Intégration native de données XML et DB2. Possibilité de traiter des suites de patterns (complex events) 13

14 Data Stream Processing Approaches : Data Stream Management Systems An example : StreamBase Data input Dev. Studio (graphical combination of Operators in streams) Engine Data processing 14 Own interfaces, adapters, clients Command Line

15 Data Stream Mining Definition : Apply data-mining algorithms to one or several stream(s) Constraints : Limited CPU and memory One-pass on the data Windowing If applied to the whole stream, incremental algorithm If applied to a sliding window : incremental + ability to forget the past Additive methods (PCA) If applied to any (past) portion of the stream : incremental + constitution of summaries Temporal summaries (clustering) : maintain k centers of clusters or maintain k clusters with statistics on their contents Problem of concept drift : evolution of distributions of data over time 15

16 Data Stream Mining an example D0 D1 D2 D3 Dn M0 M1 M2 M3 Mn-1 Mn Adaptative models Example : prediction of the number of calls within a call-center Choice of M model(s) Size of the D i Window? Content and up-dating of DB Memo (bounded size) Inclusion within the StreamBase system Pour produire Mn, on dispose de : Mn-1 Dn Et «de ce que l on veut», : DB mémo de taille bornée (par exemple un résumé de l historique des données) 16

17 17 Data Stream Mining

19 Conclusions A very active way of research Lots of practical applications in many fields Data Stream Management Systems (DSMS) and Data Stream Mining DSMS quite mature Data Stream Mining : still a lot of work to do Distributed data streams (DSMS and mining) sensor networks Data streams approaches for electric load curve analysis : issues to be completed during the next speeches 19

20 References «Querying and mining data streams : you only get one look. A tutorial», M. Garofalakis, J. Gehrke, R. Rastogi, Tutorial SIGMOD 02, Juin «Issues in data stream management», L.Golab, M.Tamer Özsu, SIGMOD Records, Vol.32, n 2, Juin «Data streams : models and algorithms», C.C. Aggarwal, Springer, «The 8 requirements of real-time stream processing», M. Stonebraker, U. Cetintemel, S. Zdonik, SIGMOD Records, Vol.34(4) : 42-47, «Mining data streams : a review», M. M. Medhat, A. Zaslavsky and S. Krishnaswamy in SIGMOD Records, Vol.34(2) pp18-26, «Birch : an efficient data clustering method for very large databases», T. Zhang, R. Ramakrishnan and M. Livny. In Proceedings of the SIGMOD 1996 Conference pp , «A framework for clustering evolving data streams», C.C. Aggarwal, J. Han, J. Wang and P.S. Yu in Proceedings of the 29th VLDB Conference, Berlin, Germany, Linear Road benchmark : 20

21 References «Statistical challenges in data stream applications», G. Hébrail, in 56 th Session of the International Statistical Institute, Lisboa, August «Data stream management and mining», G. Hébrail, International Workshop on Mining Massive Data Sets for Security, NATO school, Villa Cagnola Gazzada, Italy, September «Echantillonnage sur les flux de données : état de l art», R. Chiky, A. Dessertaine, G. Hébrail, Colloque français des sondages, Marseille, November «Echantillonnage spatio-temporel de flux de données distribués», R. Chiky, J. Cubillé, A. Dessertaine, G. Hébrail, ML Picard, Conférence Extraction et Gestion des Connaissances EGC, Nice, January 2008 «Prévision non paramétrique par «Agrégation / désagrégation» de la consommation électrique dans un contexte de flux de données», A. Dessertaine, Journées conjointe de la Société Française de la Statistique et de la Société Canadienne de la Statistique, Ottawa, May 2008 «Applications de gestion de flux de données chez EDF R&D», Workshop «Fouille de données temporelles», S. Ferrandiz, M-L Picard, EGC 2008, Sophia-Antipolis, January

22 References DSMS WebSites : ; BILab WebSite : bilab.enst.fr MIDAS ANR project : midas.enst.fr 22