Online and Scalable Data Validation in Advanced Metering Infrastructures

Size: px

Start display at page:

Download "Online and Scalable Data Validation in Advanced Metering Infrastructures"

Stanley Kelley
8 years ago
Views:

1 Online and Scalable Data Validation in Advanced Metering Infrastructures Chalmers University of technology

2 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 2

3 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 3

4 Online and Scalable Data Validation in Advanced Metering Infrastructures Demand/Response, Real-time pricing, Intrusion Detection How can we validate data given that There is a large volume of continuous data demanding for distributed and parallel analysis Validation rules depend on installation-specific features such as brands, devices, protocols, System experts should define installationspecific validation rules? Noisy and Lossy data: bad-calibrated / faulty devices, lossy communication, malicious users,

analysis Validation rules depend on installation-specific features such as brands, devices, protocols, System experts should

5 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 5

6 Data Streaming - Motivation Financial applications, sensor networks monitoring, require Continuous processing of data streams Real Time fashion Store and process is not feasible Financial markets: process millions of messages/second, take fast decisions (< 100 microseconds) Data Streaming: In memory Bounded resources Efficient one-pass analysis 6

not feasible Financial markets: process millions of messages/second, take fast

7 Data Streaming - System Model Data Stream: unbounded sequence of tuples Example: consumption readings Field House Device Time (secs) Consumption (kwh) Field text text int double A B 8:00 1 C D 8:20 2 A E 8:35 4 time 7

House Device Time (secs) Consumption (kwh) Field text

8 Data Streaming - System Model Operators: OP Stateless 1 input tuple 1 output tuple OP Stateful 1+ input tuple(s) 1 output tuple 8

9 Data Streaming - System Model Infinite sequence of tuples / bounded memory windows Example: 1 hour windows [8:00,9:00) time [8:20,9:20) [8:40,9:40) 9

10 Data Streaming - System Model Infinite sequence of tuples / bounded memory windows Example: 1 hour windows Counter: 1 Counter: 3 Counter: 2 Counter: 4 8:05 8:15 8:22 8:45 9:05 [8:00,9:00) time Output: 4 10

windows Counter: 1 Counter: 3 Counter: 2 Counter: 4

11 Data Streaming - System Model Infinite sequence of tuples / bounded memory windows Example: 1 hour windows Counter: 3 8:05 8:15 8:22 8:45 9:05 [8:20,9:20) time 11

12 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 12

13 Why Streaming-based data Validation? Expressive Online Parallel & Distributed

14 Sample Streaming-based Data Validation: Interpolate missing consumption values Expressive Online Aggregate Filter Map Field Device Time Consumption Match consecutive readings from each meter Field Device Time 1 Consumption 1 Time 2 Consumption 2 Forward if time distance exceeds a certain threshold Field Device Time 1 Consumption 1 Time 2 Consumption 2 Interpolate missing values Field Device Time Consumption

Device Time 1 Consumption 1 Time 2 Consumption 2 Forward if time distance exceeds a certain threshold

15 Sample Streaming-based Data Validation: Interpolate missing consumption values A F M Expressive Online Parallel & Distributed M A A F A F M F A F M A F A F M

16 Evaluation - Setup From a real-world AMI, data extracted from 50 meters Data covers 13 months (May 2012 June 2013) Implemented on top of Storm, a widely-used SPE (e.g., used in Twitter) Evaluated in terms of throughput (tuples/second) and processing latency (milliseconds)

17 Evaluation

18 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 18

19 Conclusions Streaming-based Data Validation Expressive / Distributed-Parallel / Online Implemented on top of the Storm SPE Evaluated with real-world AMI data

Online and Scalable Data Validation in Advanced Metering Infrastructures

Online and Scalable Data Validation in Advanced Metering Infrastructures Vincenzo Gulisano Chalmers University of Technology Gothenburg, Sweden vinmas@chalmers.se Magnus Almgren Chalmers University of