Online and Scalable Data Validation in Advanced Metering Infrastructures

Size: px

Start display at page:

Download "Online and Scalable Data Validation in Advanced Metering Infrastructures"

Stanley Kelley
10 years ago
Views:

1 Online and Scalable Data Validation in Advanced Metering Infrastructures Chalmers University of technology

2 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 2

3 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 3

4 Online and Scalable Data Validation in Advanced Metering Infrastructures Demand/Response, Real-time pricing, Intrusion Detection How can we validate data given that There is a large volume of continuous data demanding for distributed and parallel analysis Validation rules depend on installation-specific features such as brands, devices, protocols, System experts should define installationspecific validation rules? Noisy and Lossy data: bad-calibrated / faulty devices, lossy communication, malicious users,

analysis Validation rules depend on installation-specific features such as brands, devices, protocols, System experts should

5 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 5

6 Data Streaming - Motivation Financial applications, sensor networks monitoring, require Continuous processing of data streams Real Time fashion Store and process is not feasible Financial markets: process millions of messages/second, take fast decisions (< 100 microseconds) Data Streaming: In memory Bounded resources Efficient one-pass analysis 6

not feasible Financial markets: process millions of messages/second, take fast

7 Data Streaming - System Model Data Stream: unbounded sequence of tuples Example: consumption readings Field House Device Time (secs) Consumption (kwh) Field text text int double A B 8:00 1 C D 8:20 2 A E 8:35 4 time 7

House Device Time (secs) Consumption (kwh) Field text

8 Data Streaming - System Model Operators: OP Stateless 1 input tuple 1 output tuple OP Stateful 1+ input tuple(s) 1 output tuple 8

9 Data Streaming - System Model Infinite sequence of tuples / bounded memory windows Example: 1 hour windows [8:00,9:00) time [8:20,9:20) [8:40,9:40) 9

10 Data Streaming - System Model Infinite sequence of tuples / bounded memory windows Example: 1 hour windows Counter: 1 Counter: 3 Counter: 2 Counter: 4 8:05 8:15 8:22 8:45 9:05 [8:00,9:00) time Output: 4 10

windows Counter: 1 Counter: 3 Counter: 2 Counter: 4

11 Data Streaming - System Model Infinite sequence of tuples / bounded memory windows Example: 1 hour windows Counter: 3 8:05 8:15 8:22 8:45 9:05 [8:20,9:20) time 11

12 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 12

13 Why Streaming-based data Validation? Expressive Online Parallel & Distributed

14 Sample Streaming-based Data Validation: Interpolate missing consumption values Expressive Online Aggregate Filter Map Field Device Time Consumption Match consecutive readings from each meter Field Device Time 1 Consumption 1 Time 2 Consumption 2 Forward if time distance exceeds a certain threshold Field Device Time 1 Consumption 1 Time 2 Consumption 2 Interpolate missing values Field Device Time Consumption

Device Time 1 Consumption 1 Time 2 Consumption 2 Forward if time distance exceeds a certain threshold

15 Sample Streaming-based Data Validation: Interpolate missing consumption values A F M Expressive Online Parallel & Distributed M A A F A F M F A F M A F A F M

16 Evaluation - Setup From a real-world AMI, data extracted from 50 meters Data covers 13 months (May 2012 June 2013) Implemented on top of Storm, a widely-used SPE (e.g., used in Twitter) Evaluated in terms of throughput (tuples/second) and processing latency (milliseconds)

17 Evaluation

18 Agenda 1. Problem statement 2. Preliminaries Data Streaming 3. Streaming-based Data Validation 4. Conclusions 18

19 Conclusions Streaming-based Data Validation Expressive / Distributed-Parallel / Online Implemented on top of the Storm SPE Evaluated with real-world AMI data

Efficient Processing for Big Data Streams and their Context in Distributed Cyber Physical Systems

Efficient Processing for Big Data Streams and their Context in Distributed Cyber Physical Systems Department of Computer Science and Engineering Chalmers University of Technology & Gothenburg University