Distributed Data Management Summer Semester 2013 TU Kaiserslautern

Transcription

1 Distributed Data Management Summer Semester 2013 TU Kaiserslautern Dr.- Ing. Sebas4an Michel saarland.de 1

2 Lecture 8+ (DISTRIBUTED) DATA STREAM PROCESSING (INTRODUCTION) 2

3 So Far: Databases/NoSQL Datastores Data is changing, yes, but this is more due to inserts and update to stored data items Historic data is kept Queries operate on full data (tables) MapReduce is extreme, Write- once & Read- many 4mes Data warehousing, too: periodically loading data in store for deep(er) analy4cs Data mining 3

4 Data Stream Management vs. Tradi4onal Data Management Query & Results DATA Base/ Store Insert Update Delete At query 4me, data is accessed as a whole Data is persistently stored Queries are ad- hoc (mainly) 4

5 Data Stream Management vs. Tradi4onal Data Management (Cont d) results Set of queries DATA STREAM Data is moving! Con4nuously generated (assumed infinite!) At high pace Queries are (mainly) con4nuous (aka. standing). Registered once, observed forever. Answer to queries in (near) real- 4me required (o`en) Probabilis4c methods for efficiency or considering only part of the stream (sliding window) 5

6 Sensor Networks (Distributed) Sensor Networks; Smart Dust Mainly numeric measurements of (natural) phenomena Computa4on of queries like max, average, min, quan4les, value > τ 6

7 Mobile Ad- Hoc Networks Connec4on between sensors are ad- hoc Efficient and reliable rou4ng Understanding (changing) topology Power consump4on Also: Vehicular Ad- Hoc Netw. 7

8 Social Sensors Explicitly: Snow Tweets (hkp://snowcore.uwaterloo.ca/snowtweets/) #snowtweets 50.0 cm. at K1A 0A2 #snowtweets 10.0 in. at #snowtweets 4 cm at Palmerston North 4414 Implicitly: By men4oning topics, people, in social communica4on 4me 8

9 Earthquake News on Twiker source: hkp://blog.socialflow.com/ 9

10 Stock Market Real- 4me analysis of stock marked changes Compu4ng sta4s4cs over streams, e.g., for decision support Opportuni4es for reac4ng in real- 4me Even with fully automated means: algorithmic trading 10

11 DBMS vs. DSMS Database management system (DBMS) Persistent data (rela4ons) Random access One- 4me queries Data stream management system (DSMS) vola4le data streams Sequen4al access Con4nuous queries (theore4cally) unlimited secondary storage limited main memory Only the current state is relevant rela4vely low update rate Likle or no 4me requirements Assumes exact data Plannable query processing Considera4on of the order of the input poten4ally extremely high update rate Real- 4me requirements Assumes outdated/inaccurate data Variable data arrival and data characteris4cs h"p://en.wikipedia.org/wiki/data- stream_management_system 11

12 Data Stream Model Stream of data items is unbounded (available memory is not) No way to store en4re stream (how could we, its (probably) not ending) To compute query results, need to devise algorithm with likle memory consump4on 12

13 Overview of Data Stream Topics Synopses: concise representa4ons of stream content tailored to tasks, e.g., coun4ng dis4nct elements usually not exact, but approxima4ons (es4mators) of true values. Windows: focus of certain recent subset of data computa4on of func4ons/joins over window(s) content 13

14 Data Stream Mining: Teasers I tell you integer numbers between 1 and N I will tell all but one number A`er N- 1 numbers I ask: which number was missing. 14

15 Data Stream Mining: Teasers (Cont d) Keep Boolean array of length N: Mark posi4on for observed number Size required: N Computa4on at end: N to find missing number Much beker: keep sum of numbers: S Missing number is N*(N+1)/2 - S 15

16 Coun4ng Occurrences Consider a stream of elements a i, a 2, a 84, a 41, a 2, a 77, a 231, a 2, a 4, a 54, How o`en does a 2 occur? How to implement? Keep counter for each id Required space #ids (=N) Not feasible of N is very large 16

17 Probabilis4c Coun4ng: Count- Min Sketch Keep 2- dim array (h, r) h hash func4ons* that map to range 0 (r- 1) h h 2 h 3 h 4 Arriving item a For each j: array[j, h j (a)]++ Cormode, Muthukrishnan (2004). An Improved Data Stream Summary: The Count- Min Sketch and its Applica4ons. J. Algorithms 55:

18 Count- Min Sketch: Coun4ng h 1 h 2 h 3 h How o`en did we see item a? h 1 (a) = 4, h 2 (a)=5, h 3 (a)=0, h 4 (a)=2 Take minimum of the corresponding values in the 2- d array. Here: 4 Es4mate is never underes4ma4ng Overes4ma4on probabilis4cally bounded 18

19 Outlook Some more basics of stream processing Few more fundamentals of data stream mining Then, window based data stream management Then going distributed: Distributed DSMS Distributed (ad- hoc) sensor networks Scalable computa4on of massive data streams (think: Hadoop for data streams) 19