Data Stream Management

Transcription

1 Data Stream Management Yanlei Diao University of Massachusetts Amherst Slides courtesy of Michael Franklin and Jianjun Chen

2 Data Stream Management Two driving forces:! A collection of applications where data streams naturally exist but DBMS doesn t help much! Advances of sensor technologies 11/6/13 2

3 Financial Applications v Financial services Data feeds: stock tickers, foreign exchange transactions Data rate: 10 s or 100 s thousands of messages per second Applications: routing trade requests, automating trade strategies, market trend analysis Stream systems: e.g., /6/13 3

4 Network and System Monitoring v Network monitoring Packet traces, network performance measurements Data rate: gigabits per second Applications: traffic analysis, performance monitoring, router configuration, intrusion detection Stream systems: e.g., Gigascope at AT&T v System/Application monitoring Data: system log, measurements Stream systems: e.g., Ganglia 11/6/13 4

5 Wireless Sensor Networks v Wireless sensor networks Sensor devices: temperature, light, pressure, acceleration, humidity, magnetic field, A set of sensor devices auto-configure themselves into a communication network Applications: environment monitoring habitat monitoring structural monitoring vehicle tracking 11/6/13 5

6 Radio Frequency Identification v RFID Technology Tags: small devices, transmitting a tag id when brought close to an RFID reader Readers: interrogate tags with radio signal; read constantly, over a range, without line-of-sight Applications: supply chain management healthcare pharmaceuticals postal services Stream systems: e.g., E.001.F EF.0A D.01 reader_id, tag_id, timestamp 11/6/13 6

7 Execution Models: A Comparison Traditional DBMS Data Stream System Results Results Query Attr1 Attr2 Attr3 Data Queries, Rules, Event Specs, Subscriptions Data at rest One-time or periodic queries Query-driven execution Results returned once or periodically Data in motion, unending Continuous, long-running queries Data-driven execution Results returned continuously in real-time 11/6/13 7

8 Latency Time-critical Decisions Value of Data to Decision-Making Preventive/ Predictive Actionable Information Half-Life In Decision-Making Traditional Batch Business Intelligence Reactive Historical Real- Time Seconds Minutes Hours Days Time Slide Courtesy of Michael Franklin Months 11/6/13 8 8

9 History of Stream Processing v Early: Active DBs, ECA rules, Triggers, Data Broadcast v Field took off around 2002 ~10 SIGMOD/VLDB/ICDE stream papers thru since then (holding steady at 40-50/yr) v Lots of Research Systems-Building Projects AT&T Gigascope Berkeley TelegraphCQ->HIFI Brandeis, Brown, MIT - Aurora->Borealis Purdue Nile Stanford STREAM Wisconsin, Portland State NiagaraCQ v Lots of Technology Transfer: e.g., 11/6/13 9

10 Main Issues of Stream Management v Query languages v Single query processing v Multi-query optimization v Adaptive query processing v Real-time processing v Approximate answers v Out of order processing ü Sliding windows ü Non blocking, windowed operations ü Sharing ü Adaptivity ü Quality of service ü Confidence, sketching, sampling, wavelets... ü Punctuation, order-agnostic 11/6/13 10

11 Data Stream Model v A New Data Model: an infinite sequence of elements An element has a system time and an application time Elements arrive in order of system time (arrival order) If an application wants data in application time, need buffering & sorting If data comes out of order or very late, more work is needed to take care the order. (out of order processing) 11/6/13 11

12 Stream Query Languages v Extensions of SQL: SEQUIN [SLR 96], CQL [ABW03], TCQ [CCD +03], GSQL [CJS+03] v Continuous Query Language (CQL) FROM can address both streams and relations PARTITION BY creates sub-streams Window: extracts a set of tuples from an infinite stream and applies relational processing ROW, physical window in number of tuples; RANGE, logical window in terms of seconds, minutes, or hours (more generally, positions in an ordered domain) SLIDE, window movement. 11/6/13 12

13 Examples of CQL Sliding window join: Select * From S1 [Rows 5], S2 [Rows 10] Where S1.a = S2.a Streaming aggregates: Select Count(*) From S [Range 1 Minute] Select Count(*) From S [Range 1 Minute Slide 1 Minute] 11/6/13 13

14 Example Windows Tumbling: Non-overlapping windows, e.g., 5 second windows <Range 5 seconds Slide 5 seconds> Data Stream Sliding: Overlapping windows, e.g., 5 second windows move every 2 seconds < Range 5 seconds Slide 2 seconds> Data Stream Landmark: Growing windows, e.g., extend window every 2 seconds <LANDMARK ADVANCE 2 sec> [Truviso Language] Data Stream Slide Courtesy of Michael Franklin 11/6/13 14

15 Example Windows Partitioned: value-based sub-streams e.g. <Partition By attr Rows 2 Slide 2 > partitioned tumbling windows Data Stream Red Stream Blue Stream Green Stream Slide Courtesy of Michael Franklin 11/6/13 15

16 Formal Semantics v At time t, there is a window R_t, run query to obtain Q(R_t), t=1,2,3 v Collapse the results at adjacent time points based on identical values v Use Istream and Dstream to ensure that the output also conforms to the stream model 11/6/13 16

17 Main Issues of Stream Management v Query languages v Single query processing v Multi-query optimization v Adaptive query processing v Real-time processing v Approximate answers v Out of order processing ü Sliding windows ü Non blocking, windowed operations ü Sharing ü Adaptivity ü Quality of service ü Confidence, sketching, sampling, wavelets... ü Punctuation, order-agnostic 11/6/13 17

18 Single Query Processing v Blocking query operator A query operator that is unable to produce the first tuple of its output until it has seen its entire input. [BBD+02] Sorting? Sort-merge join? Nested-loops join? Hash join? v Non blocking query operator The operator does not stage data (either in memory or on disk) without producing results for a long time. [UF01] produces the initial result early returns result tuples incrementally as they become available 11/6/13 18

19 (1) Join: First A Blocking Operator Hash Table A Build Probe Source A Source B Assume first that each stream is finite and fits in memory. Traditional Hash Join blocks when one input stalls. 11/6/13 19

20 Non-Blocking Join Operator Hash Hash Hash Table ATable ATable B Build Probe Source A Source B Symmetric Hash Join processes tuples as they arrive. Produces all tuples in the join and no duplicates. Blocks only if both inputs stall. 11/6/13 20

21 Non Blocking Join with Limited Memory v SHJ requires both inputs to be memory resident. Unrealistic in stream processing v XJoin extends SHJ to work for long (but finite) streams with limited memory [UF00]. Partition each input using a hash function. When allocated memory is used up, flush a partition to disk. At each point, a partition = disk-resident data (older) + memory-resident data (newer). Join processing continues on memory-resident data. Disk-resident tuples are handled in background. 11/6/13 21

22 XJoin: Handling the Partitions M E M O R Y 1 Memory-resident partitions of source A n Memory-resident partitions of source B 1 k n flush Tuple A hash(tuple A) = 1 D I S K 1... n 1... k... n Tuple B hash(tuple B) = n SOURCE-A Disk-resident partitions of source A Disk-resident partitions of source B SOURCE-B 11/6/13 22

23 Stage 1: Memory-to-Memory Joins Output Partitions of source A Partitions of source B M E M O R Y i j i j build probe probe build Tuple A hash(record A) = i Tuple B hash(record B) = j SOURCE-A SOURCE-B 11/6/13 23

24 Stage 2: Disk-to-Memory Joins Output Partitions of source A Partitions of source B M E M O R Y i i i i..... D I S K Partitions of source A Partitions of source B 11/6/13 24

25 Three Stages of XJoin v Stage 1 - Symmetric hash join (memory-tomemory) with partitions v Stage 2- Disk-to-memory Separate thread - runs when stage 1 blocks (both input stall). Stage 1 and 2 interleave until all tuples have been received. v Stage 3 - Clean up stage Stage 1 misses pairs that were not in memory concurrently. Stage 2 misses pairs when both are on disk, and may not get to run to completion. Stage 3 joins all the partitions (memory-resident and diskresident portions) of the two sources. 11/6/13 25

26 Windowed Joins v Windowed joins Use windows to obtain a finite set from an infinite stream Basis is non-blocking symmetric hash join Add support for windows Window size: in number of tuples or logical time Sliding of window: smooth movement or hopping Maintain right state (tuples residing in the window) for the join; need to prune expired tuples from hash tables! [MAF+03, GO03, KNV03, MLA04] 11/6/13 26

27 (2) Windowed Aggregates v Windowed aggregates: Use windows to obtain a finite set from an infinite stream Apply MIN, MAX, SUM, COUNT, or AVG to each window Avoid scanning all tuples in each window for time efficiency Avoid storing all tuples in each window for space efficiency What is the time/space complexity for each aggr operator? v Windowed SUM/COUNT Symmetry between expired tuples and new tuples [SLR96] v Windowed MIN/MAX/TOP-K A naïve solution may have to store all tuples in each window How can we do better? [LMT+05, KWF+06, MBP+06, ZYC+07, YSR+11] 11/6/13 27

28 Predictability Property of Windows v When a tuple arrives, can predict 1) to which future windows the tuple will belong 2) whether the tuple belongs to the partial result (e.g., top-k) of each window These insights lead to a predicted top-k result for each of the future windows v When a new tuple arrives, it belongs to its own set of current and future windows it is also used to update the previously predicted top-k result for each relevant window 11/6/13 28

29 et. top-k ecifiw, the in the needs in the pted: ously ofall using putaategy. s topance ation. caners of be rensive tream tain a could ost of indicates its F score, while the position on the X-axis indicates the time it arrived at the system. Figure 1: (Predicted) Top-k results four consecutive windows at time of W 0 (slide size = 4 objects) Yang Yanlei et al. Diao, An Optimal University Strategy of Massachusetts for Monitoring Amherst Top-K Queries in 11/6/13 Streaming Windows. EDBT Based on the objects in W 0,wenotonlycalculatethetop-k(k=3) result in W 0,butwealsopre-determinethepotentialtop-kresults

30 updated predicted top-k results of our running example (Figure 1) after the insertion of four new objects. ject 14 is and thus f For the jects to m The size f Theore jects in pr Figure 2: Updated predicted top-k results of four consecutive windows at time of W 1 (slide size = 4 objects) Yang Yanlei et al. Diao, An Optimal University Strategy of Massachusetts for Monitoring Amherst Top-K Queries in 11/6/13 Streaming Windows. EDBT For the is simply each of th significan especially summary, C nw k Conclu fered by t keeps the monitorin very large processing no longer ment over cessing an tion are re