Aurora: a new model and architecture for data stream management

Transcription

1 Aurora: a new model and architecture for data stream management Daniel J. Abadi 1, Don Carney 2, Ugur Cetintemel 2, Mitch Cherniack 1, Christian Convey 2, Sangdon Lee 2, Michael Stonebraker 3, Nesime Tatbul 2, Stan Zdonik 2 1 Department of Computer Science, Brandeis University 2 Department of Computer Science, Brown University 3 Department of EECS and Laboratory of Computer Science, M.I.T. Presentor: YongChul Kwon(godslord@sparcs.kaist.ac.kr)

2 Table of contents One-line statement Scenario Critique

3 One-line statement They designed a new DBMS model and system specialized in data stream management

4 Scenario A.D. 201x Today, the total number of daily stock trading establishes a new record! Good evening, XXX Headline news! KAIST has announced that they developed nationwide object monitoring system Daihyun Mobis has announced that they will launch auto car diagnostic service in next month as their first telematics service!

5 Scenario Hi, this is Aha! goodday s Would you reporter. tell me the story about May I developing interview who your Rrrrr service? developed I ve heard your new about it s a quite telematics challenging service. task! Hello, Oh, Daihyun It s sure. me. Mobis research How Well, can let department, I help see you? YongChul speaking

6 Daihyun Motors car Telematics agent collects and transmits data to center Armed various sensors Telematics agent can test the car and report malfunctioning part ids RPM, temperature, pressure, oil status, All parts are RFID tagged Brightness, Pressure, exchange date,

7 Diagnostic service GPS 4G Wireless Network Service center Notify Immediate accident response Home visit service Repair center

8 Implementation - trigger Data stream : sometimes lost or delivered lately History of values : no scalable way to support latest location of the car Query register Query management : often update new triggers or queries requested by 3 rd party Trigger : they are not scalable Data Stream??? DBMS Output Update query : millions update in short time burst Data Submitter Optimization : Is it helpful doing massive optimization during high load? Messaging Systems QoS : can not ensure service for premium customers

9 Implementation - middleware Data stream : sometimes lost or delivered lately Update query : millions update in short time burst query Query register Query management : has to use new query language Resource usage : are we efficiently using the system? Data??? Stream History of values : no scalable Data way to find latest Submitter location of the car DBMS Query Optimization Processor : Can not benefit from query optimization Messaging Systems Output QoS : can not ensure service for premium customers

10 Implementation - Aurora Data stream : Data new stream : sometimes processing lost or delivered architecture lately Update queries query : millions : new stream update in short processing time burst architecture query Query management : has : intuitive to use new stream query algebra language and GUI Query register Resource usage :: train are we scheduling efficiently & feed using back the from/to system? QoS Data Stream History of of the values : no : scalable new Datastream way to processing find latest Submitter architecture location of the car DBMS Query Optimization Processor : Can : run-time not benefit optimization from query optimization Output Messaging Systems QoS : specified by QoS application administrator : can not & ensure load service shedding for premium customers

11 Implementation - Aurora Storage Manager inputs Router outputs Q 1 Q 2 Scheduler Data Stream Q m Buffer manager Persistent Store Catalog Box Processors Output Q 1 Q 2 Load Shedder QoS Monitor Q n

12 Strong points Solution approach itself Rethink about everything for the requirements Query model Data flow style query specification Optimization Dynamic runtime optimization Train scheduling QoS specification based resource management

13 Weak points Runs on a single computer Aurora* project No experiment results Train scheduling Various optimization technique

14 New ideas Q. Design looks fancy but how to embody more scalability? A. distributed aurora runtime Flux style Aurora run-time coordination Transfer aurora sub network or query to another runtime instance External QoS scheduler will help

15 Distributed Aurora runtime Storage Manager Q 1 Q 2 Buffer manager Q m Persistent Store Catalog inputs outputs inputs outputs Router Scheduler Box Processors External QoS Monitor inputs outputs Storage Manager Q 1 Q 2 Buffer manager Q m Q 1 Q 2 Q n Router Scheduler Load Shedder Box Processors QoS Monitor Storage Manager Q 1 Q 2 Buffer manager Q m Router Scheduler Box Processors Persistent Store Catalog Persistent Store Catalog Q 1 Q 2 Load Shedder QoS Monitor Q 1 Q 2 Load Shedder QoS Monitor Q n Q n

16 Supplementary Slides

17 Monitoring application VS. Traditional DBMS Typical model Managing History of values Approximate query result Real-time requirement Monitoring Application Data Active Human Passive required required required Traditional DBMS Data Passive Human Active Very hard or inefficient Not supported Not supported

18 Solution approach Rethink about DBMS System & query model Architecture System model Runtime operation Optimization Algebra

19 Runtime system Storage Manager inputs Router outputs Q 1 Q 2 Scheduler Q m Buffer manager Persistent Store Catalog Box Processors Q 1 Q 2 Load Shedder QoS Monitor Q n

20 System model User application Query spec QoS spec Aurora System Historical Storage External data source Operator boxes data flow Continuous & ad hoc queries Application administrator

21 Query model Traditional Structured Query Language Declarative query on static data Aurora Data flow model for data stream Application manager will construct queries using GUI Stream Query Algebra Queries are processed by SQuAl operators on the data stream

22 Query model QoS spec data input b1 b2 b3 app continuous query Connection point b4 b5 b6 QoS spec view b7 b8 b9 app ad-hoc query QoS spec

23 Optimization How can we fix some parts of water supply system? X X X

24 Optimization Aggregate Hold Map Filter Union Join pull data Filter Hold Continuous query Ad hoc query Filter BSort Map Aggregate Static storage Join

25 Optimization Dynamic continuous query optimization Inserting projections Combining boxes Reordering boxes Ad hoc query optimization 1 st stage : replace implementation (Filter/Join) 2 nd stage : same as continuous query

26 SQuAl Order-insensitive Filter Map Union Order-sensitive BSort Aggregate Join Resample