Conjugating data mood and tenses: Simple past, infinite present, fast continuous, simpler imperative, conditional future perfect

Transcription

1 Matteo Migliavacca School of Computing Conjugating data mood and tenses: Simple past, infinite present, fast continuous, simpler imperative, conditional future perfect

2 Simple past - Traditional Databases Database Management System (DBMS): Data relatively static but queries dynamic Queries DBMS Results Index Data Persistent relations Random access Low update rate Unbounded disk storage One-time queries Finite query result Queries exploit (static) indices 2

3 Simple past Small Data Small data OnLine Transaction Processing (OLTP) & OnLine Analytic Processing (OLAP) end of one-size fits all Transactions store data, moved to warehouse for reporting & analysis: e.g., client orders RDBMS difficult to scale Tx processing has substantial overhead for ACID properties, OLAP bottlenecked by ingestion (indexing)

4 Infinite Present - MapReduce Explosive growth of Data Volume (Big Data), e.g. clickstreams Dynamic resource availability (Cloud Computing) MapReduce much simpler data and processing model log everything, process in parallel (infinitely scalable) Input key*value pairs Input key*value pairs... Data store 1 map Data store n map (key 1, values...) (key 2, values...) (key 3, values...) (key 1, values...) (key 2, values...) (key 3, values...) == Barrier == : Aggregates intermediate values by output key key 1, intermediate values key 2, intermediate values key 3, intermediate values reduce reduce reduce final key 1 values final key 2 values final key 3 values

5 Infinite Present - MapReduce problems Costs: data still grows faster than storage and processing capacity Latency: everything is on disks, slow iteration (e.g., PageRank, Logistic Regression, Gradient descent, etc..) Map step: break page rank into even fragments to distribute to link targets Reduce step: add together fragments into next PageRank Iterate for next step... Batch processing model: stale results(e.g.,add 1 link) Page 5 View, Master, Slide Master to change this text to the title of your presentation

6 Infinite Present - Spark Raise of in-memory processing: Spark intermediate results can be stored in memory: fast iteration Can you go even lower latency? Page 6 View, Master, Slide Master to change this text to the title of your presentation

7 Fast Continuous - Stream Processing Systems Fast Data systems require quick responses (Velocity): fraud detection, high frequency trading, ad-prediction, etc... Event stream processing process one item at a time in real-time, state stored in memory, e.g., online machine learning Stream DSPS Results Transient streams Sequential access Potentially high rate Bounded main memory Continuous queries Working Storage Queries Produce time-dependant result stream Indexing?

8 Fast Continuous State in Stream Processing Data flow graph with processing operators - Apache S4, Twitter Storm, Nokia Dempsy,... Most interesting operators are stateful 8

9 Fast Continuous - Challenges Consider streaming recommender system (collaborative filtering) Input streams: user activities (item purchases, page views, clicks etc) Operators: maintain recommendation matrices Output streams: recommendations for users State: user-item matrix State: item-item matrix Activity user 1 Activity user 2 Activity user 3 Item recommendations 9 Processing needs to keep up with varying input rate acquire resources dynamically Fault-tolerance cannot restart processing from the start if there is a failure

10 Fast Continuous - Parallelising Stream Processing E Exploit data parallelism for scaling out 10

11 Fast Continuous: Elasticity and Fault tolerance Elasticity Provisioning for workload peaks unnecessarily conservative 100%" Utilisation 80%" 60%" 40%" 20%" Courtesy of MSRC 0%" 09/07" 09/08" 09/09" 09/10" 09/11" 09/12" 09/13" Date Failure tolerance Failure is a common occurrence Active fault-tolerance requires 2x resources Passive fault-tolerance leads to long recovery times 11

12 SEEP: Stream Processing System [SIGMOD 13] Dynamic scale out: increase resources when workload peaks occur need to partition the state and distribute across operators Hybrid fault-tolerance: low resource overhead with fast recovery checkpoint state periodically and reprocess a small subset of recent data Raul Castro Fernandez, Matteo Migliavacca, Evangelia Kalyvianaki, and Peter Pietzuch, "Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management, ACM International Conference on Management of Data (SIGMOD), New York, NY, June

13 Simpler Imperative dataflow programming How do we write data processing queries? Common intermediate representation is dataflow models, tedious to use directly Functional approach is popular and easy to translate to dataflows e.g. Spark Can we do something for poor imperative programmers out there?

14 pers, Java distributed e dataflow ogram and to an ex- Using prossing tasks e variablelies on the processing. An SDG te: it is a hich exeory state. large state an process ned across n local inmputation. ints to acan be rec- Simpler Imperative Imperative Big Data Processing [USENIX ATC 14] Take annotated Java, and translate to a dataflow model Algorithm 1: Online collaborative filtering Matrix useritem = new Matrix(); Matrix coocc = new Matrix(); 3 4 void addrating(int user, int item, int rating) { 5 useritem.setelement(user, item, rating); 6 Vector userrow = useritem.getrow(user); 7 for (int i= ; i < userrow.size(); i++) 8 if (userrow.get(i) > ){ 9 int count = coocc.getelement(item, i); 10 coocc.setelement(item, i, count + 1); 11 coocc.setelement(i, item, count + 1); 12 } 13 } 14 Vector getrec(int user) { 15 Vector userrow = useritem.getrow(user); Vector userrec coocc.multiply( userrow); 17 Vector rec = merge(@global userrec); 18 return rec; 19 } 20 Vector merge(@collection Vector[] alluserrec) { 21 Vector rec = new Vector(allUserRec[ ].size()); 22 for (Vector cur : alluserrec) 23 for (int i = ; i < alluserrec.length; i++) 24 rec.set(i, cur.get(i) + rec.get(i)); 25 return rec; 26 } 2Page 14 State in Data-Parallel Processing new rating rec request updateuseritem View, Master, Slide Master to change this text to the title of your presentation n 1 n 2 Task Element (TE) We describe an imperative implementation of a machine getuservec user Item State Element (SE) dataflow updatecoocc getrecvec coocc n 3 merge rec result

15 Simpler Imperative - Imperative Big Data Processing Two simple abstractions: partitioned state & partial state state merge partitioned state partial state Partition state state is partitioned across operators instances Partial state each instance has local state that can be merged when necessary Page 15 View, Master, Slide Master to change this text to the title of your presentation

16 Conditional Future Perfect One model to rule them all? Recap: Going to the zoo Transaction processing Batch processing: MapReduce Iterative processing: Spark Stream processing: Storm, Seep More: graph processing, deep learning, etc.. Can a new comprehensive model emerge? Spark can do graph and batch processing, has been extended to "micro-batch" stream processing Microsoft Naiad & our SEEP integrate stream and iterative + batch Can we integrate transaction & stream processing? Page 16 View, Master, Slide Master to change this text to the title of your presentation

17 Conditional Future Perfect Current Data Processing Architectures Online processing User Near-real-time processing Offline processing User Web Front End events Apache Kafka HDFS Web Front End transactions m m m r r r acce share Transaction Processing System Serving Layer Stream Processing System Batch Processing Unified Stream and Transactio Processing System (a) Different models, complex interoperation, duplicate code, data movement, resource sharing (b)

18 Conditional Future Perfect Unified Stream and Transaction processing Online processing User Near-real-time processing Offline processing User Web Front End events Apache Kafka HDFS Web Front End transactions m m m r r r access to shared state Transaction Processing System Serving Layer Stream Processing System Batch Processing Unified Stream and Transaction Processing System (a) (b) Page 18 View, Master, Slide Master to change this text to the title of your presentation

19 Conditional Future Perfect Unified Stream and Tx processing Why now? Renewed interest in in-memory databases, most business core DBs fit in memory! Stream processing are becoming more sophisticated in terms of state management Fault-tolerance approaches are becoming similar: deterministic databases, allow replication/ recomputation approaches Challenges Unify processing models operations (Tx with nondeterministic ordering) vs events (mostly deterministic) Synchronisation and ordering are key to scalable performance Performance vs generality tradeoffs (80% on 80%?) Page 19 View, Master, Slide Master to change this text to the title of your presentation

20 Conclusion Rapid evolution of data processing systems Transaction processing MapReduce In-Memory Processing Stream processing Each with their own strength Lack of cohesive view and interaction problems It is time to look back and map existing solutions work on integration, synergies, and tradeoffs Happy to collaborate on any of the above Page 20 View, Master, Slide Master to change this text to the title of your presentation