Real-time distributed Complex Event Processing for Big Data scenarios

Size: px

Start display at page:

Download "Real-time distributed Complex Event Processing for Big Data scenarios"

Nicholas Wade
10 years ago
Views:

1 Institute of Parallel and Distributed Systems () Universitätsstraße 38 D Stuttgart Real-time distributed Complex Event Processing for Big Data scenarios Ruben Mayer

2 Motivation: New Applications in IT Modern IT systems need to react to real-world situations Example: Enable Demand Response with Smart Grids + energy consumption energy production manage appliances High-rate event streams must be processed in real-time Gap between low level sensor readings (consumption and production rates) and high level situation (manage appliances) 2

3 Complex Event Processing Distributed Complex Event Processing (CEP) can be used to solve this problem Operator network processing of event streams switch on / off Analyze Aggregated rates Aggregate Consumption rates Aggregated rates Aggregate Production rates 3

4 IT Infrastructure Changes New, heterogeneous infrastructures Multi-core / Many-core systems Cloud Computing Computing ressources at the edge of the Internet Goal: Make CEP fit for such infrastructures Make use of multiple cores Elastic scaling in the cloud Push computing towards the edge of the network This development can make new applications possible Highly scalable Reliable Elastic 4

5 Research Problem: Reliability This talk focuses on reliability Node and communication failures Manufacturer Billing Customer Information Loss of operator state Events arrive late Event streams must still be reliable fail Delivery of a package of 3 artifacts for 300 $ No false-negatives No false-positives Source events false-negative Delivery of a package of 2 artifacts for 250 $ false-positive 5

6 Research Problem: Reliability State of the art Active/Passive Replication Rollback-Recovery with checkpoints Problem Find methods with low run-time overhead that offer real-time processing better scalability than existing approaches Approach: Develop processing model for CEP operators shows inherent operator properties better recovery methods can be developed 6

7 Operator Model All operators ω: Correlation of events is performed in steps Selection of events σ from incoming streams gets correlated A set of events (e 1,...,e n ) is deducted from that selection Correlation function f ω : σ (e 1,...,e n ) describes a correlation step σ f ω (e 1,...,e n ) incoming events ω outgoing events General observation: Processing of a selection is independent from processing of other selections Correlation function is stateless 7

8 Savepoint Recovery A rollback-recovery method that induces less run-time overhead Ensures strong reliability No false-positives, no false-negatives Works without persistent checkpoints Recovery of Incoming streams Current selection on them Incoming streams can be re-streamed from predecessors Information on current selection needs to be captured Execution model operator reveals selection information 8

9 Future Work Real-time recovery guarantees for savepoint recovery Modelling of different classes of reliability requirements Apply the optimal reliability method Find new, reliable parallelization methods Easy integration of operators Elastic parallelization degree Combine with reliability methods 9

10 End of Presentation Questions, Comments and Discussions 10