Toward a Smart Platform for Data Stream Processing in Smart Cities: Distributed Qos-aware Scheduling in Storm

Transcription

1 Toward a Smart Platform for Data Stream Processing in Smart Cities: Distributed Qos-aware Scheduling in Storm Valeria Cardellini, Vincenzo Grassi, Francesco Lo Presti, Matteo Nardelli

2 Smart City and Data Stream Processing Data Stream Processing (DSP) continuously process streams of data generated by multiple, distributed sources, to extract valuable information

3 DSP Applications conceptual model Possible (Centralised) Solutions

4 DSP Applications over Fog Compunting distant Clouds nearby Fog Devices/ Cloudlets mobile devices

5 Cloud Computing/Data Center Approach Centralized computation at large-scale data centers Distant to be accessed by mobile devices or urban sensors Assumption: enough bandwidth to collect the data Organizzazione Corso 4

6 The Fog Computing Approach Distant Cloud extended with small data centers at the edges of the network, near to the customers Organizzazione Corso 5

7 Our Vision Provide an integrated platform that facilitates development, deploying and the run time management of smart data-stream services for urban environments that are valuable for their stakeholders designers/developers development and (pre-deployment) validation tools providers QoS assurances, effective provisioning, runtime adaptation users/citizens trustable services

8 Synapsis Middleware Execution Environment Resource Abstraction/Virtualization Stream Oriented Communication Service Composition Discovery Selection Composition Qos Aware Run Time Management QoS Monitoring Run time Adaptation Service addition, removal, substitution, rewiring, 7

9 Storm Architecture 8

10 Distributed Scheduling in Storm 9

11 Storm Extension Storm Centralized scheduler (Nimbus) Developed extension Distributed scheduler (Supervisor) Round-robin scheduling policy QoS-aware scheduling policy Agnostic to network delays and other nodes attributes Not aware of environmental and stream rate changes Enhance the system with adaptation capabilities, introducing the MAPE model Analyze Knowledge Plan poor performance in geographically distributed and highly dynamic environment Sensor Monitor Execute Managed Resources Actuator 10

12 QoS-aware Scheduling in (distributed) Storm Centralized Solution (Benchmark) å C i x i,u C u "u Î V res iîv dsp Distributed Algorithm (Pietzuch et al.) Operator placement algorithm represents the application as a system of springs minimizes the elastic energy of the system å x i,u = 1 "i Î V dsp uîv i res å x i,u = y (i, j ),(u,v) j vîv res å x j,v = y (i, j ),(u,v) i uîv res "(i, j) Î E dsp,(u, v) Î V i res V j res "(i, j) Î E dsp,(u, v) Î V i res V j res x i,u Î {0,1} "i Î V dsp,u Î V i res y (i,u),( j,v) Î {0,1} "(i, j) Î E dsp,(u, v) Î V i j res V res 11

13 Conclusions & Future Works Integrated platform for DSP applications development, deployment and run time management Middleware Approach: Extending existing frameworks/platform Compatible with existing frameworks/applications Distributed implementation Heterogeneous environments Include location, privacy concerns QoS scheduling Integrate SDN-capabilities Storm Distributed scheduling Adding stateful migration Mesos Distributed implementation 12