Middleware Support for Real-Time Stream Processing Luigi Romano

Transcription

1 Middleware Support for Real-Time Stream Processing Luigi Romano Dipartimento per le Tecnologie (DiT) Università degli Studi di Napoli Parthenope HAL Software Research Report 50th Meeting of the IFIP 10.4 Working Group on Dependable Computing and Fault Tolerance Annapolis, MD, USA July 2, 2006

2 Roadmap 1. Rationale 2. Objectives of Project 3. Project Leader 4. Other Partners 5. Fundings 6. Overall Architecture 7. Application Domains 8. Preliminary Results 9. Contact Info 1

3 Rationale Presentations: Koji Manish Special session (Dependable Computing in 2031): Tom & Robert Luca The problem is not the information it is the ability to process it in a fast and reliable way 2

4 Objectives of Project Proposal To design and implement a middleware infrastructure for real-time processing of data Multiple data feeds Geographically distributed Highly heterogeneous Self-* properties Self-configuration Self-optimization Self-healing Self-protection Real-time behavior Challenging performance requirements (more later) Stream processing computing paradigm (more later) 3

5 Stream processing computing paradigm 4

6 Project Leader - 1 Dipartimento per le Tecnologie (DiT) - Università degli Studi di Napoli Parthenope docentiweb 5

7 Project Leader - 2 Number of students 6

8 Companies: Other Partners Intesis Engineering ( Sync Lab ( HAL Software Research Centers Institute for High Performance Computing and Networking - National Research Council ( 7

9 Funding Request & Mechanism Option 1 National Government European Community 0.75 * 2M Euros Regione Campania Intesis + Sync Lab + HAL 1M Euros DiT + ICAR 8

10 Regione Campania Caserta Napoli Benevento Avellino Salerno 9

11 Funding Request & Mechanism Option 2 Intesis + Sync Lab + HAL + DiT + ICAR 10

12 Overall Architecture & Application Domains Ontologies & Rules Reasoner Stream Processor EVENT CHANNEL I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor I/O Adaptor Firewall Antivirus Content IDS Phisical Access Control Video Building Automation Allarms Front End Gateway Enterprise Service Bus Enterprise Information Service PACS Server RIS Server Multi Channel Information Server Other Domains Network Bus DB Security&Safety Domain Telecommunication Domain Telecare Domain 11

13 RIS/PACS Sample Application 12

14 Additional Application Domains 1. Multi-modal transport 2. Intermodal Node Main Office GSM, UMTS to remote TRACKER TRACKER PDA/ GPS equipped Vecto (PDA equipped ) Disconnection due to a gallery on the roadmap Disconnection due to an absence of GSM /UMTS signal Disconnection due to the battery discharge Customs house Intermodal Node PORT Next Vector 13

15 Telecom Domain Case Study

16 Telecom Domain Case Study - 2 Data Feed Data Feed Data Feed Log Log Telecom Operator Log Data Feed Log Recharges Analysis System 15

17 Issues & Requirements Highly heterogeneous Log Files New Log types added at run-time Large Data Volumes/Rates Up to 300 Events/sec logged (can t achieve with current storage technologies) Soft Real-Time constraints 16

18 Old Architecture Structure of Log File impacts structure of component: Changes in Log File => Lot of Coding Data Aggregation Layer Multiple synchronization points => high overhead 17

19 New Architecture Worker Log Log Data Stream Data Stream File Monitor File Monitor DAC Grammar-Based Log Specific Parser Parser Handler Msg Filter MsgGenerator Msg Queue Msg Manager Data Aggregation Layer DB 18

20 Key improvements Parser processes a data stream: No more delays do to synchronization at buffers/queues Simple and regular log file structure: Described by means of a grammar Widely general, easy-to-configure parser: Code implementing specific parsers dynamically generated (by means of a parser generator), compiled, and integrated A parser generator is a tool that reads a grammar specification and converts it to a Java program that can recognize matches to the grammar void field0() : { } { (<ALARM> <DEBUG> <ERROR> <FATAL> <INFO> <OFF> <WARN>) {} } void field1() : { } { <LABEL> (<SPACE>)* {} } 19

21 Technologies Tomcat (in the development phase) BeaWeblogic (in the deployment phase) products/weblogic Java Compiler Compiler (JavaCC 4.0) - The Java Parser Generator Oracle 9.1 Oracle TimesTen - Main Memory DBMS x.html? FPGA? 20

22 Contact Info Luigi Romano Mobile: