Centinel: Streaming Data Handler. September 07 th, 2015

Transcription

1 Centinel: Streaming Data Handler September 07 th,

2 An Overview 2

3 Centinel: Streaming Data Handler The Centinel project aims at providing a distributed, reliable framework for efficiently collecting, aggregating and sinking streaming data across Persistence DB and stream analyzers (example: Graylog, Elastic search, Spark, Hive etc.). This framework enables SDN applications/services to receive events from multiple streaming sources (example: Syslog, Thrift, Avro, AMQP, Log4j, HTTP/REST etc) and execute actions like network configuration/batch processing/real-time analytics. Core features of Centinel framework are: Stream collector - Collecting, aggregating and sinking streaming data Log Service - Listen log stream events coming from log analyzer Log Service - Enables user to configure rules (example: alerts, diagnostic, health, dashboard etc.) Log Service - Performs event processing/analytics User Interface - Enable set-rule, search, visualize, alert, diagnostic, dashboard etc. Adaptor - Log analyzer plug-in to Graylog and a generic data-model to extend to other stream analyzers (Logstash etc.) REST Services - Northbound APIs for Log Service and Steam collector framework Leverages - TSDR persistence service, data query, purging and elastic search 3

4 Centinel: High Level Architecture 4

5 Centinel: Functional Overview Feature Streams Alerts Search and Analyze Details Mechanism to route messages into categories in real time while they are processed like stream for audit logs( install bundle etc.) Rule configuration includes message, level, source etc. Streams are generated by GrayLog server as per user-defined rules. Event-handler module handles streams from GrayLog server and persist it into Centinel Log alerts : Alerts get generated based on specific event matching in real-time Alert condition types : Message count condition, Field value condition, Field string value condition Alerts get cleared if specified condition does not persist. Alert check interval time is configurable, default is 60 seconds. Common alert operations like manual acknowledge, delete, filter, sort etc. will be supported Support for search query language. Time range for search can be specified Visualization includes histogram 5

6 Centinel: Functional Overview (continued) Feature Dashboard Diagnostics Details Build pre-defined views on data by adding widgets. Domain expert can define search query and save results on dashboard. Search result type : Search result counts, Search result histogram charts, Field value charts,quick value results Enables, user to specify/configure group of log messages (events or specific conditions ) as single rule. Order of messages and condition for each messages can also be configured. Notification will be persisted on db if log messages are coming out of order/sequence On clicking notification on UI, popup window will open which displays expected order and received order of log messages. User will be able to scroll popup window to see individual received log message. Event processor has intelligence to verify sequencing of messages and generate diagnostic reports Success notification is displayed when all specified conditions are true. Health User specify list of messages with conditions for feature. Feature health will be changed to Critical, Major, Warn on reception of all messages specified 6

7 Centinel: Project links Project Main page Project Proposal Release plan Project archives Repository 7

8 Design Approach Incremental updates on design document 8

9 DLUX LITHIUM DDoS Protection OpenStack Neutron Legend AAA: Authentication, Authorization & Accounting ALTO: Application Layer Traffic Optimization AuthN: Authentication BGP: Border Gateway Protocol CAPWAP: Control and Provisioning of Wireless Access Points COPS: Common Open Policy Service DIDM: Device Identification and Driver management DLUX: OpenDaylight User Experience DDoS: Distributed Denial Of Service SDNI Wrapper DOCSIS: Data Over Cable Service Interface Specification FRM: Forwarding Rules Manager GBP: Group Based Policy IoTDM: Internet of Things Data Broker LACP: Link Aggregation Control Protocol LISP: Locator/Identifier Separation Protocol MAPLE: Maple Programming NIC: Network Intent Proposal OVSDB: Open vswitch DataBase Protocol OPFLEX: Extensible Policy Protocol VTN Coordinator PCEP: Path Computation Element Protocol PCMM: Packet Cable MultiMedia Plugin2OC: Plugin To OpenContrail SDNI: SDN Interface (Cross-Controller Federation) SFC: Service Function Chaining SNBI: Secure Network Bootstrapping Infrastructure SNMP: Simple Network Management Protocol SXP: Source-Group Tag exchange Protocol TSDR: Time Series Data Repository TTP: Table Type Patterns USC: Unified Secure Channel VTN: Virtual Tenant Network Network Applications Orchestrations and Services Topology Inventory FRM AAA- AuthN Filter OpenDaylight APIs (REST) NB APIs ALTO DIDM GBP Service SFC SDNI Aggregator L2 Switch VTN Manager Neutron Service NIC Reservation VPN Service Applications MAPLE Topology Processing Discovery DOCSIS Abstraction BGP PCEP Plugin20C TCP- MD5 OVSDB LISP Service IoTDM USC Manager LACP Plugin Services MD-SAL / Yangtools Persistence Centinel TSDR Controller platform OpenFlow NETCONF SNBI PCMM/C OPS BGP PCEP Plugin20C SNMP OVSDB LISP IoT USC CAPWA P SXP OPFLEX SB interfaces & protocols plugins OpenFlow Enabled Devices Open vswitches Additional Virtual & Physical Devices 9

10 Introduction and Data Models Centinel framework enables SDN applications/services to receive events from multiple streaming sources (example: Syslog, Thrift, Avro, AMQP, Log4j, HTTP/REST etc) and execute actions like network configuration/batch processing/real-time analytics. The entire architecture can be divided into the following modules: User interface Centinel NB APIs Stream Analyzer Service (LAAS and Plugin) Log Service ( Configure Rules and Event handler Modules) Stream Collector Service (Flume and Sqoop based) Leverage persistence database service This design document depicts the subset of functionality (Alert) and the same is subject to follow-up add-on updates. Incremental updates on design document The stream data handler is a framework for collecting, aggregating stream data across Persistent DB and stream analyzer. To achieve the same, the entire model could be broken up into two parts: Rules Configurable from Northbound Notifications/Events Output from stream analyzer 10

11 Rules The Northbound of Centinel will allow operators to set Rules for collecting, aggregating and analysis of streaming data. The following diagram explains the Centinel data model for log services where in the BaseRule is the basic model object that is imported by BaseAlert which is further used by different types of rules. BaseRules model object is the basic data model that will be shared/imported by all the other Log service features like Alerts, Streams, Health, Dashboard and any other feature that Centinel framework may want to support in future. Incremental updates on design document 11

12 Events When a user configures a rule successfully, notifications/events corresponding to the rule will be supplied by the stream analyzer. The diagram below shows the data model for events (notifications and alert) received from stream analyzer. The base model for notifications/events is BaseEvent that contains event type and stream and is further extended by alarm that provides detailed information about the event that has occurred for example the stream which raised the alarm, alert condition, feature or module from which alert was raised etc. 12

13 Alerts Incremental updates on design document The diagram below shows the class diagram for RPCs and listeners. It defines the following interfaces and java classes for handling the Alerts: LAAS interface and its implementation that provides log analyzer abstraction layer and also publish notifications. Plugin class that provides alarm call backs from the analyzer (grey log server). AlertService interface and its configuration Implementation that defines and implements CRUD on Alert Rules and also write it in md-sal data-store. AlertServiceListener class that listens to any data change in md-sal database. ServiceAggregator and ServiceEventHandler classes that registers for notification and receive notification published by the LAAS as DTO ServicePersistData class that stores the DTO from event handler into the persistent database 13

14 North Bound APIs Feature Details NotifyAlarm() SetAlertRule() RemoveAlertRule() GetAlertRule() getalarms() RPC exposed by LAAS. Plugin use it to notify alarms Set rules for alerts Remove rules for alerts Get all alert rules Retrieve recent 300 alarm. Incremental updates on design document 14

15 Sequence diagram for Rule Handling Following is the sequence diagram for the Rule handling part. It describes the flow for How is a rule for Alert getting configured. Incremental updates on design document 15

16 Sequence diagram for Event Handling Below is the sequence diagram for the Rule handling part. It describes the flow for processing an Event (Alarm in this case) before it get stored in the persistent data store. Incremental updates on design document 16

17 Thank You 17