Event Logging - Introduction

Transcription

1 Event logging Event Logging Introduction Event Logging Listener Overview Log Listener OSGi Event Listener SAM Listener Event Logging Agent Introduction Agent Receiver part Agent Processing Part Event Logging Sender REST Sender JMS Sender Event Logging Server Overview Collector Server Preprocessing part Server Persistence and postprocessing Event Logging Service Event Logging Search Server Event Logging Search UI / Web Application Event Logging API s and Data Structures Event Data Structure Event DB Structure Event Logging Service API

2 Event Logging - Introduction The Event logging feature within the Talend ESB allows users to collect events across distributed containers and also provides the ability to index them and search through them via a Web UI. Additional to the pure collection of events the Event Logging feature support custom processing (e.g. custom filtering, customer data enrichment/reduction), aggregation, signing and also server side custom pre and post processing of event to e.g. send them also as a post processing step to a Intrusion Detection system or to any other kind of potential higher level log processing / management system. Event Logging is a Talend ESB Enterprise feature and will only be included in the Talend ESB Enterprise version and the Talend Platform (ESB) products. We also support with 5.4. the use of the SAM feature (SAM Agent + SAM Retrieval Service / TAC SAM UI) in combination with the Event Logging feature but the use of the Event Logging or the SAM Server is optional for Enterprise customers and can be defined as needed. Talend ESB SE can only be used with the SAM Server as of now. We provide a log indexing and search on Event Logs based on Elasticsearch + Kibana and we will name this as the Talend Event Logging Search Server / Search UI (see below in this document)

3 By this the primary sub parts of the Event Logging are: Event Logging Listener ( allows to get log information and pass it to the Agent for further local processing) Event Logging Agent: Receives events from listeners, buffer event, process events and sends events to the final destination using one of the configured Event Logging Appenders Event Logging Sender: sends events to Event logging backend. We provide a JMS Appender and REST Appender for the EventLogging RESTService by default but customers can add their own appenders. (e.g. to write into a local log file or to a local elastic search instance) Event Logging Service: is a RESTful Service to collect events sent from the Event Logging REST Appender and to retrieve events from the backend. (Event DB stored events only in 5.4) Event Logging Collector: Routes which listen on endpoints to receive events for further server side processing by the Event Logging Server. By default we provide a Standard Collector which expose a fixed Direct VM endpoint and a JMS Collector which takes Event out of a JMS Queue. Event Logging Server: Retrieves events from the collector and performs processing / persistence of events, with support for Event DB (RDBMS), SAM DB (SAM Events only), Elasticsearch with some custom pre, post processing extension points.

4 Event Logging Search Server (based on ElasticSearch) Event Logging Search UI (based on ElasticSearch / Kibana) The following high level architecture shows the different components and their relations: Technically the Event Logging Listener, Agent, Sender, Collector and Server are implemented using Apache Camel. The Event Logging Service will be a CXF (JAX RS) based service developed in Java (using also Apache Camel for the Receiver part) and the ElasticSearch server and Kibana based Search UI will be made available as separate deployment outside the Talend Runtime

5 container and is optional for the use of the Event Logging feature. The Event Logging DB will be supported on all databases we support also for the SAM DB in 5.4. (incl. PostgreSQL) The following chapter describe the individual components of the overall Event Logging feature and the Data Structures and public API of the feature. Event Logging - Listener Overview We will support three listener with Talend ESB 5.4. Log Listener to get all pax logging log events OSGi Event Listener to get all OSGi Events on Talend Runtime SAM Listener to get SAM Event (via SOAP) from the SAM Agent additionally the customer can also create his own custom listeners. From an architectural point the listeners are tightly coupled with the Event Logging Agent and require the Event Logging Agent to work.

6 Nevertheless, each listener can be started and stopped individually (single OSGi Bundle per Listener) Additionally to the above listeners which we provide as part of the standard product container / feature we also support custom listener which the user can implement using an Apache Camel Route (e.g. using our RouteBuilder tooling) as long as: the same Apache Camel version is used as used in our container where the listener will be deployed the direct vm component is used as producer with a fixed destination (routeid) eventloggingagent Deployed on the same JVM/Talend Runtime container as the Talend Event Logging Agent the Camel Exchange Header must contains the log information in the related event properties structure (event related camel exchange header properties ) and the log message itself within the exchange body (plain text) EventCategory is optional for the Listener but prefered to be assign/defined already in the listener

7 Log - Listener The log listener allows the user to get all the log information which typically will also be in tesb.log into the event logging and by this it is the primary listener for the Event Logging Agent. We use the pax logging which is used by Karaf / Talend Runtime as the entry point for the Event Logging Log Listener and by this the user can use the standard pax logging configuration also to configure which logs should be send to the log listener and which not. (Like for any other log appender ) To enable logging via the Log Listener the following minal entry is need in the org.ops4j.pax.logging.cfg log4j.rootlogger=info, out, osgi:vmlogappender, osgi:eventloglistener The osgi:eventloglistener entry will now also provide the log information to the Log Listener

8 Additionally/Alternatively the Log Listener will expose a second paxlogging appender named eventloglisteneraudit where all log events send through this appender will be set to audit=true and this would have the highest priority of all audit configurations. (In general if an Event has the audit flag set to true no other of our components / mappings will change it back to false but if the audit flag is false we might set it to true later on by additional mapping options). The eventloglisteneraudit can be used in the pax logging config and would automatically make all events send via this appender to Audit events. Additionally also configuration like log level filtering or package based filter as it can be configured in the standard org.ops4j.pax.logging.cfg file can be applied as for any other log destination/appender. The log listener is implemented using Apache Camel using (paxlogging: ) and provides a configurable Category mapper processor with a direct vm communication to the Talend Event Logging Agent. The paxlogging: component will have a fixed configuration to expose eventloglistener as appender name (paxlogging:eventloglistener) therefore the log4j.rootlogger=info, out, osgi:vmlogappender can be extended by osgi:eventloglistener to enable logging via the Event Logging Log Listener.

9 As pax logging is used, we can capture all the log messages via this listener which are send via different logging frameworks which we support with ESB 5.4 at least: Log4J based logging: log4jlogger.info("log4j log message"); SLF4J based logging: e.g. slf4jlogger.info("slf4j log message"); JDK based logging: e.g. jdklogger.info("jdk log message"); JCL based logging: e.g. jcllogger.info("jcl log message"); Juli based logging: e.g. julilogger.info("juli log message"); and in case the logging framework supports MDC attributes, also the MDC attributes are taken into the event as additional meta data. Which would also allow custom / business code logs to contain e.g. a business correlation id or other important meta data in a structure form, e.g. using MDC.put( CorrelationID, "abc"). The Log > Event conversion step will transform the log message format into the Event format, where essentially the Event Structure is filled as complete as possible and the additional meta data (incl. the MDC attributes) are transformed in the Event Customer Information (Key, Value) list. The Category Mapper allows to define a event logging category in a configurable way to the log message by package name from where the message comes from and if messages from this package should be treated as audit or non audit messages. The Log listener uses the following configuration file and

10 parameters for configuration: org.talend.eventlogging.listener.log.cfg category.default=system category.attribute=eventcategory audit.attribute=audit category.mapping.org.apache.cxf = service category.mapping.org.apache.cxf.rt.security = security(audit=true) category.mapping.org.apache.wss4j = security(audit=true) category.mapping.org.talend.esb.sts=security(audit=true) The configuration is primarily for the Category Mapper and will be used as follow: 1) The first priority to define which EventCategory is assigned to the current log event is defined by an (MDC) attribute, the name of this attribute can be configured under category.attribute. By default it is eventcategory. If this attribute exists the Category will be set to the value of this attribute. 2) If the attribute does not exists the Category Mapper will look for the category.mapping. configuration and will build a tree in memory and will equally to the log4j log appender package based filter configuration go from the leaves to the root of this tree to find the closed node to the given package name and will apply the

11 category and audit flag definition as defined in the config file ( only if the audit attribute is not set to true already no overwrite of true is allowed for the audit flag once it is true it will stay true.) 3) If the log event has an attribute with the name specified in the audit.attribute configuration, audit flag will be set to this value (true / false) only if the audit attribute is not set to true already. Example: in the example if a log message comes from a package org.apache.cxf.binding.soap it will be mapped to service category (with audit=false) as the org.apache.cxf is the matching node configuration. If a log message comes from org.apache.cxf.rt.security.saml it will be mapped to the security category and marked as audit message by the additional audit definition after the category. By default if (audit) is not explicitly defined in this way, event will be set to audit = false (default behaviour). After this step the log message is sent to the Event Logging Agent in a synchronous way (direct vm) to allow the agent to do a short preprocessing before the event is stored into a local buffer within the agent for final processing and sending to the backend. OSGi-Event - Listener The OSGi Event listener allows the user to get the OSGi Events from the Talend Runtime container which are published via the

12 EventAdmin service. To enable event logging via the OSGi Listener only the related listener bundle needs to be started, no other container related configuration is required by our standard Talend Runtime container. The OSGi Event listener is implemented using Apache Camel (eventadmin: ) and provides a configurable Topic Filter and Category mapper with a direct vm communication to the Talend Event Logging Agent. The eventadmin: component has a fixed configuration to listen (subscribe) to all topics (eventadmin:*) and we provide an easy to use topic filter within the OSGi Event Listener. In the configuration the user can provide a list of topics to be included and/or excluded, incl. the use of * for all. filter.include.* filter.include.org/osgi/service filter.exclude.org/osgi The topic filter allows to quickly exclude all and include just a few selected ones or include all and just exclude some selected ones.

13 As shown in the example each topic (partial topic) requires one row. And if an included topic matches to the current topic, it will be included as long as it is not matching an excluded topic (include has priority before exclude is evaluated) The example above will lead to: all events are included except the one which start with org/osgi/ but if the current topic starts with org/osgi/service/ it will be also included. The OSGi Event > Event conversion step will transform the OSGi Event into the Event (Logging) format, where essentially the Event Structure is filled as complete as possible and the additional meta data (incl. the OSGi Event Properties) are transformed in the Event Customer Information (Key, Value) list. The Category Mapper allows to define a event logging category in a configurable way for the OSGi Event based on the OSGi Event topic and if this event should be treated as audit event or not. Configuration related to the Category Mapper: category.default=osgi category.attribute=eventcategory audit.attribute=audit category.mapping.org/osgi/framework/serviceevent= service category.mapping.org/osgi/service=service(audit=true) The priority is equal to the log category mapper. After this step the OSGi Event message is sent to the Event Logging Agent in a synchronous way (direct vm) to allow the agent

14 to do a short preprocessing before the event is stored into a local buffer within the agent for final processing and sending to the backend. The OSGi Event listener uses the following configuration file and parameters for configuration: org.talend.eventlogging.listener.osgievent.cfg filter.include.* filter.include.org/osgi/service filter.exclude.org/osgi category.default=osgi category.attribute=eventcategory category.mapping.org.osgi.framework.serviceevent= service category.mapping.org.osgi.service=service(audit=true) Similar to the log listener we will allow to set the audit flag based on the audit.attribute (which is here an OSGi Property) or on the category mapping. In Talend ESB 5.4. it is the only property which can be set as part of the mapping but the notation (attribute=value) would allow us to extend this in future versions also to other attributes (attribute1=value1; attribute2=value2; custominfo={{key1, value1}{key2, value2}} ) but this is only an outlook and will not be available in Talend ESB 5.4 SAM-Listener The SAM listener allows the user to get the SAM Events from the

15 Talend SAM Agent into the Event Logging Agent with the ability to use all the features of the Event Logging also for SAM Events and also to combine easily SAM Events with other Event Logging events into a single search or audit list. To enable event logging via the SAM Listener only the related listener bundle needs to be started and as usual for SAM the related design time configuration to use SAM must be applied (or as with Talend ESB 5.4. and higher) the related SAM Custom policy must be used at runtime via the Service Registry to enable Service Activity Monitoring (SAM) for the related ESB Consumer/Provider. The SAM listener is implemented using Apache Camel (cxf: ) and expose a SOAP Service which is exactly the same as the SAM Server (SOAP Service MonitoringServiceSOAP with the putevents operation See WSDL below). In case the SAM Listener should be used it is important to stop the SAM Server first in case this service is started on the current container (tesb:stop sam) WSDL of the SAM Server (To retrieve SAM Events)

16 ( ) Essentially the SAM Listener exposes the SAM Server API and by this the SAM Agent just needs to use the right URL to send the SAM Events to the SAM Listener instead of the SAM Server. In both cases it can be a localhost or remote host. Means that also a single server side Event Logging Agent with one SAM Listener can be used as a remote replacement to the standard SAM Server, this is the only listener so far which can be use local and remote. The Event Logging SAM Listener only supports the following two configuration settings which is used in the SAM > Event converter: org.talend.eventlogging.listener.sam.cfg

17 map.category=sam map.audit=false map.subject.samltoken=false map.subject.usernametoken=false map.subject.httpbasicauth=false map.category = sam will mean that all events captured by the SAM Listener will be assigned to the Event Category sam and that all events will be set to audit=false but can be easily changed. map.audit=true if SAM events from this container would be treated as audit events. The above configuration will be used as default (category = sam ; audit = false) map.subject.. especially in the SAM Event listener we have access to information about the user / subject of the Service call and we allow with the following options to enable that this user/subject is mapped as the Event Subject or not. For performance reasons we will search from the top of the log message (exchange body) by string search to the related element name and get the values by pure string search to avoid a explicit xml document creation. map.subject.samltoken=true will map the subject of the SAML token to the subject attribute of the event (true /false) map.subject.usernametoken=true will map the user name of the Usernametoken to the subject attribute of the event (true /false) map.subject.httpbasicauth=true will map the username provided in the http header to the subject of the event

18 (true /false) Event Logging - Agent Introduction The event logging agent is responsible to collect events from the Event Logging Listeners, to buffer these events for further local processing, to process the events (incl. signing, data enrichment, custom processing) and to send out events to the central event logging server. The Agent should do only the required minimal processing before an event is stored into the local buffer to allow as less as possible impact on the listener to wait. And the Agent should work offline as long as the buffer technology supports to collect events locally and allows events and containers to continue to be collected while the needed backend services are temporarily unavailable. (Where we would currently expect that temporary is <= 8 hours but only as an estimate not a fix limit) In the following chapters we splitted the Agent into two logical parts: Receiver part to get events from the listeners until they reach the local buffer Processing part to process each event from the local buffer The two parts will be described in the following chapters in more

19 detail. Agent - Receiver part The Agent Receiver part is responsible to get the messages from the local listeners and do a minimal processing before the event get stored in the local buffer. 1) The Event Logging Listener sends each event via in memory synchronous communication (direct vm component) to the Agent using a fixed routeid. eventloggingagent 2) The user can optionally configure a custom route at this early stage to filter, to shorten, to enrich meta data etc. the log event as it was received by the listener.

20 The custom route must be deployed on the same JVM / same Container and must expose a unique routeid via the direct vm component. The custom route has full access to the Event data. The configuration part in the agent config ( org.talend.eventlogging.agent.cfg ) is as following for this part: agent.receiver.custom.routeid.default=mycustomreceiverroute agent.receiver.custom.routeid.audit=mycustomreceiverroute agent.receiver.custom.routeid.security=mycustomreceiversecurityroute The agent.receiver.custom.routeid. is the fixed part followed by the event category. default and audit are reserved categories and can not be used as normal category names. Default will be used if no specific mapping for the current event category is found and the default custom route will be called. If the Event is already marked as audit event and if a audit custom route is defined this one will be called. Even so we do not limit what the user can do at this stage with the custom route, it is strongly advised that these early processing routes are designed for max performance as we are still in a synchronous processing with the listener at this stage. By default no custom route for none of the categories is defined. (the above example will be commented with a # in our default agent config file for reference but it will be commented and no custom route will be called by default) 3) The data enrichment component in the agent will add the minimal required system data to the event before it is stored for further processing in the local buffer. If one of the following attributes of the Event Data Structure is not already filled (e.g. by the listener, customer route) the Data Enrichment at this stage will fill the values as follow:

21 Attribute Value Remark id string (UUID ) generated within agent category agentid agent_time stamp string (default= system ) string (Unique agentid ) timestamp If not empty, we will try to find an attribute equal to the value defined in event.category.map.attribute if the attribute does not exists in the custominfo (key/value) collection of the current event we will set the category as defined in the agent.cfg eventcategory=xxx as fallback. This attribute must be defined in the agent.cfg (mandatory) Default: eventcategory=system from agent config file (property agentid=agent1) local machine date / time (converted to UTC time see event structure e.g T08:45:30.453Z) hostname string (hostname) hostname of the current machine ( if hostname can t be resolved via java api we will use the ip address as fallback. ) processid string (processid) current JVM process id

22 audit auditsequen ceno boolean (default=false if not set at this stage) long unique sequence number for audit event only if the event is marked as audit a unique sequence id will be assigned (the sequence id is a long number starting by 1 and increased by 1 for each audit event from this agent (for non reputation agentid + auditseqenceno should show on the backend Event DB tables no gaps). The auditsequenceno will be persisted. in a local file and saved each time a new number is used subject string if not empty, the subject will be set to the value of an attribute with the name defined in event.subject.map.attribute by this the listener or even the user can provide this value in his logs as attributes (e.g. for log events as MDC attribute) If no attribute exists we use the event.subject.default value (which if not defined will lead to an empty subject)

23 correlationi d string if not empty, we will use event.correlationid.map.attribute configuration to find an attribute (in the eventlogging_custominfo attribute collection) Configuration parts for this step in the agent.cfg are: agentid=agent1 agentsequencefile=${tesb.home}\etc\eventlogagent.seq event.category.default=system event.category.map.attribute=category event.subject.default=unknown event.subject.map.attribute=subject event.correlationid.map.attribute=correlationid event.subject.map.attribute=subject 4) The Router will send the Event to the appropriate / configured local buffer. With Talend ESb 5.4. we support two buffer jms JMS (in 5.4 only ActiveMQ) based buffer (with or without persistent queue) intended to be a local ActiveMQ Broker for the local buffer. memory where we so far plan to use the in memory queue as provided by the vm: component of Apache Camel file Not confirmed for 5.4: a file based FIFO buffer which is less reliable as ActiveMQ but more reliable as in memory The router configuration in the agent is as follow:

24 # Receiver buffer config agent.buffer.jms.url (url of the local file buffer message broker using the camel activemq component parameter definition) agent.buffer.jms.queue =event.logging.${agentid} # Router definition agent.receiver.buffer.default=memory #agent.receiver.buffer.audit=jms #agent.receiver.buffer.system=jms By default all events will go via a memory buffer but the user can easily change this and e.g. use jms for audit event and also other event categories if need in the example above. If the user comments the agent.receiver.buffer.system=jms all event of category system (value after the static part: agent.receiver.buffer.) will be also stored via the jms buffer. Agent - Processing Part The Agent Processing Part is responsible to get the messages from the local buffer and process them up to the point where they are ready to be sent to the final destination (to the Event Logging Collector Service (via HTTP/HTTPS REST) or to the Server JMS Broker Queue.

25 Consumer of the local buffer will read existing buffered event from the buffer and start processing via a synchronous processing and send the event via direct: communication (transactional in the JMS case) to the core processing route which will start a custom processing route if one is defined for the given event category in the agent.cfg Example agent.processing.custom.routeid.default=mycustomprocroute agent.processing.custom.routeid.audit=mycustomprocroute agent.processing.custom.routeid.security=mycustomprocsecurityroute which would mean for security category the mycustomprocsecurityroute will started and for audit and all others (default) the mycustomprocroute will be called in the case of the above

26 example. By default no custom processing route will be called. In the next step the Event Log message (as stored as plain text in the exchange body) will be signed and a signedlogmesage header property will be created which contains the logmessage in a XML Digital Signature (enveloped) after this step. We use the camel xmlsecurity component (based on Apache Santuario) for the XML DSIG complaint signature creation. If signing is required it can be defined by category within the agent.cfg: Example: agent.processing.signing.default=false agent.processing.signing.audit=true agent.processing.signing.security=true The above example would say that audit events will be signed and also events in the security category (which are not marked as audit) but all others (by the default definition) will not be signed. The default is: agent.processing.signing.default=false agent.processing.signing.audit=true We will use a local keystore (trun.jks) for the private key which will be used to sign the log message. Additional configuration to find the truststore, signing parameters as needed will also be configurable by the agent.cfg file.

27 After the signing step the Camel Exchange (Event Structure ) is treated as ready and the event is send with the direct component synchronously (transactional in the JMS case) to the sender part of the Agent. The last part of the processing is a direct vm component which gets configured per event category as follow: agent.sender.destination.default=eventlogsenderrest agent.sender.destination.audit=eventlogsenderjms and by this the event will be send to the Event Logging Sender which is configured for the related category. We provide two sender destinations as product features eventlogsenderrest eventlogsenderjms Even so these values are just the routeid s where the direct vm will send the event to, by this the customer can create custom sender easily as routes with a Direct VM endpoint and by configuring the related route id of the exposed direct vm endpoint in this configuration e.g. agent.sender.destination.system=eventlogsendermysender would send the event to the route with the route id eventlogsendermysender (and this route can put the log event wherever it likes. Event Logging - Sender We support two Sender for the Agent by default:

28 a JMS based sender a REST based sender The Event Logging Sender are responsible to get the messages from the processing part of the Agent and send them to the final destination which can be either a JMS Broker Queue (in Talend ESB 5.4. we only support ActiveMQ as broker for Event Logging Events, which might change in future versions ) or the Event Logging Collector Service (http/https RESTFul service) REST Sender

29 The sender receives the event from the processing part via the direct vm: component and does a technical conversion from the exchange header and body to a json format which is stored in the exchange body and all header field of the event will be removed. After the conversion from the Exchane Event Structure to the Json structure, the aggregator will be used to optimize the network transfer with the following aggregation strategy: org.talend.eventlogging.sender.rest.cfg sender.aggregation.eventcount=10 sender.aggregation.eventsize=1024 sender.aggregation.sendtimeout=60 # in KB # in seconds

30 Which will be interpreted as follow if the amount of individual aggregated events reach 10 the collection of events will be send as one REST (POST) to the backend remote Event Logging Service. If the count is not reached but the total size of the event collection within the aggregator reach the max size of 1024 KB, the collection will also send even if the count is not reached. and finally if the count is not reached and also the size is still below the max threshold, the event will be send if the timeout (in seconds) is reached. In the above case after 1 minute (60 seconds). An event count = 1 in the above configuration would essentially mean that no real aggregation is done even so all events will be handed by the aggregator. By this events will be send in an network optimized way while still be send in a timely manner. As the aggregator collects events in memory this transport destination is not as reliable as the jms option and audit events should be preferable send always via jms even so our default will be service for all events to limit the initial setup effort. The jms and service destination can be configured in the org.talend.eventlogging.sender.rest.cfg as follow: sender.destination.service.url = sender.destination. service.authentication=basic # NO, BASIC sender.destination. service.username=tadmin sender.destination. service.password=tadmin

31 JMS Sender The sender receives the event from the processing part via the direct vm: component and does a technical conversion from the exchange header and body to a json format which is stored in the exchange body and all header field of the event will be removed from the exchange. After the conversion from the Exchange Event Structure to the JSON structure, the event (exchange body = JMS message) will be send via the jms component (so far only ActiveMQ Broker will be support in Talend ESB 5.4.) : org.talend.eventlogging.sender.jms.cfg The jms destination can be configured in the org.talend.eventlogging.sender.jms.cfg as follow:

32 sender.destination.jms.url = (camel Url of JMS component with all options ) sender.destination.jms.queue=event.logging.server Event Logging - Server Overview Within the overall Event Logging architecture the Event logging Server will collect/receive all the events from the agents and send them to the defined destination. Primarily the service will be able to save the events into the new EventLogging DB (RDBMS), the SAM DB (for SAM Events only), a custom persistence destination and also additionally to ElasticSearch for indexing. Overall the server will not process much to be able to handle a large amount of events as fast as possible. In the following chapters we will describe the details of the Collection Server and how it can be configured. Technically the collection server is implemented using Apache Camel and is preinstalled on Talend Runtime (Enterprise version only).

33 Collector In the part of the Server we receive events (via direct vm / jms) Each collector is an own bundle and additional collectors can be added by the user if needed. The product supports the Direct Collector (which exposes a direct vm endpoint and which is used by the Event Logging Service for new events (POST) ) JMS Collector (which able to read / get events from a JMS Broker ) The collector Direct has no configuration and exposes a direct vm endpoint: eventlogcollector and uses the direct vm fix endpoint eventlogserver to put the event into the server processing part. The collector JMS has a configuration in the

34 org.talend.eventlogging.collector.jms.cfg configuration file collector.jms.url = (camel Url of JMS component with all options ) collector.jms.queue=event.logging.server The queue and JMS configuration parameter to connect to the remote broker can be configured and by default the event.logging.server queue will be used but can be changed. In Talend ESB 5.4. we only support ActiveMQ as broker. Events received via the Direct Collector Service are send in collections (1..more) events per request. Therefore we split them after receive and process each event separately afterwards. Event received via the JMS collector are always one event per read (therefore no splitting is required in this case). Both API s expects that the Event is provided by the Event Logging Agent even so events can technically be also directly send to the Collector Service if they fulfil the format and completeness as required by the event structure. Server - Preprocessing part

35 After one of the collector routes receives the event(s) and sends it via direct vm communication to the server route the preprocessing will be done as the first step The system preprocessing step will do the following: checks if the category of the event is ping and if this is the case it will discard the event without any further processing. (Ping events are sent by the agent to check the Broker / Queue availability during the period where network or broker communication is not working as expected but as defined in the Agent, ping category is the third reserved category. These events are no real events and processing will stop immediately on this stage for ping events. convert the json body back to the Event Exchange Structure (as defined in the Event Structure Camel Exchange mapping below) Check if an Event Category exists and if not applies a default one to the event. Set the server timestamp to the Event The following configuration in the

36 org.talend.eventlogging.server.cfg applies to this part: preprocessing.eventcategory=system Which would set the event to the system category if no Category exists until this stage. (Which can only be the case if the event was not sent via the Event Logging Agent) After this step the user can configure user defined preprocessing using a customer route which can be configured as follow for each individual category in the.collector.server.cfg : preprocessing.custom.routeid.default=mycustompreprocroute preprocessing.custom.routeid.audit=mycustomrauditpreprocroute preprocessing.custom.routeid.sam=mycustomsampreprocroute same as in the Event Logging Agent the preprocessing.custom.routeid. will be followed by the category name where default and audit are reserved Where default configuration will be applied to all categories which are not explicitly mapped and audit will be applied to all event which have the audit flag set independent on the event category. In the above example audit and sam event (when not marked as audit) will be preprocessed by the related custom routes while all other will be preprocessed by the default route mycustompreprocroute. By default no custom processing is required and also none is configured by default. The custom processing will be started using the configured route id via a direct vm communication between routes. By this the customer route needs to be deployed on the same Talend Runtime

37 / JVM as the Collector Server is running. Server - Persistence and postprocessing After the preprocessing part the event goes into the persistence, search and post processing steps which are shown in the following diagram: An event can be handed by one ore more persistence backends 1) the Event Logging DB 2) the SAM DB (only applicable for event which was created by the SAM Agent) 3) the custom persistent step

38 None of these step is mandatory (even not the Event Logging DB) and each step can be activated or deactivated by category. Example: persistence.event.db.active.default=false persistence.event.db.active.audit=true persistence.event.db.active.sam=false persistence.sam.db.active.default=false persistence.sam.db.active.sam=true persistence. custom.active.default=false persistence. custom.active.system=false persistence.custom.routeid.system=mycustomserverpersiste nceroute In the above example configuration events with audit = true will be stored in the Event DB, Event with category sam will be stored in the SAM DB and event with category system will be handled by the custom route (called via direct vm with the name mycustomserverpersistenceroute ). Even so what the custom route does with the event is up to the custom route. The Event DB has the database structure as defined in the Event Logging API s and Data Structures Event Db Structure chapter below and accessed via JDBC using a predefined datasource: event.logging.db.datasource=el ds mysql

39 event.logging.db.dialect=mysqldialect The SAM DB is exactly the same database as used by the SAM Server and the SAM DB step will remap the Event Structure to the SAM Event structure and will try to map custom info fields to the fix columns of the SAM Event structure (e.g. flowid, servicename etc.) as this data is provided by the SAM Agent and just in between converted to the Event Logging Structure all information is available and the SAM Db will be filled consistently in the same way as with the SAM Server and by this records written by the Event Logging SAM DB step can be retrieved and viewed also via the SAM Retrieval Service (part of the SAM Feature) and the SAM UI in the Talend Administration Server (TAC) The SAM DB has the database structure as defined by the SAM Feature (see ESB Infrastructure Services Guide Service Activity Monitoring ) and accessed via JDBC using a predefined datasource: sam.db.datasource=ds mysql sam.db.dialect=mysqldialect (if technically possible, we plan to use the same data source as also used by the SAM Server. By this configuration and setup of the DB driver is exactly the same as described in the SAM Chapter in the Infrastructure Guide.) It is important to note that an event can be handled by each of the persistent steps, means a single event can be saved in the Event

40 DB, SAM DB and handled by the custom route. Even so this will for sure add some performance reduction to the overall handling of a single event but is technically possible by the feature. After this the synchronous processing is done, an async seda communication from this point on will be used. The last step is to send events to search indexing (Event Logging Search Server based on ElasticSearch) a final custom post processing (e.g. to reformat the event and send it to a intrusion detection system (IDS), stored on Hadoop HDFS (e.g. to process this event further with Big Data technologies), send it to a Complex Event Processing engine (CEP) or to a larger scale log analysing system like splunk, or to whatever destination it might be send to. The search indexing step is optional and can be configured as following in the.collector.server.cfg per category: Example: search.active.default=false search.active.sam=true Would essentially mean that only sam Event category events will be indexed by the Event Logging Search server but no other event. Example: search.active.default=true search.active.sam=false Would mean all events will be indexed except events with category

41 sam, in case they have the audit flag set to true they would also be indexed as in this case the default is used. To avoid this search.active.audit=false must be configured to also exclude audit events from indexing. The Search Indexing step would convert the Event Exchange structure to a JSON format (equal as the Talend Event Logging Agent does it for sending it to the Event Logging Collector Server) The Event Logging Search Service is based on ElasticSearch and can be configured with the following configuration in the.collector.server.cfg elasticsearch.available=true elasticsearch.clustername=local elasticsearch.operation=index elasticsearch.inddexname=eventloggingindex elasticsearch.indextype=log With the elasticsearch.available the entire feature can be activated or deactivated. If it is deactivated, the category based configuration shown above will be not used at all. The elasticsearch.clustername specifies how the ElasticSearch Server/Cluster can be reached, local on the same machine or remote. The other parameters are specific to the elasticsearch server. The last step in the Collector Service is the Custom Post processing step which allows the user to send the message to any kind of destination he would like this event to also go to.

42 E.g. he might reformat the event and send it to an intrusion detection system (IDS), store it on Hadoop HDFS (to process this event further with Big Data technologies, send it to a Complex Event Processing Engine (CEP) or to a larger scale log analysing system like splunk>, or to whatever destination it might also be send to. Example postprocessing.custom.routeid.default=mycustomserverpost Route postprocessing.custom.routeid.audit=mycustomserverauditp ostroute postprocessing.custom.routeid.security=mycustomserversec uritypostroute As in many other configuration the related route will be called via direct vm and must be deployed on the same Talend Runtime / JVM as the Talend Event Logging Collector Server. Event Logging - Service The Event Logging Service can be used to query and read events from the Event DB (and only from the Event DB) and to post events via REST to the Event Logging Server The service will be implemented in Java and will provide a RESTFul API as described in the Event Logging: API s and Data

43 Structures Service API The Retrieval Service can be configured via the org.talend.eventlogging.service.cfg db.datasource=el ds derby db.dialect=derbydialect authentication = BASIC # NO, BASIC where the db setting will be used to find the related data source definitions of the Event DB and the related dialect and the retrieval.authentication can be used to defined the authentication method for the RESTful service NO = no authentication BASIC = HTTP Basic Authentication (plain text) Additionally via the standard container configuration the Retrieval service can be made accessible via http and/or https. Event Logging - Search Server The Event Logging Search server is based on Elasticsearch. Elasticsearch is a open source search and analytics engine that makes data easy to explore. The Event Logging Search server will be installed outside the Talend Runtime container and is optional for the Event Logging feature but when the event logging search server is installed the log data even in large amounts of log data can much easier and fast search though and in the combination with the Event Logging Search Web Application it is easier for users to find related data

44 by different criteria incl. full text search. Some of the highlights of the elasticsearch product are as follows: Apache 2 Open Source License and based on Apache Lucene Real time data & analytics Elasticsearch provides data search and analytics almost in real time. Distributed Elasticsearch allows to start small, but will grow with your business. It is built to scale horizontally out of the box. As you need more capacity, just add more nodes, and let the cluster reorganize itself to take advantage of the extra hardware. High availability Elasticsearch clusters are resilient they will detect and remove failed nodes, and reorganize themselves to ensure that your data is safe and accessible. Full text search Elasticsearch uses Lucene under the covers to provide the most powerful full text search capabilities available in any open source product. Search comes with multi language support, a powerful query language, support for geolocation, context aware did you mean suggestions, autocomplete and search snippets. Document oriented It allows to store complex real world entities as structured JSON

45 documents. All fields are indexed by default, and all the indices can be used in a single query, to return results. RESTful api Elasticsearch is API driven. All actions can be performed using a simple RESTful API using JSON over HTTP. Event Logging - Search UI / Web-Application The Event Logging Web based Search UI is based on (Elasticsearch ) Kibana 3. Kibana is an open source (Apache licensed), browser based analytics and search interface to Log data and other time stamped data sets stored in ElasticSearch. Its point and click composition lets customers/users easily design custom dashboards. The Search Web application allows users to explore and make use of a large amount of log data by optimized search and visualization support.

46 The use of the Event Logging Search UI is optional and the Event Logging Search UI will be deployed within a Web Server. Event Logging - API s and Data Structures Event Data - Structure The event structure is the primary data structure used by all components of the Event Logging feature and consists of the following attributes. The table shows also how the event structure is represented in the Apache Camel Exchange structure (header / body attributes) Attribute Name Type Camel Exchange Description

47 Mapping id String header.id Unique id associated with every event. category String header.category Each event can be assigned to a particular category. Category will help to group and organize events which can be further used to search & filter them. ('default', 'audit' and 'ping' are reserved categories and can't be used as category names). Initial used categories might be 'system', 'ogi', 'service', 'route', 'bam', 'sam' and 'security but the user can define and use easily additional categories (but only A Z, a z, 0 9 and _ are supported in the category name and only <= 32 characters are allowed. eventtype String header.eventtype Attribute to identify the type of Event. For example for OSGI events, the eventtype is equal to the eventtopic. logmessage String body.logmessage The log message associated with this event. severity String header.severity Describes the severity of the event. Example log4j log levels. logsource String header.source The source is for example class, package or the component name which created this event. agenttimestamp Date header.agenttimes tamp Timestamp applied on the event when the event logging agent receives the event. The timestamp is the local machine date / time converted to UTC date time with milliseconds and the local timezone of the machine. e.g.

48 T08:45:30.453Z servertimestamp Date header.servertime stamp logtimestamp Date header.logtimesta mp Timestamp applied on the event when the event is received by the Event logging collector service. The timestamp is the server machine date / time converted to UTC date time with milliseconds and the timezone of the server. e.g T08:45:30.453Z Timestamp of the log, when the log was created. The format of the timestamp completely depends on the logging framework, it will not be converted or transformed by the agent. If no timestamp exists in the log this attribute will be empty. hostname String header.hostname The hostname of the machine where this event was created. In case hostname cannot be resolved, IP will be used. processid int header.processid Id of the process in which this event was created. audit boolean header.audit A flag to indicate if the ehvent needs to be audited. Default value for this attribute is false. agentid String header.agendid The local agent id which handled this event. Every agent should have a unique agent id auditsequenceno long header.auditseque nceno Sequence number is continuously increased by +1 on the agent for each Audit Event (and only for Audit events) this allows to validate non reputation within the EventLogging Database for audit events later on.

49 signedlogmessage String header.signedlog message correlationid String header.correlationi d The log message in xml format including the signature info. An Id used to correlate different events. (the Business Correlation ID) (in case of SAM Event the technical FlowID will be stored as eventlog_custominfo key / value) subject String header.subject User which created this event. custominfo Map header.custominfo Is a collection of key value pairs. This attribute can be used to propagate custom info. Event DB Structure The event logging database schema consists out of two primary tables the EVENTLOG and EVENTLOG_CUSTOMINFO Scripts will be provided for the following supported databases MySQL Oracle DB2 PostgreSQL MS SQLServer Derby (only for Test and Development) Table: EVENTLOG

50 Field ID CATEGORY EVENT_TYPE LOG_MESSAGE SEVERITY LOG_SOURCE AGENT_TIMESTAMP SERVER_TIMESTAMP LOG_TIMESTAMP HOSTNAME PROCESS_ID AUDIT Type varchar(255) varchar(255) varchar(255) clob varchar(255) varchar(255) datetime datetime datetime varchar(255) varchar(255) char(1) AGENT_ID varchar(255) AUDIT_SEQUENCE_NO SIGNED_LOG_MESSAGE CORRELATION_ID SUBJECT varchar(255) blob varchar(255) varchar(255) Table: EVENTLOG_CUSTOMINFO Field Type Description ID bigint(20) Unique id of the CustomInfo EVENT_ID varchar(255) Relative EVENT's ID value CUST_KEY varchar(255) Custom key

51 CUST_VALUE varchar(255) Custom value Event Logging Service API Resource and URI: This section describes the Event logging REST service resources and URI. The base URI for the service will be GET / Resource to check if Event logging REST service is online. On success will return an HTTP code 200. Example request GET GET /events/{id} Returns an event with the given id. Example request GET d3 9adf 1d47dfa52e07