Event based Enterprise Service Bus (ESB) By: Kasun Indrasiri 128213m Supervised By: Dr. Srinath Perera Dr. Sanjiva Weerawarna
Abstract With the increasing adaptation of Service Oriented Architecture for Enterprise Applications, Enterprise Service Bus becomes the de facto integration solution for most of the Enterprise Integration problems. On a different context, Complex Event Processing Engines also becoming increasingly popular owing to the high demand for real time processing of avalanche of moving data. This research project is an effort to implement a light weight Enterprise Service Bus based on the Even Stream Processing Architecture.
Research Context SOA Service-Oriented Architecture (SOA) is increasingly used as the Enterprise Architecture of many Enterprise Applications. SOA is often defined as a style of software architecture that is modular, distributed and loosely coupled. Componentization is a main driver of SOA, and disparate business functionalities are implemented in different components. These business components provide its functionality to its consumers as a service. The service interfaces are well-defined based on various Service Definition standards and the underlying implementation is completely independent from the service interfaces. The most common realization of SOA is Web services, while any service-based technology can be a candidate for a SOA implementation. Owing to the growing complexity of the Enterprise Applications, most organizations adopt SOA as the Enterprise Architecture. SOA enables the enterprises to share the wide range of functionalities available across the enterprise as services which any other business component in the enterprise can consume. These services must have well-defined interfaces and the underlying implementation has to be completely independent. Hence the proprietary or legacy systems also needs to be wrapped form a service interface which other applications can consume using standard protocols. Also, by integrating different services with diverse business functionality, the organizations can build new applications upon existing applications. Therefore, integration becomes the main and the most common requirement of any SOA implementation where we require a middleware that enables stateless and seamless integration of diverse and heterogeneous services across the enterprise. The middleware solution that enables the seamless service integration is the Enterprise Service Bus (ESB). An ESB is a middleware solution that enables interoperability among heterogeneous environments using a Service-Oriented Model.
ESB An ESB has to support various standard protocols such as SOAP, REST and JSON etc. and transports such as HTTP/S, JMS, and FTP etc. In addition to that, the latency added from ESB should be minimal so that it can process a very large number of messages in parallel. And the extension points and the support for various business connectors such as SAP, Sales Force, and LinkedIn etc. is also a critical requirement. Figure 1: ESB in Action With all these complex and overlapping requirement, most ESB implementations become so complex and bulky. As the main functionality of an ESB is configured via a mediation language, different ESB vendors came up with totally different mediation languages and with all those requirements mentioned above, the languages also becomes complex so as its architecture and implementation.
Figure 2: Content Based and Pattern Based Routing with ESB CEP Similar to the context of SOA and ESB, there are many enterprise scenarios where we need to have the realtime processing of an avalanche of data. Those data can be categorized as data at rest and data in move. Many enterprise use cases increasingly demands the real time processing of data in move. Complex Event Processing (CEP) is one of the most rapidly emerging technology solution to carter the needs or real-time processing of moving data. A CEP engine is designed and implemented such that to provide the runtime to perform Complex Event Processing, which can evaluate the queries provided from a given user and match those queries against continuous event streams and trigger an event when the conditions mentioned in the queries are satisfied. Siddhi A CEP Engine Siddhi is an Apache-2.0 Licensed Complex Event Processing Engine which addresses some of the main concerns of event processing world where there is absolute need to have an open-source variant with the ability of processing huge flood of events that may go well over one hundred thousand events per second with a nearzero latency. This needed careful design of generic concepts of a CEP. Siddhi was designed after doing an indetail literature review focusing on each and every concept separately. Current Siddhi implementation provides an extendable, scalable framework for the open-source community for extending Siddhi to match specific business needs.
In what follows is the high level overview of the Siddhi architecture. Siddhi receives events through input adapters and the major task of input adapters is to provide an interface for event sources to send events to Siddhi. Also Siddhi has several types of input adapters where each accepts different types of events. For instance Siddhi accepts XML events, POJO Events, JSON Events and convert all of those different types of events in to a particular data structure for internal processing. This data structure is simply a tuple. Figure 3: Siddhi High Level Architecture Siddhi-core is the heart of the Siddhi complex event processing engine, where all the processing is done. Core consists of many sub components such as Processors and Event Queues. As indicated in the diagram input events are placed on input queues, the processors then fetch the events to process them, and after processing those events are places in to the output queues. And the output adaptors are responsible for notifying the appropriate clients with the detected complex events. The compiler component takes the user queries, compiles them and builds the object model according to the query.
Figure 4: Siddhi in Actions and Event Structure As per the architectural aspects of an ESB and CEP engine, we can clearly see that both share some of the similar implementation and architectural aspects. The main object of this research projects lies along the lines of implementing an ESB based on the Event Processing architecture. Research Problem Statement & Research Objectives As we have discussed the research context related to both ESB and CEP Engine, the research problem that we are trying to solve is to implement an ESB based on CEP architecture. Although, ESB and CEP are designed for two distinct purposes, as we have discussed above, there are similar architectural aspects of CEP that we can leverage to implement an ESB based on event stream. The objectives of an Event Stream based ESB implementations are listed below. Implementation of a light weight ESB based on Event Streaming Architecture. Mediation Language implementation based on Siddhi Query language. High Performance: Non-Blocking IO based on Java NIO- Integration of high performance Non- Blocking HTTP/S transports as transport layer. As Siddhi CEP is a very high performance CEP, the objective is to leverage its efficient event processing architecture and implementation in the implementation of ESB mediation engine.
Proposed Approach The proposed approach to implement an ESB based on Event Stream processing architecture is that to use Siddhi as the core mediation engine. As depicted in figure 5, the main in flow and out flow are consists of transport listeners and senders. The incoming events comes via input adapters and even stream processing is done inside the mediation engine. The mediation engine can be implemented using Siddhi core architecture and we may possibly reuse some implementation details if possible. The mediation logic is defined inside the mediation language and the compiler evaluates the language and message mediation takes place based on the mediation logic given in the mediation language. Finally the messages are mediated via a set of output adapters which eventually goes to the wire via a transport sender. Mediation Lnaguage Transport Listeners Mediation Language Compiler Transport Senders HTTP/S JMS Input Adapters Mediation Engine Output Adapters HTTP/S JMS VFS VFS Business Adapters Business Adapters Figure 5: ESB implementation based on Event Streaming Processing Architecture Scope and Constrains Enterprise Service Bus is a software product with a vast scope and often contains heaps of features. Therefore, in this research project where we are planning to implement a light weight ESB with Event Stream Processing Architecture, we need to define a scope which we can realize the design and implementation with in the time frame of the MSc project. Hence, we will mainly focus on set of pre-defined ESB integration scenarios. Pass-Thru ESB purely acts as the pass through bridge between the client and the backend service Content Based Routing Mediate the messages based on its content. Support for Basic message types SOAP/REST message mediation. Stateless Service Integration Scenarios
References [1] S. Suhothayan, Isuru Loku Narangoda C. Subash, K. Gajasinghe, Srinath Perera, Vishaka Nanayakkara, "Siddhi: A Second Look at Complex Event Processing," in GCE '11, New York, 2011. [2] L. Garc es-erice, Building an Enterprise Service Bus for Real-Time SOA: A Messaging Middleware Stack. 33rd Annual IEEE International Computer Software and Applications Conference, 2009. [3] Payam Nabhani, Amir Massoud Bidgoli, Intelligent Conceptual Message Routing in Enterprise Service Bus (ESB). International Conference on Convergence and Hybrid Information Technology, Washington, DC. 2008.