ESB as a SOA mediator: Minimizing Communications Complexity

ESB as a SOA mediator: Minimizing Communications Complexity Nadya Alexandra Calderón R., Sergio Daniel Moreno P. Universidad de los Andes. Ingeniería de Sistemas y Computación. Bogotá, Colombia n-calder@uniandes.edu.co, serg-mor@uniandes.edu.co Abstract. IT within enterprises has evolved into a complex ecosystem that generally presents itself as legacy systems being organized as application silos that solve specific problems in the business core. Modern enterprises look at their core business as integrated processes involving several dependencies (business entities) instead of independent solutions given by each dependency (business entity). Thus, the new challenge faced today is the ability to integrate these application silos minimizing the impact on legacy applications while maintaining non-functional requirements. As a possible solution to this challenge, Enterprise Service Bus (ESB) technologies emerge, providing a standard communication platform available to companies that intend to use a Service Oriented Architecture (SOA) approach as a solution to the (fore mentioned) challenge. We present the benefices and limitations of ESB's in this architectural context based on case studies. We conclude describing different contexts where ESB's provide a real solution to application integration problems. 1 INTRODUCTION Organizations have relied on IT systems for almost forty years in order to manage different aspects of their operation. This has left modern organizations with legacy systems derived from different technological evolutions. These evolutions were focused only on technology, and organizations found themselves limited by technical shortcomings, thus having to adapt quickly to constant changing solutions provided by the market. As a consequence, organizations molded their business to fit the IT available at that moment, instead of having technological solutions that could accurately represent the relationships and interactions defined by business logic. Legacy systems result from years of developing technology guided solutions, focused on solving specific tasks without knowledge of the business environment. This means that systems were unaware of the relations between business entities and that integration was made at a data level (e.g. batch processing), introducing errors, data replication, time overhead and inconsistencies on the results obtained. The overall result of this operation is organizations where technology is not aligned with business, making business evolution slower. The business process redesign (BPR) or reengineering reappeared around the nineties when organizations demonstrated the problem given by the growth of their complex IT systems (composed along time by isolated applications and vast data

2 Nadya Alexandra Calderón R., Sergio Daniel Moreno P. repositories). The idea of achieving maximum efficiency, across different but logically related functions offered by each entity, supports applications that could interact with other business entity applications [8]. This intention was widely promoted and technology trends started to improve solutions for the business processes oriented architectures designed into organizations. The time and cost to redesign and implement legacy systems, trying to include communication protocols and gateways that processes need to express the relationships between entities is too high. The constant change of these processes requires a technological solution where all improvements could be developed under an acceptable cost while maintaining a non-functional requirements layer that support the processes quality. These silos of legacy systems supporting the organization business have marked the era of vertical design where independent and specific tasks applications had control over business logic and organizational information (distributed over each application). Business processes oriented design brings horizontal conceptualization where applications can work in the context of their supported entities creating new needs in communications and interaction behavior amongst them. The emerging interaction between different business entities composes diverse services. A process orchestrates the services: the automated management, arrangement and coordination of the overall IT system including services, middleware platforms and applications. Each entity (described as a process area, organization department, or resource) relates logically to one or several services, the communication and interaction needed between applications is the technology challenge in order to continue with the alignment of technical solutions to business intelligence developments within organizations. Communication solutions must deal with a variety of heterogeneous and distributed functions and information sources. Security, consistency, time processing latency and every non-functional requirement, have a higher complexity and importance, every detail further designed to be local and atomic, must support the characteristics of this new distributed, polymorphic ecosystem. Enterprise Application Integration (EAI) is the technology area that tries to tackle this challenge. EAI products aim at facilitating the interaction between applications and support certain workflow needs [1]. One of the approaches for application integration is the service-oriented solutions described as software components whose descriptions are published to allow for reusing services over interconnected networks [6]. An Enterprise Service Bus (ESB) provides a communication platform where service discovery and service communication can be achieved using several standard protocols provided by the bus. It is not the service s responsibility to make sure its message is understood or delivered; it is the Bus s responsibility. Such responsibilities will be discussed later on, along with basic concepts on how an ESB may work internally and environments where it could be most productive. This article introduces SOA concepts that are relevant in describing ESB-based SOA solutions. In the second section, we summarize these concepts and present the importance of developing an ability to design a business process as services and the architectural layers involved. The third section defines the basic layers used in the ESB to achieve the VOL. 1

ESB as a SOA mediator: Minimizing Communications Complexity 3 high level services of its integration model, we present a clear environment where ESB's are most useful and clearly define high level services that an ESB must provide. Finally we will give an overview that reflects the impact on SOA architecture when it is empowered with an ESB and present our conclusions. 2 SOA IMPORTANCE AND BASICS Importance of BPM. Developing an ability to design a business process as services and the layers involved. In time, after organizations realized that reengineering was not enough to maintain a constant evolution in their business processes, they turned to Business Process Management (BPM) as a new approach. Processes are conceived as evolving entities in continuous development. A process life cycle always starts at design, then it is executed and thoroughly monitored in order to identify weak points to correct it, the process is redesigned constantly to achieve a desired service level [8]. This approach benefits from the fact that it is supported by different IT technologies, hence aligning technology with business. In order for this alignment to be genuine, IT systems must bring new flexibility assets to the organizations, so that technological knowledge and bases in organizations (as legacy systems and processes definitions) can be integrated into the new BPM approach instead of changing the whole technological platform. A widely known approach to provide this flexibility is the Service Oriented Architecture (SOA) paradigm, where loosely coupled services from different business entities run across the organization interacting to create business processes. This approach implies different technologies, as merging heterogeneous legacy systems imply dealing with different communication platforms, different programming languages and architectures. The objective is to provide mechanisms to communicate and interact. These mechanisms are platform independent so that enterprise connectivity is seen as transparent by all systems removing redundancies, generating unified collaboration tools and oriented to IT processes [2]. SOA follows certain principles to be effective: reuse, high granularity, modularity, composability, componentization, and interoperability [5]. The organization must learn how to group its activities into services that comply with SOA design principles. This ability needs to be developed in time and is the reason why SOA is complimentary to the BPM approach. SOA will enable the organization to respond in a quicker fashion to market changes and will reduce the inherent costs. Service Interaction Orchestration is widely used for service interaction; it describes automated ways for components to interact. Service Orchestration defines a way to describe business interaction amongst its services to create a business process. Service orchestration deals with the way the organization describes and designs the high-level processes (business processes). Several process definition languages such as BPEL, BPMN, or YAWL VOL. 1

4 Nadya Alexandra Calderón R., Sergio Daniel Moreno P. support organizations in this task. Most of them serve the same purpose, there are differences regarding the expression capabilities of each one, commonly compared by the ability to express the different workflow patterns defined by Van der Aalst in [10]. Communication Communication solutions are never straightforward and SOA is not the exception. Different technologies that cover specific aspects must be used in order to achieve SOA's communication goals; Web Services, XML, XSLT and others are commonly used technologies. Yet a common communication channel that reduces complexity is missing. ESB's respond explicitly to that requirement by providing standard based communication types and protocols (messaging engine), thus decoupling services called and the transport medium. The next two sections introduce basic concepts on ESB's and present application environments where the use of ESB's in SOA architecture is most useful. 3 PRELIMINARIES ABOUT ESB S Why are ESB's useful to deal with Enterprise Application Integration (EAI). An overview of high level services that an ESB must provide. Organizations can deal with integration at different architectural domains (data, applications, processes, user interfaces and analytic systems), ESB's are an approach to the application integration problem supporting communication between independent, heterogeneous and distributed sources that makeup organizations IT environment. An Enterprise Service Bus is an infrastructure where service discovery and service communication can be done using several standard protocols. Based on meta-data that describes services, requesters, providers, mediations and their relations to discover endpoints (as any party that needs or provides services), interchange information (service requests or messages), routing and other characteristics, ESB technology provides tools to perform SOA interactions in the "publish - find - bind" model [1]. This infrastructure enables SOA as the layer that interconnects services and formalizes their publication, by providing a registry of the services available and the requesters that could connect to them. Definitions and relationships of service requesters and providers (being equal partners in the interaction with the requester as the endpoint that initiates this interaction) are managed through meta-data allowing operation, visualization, and update of those endpoints [1]. It is important to notice that the bus does not support business logic of the service providers nor requesters and it is not part of its job being the services container. After an ESB has delivered the messages to a container, its responsibilities are done, within hosting containers service invocation is redirected and operated. The importance of ESB's performance is the ability to present services that are available through the bus meaning the ability to offers transparent localization and mediation of the services to be used. VOL. 1

ESB as a SOA mediator: Minimizing Communications Complexity 5 After the registry service, the mediations are "the means by which the ESB can ensure that a service requester can connect successfully to a service provider" [1]. As part of the flexibility demanded by the nature of SOA, ESB technology must be able to accept different requests as messages then operate on them and finally deliver them. Requesters and possibly providers could come in different implementation languages, different data formats, with inconsistent or incomplete data values and other inconsistencies that require to be attending before binding requests and providers. The mediations provided by the bus ensure the correctness before the bind stage, summarizing, ESB's transform messages (through the mediations) to match the requester formats to those of the provider. Mediations themselves are involved with the dynamically loosely coupled intention of SOA, they could be provided by third parties and be deployed on the ESB without changing services requesters or providers. The mediations enable different operational requirements within ESB infrastructure, they are useful monitoring, validating tasks and translations, as in cache functions or routing requirements (selecting between the service providers associated with the mediation) [1]. Thus, the bus through the model of registry and mediation between services publishers and consumers achieves fundamental EAI bases. The value of an ESB is in its ability to free up IT resources to concentrate on the higher-level issues associated with rolling out a SOA [7]. 4 BASIC CONCEPTS IN ESB'S This section describes basic services and concepts of ESB s. This section deepens on the services an ESB must provide in order to aid an organization either as EAI technology, or as a fundamental piece for its SOA implementation. These services are presented as components described in [9]. The ESB must decouple communication in such a fashion that it is not necessary to connect directly providers and consumers; instead, the bus takes over communication, as seen in figure 1. ESB simplifies the number of connections necessary also as a communication platform, it must support different communication abstractions (event-driven, publish/subscribe ) either synchronically or asynchronically. In addition, the bus must provide the user with different connection types, for the messages to go in or out the bus, thus making the bus protocol independent [3]. VOL. 1

6 Nadya Alexandra Calderón R., Sergio Daniel Moreno P. figure 1 Solving communication links through the bus [3]. Once messages are received, routing services based on subject, content, and itinerary are needed. In addition, a services version handler is useful under this category for the user to manage different service versions running on the bus. [9] As discussed in section three, mediation is a big service provided by the ESB. Mediation facilitates translation from different types of messages, they enable the user to enrich the message according to different criteria, decomposing, and composing messages based on content. [3][9]. Communication across the bus can be secure if desired, thus the bus needs to provide security mechanisms or the ability to interact with existing security systems in order to authenticate, control access to certain messages, and provide access to certain services. In order to register these services the bus must also provide a directory service for reuse and ease of access and service metadata repositories. VOL. 1

ESB as a SOA mediator: Minimizing Communications Complexity 7 figure 2 High level services offered by the bus. Default implementations of these services and interfaces definitions are flexible enough to support the SOA nature [3]. Another important concern is Quality of Service (QoS), determined in this case by the service user and the Service Level Agreements (SLA s) of the different services deployed across the organization. Thus the bus may implement different mechanisms in order to persist messages, resend messages if errors occur, guarantee message delivery, etc. These mechanisms support the quality at different levels defined in the SLA. Finally, an interface for monitoring and management of the bus is to be provided in order to find problems, identify causes of the problems, and audit different aspects of the bus. In addition, automated tasks in monitoring and management are desirable in order to save the end user time. Figure 2 presents a categorized view of an ESB capability, in order to illustrate the wide range of services an ESB could provide. Different implementations include most of the capabilities shown. In addition, in [9] presents a ranking of different ESB as the result of a qualitative evaluation of the product and vendor; we do not discuss this ranking. Application Environment and Usage Patterns VOL. 1

8 Nadya Alexandra Calderón R., Sergio Daniel Moreno P. Along this article, we have been describing the problems faced by organizations to integrate legacy applications and new IT systems that support process flow in the current business process oriented models within enterprises. These characteristics are related with complex enterprises, where data and applications are really distributed, even isolated in several cases. The group of small and medium enterprises could satisfy their IT needs with application containers, basic (Enterprise Information Integration) EII solutions to integrate the information at an analytic level or manage data integration through more simple solutions in terms of the number of entities involved. With this context in mind and the concepts described along this section, Schmidt proposes seven usage patterns in his studies of ESB technology. These patterns enable and facilitate the implementation of successful solutions through the reuse of components and solution elements from previous experiences [7]. 1. Broker application pattern: Based on the principle that distribution rules are separated from applications. 2. Service and event-oriented routing pattern: Useful when dealing with an event distributed across more than one target provider. It is important where the selection of appropriate destinations could be done based on availability, workload, or error detections. 3. Protocol switch pattern: Presented as solution where requesters and providers use different network protocols. 4. Proxy or gateway pattern: Maps services endpoints providing security functions such as authorization and access control or logging and auditing capabilities. 5. Event distribution pattern: Events may be distributed to more than one target provider. 6. Service transformation pattern: The ESB provides necessary translations between service requesters and providers due to the use of different interfaces among them. 7. Matchmarking pattern: Target services could be discovered dynamically base on policies definitions. These basic interaction patterns guide the construction of solutions for the ESB, they should be able to interact with process-oriented interaction patterns like a workflow definition using BPEL that could extend and connect several of the mentioned patterns to define a complex solution. 5 HOW ESB'S HELP MAKE SOA REAL The impact on SOA design when it is empowered with an ESB. As we described previously, SOA s have evolved and the number of projects incurring in this field is growing fast. Reliability, security, performance among others, are real challenges today in application integration solutions. An ESB as the infrastructure that we have presented in this article supports the SOA implementation within enterprises, making real several aspects of this widely accepted architectural proposal. VOL. 1

ESB as a SOA mediator: Minimizing Communications Complexity 9 The most important technical contributions of ESB s rely on the ability to decouple service publication from different implementations and decoupling technical aspects of service interactions. ESBs also contribute with the possibility to integrate and manage services, supporting data transformations, promoting communication messaging standardization and mediating the binding between endpoints, (requesters and providers) while offering all the non-functional requirements that distributed systems must include (reliability, security, synchronous and asynchronous delivering) [6]. James Pasley in his studies published on the SOA Journal [6] describes the importance of the ESB when designing the first SOA projects, due to inexperience and time needs of major organizational challenges. Although implementing all non-functional requirements could be done without the bus infrastructure, there is an earned value. This value comes when all those resources (time, IT specialists, cost) can be assigned to design complete services, training existing staff, driving the adoption of shared message formats, coordinating team activities and testing, leaving all technical issues in the already implemented solutions offered by the bus and not implementing these from the ground up (reinventing the wheel!) [6]. Case Study: QualDev Dashboard As a case study, we present the QualDev SPCC (Software Project Control Center), an application designed to monitor GSD (Global Software Development) projects. This dashboard integrates information from different QualDev systems and presents the client with a unified view of the process in order to manage resources assigned to the process and support the improvement on analytic business decisions. Figure 3 QualDev SPCC The QualDev dashboard presents (at a study level) characteristics of an environment where multiple sources and functional applications support business requirements and the VOL. 1

10 Nadya Alexandra Calderón R., Sergio Daniel Moreno P. lack of interaction over them is the problem due to recent needs of project control and analytic decision support that must be taken. Every application has the ability to offer services that can cooperate to achieve this goal; a SOA perspective presents itself as a natural way to design a solution to the problem. The challenge raise was the decision of EAI middleware that would support the interaction and the services design as Pasley [6] suggest. In this case, event driven messaging was needed and message mediation in order to translate from messages sent by the different applications to messages understood by the dashboard while minimizing changes to be made to the different applications. This scenario was implemented in an early beta version of the JBossESB implementation that lacked a lot of performance and documentation at the time, thus making the learning curve very high. Two months later the scenario was implemented after the final release was made, and this time it was much simpler and there was a big difference in the experience. The advantages the ESB provided were notorious. The ESB made the interaction with web services used by the dashboard and provided by the other applications transparent by the use of mediations. In addition, the ESB had to listen to message queues in order to determine changes and translate the messages into WS that the dashboard could understand. The ESB rendered a big help in that process and completely managed the communication translation overhead by only implementing the message mediation needed. 6 CONCLUSIONS In an era of fast changes within organizations, when the major challenge is the ability to express services as sets of relationships between the overall enterprise entities; technological demands must be aligned with the intention to satisfy horizontal conceptualized business services that integrate and communicate heterogeneous, isolated and independent applications, that currently support the business areas. ESB s come as a big help in isolating the communication problem to a middleware platform that provides services that help diminish the complexity of the integration. This makes ESB a good entry point for SOA that would render the organization in a shorter time to market reusing services. In addition, IT systems must bring new flexibility assets to organizations, so that technological knowledge within (legacy systems) can be integrated into the new BPM approach instead of changing completely the technological platform as before. In addition, improvements must be developed under an acceptable cost while maintaining a non-functional requirements layer that supports the processes quality. The emerging ESB technology comes as a contribution to complex and challenging SOA projects. It brings high level services that allow entities (as requesters and providers) to interact and communicate, presenting a low coupled and dynamically model solution for some of the most important non-functional requirements and allowing VOL. 1

ESB as a SOA mediator: Minimizing Communications Complexity 11 organizations to invest its resources in the higher-level issues associated with implementing a SOA. REFERENCES [1] Ashish, Naveen, et al. Enterprise Information Integration: Successes, Challenges and Controversies. ACM SIGMOD International Conference on Management of Data. Session: Industrial papers: enterprise information integration Pages: 778-787. [2] Bieberstein, Norbert, et al. Impact of service-oriented architecture on enterprise systems, organizational structures, and individuals. IBM Systems Journal, Vol 44, # 4, 2005 [3] Keen, Martin, et al. Getting Started with WebSphere Enterprise Service Bus V6. ibm.com/redbooks 2006 [4] Koch, Christopher A New Blueprint For The Enterprise, CIO Magazine, Mar 1 2005. [5] Papazoglou, Mike. Van den Heuvel, Willem-Jan. Service oriented architectures: approaches, technologies, and research issues. The VLDB Journal. Springer- Verlag. 2007. [6] Pasley, James. The ESB in Your SOA. SOA Web Services Journal. Volume: 6 Issue 11. November 2006. Pages 16-20. Available at www.webservices.sys-con.com [7] Schmidt, Marc-Thomas, et al. The Enterprise Service Bus: Making service-oriented architecture real. IBM Systems Journal. Vol 44. October 2005. Available at: http://www.research.ibm.com/journal/sj/444/schmidt.html. [8] Teng, James et al. Business Process Redesign and Information Architecture: Exploring the Relationship. ACM SIGMIS. Volume 26, Issue 1, Pages30-42. February 1995. [9] Vollmer, Ken. Gilpin, Mike. The Forrester Wave: Enterprise Service Bus, Q2 2006, Tech Choices, June 30, 2006 [10] W.M.P. van der Aalst., A.H.M. ter Hofstede, B. Kiepuszewski, and A.P. Barros. Workflow Patterns. Distrib. Parallel Databases, 2003, 14, 5-51 VOL. 1