July 11, 2008 Selecting An Enterprise Service Bus. by Larry Fulton for Enterprise Architecture Professionals. Making Leaders Successful Every Day

Transcription

1 July 11, 2008 Selecting An Enterprise Service Bus by Larry Fulton for Enterprise Architecture Professionals Making Leaders Successful Every Day

2 July 11, 2008 Selecting An Enterprise Service Bus Understanding The Intersection Of Needs And Capabilities by Larry Fulton with John R. Rymer and Kahini Ranade EXECUTIVE SUMMARY Selecting an enterprise service bus (ESB) is challenging because the products themselves are complex; they combine many discrete capabilities, there are many products available, and the definition of an ESB continues to evolve. The core functions are common, so differentiators include such issues as the deployment topology, environmental dependencies, the style of included tooling, and the specific lowlevel protocols and other details associated with each individual product. At the same time, open source options are functionally viable, and their surrounding tools and features are maturing rapidly. Some lock-in is inevitable, so migrating between products is not a trivial undertaking. Understanding the function of an ESB and its role and fit in your particular environment is essential to making a sound selection. TABLE OF CONTENTS 2 Understanding ESB Functionality Is An Essential Step In Selection What Does An ESB Really Do? Distributed Or Brokered ESB? It Depends On Your Scenario(s) ESB Dependencies Can Mask Additional Costs Tooling Fit Will Drive Long-Term Productivity The Low-Level Details Matter Open Source Options May Be Attractive, But They Lack Traditional Offerings Fit And Finish 7 Distributed Topologies Offer The Most Flexibility Today Topologies Will Matter Less In The Future NOTES & RESOURCES Forrester interviewed 9 ESB vendor companies, collectively representing all of the major commercial and open source ESB offerings available today. Related Research Documents ESB Lessons Learned August 21, 2007 Shaping Your Middleware Strategy To Benefit From ESBs May 21, 2007 RECOMMENDATIONS 8 Understand The Strategic Fit Of The ESB In Your Organization 2008, Forrester Research, Inc. All rights reserved. Forrester, Forrester Wave, RoleView, Technographics, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. Forrester clients may make one attributed copy or slide of each figure contained herein. Additional reproduction is strictly prohibited. For additional reproduction rights and usage information, go to Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. To purchase reprints of this document, please resourcecenter@forrester.com.

3 2 Selecting An Enterprise Service Bus UNDERSTANDING ESB FUNCTIONALITY IS AN ESSENTIAL STEP IN SELECTION ESB demand increases alongside service-oriented architecture (SOA) adoption, as an ESB is often among the first serious SOA infrastructure purchases. However, selecting an ESB is challenging because the very definition of an ESB is evolving, ESBs include a long list of capabilities, and there are many viable products available in today s market. What Does An ESB Really Do? In the very simplest terms, an ESB s job is to give service consumers access to services, regardless of location, protocol, transport, interface technology, security domains, syntactical mismatches, and semantic differences. This means that ESBs must support many interface and transport protocols and data formats, and it also means they must provide conversions between interface protocols, between transport protocols, and between data formats. ESBs also route requests based on data; support request/response, notification, publish and subscribe, and other interaction styles; and integrate with a range of commercial directories and security models. The real power of an ESB comes from its ability to chain operations together to create individual message flows (sometimes called itineraries, lightweight orchestration, or just flows ). Developers can string services together, the results of one service providing the input to the next, without having to build the flow logic into the services themselves. This assembly of existing service interfaces into larger composite services provides a number of opportunities including: Unified handling of nonfunctional requirements. Your organization has unique security authorization, auditing, regulatory, and other nonfunctional business requirements. You will want to separate concerns by creating dedicated services to handle these needs, keeping them detached from individual functional business services. But, the nonfunctional and functional capabilities must be invoked together, so an ESB allows you to assemble them into valuable composite services that incorporate both. This gives you the benefit of using both together while also managing and maintaining them separately, which makes it easier to accommodate future changes. Rapid assembly. You will, over time, accumulate an existing portfolio of proven business services, many of which will be assembled into existing composite services. You can create new or subtly altered capabilities just by assembling new composite services from the existing services. Rapid assembly not only saves time and money by creating the composite services you know you need today but also maintains future agility by making future changes easier, too. Distributed Or Brokered ESB? It Depends On Your Scenario(s) ESBs come in two basic topologies: distributed and broker-based. It may be helpful to think of the distributed-topology products as enterprise service agents and the broker-based products as enterprise service brokers. Vendors will talk at great length about the advantages and disadvantages July 11, , Forrester Research, Inc. Reproduction Prohibited

4 Selecting An Enterprise Service Bus 3 of each approach, naturally favoring the approach embodied in their particular offering. We have concluded that: Logically, there is no meaningful difference between the two approaches. Your ESB makes an interface available so that outside service consumers can access it; it performs the expected operations in the expected order, including the potential invoking of one or more external services through their interfaces. From this functional standpoint, it really doesn t matter whether all of the hops between activities happen in a dedicated broker or on a series of distributed components. Distributed ESB vendors low latency claims can be misleading. One of the purported benefits of a distributed ESB is the opportunity to reduce latency in any given service invocation chain by co-locating ESB services and business services. While it s true that this kind of colocation can reduce latency, taking this approach is not always possible or even desirable. Also, an invocation chain linked by asynchronous messaging to simplify scalability may provide greater throughput at the expense of latency. Broker-based ESB vendors ease of scalability claims can be misleading, too. A professed benefit of a broker-based approach is straightforward scalability, achieved through the addition of more physical instances running in parallel, which presupposes that any composite services in question are stateless and that each ESB image is complete on its own. This kind of solution also requires a means of load balancing the ESB instances. This is fine for spreading work from service consumers across the ESB instances but doesn t address the possible need for load balancing from the ESB to multiple instances of external services. In fairness, this load balancing of external services is a feature provided by many ESBs regardless of topology. The nature of inbound work will also play a role; IP load balancers can handle HTTP requests, but the solution for JMS messages is shared queues. Some ESB services flat-out require a broker. Any infrastructure service maintaining state falls into this category. Examples include business process execution language (BPEL) engines and any service that supports a long-running business activity, store-and-forward-messaging facilities that provide guaranteed delivery of messages, and even basic file systems where FTP will be used to drop files that the ESB must consume. This category also includes facilities that have special processing requirements and that therefore may demand dedicated server resources such as business rules engines or data caching facilities to operate. While the broker-based ESBs would seem to have an edge here, the distributed ESBs support these models by treating the broker nodes as part of their distributed ESB network. Distributed ESBs can fill the role of a brokered ESB. In cases where a broker approach is the way to go, distributed ESBs will let you configure them into a broker topology. This is not a trick that works in reverse: A purely broker-based solution cannot masquerade as a distributed ESB. 2008, Forrester Research, Inc. Reproduction Prohibited July 11, 2008

5 4 Selecting An Enterprise Service Bus Special cases favor distributed approaches. The majority of ESB implementations support integration of applications located in large data centers. There is a sizeable minority of cases where the ESB spans a large physical environment. Examples include using an ESB to support a large number of retail outlets, banking branches, or manufacturing facilities and using an ESB to provide multiple business processes with on-premise access to remote business services. While there is no particular reason that you can t address these cases with multiple instances of a broker-based ESB, these situations are much closer to a distributed ESB s typical configuration. Both sides are slowly migrating to a hybrid approach. Distributed ESB vendors recognize the need to support broker configurations in specific situations. At the same time, broker-based ESB vendors are recognizing the value of hosting business services within or adjacent to the ESB on the same host. Over time, this mutual recognition of the value of the other approach will erode the advantages of one approach over the other and will push most ESB vendors toward offering a grid model that also supports service provisioning. We are already seeing a convergence of topology models: Distributed ESBs already accommodate broker deployment patterns as needed, and brokered ESBs are starting to support new service container types and already have the ability to support multiple scaling models beyond simple horizontal scaling. The lines between distributed and brokered ESBs will continue to blur with the growth of other trends, such as: Closer integration of ESBs and SOA management tools. In those cases where services are not hosted by the ESB, closer integration with SOA management tools means the same policy and instrumentation benefits will extend to the endpoints, anyway. Incorporation of grid technology with ESBs. Though many ESBs can host services directly, more than one of the ESB products available today also include the ability to use a grid architecture, which supports policy-based provisioning of service instances within a pool of available servers. Increased availability of data caching solutions alongside ESBs. A classic enterprise application integration (EAI) design model is to carry all of the data about a business activity within the messages that execute it, which has many advantages but also has the disadvantage of needing to read, move, and write the same information multiple times. Many ESB vendors have available data caching technologies that make it realistic for messages to carry only references to the data, with more information available via the cache as needed. ESB Dependencies Can Mask Additional Costs ESBs can have operational dependencies that, depending on your specific situation, represent either significant advantages or significant unexpected challenges. Here are some examples of possible requirements of a specific ESB: July 11, , Forrester Research, Inc. Reproduction Prohibited

6 Selecting An Enterprise Service Bus 5 A JEE application server. Some ESBs are themselves JEE applications, which therefore run on top of an application server. Some such ESBs require a specific application server, while others will work with any vendor s application server. Most of these ESBs use the high-availability, scalability, and JMS features that the application server provides. Organizations with significant JEE installations will find this to be a benefit since they are accustomed to this administration and value the consistency in the underlying software infrastructure. Other organizations will see this application server as an additional layer to understand and manage. Clustered hardware. Some ESBs achieve high availability through hardware clustering. If this is the case for your ESB, you will have fewer choices in hardware as well as the additional cost of clustering hardware, software, and administration. These solutions will make more sense in environments where there is already a lot of hardware clustering in place for other solutions. Database or clustered database. For any operation requiring the maintenance of state or configuration information, some kind of persistent data store is required. Some ESBs do this using their own file formats, some use features of an underlying JEE application server, and others require a relational database to fill this need. Cases requiring a relational database may need to use hardware clustering to achieve high availability. In some environments, this is no big deal; in others, it adds significant cost and complexity. Tooling Fit Will Drive Long-Term Productivity Like all middleware, an ESB is largely invisible to the business users who rely on it. Developers and support staff, however, will see your ESB through the lens of its tooling. Tooling varies considerably in focus and capability from vendor to vendor. To assist in your selection process, you will want to understand how each vendor under consideration addresses: Development tooling. Your integration and composite service developers will spend much of their time working with an ESB s developer tooling. The user experience varies widely, ranging from working directly with XML files and command-line scripts to working in integrated development environments (IDEs), very often based on Eclipse, which let developers work with graphical models of event flows. You will need to evaluate your integration community s skills and preferences to determine the best fit for your environment. Monitoring and support. Some ESBs provide extensive support for monitoring and support, including the ability to view and manage in-progress business activities within the ESB (both BPEL and shorter-running message flows). Other vendors take a different approach, either providing these capabilities in separate business activity monitoring (BAM) and system management environments or expecting that customers will already have those capabilities in house. Organizations without extensive support infrastructure in place must give more serious consideration to solutions that include these features. 2008, Forrester Research, Inc. Reproduction Prohibited July 11, 2008

7 6 Selecting An Enterprise Service Bus The Low-Level Details Matter A major feature of and justification for ESBs is their ability to work together with many things in your environment: operating with many protocols, transports, data stores, file formats, and user directories; supporting integration with other tools, such as SOA service life-cycle management solutions or commercial applications; and federating with other ESBs. It is unlikely that you ll need everything a particular vendor offers in this area, but you should ensure that the ESB you select supports the areas that are important to your business. There are many specific capabilities you will want to inventory from both a requirements perspective and as part of any ESB evaluation (see Figure 1). Figure 1 A Sampling Of Low-Level Enterprise Service Bus (ESB) Capabilities To Inventory Protocols ESB federations Application adapters Service container support Routing options This includes Web services (WS-*) protocols as well as other protocols required for planned legacy integration activities. Vendors can provide lists of third-party ESBs with existing federations in production environments. You will want to talk to customer references to understand any challenges they may have experienced. You will want to be sure your vendor of choice has available adapters for the specific applications you will be using. Some ESBs can host business services in their own address space. Container types may include EJBs, Spring, Hibernate,.NET, and others. This usually means that the ESB manages the deployment and provisioning of the implementation, which may reduce latency. You will want to be sure that the vendor provides the specific service containers you will need. Larger enterprises in particular will want the flexibility of many routing options, both to meet specific requirements they know about now and to ensure the broadest possible flexibility in the future. You need to understand the available routing options for each ESB under consideration and compare these options to your known and anticipated needs Source: Forrester Research, Inc. Open Source Options May Be Attractive, But They Lack Traditional Offerings Fit And Finish The open source ESBs are functional, reliable, and cost effective. All of them today support missioncritical, high-performance environments. Supporting vendors offer subscription plans that provide higher levels of support and more advanced tooling than is available in the no-cost community editions. In general, open source ESBs have limited tooling compared with commercially licensed products; they typically require direct manipulation of XML files, though this is changing rapidly as supporting vendors drive the creation of graphical development tools, support tools, and add-ons. July 11, , Forrester Research, Inc. Reproduction Prohibited

8 Selecting An Enterprise Service Bus 7 Commercially licensed products also enjoy broader surrounding SOA ecosystems collections of products from the same vendor or business partners that provide other SOA capabilities but this, too, is changing rapidly. On the other hand, large, distributed deployments can be very expensive using traditionally licensed ESBs, making open source solutions particularly attractive. DISTRIBUTED TOPOLOGIES OFFER THE MOST FLEXIBILITY TODAY The most important differentiator of the distributed ESBs is not their ability to operate in a physically distributed fashion, though that is of course important in many situations; rather, it is their service hosting option, which puts the ESB directly under the business services it is invoking. A number of ESB vendors have asked themselves, If my software is already all the way out at each endpoint, what other value can be added? This raises a number of possibilities that are just now finding their way into commercial ESBs, including: Converged movement and management of service metadata and service implementations. ESBs are much more effective at managing the metadata that defines services and their locations than at managing the distribution and execution of the service implementations themselves. This made sense in the early days of ESBs, when they primarily provided integration among existing capabilities exposed as service interfaces. However, as SOA matures and more services are built as standalone components, separately managing, designing, and executing the distribution of metadata and implementations is just an unnecessary challenge. We are already seeing ESBs with the ability to provide load balancing and monitoring of independent services as well as the ability to host those services, providing policy-based distribution and provisioning. Over time, we expect this approach to become more popular both because it is simpler to have the ESB do this and because it eliminates dependencies on other components. Policy-driven control of quality of service in one place. Managing quality of service (QoS) requires the involvement of multiple points of control. Many ESBs can provide throttling, load balancing, and similar features with both hosted and external services. Managing security and provisioning requires software at the endpoint, so distributed ESBs that provide service hosting can do this, too. As the ESB essentially controls the entire communications path from initial invocation through the ESB and on to the service endpoints, it has the opportunity to control all of these parameters with a single set of policies. End-to-end instrumentation in one place. Network- and OS-based instrumentation is valuable but has the fundamental limitation of not quite reaching the application level. Middleware, on the other hand, may actually run within an application s address space. This is certainly true of an ESB s hosting services, which run in the ESB s address space. This means that the ESB has end-to-end visibility from the point of invocation through delivery and on to every participating service. 2008, Forrester Research, Inc. Reproduction Prohibited July 11, 2008

9 8 Selecting An Enterprise Service Bus Topologies Will Matter Less In The Future In the future, we expect to see few ESBs retain the pure broker-based model, for the simple reason that the shift toward SOA makes it attractive to have the ESB fill a larger role in managing service implementations. These opportunities are clear to vendors of brokered ESBs as well, which is why many are adding hosting options. This effectively moves them toward a more distributed ESB model and somewhat blurs the line between the two approaches. To make an appropriate decision for your organization, you will want to understand your vendor s strategy and future road map with respect to service hosting and topology. R E C O M M E N D A T I O N S UNDERSTAND THE STRATEGIC FIT OF THE ESB IN YOUR ORGANIZATION Competing ESBs are not that different when it comes to operating features. The real differences emerge when you look at the environment the ESB must support, the ESB s tooling and how it fits with your organization s skills, and how the ESB achieves high availability and required scalability. Differences also become clear when you try to match a product s instrumentation, monitoring, and support to your existing support infrastructure. You need to understand how an ESB will connect to your infrastructure, people, and processes to make a good decision. When selecting an ESB: Look at a range of available solutions. You owe it to your organization to take a good look at all of the major options: open source versus commercially licensed, distributed versus brokered, and developer-centric versus highly graphical. It costs nothing to invite a few more vendors to demo their products, and many vendors (both open source and commercial) will let you download trial versions of their products to explore in your own lab. Challenge each vendor to clearly describe how its solution will address your specific problems, how it will meet your performance and availability expectations, and how you will achieve success in using the tools effectively over time. Include a proof of concept in your evaluation process. It has become common for ESB requests for proposals (RFPs) to include the provision that each vendor on your short list complete a proof-of-concept implementation to demonstrate both the product s capabilities and its ease of use. Architects typically learn a great deal during these exercises, so be sure to include one in your selection process. Develop a clear idea of the internal processes on which the ESB will have an impact. How you use the ESB for example, if there is already a separate ESB in place acting as a B2B gateway or if there will be a separate ESB for every division will dictate many of the surrounding governance processes. You need to be sure that you understand integration requirements among the new ESB and existing ESBs, life-cycle management solutions, SOA management solutions, and other components of your existing and planned infrastructure. July 11, , Forrester Research, Inc. Reproduction Prohibited

10 Selecting An Enterprise Service Bus 9 Match your ESB to your most important scenarios. Finding the perfect one-size-fits-all ESB is not realistic, so identify the most important considerations for your organization in those areas where ESBs really differ. Be realistic about vendor lock-in. Some level of vendor lock-in is a fact of life in the ESB world. This is because there is no standardization (nor is there any emerging standard) in the artifacts defining ESB operation they are all completely proprietary XML files. This means that replacing an ESB with a competing product is not a simple undertaking, and there is little indication that this will change in the coming years. Don t skimp on developer and administrator tools. Remember that up-front development happens once, whereas maintenance and support happen forever. While the team doing the initial work may be fully capable of working with XML files directly, consider the skill level and business domain expertise of maintenance and support teams when you determine your tooling requirements. Pick the right topology. Data center integration is a dominant use case that brokered or distributed ESBs easily support. Plus, a particular brokered solution may provide the best fit with your needs and your surrounding technology ecosystem. If your organization s needs extend further, you should inform your decision with an understanding of your potential ESB vendors long-term plans regarding service hosting, policy-based provisioning, and support of distributed implementations. 2008, Forrester Research, Inc. Reproduction Prohibited July 11, 2008

11 M a k i n g L e a d e r s S u c c e s s f u l E v e r y D a y Headquarters Forrester Research, Inc. 400 Technology Square Cambridge, MA USA Tel: Fax: forrester@forrester.com Nasdaq symbol: FORR Research and Sales Offices Australia Israel Brazil Japan Canada Korea Denmark The Netherlands France Switzerland Germany United Kingdom Hong Kong United States India For a complete list of worldwide locations, visit For information on hard-copy or electronic reprints, please contact the Client Resource Center at , , or resourcecenter@forrester.com. We offer quantity discounts and special pricing for academic and nonprofit institutions. Forrester Research, Inc. (Nasdaq: FORR) is an independent technology and market research company that provides pragmatic and forward-thinking advice to global leaders in business and technology. For more than 24 years, Forrester has been making leaders successful every day through its proprietary research, consulting, events, and peer-to-peer executive programs. For more information, visit