Empowering the Discipline of Cloud Integration - Part IV

Transcription

1 by Pethuru Cheliah, SOA Specialist, Wipro Technologies SERVICE TECHNOLOGY MAGAZINE Issue LIII August 2011 This is the fourth in a multi-part article series. The first part is published at Empowering the Discipline of Cloud Integration - Part I, the second part is published at Empowering the Discipline of Cloud Integration - Part II, and the third part is published at Empowering the Discipline of Cloud Integration - Part III. Depositing Integration Hubs and Messaging Middleware on Clouds Integration as a Service (IaaS) is all about taking the functionality of business integration (intra as well as inter-enterprise integration) and putting it into the cloud, providing for a smooth data and message transport between any enterprise application and SaaS applications in the online, on-demand, and off-premise cloud. Users subscribe to IaaS as they would do for any other SaaS application. Cloud middleware, alternatively cloud integration bus (CSB), is the next advancement of traditional middleware solutions. Due to varying integration requirements and scenarios, there are myriad types of middleware and broker technologies, and products such as JMS, APIs, EAI, EII, ESB etc. Enterprise service bus (ESB) is for service integration. Composite data services are for data integration. There are message oriented middleware (MOM) and message brokers for integrating decoupled applications through message passing and pick up. For capturing, processing, and extracting any actionable information, insights, trends, threats, tips, prescribing patterns and usable associations out of streams of events from diverse, distributed and decentralized sources and resources, there are event processing engines that receive proactively event messages, process them quickly to extract the encapsulated knowledge and accordingly empower the concerned applications to activate the next course of actions. Thereby a kind of light connectivity and integration occurs between the initiating and the destination applications and services. The fast-growing service orchestration and choreography disciplines enable process integration and composition via the standards-compliant BPEL managers. Service interaction through ESB integrates loosely coupled systems. Besides data services, mashups perform and provide composite business applications, data and views. Therefore, at every layer or tier in the enterprise IT stack, there are competent integration modules and guidelines for bringing up the dynamic integration. With the unprecedented rise in cloud usage, all these integration software solutions are bound to move to clouds. Amazon s Simple Queue Service (SQS) provides a straightforward way for applications to exchange messages via queues in the cloud. SQS is an example for understanding what happens when a familiar onpremise service is recast as a cloud service. Because SQS replicates messages across multiple queues, an application reading from a queue isn t guaranteed to see all messages from all queues on a particular read request. SQS also doesn t promise in-order and exactly-once delivery. These simplifications let Amazon make SQS more scalable, but they also mean that developers must use SQS differently from an on-premise message queuing technology. In this section, you will find coverage on a couple of mainstream cloud middleware. Hosted Mule ESB [REF-3] BizTalk Internet Service Bus (ISB), a Cloud-hosted ESB from Microsoft OnlineMQ on the Clouds [REF-2] Niche Cloud Integration Product Vendors Copyright Arcitura Education Inc. 1

2 On-Demand Mule ESB ESB is the core integration backbone for service-oriented application integration. As cloud IT matures, ESB vendors are doing the relevant changes at the architecture and functionality levels to hoist and host their products in clouds. Mule ESB is an on-premise, dependable, and open source ESB steadily storing a large user base and is entering the on-demand and off-premise space with the hosted version MuleOnDemand, which supports a multitude of APIs, frameworks and transports-such as web services, MQ Series, JMS, HTTP, REST, SOAP and for seamless integrating systems with enterprise applications. Therefore, MuleOnDemand is the corresponding on-demand ESB that could be deployed in any cloud infrastructure. MuleOnDemand is the first open source integration as a service (IaaS) product in the marketplace. This has been a worthwhile offering for numerous siloed applications. MuleOnDemand acts as the mediator between applications and data sources regardless of where they physically or virtually reside. An example would be an e-commerce site that processes orders but has order fulfillment provided by a thirdparty fulfillment house. In this case, the order is taken from the retail site and logged internally. The internal system sends order data to MuleOnDemand, which parses the order accordingly (for instance, converting XML or switching from Java messaging Service (JMS) message to CSV) and passes it on to the fulfillment house. Fulfillment can subscribe to a web service that manages the transactions from the retailer. MuleOnDemand resolves such kinds of integration challenges. It allows businesses to leverage their existing systems through loosely coupled integration using a broad range of protocols such as , JMS, MQ Series, Representational State Transfer (REST), Simple Object Access Protocol (SOAP), and Web services. MuleOnDemand provides the ability to send and consume data and services securely. For example, if a business needs to transfer data from an enterprise application like SAP to Salesforce.com, it can do so with MuleOnDemand, which can be used alone for Integration-in-the-cloud or connected to a Mule Gateway server for deeper enterprise integration. Figure 2 SOA-BI Acquisition, Integration, and Distribution Architecture Copyright Arcitura Education Inc. 2

3 Deployed inside an enterprise and configured to receive data and services from internal applications via any of Mule s transport methods, the Mule Gateway server can send data and services to MuleOnDemand, which mediates the transactions and performs transformations, routing, and other integration tasks. MuleOnDemand then shares data and services securely with partner sites (suppliers and customers). With this Mule-to-Mule communication, complex transactions occur seamlessly across the network both inside and outside the company firewall. When used for stand-alone integration-in-the-cloud, MuleOnDemand allows enterprises to move and manage data regardless of where the data resides. Complex transactions occur across the network, allowing enterprises to share data and services with SaaS applications like Salesforce.com and internal applications such as SAP or third-party payment processing. There are other traditional ESBs steadily moving to leverage the fast-growing market for IaaS. BizTalk Internet Service Bus (ISB) BizTalk Server is the business integration technology and product for Windows environments by Microsoft. Microsoft has taken the enterprise service bus (ESB), a constructive component of SOA-compliant BizTalk server, and web-enabled it to be hosted in clouds in order to facilitate application integration services from clouds. The ESB moves to its new home in the clouds and is being termed as the internet service bus (ISB). All the distinct functionalities of enterprise buses will be replicated in the ISB. In addition, there will be extra and value-added services in order to mitigate and moderate the cloud-induced complexity. As there are some issues delaying the transfer of data among onsite and online IT resources, designing, developing and enhancing ISBs are complex when compared with earthly ESBs. ISB is a cloud-based messaging middleware receiving messages from the source system, transform, filter, and route them to appropriate target systems. ISB enables users to deposit their business applications in clouds, and these applications are capable of interacting with local systems seamlessly. A notable consideration here is the security and authentication that is needed to use the ISB. The ISB is a part of its Azure cloud offering, and is a critical piece of the infrastructure in brokering communication and managing workflow. Microsoft has incorporated some vital services (known as BizTalk Services) for supplying cloud-based integration. Rather than using message queuing, BizTalk Services implements a relay service in the cloud that lets applications communicate through firewalls. Cloud-based integration, such as connecting applications in different organizations, typically requires traversing firewalls. BizTalk Services also provide workflow support along with a way for an application to register the services it exposes, and then let those services be invoked by any other application that has permission to do so. Other prominent BizTalk services are: Service Registry Naming Identity and Access Control Connectivity Service Registry - This is a common registry for storing services hosted by the ISB. This helps mediate the brokering process. Naming - Applying human-readable naming conventions such as a URI to easily identify a particular service. Identity and Access Control - This helps identify what kinds of security mechanisms will be available (for example, Active Directory, Live ID, OpenID, etc) and how to manage the authentication and authorization of various services. Copyright Arcitura Education Inc. 3

4 Connectivity - The ISB needs to provide brokering services, but also needs to be able to step out of the picture once a connection is established. However, sometimes having all communications going through the relay can help optimize performance. On-demand Message Oriented Middleware (MOM) We have a number of standard (JMS)-compliant messaging middleware (messaging queue) such as IBM MQ Series, Advanced Queue from Oracle etc. The Apache open source community also has a message queue product. Message queuing has been an important module for application integration. That is, MQ enables varied applications running on heterogeneous platforms situated in geographically different places to communicate with each other across diverse networks that may be temporarily offline. It provides a form of communication between applications without dealing with low-level communication interfaces/protocols. Without queuing, sending an electronic message from one application to another, over a network, requires every node on the route to be available for forwarding messages, as should be the receiving application. In a queuing environment, messages between applications are stored in the queuing system, which has to be available all the time so not to lose even a single incoming message. After a message is put in the queuing system, it is made available by the queuing system to the receiving applications. All efforts are on taking these message queuing products into the clouds in order to realize the messaging as a service feature. The messaging services are made available to the service consumer (authenticated and authorized) with public URIs. The service implementation may include JMS message service as the backend routing service. The cloud concept could change the messaging development process in the corporate IT. The following issues are usually required to be included in the design and deployment for a conventional messaging system. Hardware concerns Messaging server/service installation, configuration, upgrade, and maintenance issues Cluster and High-availability challenges Scalability issues These problems are now vested with the cloud service provider. IT software developers can simply use the new web API (such as the RESTful API) to solve the messaging requirement between applications. The cloud service provider is responsible to have the messaging service available all the time. Security has been a perpetual issue in network computing. As cloud moves to virtualized and shared infrastructures and the open Internet is the message and data carrier, the boiling security problem gets bigger. Security technologies are taking into account the specifics of cloud computing. Client applications could use https, certificates, and cryptographic libraries to resolve the security issues. The cloud service provider also needs firewalls, IDS, IPS and other monitoring agents in order to proactively ward off all kinds of vulnerabilities, external and insider threats, internal holes, etc. FioranoMQ - is the world s first cloud-enabled, peer-to-peer Java Message Service (JMS) messaging platform with powerful and unique features including dynamic routing, distributed debugging of message flows, dynamic deployment of JMS client applications, unbound linear scalability, direct invocation of standards-based Java Component Architecture (JCA) components and unparalleled ease-of-management with Java Management extensions (JMX) standards. FioranoMQ is the industry s fastest, standalone standards-based messaging server, with speed of 50,000 messages per second. Amazon SQS - Messaging in the cloud is an alternative for many scenarios. It allows for newer applications to be developed, tested and deployed immediately without the upfront hardware and software cost. Amazon provides the Simple Queue Service (Amazon SQS) that address some of the queue messaging requirements. Copyright Arcitura Education Inc. 4

5 Online MQ - puts the traditional messages queuing services in cloud infrastructures. The standard web service protocols are being utilized to communicate to and from Online MQ. The MQ service is secure (SSL), reliable and has a simple API (REST, SOAP, Pox). Online MQ is an Internet-based queuing system providing a secure online messaging solution. As a messaging middleware, Online MQ is a stable platform and framework for sending and receiving messages. Figure 2 The Information Flow Architecture in Online MQ Why Online MQ? When compared with the on-premise messaging middleware, a cloud-hosted messaging middleware carries a number of measurable advantages; there is a movement from the current capital expenditure (Capex) model to the futuristic operation expenditure (Opex) model. There is no initial and upfront investment for buying and installing messaging products. There is no cost involved for integration and support. There is no special server needed to deploy the messaging middleware. The cloud architecture is fully redundant and intrinsically elastic to take care of service and operation level requirements. Setting up middleware is easier in any cloud infrastructure. This is why new products and services could be rapidly realized. There is no need to take care of the communication route between dispersed business partners. They can be anywhere in the world, yet be in touch if they have an Internet connectivity. All the capabilities of conventional messaging middleware are also made available in cloud-based middleware. Online MQ enables to the construction of newer business and pricing models, delivery systems, fresh problemresolution approaches, and revenue-generation opportunities. As it is deployed in clouds, non-functional characteristics such as high availability and scalability are being guaranteed. Online MQ can be used as an SOAP service, and as a REST service. It is fully JMS-compliant and therefore, it can integrate easily with any Java applications and JEE application servers. Online MQ is not limited to any specific platform, programming language or communication protocol. Online MQ fully supports SSL, user-authentication and transactions. Integration Appliances: A Better Choice for IaaS Integration Appliances: A Better Choice for IaaS - The ease of using the integration appliance matches the simplicity delivered by a SaaS application. Both eliminate the need to write any code, do not require installation or deployment of software and provide configuration capability. The benefits of integration appliances line up well with the companies expectations of a SaaS environment. Rapid Integration - Integration appliances are plug-and-play devices that enable companies to integrate their scattered applications. This aligns with the rapid provisioning capability of a SaaS application. Taken together, users can access business-critical information quickly and businesses can provide new services and products. Copyright Arcitura Education Inc. 5

6 Integration through Configuration, not Coding - The integration appliance is a self-contained device with nothing else for IT to add or buy. Appliances do not require complex coordination or specialized IT skills for deployment. Ongoing management and maintenance of an appliance imposes minimal requirements on IT, but the impact on business efficiency is noteworthy. Less on IT but more on Business - The integration appliance is a self-contained device with nothing else for IT to add or buy. Appliances do not require complex coordination or specialized IT skills for deployment. Ongoing management and maintenance of an appliance imposes minimal requirements on IT, but the impact on business efficiency is noteworthy. Novel Pricing Model - SaaS follows a subscription-based pricing model and integration appliances can be obtained for a monthly charge. By eliminating upfront costs, the integration appliance delivers the same type of economic benefit that leads companies to choose SaaS solutions. Easy to Change - An integration appliance allows IT to change connectivity, revise transformations and modify workflows using point-and-click functions in a visual user interface. Easy changes enable integrations to better serve the evolving needs of the business. WebSphere Cast Iron Cloud integration ( provides pre-configured integration solutions for hundreds of leading enterprise and cloud applications. This Cloud integration product, built on the IBM WebSphere platform, uses our configuration, not coding approach to solve the entire lifecycle of your integration needs. Niche and Nice SaaS Integration Products There are several vendors coming up with unique cloud integration products as listed below. Boomi [14] says it has a patent-pending technology that lets it customers integrate SaaS and on-premises software via the Web. Once an application is connected to Boomi On Demand, it can operate with hundreds of other SaaS and on-premises apps that Boomi supports. Boomi offers a software development kit that will let software vendors create connections to Boomi. A user, typically a systems analyst, signs on to the Web-based Boomi service and uses pre-built connectors and process maps to link an on-site application with a software service, or link two software services. Boomi offers connectors for Intuit, Taleo, dozens of SaaS vendors, and generic connectors for common systems a customer might have on site, such as Oracle databases. Once a process is mapped, it s loaded into a runtime engine called an Atom, which is downloaded at a customer site, or hosted by Boomi if it s a SaaS-to-SaaS connection. SnapLogic offers open source data-integration tools that map data from one source to another, letting companies create new types of Web applications and mashups. It also provides connectors to some SaaS sources, including Salesforce and SugarCRM. SnapLogic for EC2, a version of its technology that s designed to work for companies using Amazon.com s hosted computer processing service is also available. The consulting company, Bluewolf, is one of a growing group of smaller firms shifting more focus to Salesforce integrations. This includes selling its Bluewolf Integrator, a wizard-based integration tool, via Salesforce s Force.com AppExchange portal, as well as offering SaaS applications for various industries, such as insurance and advertising, on the Force.com platform. Jitterbit makes connections between data and the cloud, and on-premise systems. Jitterbit SaaS integration enables organizations to replicate, cleanse, and synchronize their SaaS application data securely with their onpremise enterprise business applications and databases it is designed to connect to all modern Web Services APIs, Jitterbit s intuitive UI and make connecting cloud data with on-premise data. Jitterbit supports SOA, event-driven architectures, and traditional data integration methods, and can easily scale to fit any enterprise data initiative involving SaaS applications. Copyright Arcitura Education Inc. 6

7 Figure 5 Investment returns accumulate when a service is reused, while cost remains low and can be amortized Characteristics of IaaS Solutions If we go for a SaaS scheme, it might sound like a good idea to leverage IaaS offerings [REF-1] as well. After all, why build our own custom integration links to share data with other enterprise apps when we can place those integration problems in the cloud, alongside the application that needs them? But there are a few things we should know before making that choice. It s clear that IT environments in the future will include a mix of SaaS and on-premise applications. Different companies will select various combinations of these systems to maintain true differentiation and be responsive to the needs of the business. To realize the full benefit of SaaS applications, integration must be simplified. Copyright Arcitura Education Inc. 7

8 Requirement Comprehensive Support for Application Interfaces Extensibility Ease of Use Flexibility in Deployment Security Reusability Consistent with the Benefits of SaaS Description Most SaaS applications feature SOAP Web services interfaces for exchanging data with other applications. SOAP and the related WS-* standards were designed to provide secure and reliable software services, but they are difficult to use. A SaaS integration solution needs to provide support for these application interfaces, while providing a simpler and more approachable programming model. SaaS integration product should be extensible and capable of supporting current and future applications. It should feature a plug-in architecture, so that new features and interfaces can be added incrementally, without requiring a major release or its re-deployment. While supporting SOAP, a SaaS integration solution must be easy by embedding advanced features such as rich client interfaces, and by supporting common desktop functions such as search, drag-and-drop, etc. SaaS-to-SaaS integration might run entirely as a hosted operation. Other integrations might require some code running internally behind the firewall. A SaaS integration solution must have a flexible architecture that supports deployment in the cloud or behind the firewall, depending on where it is needed Because business data will be flowing through the integration solution, it must be secure. Besides supporting high-end security capabilities, the basic authentication and authorization mechanisms have to be in place. Integration is a horizontal activity and hence integration solution and service can be configured quickly to different domains and problems. SaaS integration complexity cannot overwhelm the benefits of adopting SaaS in the first place. SaaS integration solutions must be able to be quickly deployed, affordable, and flexible. Software Upgrade While on-premise software tends to go through upgrades every 12 to 18 months, SaaS vendors may revise their software three or more times each year. IaaS vendors help to ensure that customizations done for customers continue to work. High-Performance Using IaaS turns a two-way communication between two applications into a three-way conversation with the hosted integration service provider. This is a potential point of failure. With some hosted services, all of the data moves through the vendor's servers. In others, such as Informatica's offering, the data moves directly between the applications; only metadata resides on the IaaS vendor's servers in the cloud. Appliance-like Solution Companies need a solution that can run anywhere, connect applications anywhere, be managed from anywhere and require no specialist integration skills or IT infrastructure. These solutions must be easily configurable, flexible and scalable. And, the integration solution must implement projects within days, not weeks or months. Copyright Arcitura Education Inc. 8

9 Conclusion The Fiorano Cloud Platform integrates applications across SaaS, PaaS and on-premise worlds. Fiorano blends cloud-based integration with general business-process integration providing the benefits of centralized management while maintaining the efficiency of peer-to-peer architectures, ensuring seamless, real-time and flexible integration. Fiorano Cloud utilizes the Fiorano ESB as the Hybrid Integration infrastructure to integrate applications across Software as a Service (SaaS), Platform as a Service (PaaS) and In-House worlds. Fiorano s unique and powerful peer-to-peer architecture enables efficient, scalable implementations of on-premise to cloud, inter-cloud or intra-cloud integration scenarios. The Fiorano Cloud Platform is currently available on Amazon EC2. The cloud idea has brought in a number of tectonic and terrific shifts on the total IT landscape. Service orientation simplifies the complicated process of system design and facilitates in implementing enterprisescale and mission-critical business applications. Further on, the service-orientation concept has established services as the exposable entity to be found and accessed over any network. This has laid the solid foundation for software as a service (SaaS). But in every company, there is a large quantity of corporate data in their onpremise systems and SaaS solutions (newer as well as SaaS-enabled enterprise applications) that are getting deployed in third-party clouds and do not have the access to the company back-end systems that are behind the company firewalls. Therefore the demands for robust and resilient solutions in effecting enterprise-to-saas as well as SaaS-to-SaaS integration requirements have gone up. With the idea of integration as a service, the hosting systems have to accommodate integration backbones, middleware, containers, connectors, etc. References [REF-1] Hybrid Data Integration - Open Data Integration that unites structured data with the industry s best architecture for the web [REF-2] Online MQ Cloud Message Queuing Service [REF-3] Mule On Demand, the Hosted Version of Mule ESB Copyright Arcitura Education Inc. 9

10 About the Author: Pethuru Cheliah Dr. Pethuru Raj has been working as an enterprise Architect in Sify Software Ltd. ( sifysoftware.com/), a fast-growing and enterprise-class IT solutions company at TIDEL Park, Chennai, India. Before joining in the present company, he worked in two leading IT services and solutions companies: as a lead architect in Robert Bosch Engineering and Business Solutions (RBEI), Bangalore and as a senior consultant in Wipro Technologies, Bangalore. for five years. He has been focusing on a few emerging and evolving technologies such as SOA, Cloud Computing and the Internet of Things (IoT). He was selected as a JSPS Research Fellow and again as a JST Research Scientist to focus on promising and potential IT research disciplines (Distributed Computing and Quantum Computing) in two leading Japanese universities for 3 years. Totally Pethuru Raj has 10 years of industry experience in business integration technologies and tools and 3 years of post-doctoral research experience. Also he had a very fruitful expertise on enterprise-scale and distributed systems frameworks such as enterprise Java (JEE) and Microsoft.NET framework. Currently his research areas include embedded SOA for service-based device-to-device (D2D) integration, Cloud computing for IT optimization, augmentation and Transformation, and device-to-cloud (D2C) and sensor-tocloud (S2C) integration towards the vision of IoT that intrinsically involve pervasive computing, autonomic communication, ambient intelligence (AmI), cyber-physical systems (CPS), smart sensor networks, and smarter environments. He finished his PhD degree from Anna University, Chennai and worked as an UGC-sponsored research associate (RA) in the department of computer science and automation (CSA), Indian Institute of Science, Bangalore. He has been contributing book chapters in BPM, SOA, Cloud Computing, Ubiquitous Computing, Enterprise Architecture, Smart Enterprises, etc. Finally on his own, he has been writing two books: the Cloud-enabled Smart Enterprises and the Internet of Things (IoT). He has a personal site at Copyright Arcitura Education Inc. 10