Adoption of XTP (extreme Transaction Processing) leveraging grid-enabled Federated Enterprise Service Bus (FESB)

Transcription

1 Adoption of XTP (extreme Transaction Processing) leveraging grid-enabled Federated Enterprise Service Bus (FESB) Shivaji Sarkar TATA Consultancy Services Peter Mendis TATA Consultancy Services ABSTRACT Gartner predicts that By 2011, more than 20% of new TP application projects will require support for more than 10,000 concurrent accesses or more than 500 transactions per second, up from less than 10% in 2006 (0.7 probability). To avoid vertical scaling on expensive and proprietary hardware and software, extreme Transaction Processing applications are architected to scale horizontally to minimize initial deployment costs and optimize the cost of scaling up. One such foundational technology for XTP is Enterprise Service Bus (high-performance interaction with distributed resources) given the distributed, serviceoriented and event-driven nature of most new transactional applications. However, the shortfall of scalability and performance even with the combination of Enterprise Application Servers (EAS) and ESB technology to meet the requirement of the most extreme scenarios, forces enterprises to adopt incremental (Computing Appliances e.g. Xi150) and emerging technologies (Enterprise Grid Architectures) without forcing substantial application redesign. This article aims to establish the options of adopting the optimum solution for extreme Transaction Processing leveraging service virtualization, federated enterprise service bus and grid-architecture. 1. INTRODUCTION EXTREME TRANSACTION PROCESSING WHAT & WHY Extreme Transaction Processing (XTP) is an exceptionally demanding form of transaction processing. Transactions of most high-end (more than 10,000 concurrent accesses or 500 transaction per second) or ultra-high-end (more than 100,000 concurrent accesses or 5,000 transaction per second) requirements or more would require this form of processing. Gartner defines XTP as an application style aimed at supporting the design, development, deployment, management and maintenance of distributed TP applications characterized by exceptionally demanding performance, scalability, availability, security, manageability and dependability requirements. Very much like traditional TP systems, XTP applications are aimed at enabling efficient, reliable concurrent and real-time access (read/update) to a shared database by executing application programs commonly referred to as "transactions." The first and foremost adoption of XTP can be observed amongst the financial institutions (A 1-millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm, by one estimate), whose prime requirements are more processing capability, but without requiring exponential increase in hardware costs, in areas such as fraud detection, risk computation, and stock trade resolution, to profit from minute, fleeting price anomalies and to mask their intentions via "time-slicing," or carving huge orders into smaller batches so as not to move the market. FEDERATED ESB (ENTERPRISE SERVICE BUS) - WHAT Federated ESB (FESB) provides the ability to use multiple service registries and administration domains while mapping the disparate registries into a federated set of services. The federated ESB facilitates service interactions with multiple ESB implementations. One master ESB to which several dependent ESBs are federated to provide a subset of services that are applicable throughout the enterprise.

2 Fig. Federated Enterprise Service Bus GRID ARCHITECTURE - WHAT Grid Architecture aim to integrate, virtualize, and manage resources and services within distributed, heterogeneous, dynamic virtual organizations. The realization of this goal requires the disintegration of computing systems within and across organizations for resources to be accessed as and when required, regardless of physical location. The two key players in Grid environment are resource providers and resource consumers. Resource consumers adopt the strategy of solving their problems at low cost within a required timeframe and resource providers adopt the strategy of obtaining best possible return on their investment. The Supply-Demand between Resource Providers and Resource Consumers are guided by the following key functional and nonfunctional requirements: i. Interoperability and Support for Dynamic and Heterogeneous Environments ii. Optimization, Ease of Use and Extensibility iii. Quality of Service (QOS) Assurance iv. Security v. Scalability & Availability The utility provided by such Grid infrastructure is realized as a set of capabilities: Request Management & Optimzation, Resource Management & Optimization, Infrastructure Optimization & Fedreration. 2. Challenges in Adoption of XTP The fundamental challenge of XTP is supporting business-critical requirements comparable (or even more demanding) to those of classic TP systems, but in a more distributed, interoperable, standards-based and global-class environment. XTP requirements are fostering a new cycle of potentially disruptive innovation in application platforms involving: Atomicity The effect of the execution of a transaction cannot be partial. A transaction must be fully executed or its partial effects must be rolled back in case of failure. Consistency Execution of a transaction cannot break the integrity rules associated with the database. Isolation While a transaction is being executed, the data it is working on is not visible by other applications, so no other application can have access to data in an intermediate state. Durability Once a transaction has been completed (or "committed"), the corresponding database state change is persisted and cannot be rolled back. Availability Minimal planned system downtime for application or system upgrades Reliability Fast and transparent recovery from hardware/software failures Manageability Installation, configuration, administration, monitoring, maintenance and change management Security Authentication, authorization, access control and communication encryption

3 Performance Execution of transactions within a pre-defined time frame (typically subseconds), irrespective of the number of users concurrently accessing the system Scalability Support to large and fluctuating workloads by optimally and dynamically allocating resources accordingly Accountability Tracking and tracing operations for auditing or other management reasons Extensibility Adding new transaction types rapidly without stopping operations Faster Infrastructure Dedicated Compute Resources, Garbage Collection Pauses, Memory Heap Size, Lock Contentions Limit Scalable Performance Costs Utilization, Footprint, Power and Cooling 3. Leveraging Technology for XTP From traditional TPMs and database management systems (DBMSs) to ESB technology of today, technology has evolved dramatically to support high scalability and performance, optimized resource allocation, fast recovery and transaction integrity on top of the bare operating system and file system. However, the combination of EAS and ESB technology cannot provide the scalability and performance required by the most extreme scenarios without forcing substantial application redesign. Fig. Technology options for XTP Incremental Technologies Fig. Incremental Technologies

4 Emerging Technologies Fig. Emerging Technologies 4. Service Virtualization through Federated ESB in Grid-enabled environment What is Service Virtualization? Service Virtualization is the ability to hide operating system, platform technology and network connection or transport and helps the end-user to work with simplified resources. In other words, one has to just know and implement business logic without any knowledge of other underlying pieces. Polymorphism is best suited example for Code-level virtualization. When implementing Business services there are four things that one needs to look at: A. Business Logic B. Deployment/Runtime Configuration C. Security/Compliance Configuration D. Messaging Infrastructure Details Another aspect what one needs to know is the reuse and repeatable code that is churned out in Development lifecycle of a product/project. In any given scenario during development there will be larger code reuse applicable from Deployment, Security and Messaging standpoint. Such implementation shouldn t be duplicated anywhere in a typical Enterprise SOA Model. Instead in the above scenario, the set of configuration has to be coded only one time at one place. So in any example if there are three services that are needed to be developed and deployed separately, then what means by service virtualization is that only Business logic (A) need to be written for disparate services while (B) (C) (D) needs to be just configured and reused. Characteristics and Benefits of Service Virtualization:- 1) Enables factory development Model giving way for high Productivity. 2) Packaging & Container Deployment 3) Significant Code Reduction with reconfigure ability on Service Binding, Security and Policy 4) Minimal effort on Service Administration & Code Release. 5) Technology Independence. So in any SOA Enterprise, what means by Service Virtualization is Developing Service with keeping Deployment and Services Management to minimum and constant. Service Virtualization through Federated ESB: Enterprise Services Bus (ESB) is the backbone for any SOA enterprise. ESB helps Virtual zing the services. In a larger enterprise you may need Multiple ESBs based on LOBs to rollout huge SOA initiatives. This is because of growing demand on Performance and SLAs. Different ESBs emerge with different functions. These ESBs might be similar or different based on one or multiple vendor selection.

5 In today s ESB's evolution we see that ESB has come miles from Foundational to Transformational and now with one that can adapt dynamically called as Federated ESB (FESB). (Pronounced as fis-bee) In Transformational environment ESB would use Service Registry, do BPM Service Orchestration, Service Activity and Systems Monitoring with or without using Computing Appliance Systems. Federated ESB (FESB) would not only use above features but also have services hosted on multiple ESBs. FESB may be owned by multiple LOBs. They are managed separately but they work together as Single Golden Source for Enterprise Services. These FESB need not proliferate from same vendor. Dependencies:- Single point of source for Registry Source. Autonomous Manager managing and mediating service across FESB. How Grid Environments enables optimization of Service Virtualization in Federated ESB? FESB enabled by Grid enables high end performance based dynamically configurable environment. In a Federated ESB Grid Environment, the Provider and Consumer had to scale and reconfigure based on runtime demand vs. supply footprint. The Autonomous Manager would start or stop services across ESB. Know the health status of the services on distributed Grid Environment across Federated ESB environments. In other words the Services Deployed in Federated ESB Grid Environment should behave in manner of self implied Artificial Intelligence System. Here are the bases for any SOA infrastructure to attain maturity on Federated ESB Grid: 1. Self Managed Autonomous Domain 2. Event Driven Application 3. Service Mediation 4. Heterogeneous Platform Support Self Managed Autonomous Domain: In Federated ESB Grid environment, there will be one Master ESB and several other Slave ESBs. Both master & slave will be workers except that the Master will also manage the Services deployed in other ESB for throttling (Slow Consumer/ Fast Provider) or Service Recovery. At the moment of Master ESB failure, one of the Slaves will be promoted as Master and would start Managing until the Original Master comes back online. This entire setup is called as Self Managed Autonomous Domain setup. In an extreme Transaction Procession environment this setup would ensure high system performance since the Autonomous domain would manage the services in such a manner that the least used ESB would be leveraged by scaling the services on it. Event Driven Application Event Driven Application Platforms support event centric programming models such as event handlers which are triggered by process business events. The Service application that is exposed as part of ESB should support Event-Driven-Application (EDA) models. Service Mediation: The ESB should able to intercept and modify service messages that are passed between the services. It should process message based on primitive or complex logic. It should able to transform, route or augment message within the grid environment. Heterogeneous Platform Support: Services that are hosed in the Grid enabled architecture should not be technology agnostic. It should be able to support JAVA,.net, python, etc. What Services need to be optimized? Identification of services for optimization would be based on most used ones in terms of number of hits or services that conform to SLA commitments. These services would have to be configured and deployed on various ESBs. At runtime these services would be dynamically scale based on the policies written within the grid environment. Services that need to process extreme transaction would leverage this underlying grid enabled FESB environment.

6 Fig. Federated ESB (FESB) in Grid Enabled Environment 5. CONCLUSION XTP is a complex problem and cannot be solved with one specific product or technology. Organizations should follow an incremental approach by complementing established and proven products with the appropriate combination of innovative technologies as needed for supporting emerging business requirements or adopt the riskier and more-disruptive technologies to grab greater business benefits and sustainable competitive advantage e.g. the combination of proven and advanced technologies is, in many cases, the most pragmatic way to take advantage of innovation by keeping risk at a tolerable level. Fig. Roadmap for Hybrid Technology Adoption

7 6. REFERENCES [1] Gartner RAS Core Research Note G , G , G [2] Wall Street's Quest To Process Data At The Speed Of Light By Richard Martin [3] Open Grid Services Architecture (OGSA) [Copyright Global Grid Forum ( )] [4] A Case for Economy Grid Architecture for Service Oriented Grid Computing By Rajkumar Buyya, David Abramson, and Jonathan Giddy [5] Service Virtualization:The Road to Simplification by Art Sedigh [6] Enterprise Service Bus Roy Schutte.