A Unified View Across WebSphere DataPower and MQ, Solace and TIBCO Messaging Get a single point of control for: message auditing, tracking, transformation, routing and performance P/2 EXECUTIVE SUMMARY P/2 FINANCIAL SERVICES SCENARIO P/9 THE CASE FOR MESSAGE TRACKING P/13 PERFORMANCE AND AVAILABILITY OF AUTOPILOT P/14 CONCLUSION 2012 Nastel Technologies, Inc.
Executive Summary Many companies have a pressing need to establish a single point of control over their multi-vendor messaging infrastructures. They are frequently confronted with: Acquisitions that expand technology diversity and footprint Lack of visibility across multiple middleware platforms The need for self-service Shortage of human capital Continuously rising cost for support and maintenance Increasing urgency to determine the root-cause of problems The complexity of one management system per messaging infrastructure is too costly and when these infrastructures are interconnected, the management systems become ineffective. These firms require one place to monitor, visualize and analyze the status of applications and services shared across the enterprise environment. However, the skill set required to manage each middleware technology is often different and a single business flow can traverse multiple different transports and support teams. This makes unified monitoring of messages across the infrastructure and the ability to track them and rapidly identify and correct problems anywhere in that flow an essential element in improving service and reducing cost perhaps, even a necessity for survival From the business side there is pressure to adjust to the realities of our economy, reduce operational and capital expenditures essentially, do more for less. In addition there is a strong trend toward outsourcing support functions or moving them off shore which puts pressure on operating procedures. Economic pressure on enterprise organizations to consolidate has resulted in significant merger and acquisition activity, where companies are forced to use the IT assets and applications inherited through these consolidations. However, those inherited applications do not use the same interfaces essentially, they do not speak the same language as those already in use at the acquiring organization. Additionally, these pressures have accelerated the move to the private cloud, where enterprises hope to save additional costs. In many of these enterprises, appliances such as DataPower and Solace are used to augment traditional messaging middleware. In addition, tools such as an enterprise service bus or SOA appliances are used as universal translators to transform the content from one service to the other and route traffic between them. Much of that traffic comes from Message Oriented Middleware (MOM) as it has been accepted as the most common way of linking systems in enterprise environments. The combination of multiple brands of traditional middleware, appliances and an enterprise service bus creates a great deal of complexity to contend with. While the flexibility and power gained from this approach can be essential, it remains a significant challenge to manage this complexity without a single point of control. P/2
A Financial Services Scenario: ONE UNIFIED GLOBAL POINT OF CONTROL One of Nastel s clients, a global banking conglomerate, had the challenge of monitoring and managing the WebSphere MQ family of products, DataPower, Solace, and TIBCO EMS and Rendezvous (RV) as well as a homegrown messaging solution. Historically, their applications ran on WebSphere MQ. However, when the bank acquired another bank with it s own IT infrastructure using TIBCO EMS and Rendezvous rather than WebSphere MQ it realized that it had a requirement to integrate the disparate systems. Many of the new systems acquired leveraged TIBCO EMS and Rendezvous for the middleware layer. The bank decided to leverage DataPower appliances to act as the bridge to transform and route the content of messages coming from applications using WebSphere MQ to the format that their acquired banks applications using TIBCO EMS required and then route them to TIBCO. In order to monitor the performance of the messaging infrastructure end-to-end, they needed visibility for their custom applications, WebSphere DataPower, Solace Message and Content Routers, WebSphere MQ and TIBCO EMS and RV from a single monitoring tool. The bank realized that vendor monitoring products do extend across technology boundaries and their home grown tools can no longer keep up with their technology evolution. They needed better tooling to maintain productivity and keep the costs associated with problem resolution at acceptable levels. The IT team began searching for a global monitoring solution that unified the common operational procedures that existing across the multiple technologies, supported them operationally; including run-time operations, WebSphere MQ administration, rapid problem detection and isolation and operational review and planning. But it also had to support the increasing interoperability between WebSphere DataPower, Solace, WebSphere MQ and TIBCO. Plus, they felt it was time to begin message tracking through JMS, WebSphere MQ and Data- Power. FIRST STEP: EXPANDING MONITORING AND MANAGEMENT CAPABILITIES The bank s IT department had been using a well-known vendor s event monitoring product to monitor WebSphere MQ. They replaced this with Nastel AutoPilot to monitor and manage WebSphere MQ on distributed and z/os platforms as AutoPilot gave them the ability to watch the homegrown message broker and internal business applications as well. They also used TIBCO Hawk to monitor the TIBCO EMS system. Business requirements soon dictated that integration across their environment was necessary. But at this point, they had two separate management systems. As the bank moved to integrated their messaging systems they decided to do likewise with their systems for middleware monitoring. They placed responsibility for all middleware under one team chartered with integrating front-end and back-end applications. It soon became impossible P/3
for this new middleware team to meet the performance service level agreements mandated by the business. They were running mission-critical applications that relied on IBM and TIBCO products, including a high-volume trading application, but couldn t handle the demands of a full production workload, which is millions of messages per second over hundreds of queue managers. In addition they did not have integrated monitoring for DataPower or Solace appliances. Having to deal with multiple monitoring solutions along with a need for a third or fourth prevented them from meeting performance standards. The group determined that they needed a single monitoring solution that met the following requirements: Provides real-time visibility across DataPower, Solace, TIBCO, WebSphere MQ and their homegrown middleware solution Finds stuck messages Delivers early notification of faults, bottlenecks and slowdowns Monitors the performance of messaging and compares to SLAs Provides administrative control over WebSphere MQ Can provide message tracking through JMS, WebSphere MQ and DataPower Easily integrates with the bank's enterprise management systems AUTOPILOT: End of the Hawk Nastel s AutoPilot was already effectively monitoring the bank s WebSphere MQ and homegrown MOM systems. However, while the bank trusted TIBCO, it became desirable to integrate the TIBCO messaging products into the same monitoring framework. In large enterprise environments like this one, with many interdependent applications constantly in flux, the speed and ease of adapting to change was a key factor in the decision process. To configure TIBCO Hawk to monitor a new EMS server or Rendezvous daemon would typically take the team considerable effort. With AutoPilot, it takes minutes. With the size of the bank s environment, AutoPilot emerged as the obvious choice. As an added benefit and cost saving, training for team members could be limited to just a single monitoring environment. The middleware team decided to standardize all application performance monitoring under Nastel s AutoPilot. The middleware team felt that AutoPilot would allow them to: Unify the monitoring and management procedures across all middleware technologies Create a single portal for administration and certification of user permissions and access Create valuable rules that can be applied to all middleware transports for end-to-end monitoring P/4
Figure 1: This screenshot displays AutoPilot s real-time monitoring for the Solace appliance. NASTEL S MONITORING SOLUTION for the Solace Message Appliance Nastel s solution for monitoring the Solace messaging appliances was welcomed by the bank as it was an integrated component within its unified middleware monitoring solution and not a standalone tool. AutoPilot for Solace can monitor multiple appliances harvesting their events and analyzing them in correlation with other middleware and system events in the AutoPilot Complex Event Processing engine. In addition it combines monitoring of software components of the Solace environment with the hardware inventory and assets (blades, power modules, fans) to provide a complete monitoring solution for appliance-based technology The collected analysis includes: Appliance specifications Appliance Incoming/outgoing data statistics Cache and cache-level statistics Router clients and queues Hardware properties and specifications List of VPNs and VPN-level statistics Current/max messages spooled P/5
Figure 2: AutoPilot Expert for DataPower: In the situation above it has detected a configuration error. AutoPilot can create robust sensors which monitor multiple factors within DataPower before triggering alerts and warnings and offers a single monitoring interface across all of a customer s appliances. Nastel s DataPower Monitoring Solution The bank decided to use WebSphere DataPower appliances for message transformation and routing between the different middleware messaging systems. AutoPilot's ability to provide deep operational monitoring for DataPower and provide a unified view across all their middleware messaging solutions was important to the bank. The AutoPilot Expert for DataPower analyzes: DataPower's consumption of system resources over time DataPower metrics DataPower s performance of key business activities Routing and transformation history Correlate DataPower performance data with events from other systems NASTEL S TIBCO EMS MONITORING SOLUTION A major driver of the bank s decision to standardize on AutoPilot was its ability to monitor all of its middleware systems. With AutoPilot s TIBCO EMS plug-in expert, the middleware team can easily monitor TIBCO EMS servers and components including: queues, topics, consumers, producers from a single, consolidated vantage point. There is no limit to the number of TIBCO EMS servers they can simultaneously monitor. P/6
The TIBCO EMS Plug-in expert is modularized based on the bank s EMS servers and their components, for example: queue expert, topic expert, consumer and producer experts. The bank s specific requirements for performance monitoring activities will determine how the experts are deployed -- they can be deployed individually or in any combination of multiple deployments. In addition, the TIBCO EMS expert automatically measures and collects EMS performance metrics. The expert periodically samples information about TIBCO EMS servers and their components based on any interval defined by the bank s middleware team. Some of the key performance metrics they can collect include: inbound, outbound, acknowledged and pending message volumes, message rates per second, average response times, message tracking and age of oldest messages. These metrics are collected for servers, queues, routes, groups, consumers, producers, transports, Figure 3: TIBCO EMS Metrics Collected by AutoPilot s TIBCO EMS Experts durables and events. AutoPilot s powerful Complex Event Processing (CEP) engine uses these metrics in policies the middleware team defines for prediction, correlation, validation, automation, notifications, logging and alerts. AutoPilot s CEP engine can also combine metrics and data from many diverse systems within the bank s infrastructure. One of the most important features for the bank is that the AutoPilot TIBCO/EMS plug-in monitors multiple TIBCO EMS servers and the AutoPilot CEP engine relates the EMS metrics with metrics from other infrastructure components such as DataPower, Solace, WebSphere MQ, Message Broker, ESB and other supported technologies to dynamically create events and alerts as needed. P/7
NASTEL S TIBCO RV MONITORING SOLUTION An additional requirement the bank had for AutoPilot was to provide performance monitoring for TIBCO RV and provide a service view of TIBCO multicast services. Using AutoPilot s TIBCO RV plug-in expert, the middleware team can monitor TIBCO RV performance statistics via the plug-in expert s automatic collection of data about the TIBCO RV server and its components such as services, routers and transports. The expert collects TIBRV statistics by subscribing to and processing the advisory messages. The TIBCO RV expert analyzes the collected data providing performance analysis for TIBCO RV Server Info, Client Transport, Service, Router Info and Security Info. It was important to the bank that this was functional in both agent and agent free deployments. The metrics that the expert provides come from the RV server and this data is used in AutoPilot in business views for validation, automation, notification, logging and alerts. A Business View showing the health of the RV Server environment can display RV status including: Server health state, inbound and outbound Multicast lost messages, inbound and outbound Point-to-Point lost messages, reports number of clients using a particular service, reports number of hosts using a particular service, rate of inbound and outbound traffic, amount of inbound and outbound data and the number of missed inbound messages. Figure 4: : TIBCO RV Health Business View P/8
THE CASE FOR MESSAGE TRACKING Messaging middleware has transformed the way applications communicate and exchange information. Application integration has brought about significant advantages to the development of composite applications; however, it has also brought about a need to track messages, end-to-end through JMS, WebSphere MQ, DataPower, brokers and other middleware. As IT complexity grows, compliance and integration needs evolve and so does the need for tracking the exchange between IT services and their corresponding business services. Most professionals who support critical business applications have dealt with the problematic consequences of performance degradation or messages that have not made it to their destination. These problems are difficult to resolve and take countless hours to determine the root-cause. Figure 5: AutoPilot tracks WMQ transaction compliance including: number of transactions, in-flight transactions, violations, violation rate, failures, failure rate, maximum and average duration. P/9
Figure 6: This screenshot shows the results of AutoPilot s automated discovery of WebSphere MQ and Message Broker, the bank s applications and their WMQ transactions. In real-time AutoPilot tracks the flow of WebSphere MQ messages and monitors their performance and availability. P/10
Figure 7: This screenshot shows an alternative view of WebSphere MQ transactions. In this case we are comparing discovered transactions to customer specified milestones or progress through a business process P/11
Real-time, message tracking provides visibility into message flows enabling organizations to prioritize problems appropriately and work on the issues that have the most impact to the business, such as: Where is my message?, Where is my order?, Are orders flowing as expected? Many of these problems may turn out to be false alerts. These false alerts can often be suppressed using AutoPilot CEP which can also provide proactive notification about disruption in message flows and stuck or hung messages. Next Step: GIVING THE MIDDLEWARE TEAM MESSAGE TRACKING The middleware team needed message tracking in order to more rapidly resolve the issues their newer applications were facing. However, these teams also used middleware monitoring and administration and required a unified approach to managing all of this. The team took advantage of AutoPilot s Message tracking capabilities which provided the following benefits: Real-time, tracking of JMS, WMQ and DataPower middleware message flows (transactions) Dope and Trace and Recognize and Trace technologies are used with unique Transaction IDs for stitching and inferencing of the message stream Instantly detect delayed, compromised or incomplete JMS, WMQ and DataPower middleware message flows (transactions) Provide notification and automated actions based on user-defined criteria Can use middleware payload data to stitch messages together based on business policy Analyzes WMQ transaction throughput and message volume in real-time Stores all of the performance-related information in an easily accessible SQL database Provides an audit trail for all transactions and messages Generates reports about WMQ transaction activity on a daily, weekly, monthly basis Policy-driven monitoring and automated notification Identifies the root cause of performance bottlenecks P/12
PERFORMANCE AND SCALABILITY OF THE AUTOPILOT SOLUTION Horizontal scalability: On the agent level: ability to monitor a large number of applications across thousands of servers On the server level: AutoPilot s Virtual Complex Event Processing engine (CEP) engine can span multiple servers utilizing its built-in grid On the user level: AutoPilot can handle thousands of users Vertical scalability: Ability to process large volume of data from one source (one server that may have multiple applications). For example: situations where one Queue manager had more than 20 thousands queues and 10K channels, which translated to hundred of thousands of data metrics. Transactional metrics: The performance impact on applications tested with JSpec with an impact of less than 2.5% Input Speed: The capability to process consistently high volumes of KPIs (key performance indicators) and tens of thousands of transactions per second, per CEP instance. Because CEP engines can be added on-demand, processing capacity can be elastically increased as needed. Rules processing speed: The single CEP engine running on 64 bit quad CPU server with 4 GB of memory can process 2M rules per second. Because CEP is a virtual machine it can scale up linearly. When adding an additional CEP engine, the throughput can double. High Availability: The GRID-based infrastructure provides a smooth failover. Support for monitoring a wide range of messaging middleware platforms: Including: WebSphere DataPower, WebSphere MQ, WebSphere ESB, Message Broker, WBI, TIBCO RV and EMS, Solace Systems Messaging Appliances, CICS and JMS. P/13
CONCLUSION: A POWERFUL APPLICATION PERFORMANCE MONITORING SOLUTION FOR LARGE ENTERPRISES AutoPilot has proven to be a highly effective, comprehensive and proactive solution for the bank s complex, heterogeneous middleware environment. It is the only monitoring solution that supports traditional software-based messaging technology as well as appliance-based technology. It also enables the same operating procedures to be applied for all messaging products. Since standardizing on AutoPilot, the middleware team has real-time, end-to-end visibility and is now able to pinpoint problem areas and also the source, or root cause, of a problem anywhere across its various middleware systems. AutoPilot s Complex Event Processing engine automatically eliminates false alerts, identifies potential problems and provides early warning in order to avoid impacting users and disrupting business processes. These capabilities dramatically reduce the mean time to problem resolution, increases overall uptime and reduces user support requirements by cutting down significantly on calls from business users to the middleware team. It is also the only monitoring solution that unifies user permissioning and certification of access across the middleware estate to ensure regulatory compliance is met. Real-time, visibility from message tracking lets organizations not only measure how well their business services perform at any given moment, but also optimize them end-to-end. The most striking outcome is that it enhances an organization s responsiveness by focusing them at the right time and the right place. This can be clearly demonstrated when tracking messages across the infrastructure and can pin-point where performance bottlenecks are and why, so that organizations, which have previously embarked on a wild goose chase, can focus on problem resolution with precision. As a result, AutoPilot is now the foundation that enables the monitoring, automation, visualization and fault recovery for all of the bank s multi-vendor middleware, SOA and messaging appliances and brokers end-to-end. AutoPilot lets the middleware team keep their entire infrastructure governed, visible and manageable from a single pane of glass. P/14