webmethods Insight for webmethods Integration Server Gain Real-Time Visibility and Minimize Disruptions Ann Marie Bond Systems Engineer, Software AG Technical White Paper October 2009
2
Contents introduction 4 Meeting the Challenges of Transaction Management 5 webmethods Insight: How it Works 7 Webmethods Insight: technical Details 12 webmethods Insight: Architecture Advantages 14 Conclusion: Benefitting the Entire Community of Users 15 3
INTRODUCTION With webmethods Insight you can quickly pinpoint business and technical exceptions and disruptions, identify root causes, and fix problems. IT organizations that use webmethods Integration Server have access to the most complete integration infrastructure in the market. Its ability to tie together existing applications and partners, and to enable developers to efficiently develop new applications, provides businesses with an agility that s hard to match. Now, webmethods Integration Server users can take even more control over today s heterogeneous IT environment with webmethods Insight TM. webmethods Insight is a business transaction management tool that provides real-time visibility into every business transaction across every siloed system and application that touches the webmethods Integration Server TM. With webmethods Insight you can quickly pinpoint business and technical exceptions and disruptions, identify root causes, and fix problems. webmethods Insight not only reduces the time and costs of troubleshooting, it also shortcircuits the blame game that typically takes place when problems occur. And it enables IT to be an even better partner to the business by maintaining the highest availability of interconnected customer-facing systems and applications. In this white paper you will learn: How webmethods Insight addresses the key challenges of running an integration infrastructure The capabilities and benefits webmethods Insight provides for integration teams and business application owners Together, webmethods Integration Server and webmethods Insight set the standard for the combined power of integration and end-to-end business transaction management. How the webmethods Insight Server and the webmethods Integration Server work together to provide unparalleled visibility across a heterogeneous environment How the architecture of webmethods Insight provides superior performance, security, and visibility Fast to deploy, with lower overhead and lower total cost of ownership, webmethods Insight is lightweight and non-invasive. It provides a proven, best-of-breed solution for monitoring SLA management and for troubleshooting services and business transactions. Together, webmethods Integration Server and webmethods Insight set the standard for the combined power of integration and end-to-end business transaction management. 4
MEETING THE CHALLENGES OF TRANSACTION MANAGEMENT As an integration team member or business application owner, you re expected to have more transparency and accountability than ever for your distributed and heterogeneous environments. webmethods Integration Server has provided the application integration capabilities, high-speed messaging, support for B2B partner integration, legacy applications and eventdriven technologies needed to tie together the many disparate components of these environments. You know you can rely on webmethods Integration Server to optimally link your siloed applications, orchestrate services and expose data for internal and external consumers (Figure 1). Now, with webmethods Insight, you can take the performance of your webmethods Integration Server to the next level. webmethods Integration Server With webmethods Insight, you can take the performance of your webmethods Integration Server to the next level: Implement error handling framework without coding Isolate performance issues non-invasively Find problems before encountering them Create run-time dependency tree Generate alerts Databases Custom & Packaged Applications B2B Trading Partners Software As Mainframe a Service (SaaS) & Legacy Systems Figure 1. webmethods Integration Server links your siloed applications, orchestrates services and exposes data for inernal and external consumers. Manage SLAs Capture payload and context By providing real-time visibility into every business transaction across every siloed system and application that touches the webmethods Integration Server, webmethods Insight helps you address the key challenges of integration management: Uptime: Mission-critical applications must not only be up, but also functioning properly all the time. A webmethods Integration Server application can be dependent on packaged applications, databases, even applications outside the firewall. When problems arise it is costly, inefficient, and time-consuming to identify the problem s source and correlate pieces of transactional information. Research shows it can take up to 20 people about six hours each to perform root-cause analysis, fix the problem and test the resolution. 1 But with the deep visibility into service transactions and automatic transaction correlation provided by webmethods Insight, you can quickly react to, identify, and resolve the root cause of problems, and thereby: > Reduce the time/costs of problem-solving by 85% > Improve problem avoidance by 25% > Increase your team s productivity Performance: Developers and operators need to be able to understand what s going on both at design-time and at run-time. With webmethods Insight, you know how transactions flow through webmethods Integration Server and where the bottlenecks are that might affect your consumers. You know how many calls you re making or how much data is being passed. You can aggregate statistics across an end-to-end transaction or view time-bounded snapshots with average response times. Research shows it can take up to 20 people about six hours each to perform root-cause analysis, fix the problem and test the resolution. But with the deep visibility into service transactions and automatic transaction correlation provided by webmethods Insight, you can quickly react to, identify, and resolve the root cause of problems. 1 Source: Forrester Consulting: The Total Economic Impact TM of Progress Actional Management for Interconnected Applications, February, 2009. 5
webmethods Insight helps you meet the challenges of transaction management: Increase uptime Improve performance Establish governance Manage chargebacks Develop rapidly Effective planning Governance and Compliance: Governing your ecosystem and monitoring compliance is difficult without the right level of visibility. With webmethods Insight, you can see who consumes the services offered by your webmethods Integration Server and how. You can identify unexpected or rogue consumers of your ESB services and ensure that applications are operating in compliance with organizational processes and policies. Chargeback: Organizations commit over 80% of their IT budget to run, debug, and maintain their systems, applications and services. As organizations grow Service-Oriented Architecture (SOA) adoption or complex distributed infrastructures, the ability to implement utilization chargebacks becomes even more challenging. webmethods Insight collects and aggregates transaction and service usage to make chargebacks easy, accurate and defensible. Rapid Development: webmethods Insight enables you to reduce errors and maximize productivity during development. For example, you could avoid common mistakes like promoting a webmethods Integration Server package to production that s still calling a development service or database. You can also analyze the performance of services while still in development, when problems can be addressed easily. Planning: webmethods Insight helps you understand the current requirements of the webmethods Integration Server from an IT standpoint and extrapolate future requirements. Historical performance metrics enable you to analyze the effectiveness of your server infrastructure and recommend changes that will improve performance and business results. Key Benefits For integration teams and application owners, the benefits that webmethods Insight offers and the capabilities that it puts in their hands are unsurpassed: Provides Deep Visibility Gain insight into transaction and service performance Automatically discover calling patterns and dependencies Discover and manage service transactions Minimizes Disruptions Actively manage transactions and services with detailed real-time metrics Implement alerts or alarms to prevent performance problems before you reach SLA limits Isolate and diagnose the root cause of problems more quickly Ensures Compliance Track and enforce compliance with governance policies Detect rogue services and bring them into the governance process Ensure design-time and run-time policies are not avoided or evaded With webmethods Insight there are no more lengthy troubleshooting sessions or manual looking at logs and laboriously trying to correlate transactions. Just as importantly, webmethods Insight puts an end to finger-pointing, which typically finds the integration team at the center of an unproductive blame game. webmethods Insight changes the conversation instantly, transforming the people who have been unfairly regarded as the problem into the solution. 6
WEBMETHODS INSIGHT: HOW IT WORKS The webmethods Insight Network Overview in Figure 2, a screen shot from webmethods Insight, shows a complete picture of an organization s IT infrastructure. webmethods Insight will dynamically create updates. When new systems are deployed webmethods Insight detects and displays them, making obsolete the huge network diagrams hanging on walls in IT shops. In this example, two webmethods Integration Server instances are at the center of a typical implementation, including packaged applications, an application server, a database, and more. The webmethods Integration Server instances are orchestrating services across SAP, Layer 7, Oracle,.NET, and a custom portal application. The other servers have been auto-detected as consumers or providers of services on these managed nodes. Figure 2. webmethods Insight Network Overview: In this example, two webmethods Integration Server instances are at the center of a typical implementation, including packaged applications, an application server, a database, and more. 7
Two primary elements of webmethods Insight webmethods Insight Server The server is installed on a standalone system and is highly scalable. The server runs standalone or in any number of application platforms. It requires a database and supports many popular databases. Figure 3 shows how policies are distributed from the server to the agents and statistics and alerts reported back. webmethods Insight is lightweight and non-invasive, so neither installation nor operation of the agent requires making changes to the application. Figure 3. webmethods Insight Server-Agent Communication: Policies are distributed from the server to the agent and statistics and alerts are reported back. webmethods Insight Agent The agent is installed on a platform, turning it into an instrumented platform. The term instrumented means that the platform can be managed by the server. webmethods Insight is lightweight and non-invasive, so neither installation nor operation of the agent requires making changes to the application. An interceptor is a lightweight component that plugs into a particular protocol stack or application platform to gather information about calls being passed by the system. Interceptors are specific to the application that they instrument. Integration Server and a number of other protocol stacks and platforms are currently supported. Instrumented Platform Components An agent is made up of two types of components: at least one interceptor and an analyzer. An interceptor 2 is a lightweight component that plugs into a particular protocol stack or application platform to gather information about calls being passed by the system. Interceptors are specific to the application that they instrument. webmethods Integration Server and a number of other protocol stacks and platforms are currently supported. There is one interceptor for each application that is being tracked, or managed. An analyzer processes events by reading them from the in-memory buffer, called the Flight Data Recorder. webmethods Insight agents plug into the container, leveraging the normal container parsing of XML. Because the agents are running in the container, the applications within do not need to be modified. What s more, the container is session/thread aware and can use the container s session context to correlate across the session as it goes from input on one protocol to one or more output calls on one or more protocols. webmethods Insight provides support for multiple protocols. All this adds up to much better performance, auto-discovery of services, automatic correlation across transactions, and no impact to applications. 2 See page 12 for technical details on the Interceptor for webmethods Integration Server. 8
SLA Management and Root Cause Isolation Service Level Agreements (SLA) and Transaction Level Agreements (TLA) in webmethods Insight can be defined to detect a range of issues from pure outages to non-responding transactions to performance slowdowns to business exceptions. When SLA violations are reported, users can review the related transaction in a visual way by tracing through each step to isolate the root cause. webmethods Insight collects and automatically correlates data across all webmethods Integration Server instances and 3rd-party platforms, extending out to consumers or providers one hop away from managed nodes. webmethods Insight provides automatic correlation that requires no changes to payload it uses a real-time decentralized process. During this process, it can capture a payload transaction snapshot for root-cause analysis. Figure 4. webmethods Insight Root Cause Analysis: This screen shows an alert generated by an SLA. The webmethods Insight agent provides this information by buffering in-flight transactions in memory as the transactions flow through the enterprise. If there is a policy/sla violation, the server gathers transaction history from all the involved agents for the specific problem transaction. Collecting the full transaction history only on-demand greatly reduces the amount of information the agent has to share with the server. However, the sliding window of buffered data maintained by the agent ensures that a complete set of transaction statistics will be available when needed for an end-to-end analysis of problem transactions. Operations teams can then drill into reported violations and evaluate both the content and the context of a transaction to take action on problems and minimize disruptions to consumers. 9
Dashboards All the metrics collected by webmethods Insight can be aggregated, filtered and organized to create custom dashboards to provide management and accountability. By default, transactional metrics such as SOAP faults, service response time and data transmitted are collected. Administrators can also request pipeline information from webmethods Integration Server calls or web services and combine them with transactional metrics to build chart dimensions that reflect mission or business transaction criteria. Operators, administrators, project managers and architects may each have different dashboards focused on their requirements, as shown in Figure 5 below. Figure 5. webmethods Insight Server Dashboard: Operators, administrators, project managers and architects may each have different dashboards focused on their requirements. 10
Deep And Wide Visibility webmethods Insight s powerful capabilities provide unparalleled visibility into webmethods Integration Server and beyond, including: Internal Visibility: webmethods Insight traces application transactions from consumer to provider across all webmethods Integration Server protocols, eliminating costly manual searches and logging on multiple machines. Even complex implementations can be traced as they execute and follow application logic. And because webmethods Insight collects performance metrics for all service invocations, it s easy to determine how long a service takes at run-time or how much data it s transmitting. Cross-Platform Visibility: Automatic correlation of transactions across many common application platforms enables end-to-end monitoring and management of transactions that reach beyond webmethods Integration Server. With a visual representation of a transaction linked to performance metrics, debugging multi-level, cross-platform transactions is much easier. Automatically Discovered Services: webmethods Insight runs in the webmethods Integration Server container, not in the application, so it automatically detects whatever services are running. One-hop Visibility: webmethods Insight tracks both service consumers and outbound requests. So when webmethods Insight is installed on a single webmethods Integration Server it will discover all inbound and outbound service requests and expand your visibility without requiring additional agents. Reconnecting Flows: webmethods Insight provides support for auto-correlation across an un-instrumental node which is sandwiched between two instrumental nodes. Data Capture: Automatic capture of performance data and pipeline data from webmethods Integration Server provides a powerful debugging and analysis tool. You can: Drill into pipeline content and service context for quick root cause analysis. Use historical performance data to aggregate metrics across transactions, servers, time and many other dimensions. Pull out pieces of message data from the pipeline and aggregate them to derive mission-oriented metrics. Service-Level Agreements (SLAs): webmethods Insight has the ability to detect both content issues via pipeline data capture from webmethods Integration Server services and technical exceptions via infrastructure metrics. SLAs can be set on technical exceptions and mission-oriented exceptions. 11
WEBMETHODS INSIGHT: technical DETAILS The Role of Interceptor Plug-in for Monitoring webmethods Insight is the only service and transaction monitoring product designed specifically for webmethods Integration Server. The Interceptor for webmethods Integration Server is the critical monitoring component of webmethods Insight s architecture and has been designed to work seamlessly with webmethods Integration Server. The Interceptor for webmethods Integration Server provides unique visibility into Flow services, with automatic discovery and dynamic handling of Flow changes at run-time. By instrumenting the webmethods Integration Server container, webmethods Insight requires no changes to applications. Options with the installation include deciding which protocols, services, and packages to show or hide to fine-tune the network and service path views. Visibility The webmethods Integration Server Interceptor has visibility into servlet traffic hosted in the Tomcat package, which hosts JSP and DSP pages. It also tracks JDBC and JMS traffic going in/out of the webmethods Integration Server. And it records all HTTP client traffic going out of webmethods Integration Server. The Interceptor shows service traffic between: Java services XML services Internal services XSLT services Flat file services JMS trigger services MIME/SMIME services Flow services The Interceptor also correlates inbound requests through SOAP via the web services handler (/ws/...) and the built-in SOAP handler (/soap and /soap/rpc). For outbound requests, the Interceptor collects the service, operation, steps and metrics for services using the HTTP, SOAP, FTP, SMTP and Remote Invoke protocols. Instrumentation The Interceptor implements a style of instrumentation called staged classloading whereby certain interceptors (JDBC, servlet, Axis) are only deployed in the webmethods Integration Server package classloader, while others (such as the base and JMS interceptors) are deployed in the server s main classloader. This is done in order to avoid intercepting purely internal calls. Protocols webmethods Integration Server protocols that are instrumented by the Interceptor include SOAP, HTTP, EJB, JDBC, RMI, Axis and JMS. The Interceptor supports webmethods Integration Server versions 6.5, 7.1.1 and 7.1.2. 12
Fine-Tuning the Display The webmethods Integration Server Transaction Path (Figure 6) shows a typical detailed path view. Path displays can be filtered and adjusted to focus on key transactions in several different ways: webmethods Integration Server packages can be omitted from reporting. Specific services can be omitted from reporting. Flow steps are not reported by default. However, they can be enabled and displayed for more detailed analysis. System-wide visibility of JDBC adapter database transactions can be enabled. Installation webmethods Insight is installed by instrumenting the webmethods Integration Server container and is non-invasive. It requires no customization or application changes. Figure 6. Flow Service Visibility: This screen shows the path taken by the transaction as it calls one flow-service after another. The Interceptor is what makes this happen. 13
WEBMETHODS INSIGHT: ARCHITECTURE ADVANTAGES webmethods Insight is built from the ground up for enterprise-level deployments and is designed to manage mission-critical distributed applications like those handled by webmethods Integration Server. Unique Visibility into Integrations at Multiple Protocol Levels Integration services do not depend only on HTTP(s) and SOAP traffic but incorporate a multitude of protocols such as JMS, RMI, JDBC, and with webmethods Integration Server, XML-based proprietary protocols as well. Regardless of the protocols used, webmethods Insight can provide a complete end-to-end view of a webmethods Integration Server network something no other management solution can do. What s more, this network can be extended to many other applications and protocols, seamlessly tracking applications across platforms. KEY ADVANTAGES Rapid Deployment Easy to install and manage using webmethods Insight server console Deploying agents requires no impact on your applications and services Smart Monitoring Dynamic auto-correlation without modifying the SOA environment Agents give you one-hop visibility, uncovering underlying service dependencies Minimal Overhead Minimal overhead even on highvolume transaction environments Snapshot problem transactions quickly without flooding the network Designed and Architected for Performance, Security and Visibility Automatic service discovery: webmethods Insight can automatically discover service producers, service consumers, and the dependencies between them. This auto-discovered interconnection can also be viewed graphically. Message flows between nodes are identified down to the operation level. In addition, there is no need to explicitly configure the services in advance for them to be discovered or managed. Simply installing the webmethods Insight agent enables operators to deploy policy that takes effect on the first service call to the node. High scalability: webmethods Insight s architecture uniquely blends distributed policy evaluation with centralized command and control, enabling very high scalability. The webmethods Insight management server is designed to be able to handle at least 1,000 managed nodes and 50,000 service interdependencies. Also, the server is never a single point of failure. Policies are evaluated at the agents, so a server failure does not affect policy evaluation. High performance with minimum impact: With less than 50 microseconds of latency added to each message, webmethods Insight has the highest performance of any solution on the market. This excellent performance enables webmethods Insight to be run all the time in production. In addition, because the server is never in the flow of messages, the server or network will never detract from management performance. On-site tests have shown that it is possible to stay under 2% of CPU utilization and to achieve message throughput of more than 3.5 million messages per hour on webmethods Insight-managed nodes even though this is where policy processing occurs. Rapid Deployment with No Application Changes Using webmethods Insight to manage an integration ecosystem does not require any application changes. webmethods Insight can be installed into an existing application environment without modifications to the application. Additionally, there is no need to deploy webmethods Insight agents everywhere to take advantage of the product; even a single instrumented (managed) node can bring value because of the automatic discovery of consumers and services one hop away from each instrumented node. 14
CONCLUSION: BENEFITTING THE ENTIRE COMMUNITY OF USERS webmethods Insight provides extraordinary visibility, management, and control in a very efficient way to a broad spectrum of users: Business Managers: They benefit from the ability to extract payload or pipeline information as it passes through the webmethods Integration Server. This expands the management capabilities of the platform to analysis at a business transaction level. While it s certainly possible to install network sniffers and collect data from traffic passing through a network, correlating and modeling that data in business context for business users is nearly impossible. Architects: Implementing webmethods Insight allows architects to transform from the point-to-point, or just-a-bunch-of-services structures that are often found in webmethods Integration Server implementations toward a true SOA. Using webmethods Insight, architects get the visibility into services, complete transactions and run-time dependencies. This gives them a greater awareness into what is really happening in their enterprise and enables them to optimize their plans for maturing SOA strategically. What s more, information from web- Methods Insight can be transferred directly into CentraSite ActiveSOA or another registry/ repository, empowering architects to define policies and lifecycles that would return the benefits of a disciplined, governed service-oriented architecture. Operations and IT Managers: IT responsiveness is critical to their ability to support the business side of the house. webmethods Insight gives the IT operations team the ability to detect problems quickly and instantly and to accurately isolate the root cause, and prevent some problems before they happen. Additionally, webmethods Insight is an invaluable tool for proper capacity planning with SOA, since capacity requirements can vary for each service consumer and provider. Operations and Security Managers will also appreciate the quick identification of rogue services. About the Author: Ann Marie Bond is a Systems Engineer with the SOA Practice at Software AG. She has been working with webmethods Insight since the inception of the product to provide technical expertise for Sales. Prior to working in the Sales organization, Ms. Bond led the Integration Competency Center at webmethods, using webmethods own software to solve business problems. Ms. Bond is a certified Zapthink SOA Architect and has a PMP. She also worked in software development for 14 years on projects ranging in size from very small to over 400 developers. Ms. Bond has a B.S. in Computer Science from George Mason University. Developers: webmethods Insight enables developers to gain deeper knowledge into how their services or application behave and execute in the context of a business transaction along with all interconnected dependencies. Performance benchmarking can now be extended to the transaction level with performance statistics shown for each invocation or hop. The entire community of users will benefit from the increased uptime, faster problem resolution, and more stable deployments made possible by webmethods Insight. Using webmethods Insight, your organization can proactively manage every single important transaction. Trying out webmethods Insight is easy. Schedule a free ½ day on-site workshop or pilot it in your environment. Deployment is fast and non-invasive. You ll surely gain visibility into something new. Find out more at www.softwareag.com/insightworkshop. 15
TO FIND THE SOFTWARE AG OFFICE NEAREST YOU, PLEASE VISIT WWW.SOFTWAREAG.COM Take the next step to get there faster. ABOUT SOFTWARE AG Software AG s 4,000 global customers use our software to improve business processes and drive an agile IT infrastructure. Our customers goals are to reduce costs and increase flexibility and efficiency. We help them do this by optimizing and governing their operations and aligning IT with the business goals. Our leading Business Infrastructure Software portfolio is used for data and system integration and modernization. It fosters new levels of IT agility through SOA and allows the rapid creation of new business processes with BPM. Our 40-year history of success ensures our customers have a reliable platform for driving future business results faster. Software AG Get There Faster Copyright 2009 Software AG, Darmstadt, Germany and/or Software AG USA, Inc., Reston, VA, United States of America, and/or their suppliers. All rights reserved. SAG_Insight_IS_Tech_WP_Oct09