An Innovative Approach to SOAP Monitoring Written By Quest Software
Contents Introduction...2 SOAP Overview...3 The SOAP Monitoring Challenge...6 From the Service Consumer Perspective...6 From the Service Provider Perspective...6 Foglight Meets the Challenge...8 Comprehensive Metrics...8 Meaningful Presentation of Actionable Information...10 Beyond SOAP Monitoring...14 Conclusion...15 1
Introduction Business applications have followed an evolutionary path similar to that of the manufacturing industry. Initially, manufacturers were vertically integrated. They manufactured and tested most of the components that went into their products themselves. Some even produced their own raw materials. Today, most manufacturers have evolved into a horizontally-integrated approach. They work closely with a number of supplier partners who provide subassemblies and other components, which the manufacturer combines into final products. It s imperative that partners deliver on time. Any delays could slow the manufacturing line and expose both the manufacturer and its supplier partners to significant business risk. Consequently, manufacturers and partners must closely monitor the links between them, and ensure that all orders are delivered on time. When business applications were vertically integrated, each application was self-contained and performed all required functions. Today s horizontally-integrated applications, which are usually based on a service-oriented architecture, combine services from different sources, such as outside service providers. For example, an online retail application may use a credit card verification service from an outside provider. The services-oriented approach requires tight coordination between service consumers and service providers to ensure timely delivery of services. To make this happen, consumers and their providers have to closely monitor the link between them. Like most enterprises today, you probably have web applications that are based on a web services architecture. These applications typically use SOAP messages for the link between service consumers and service providers. Consequently, it s imperative that, whether you are a service provider or a service consumer, you closely monitor SOAP messages. That presents a technological challenge. The SOAP monitor must analyze every message and capture detailed information buried within each one without introducing delays. This presents quite a challenge, given the massive volume of SOAP messages to be monitored. Quest Software answers that challenge with Foglight, an innovative and unique solution that monitors SOAP traffic, capturing and analyzing comprehensive information in near real time, without adding latency to your applications. Whether you are a service consumer or a service provider, you can use Foglight to keep close tabs on all SOAP communications. Consumers can monitor how providers are performing relative to their commitments. Service providers can monitor and prove their performance to consumers. Both consumers and providers are notified of problems early so they can address them promptly. In addition, providers can drill down into the monitored information to identify the cause. This paper presents an overview of SOAP messages, the challenges involved with monitoring them, and how Foglight helps meet those challenges. 2
SOAP Overview In service-oriented architecture, the service provider implements an application programming interface (API) that allows a service consumer s application to request and obtain services offered by the provider. The services offered are many and varied, such as: A map provider offers a service that provides directions and a map to a retailer s closest store A financial services provider provides stock price quotes A credit verification firm provides a credit check service A package delivery company provides a delivery time estimate service The API typically specifies that the SOAP standard be used for communicating between the consumer and the provider. SOAP is a simple XML-based communication protocol. It enables applications that may be running on different operating systems, with different technologies and programming languages, to communicate over the Internet. A SOAP message is an ordinary XML document containing three major elements: Envelope Contains both the Header and the Body; identifies the XML document as a SOAP message Header Contains the URLs of both the sender and receiver Body Contains the message details, including what service is being requested, as well as fault and status information SOAP messages provide the service request and service response mechanism between service consumers and providers. In a request message, the consumer requests a particular service from the provider, such as a stock price quote. In the response message, the provider forwards the results of the requested service to the consumer. 3
The following is an example of a SOAP request message and corresponding response message for a stock price quote: Stock Quote Request <?xml version="1.0" encoding="utf-8"?> <soap12:envelope xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns:xsd="http://www.w3.org/2001/xmlschema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope"> <soap12:body> <GetQuote xmlns="http://www.webservicex.net/"> <symbol>qsft</symbol> </GetQuote> </soap12:body> </soap12:envelope> Stock Quote Response <?xml version="1.0" encoding="utf-8"?> <soap12:envelope xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns:xsd="http://www.w3.org/2001/xmlschema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope"> <soap12:body> <GetQuoteResponse xmlns="http://www.webservicex.net/"> <GetQuoteResult>14.99</GetQuoteResult> </GetQuoteResponse> </soap12:body> </soap12:envelope> A SOAP-initiated action can be a single SOAP operation or a combination of SOAP operations. A credit card verification is an example of a single SOAP operation. A Google search request triggers a combination of two SOAP operations: a spell check operation and a search operation. 4
Figure 1. SOAP message example Figure 1 illustrates how SOAP messages are used. In this example, a travel web site uses external service providers to obtain airline ticket prices. In this case, the providers are the airlines with which the travel site has partnered. The customer enters flight information, such as flight origin, destination, and travel dates, and submits the ticket price request via the travel site web application. The application then requests current ticket prices from the partner airlines by sending a SOAP message to each airline. The request message includes the flight information entered by the customer. Each service provider (airline) responds with a SOAP message that contains the current ticket price. The travel site application then presents the price information to the customer. 5
The SOAP Monitoring Challenge Because of the critical role SOAP messages play in web applications, it s important that both consumers and providers closely monitor SOAP services. The monitoring solution has to meet stringent criteria to satisfy the needs of both consumers and providers. From the Service Consumer Perspective Service consumers hire out business-critical services to external service providers. Consequently, consumers are highly dependent on timely and accurate delivery of these services. Any provider delays in responding to consumer requests slow the consumer s application performance, and can lead to serious business consequences. Customers may abandon transactions, resulting in low customer satisfaction and loss of revenue. Even worse, customers may defect to competitive sites, ultimately resulting in loss of customers. But consumers have no control of provider s infrastructure. That s why, in most cases, consumers require that providers commit contractually to minimum performance service levels through service level agreements (SLAs). For example, each service must be delivered within 500 milliseconds of the request for that service. SLAs often include financial penalties for non-compliance. It is therefore important that the consumer closely monitor the provider s performance relative to the applicable SLAs. Another factor is that, on the consumer side, the individual that is responsible for monitoring service provider performance is often a management-level employee, rather than a technical person with deep infrastructure knowledge. Consequently, the SOAP monitor must not only provide comprehensive information, but also present that information in a way that is clear and meaningful to management-level staff. From the Service Provider Perspective Because service providers are embedded in consumers business processes, they typically have to commit to SLAs. In some cases, providers establish two-way SLAs by placing a caveat on the consumer s SLA. For example, a consumer s SLA specifies that a response must be received within 500 ms of a request. The provider s caveat establishes a volume cap on that SLA. For example, the provider guarantees 500 ms response only up to a volume of 500 requests per second. In this case, it s important that providers understand not only service performance levels but also service request volumes. Not meeting SLAs can have serious business consequences for providers, including financial penalties and the defection of consumers to other providers. Therefore, it s important that providers closely monitor their SOAP service performance levels. This requires the ability to audit SOAP service performance in real time to maintain compliance with SLAs, as well as the ability to demonstrate compliance to consumers. Providers also must be able to quickly detect performance problems and move proactively to resolve them before consumers are affected. There are many possible causes, such as a problem in an individual server, a network problem, a problem with a particular service, or a problem with a particular consumer. 6
An effective SOAP monitor must not only provide an early warning of problems, but also include detailed information to help the provider s technicians quickly identify the cause. Capturing the required information presents a technological challenge. The SOAP monitor has to capture detailed information on all SOAP transactions, which may be in excess of thousands of transactions per second. To get the needed details, the monitor has to parse each SOAP message in its entirety. That means digging down into the body of the message. This extensive processing has to be performed in near real time to ensure that SOAP messages are not delayed in any way. 7
Foglight Meets the Challenge Foglight provides a solution that meets the SOAP monitoring challenge for both service providers and service consumers. As Figure 2 illustrates, Foglight is a self-contained physical appliance that plugs into either the consumer or provider network. Figure 2. The Foglight monitor Foglight offers a host of industry-leading features. It has zero overhead, there are no agents to install or maintain, and you can access Foglight through a simple web browser. By passively monitoring SOAP messages in near real time, Foglight does not increase network traffic or delay SOAP messages. Foglight is also highly scalable. You can install and cluster multiple appliances to provide a broad perspective of SOAP service performance across your enterprise, and across multiple datacenters, if needed. Comprehensive Metrics Foglight captures and aggregates a rich array of metrics that gives both service providers and consumers the information they need to ensure web service performance commitments are met. It monitors at the point of offering, rather than at an individual web server, to provide a near real time view of your environment from your customers perspective. 8
With Foglight, you can group metrics in different ways to gain a variety of perspectives in analyzing the aggregated data. Foglight provides seven (7) standard groupings: SOAP web service. Tracks SOAP-based web services that are executed in the same service group. SOAP operations. The building blocks of SOAP transactions, these are performed by the SOAP provider in responding to a request for a service. SOAP transactions. Defined as a sequence of SOAP operations that the SOAP consumer must execute to complete an end-user-initiated task. SOAP transaction steps. Provides metrics for each step of a SOAP transaction. SOAP applications. Groups soap operations that are related. These can be either a single function (account login method) or an entire set of functions (all account management methods). SOAP consumers. Groups metrics by consumer. SOAP servers. Groups individual servers that are servicing specific SOAP requests. SOAP faults. Provides a real-time log of SOAP faults that are observed in the monitored SOAP traffic. Table 1. shows the relevance of the information presented by each group. 9
Group Questions Answered SOAP web service How many requests did the web service receive? What was the success rate for calls to the web service? What is the performance of the web service? SOAP operations SOAP transactions SOAP transaction steps SOAP applications SOAP consumers SOAP servers SOAP faults What volume of traffic is generated by the web service? How often is the operation called? What is the success rate of calls to the operation? What kind of performance are consumers of the operation experiencing? What volume of traffic is generated by the operation? What faults are occurring on the operation and what are the details of those faults? What is the total end-user response time for the transaction? What volume of network traffic is generated by the transaction? How often is the transaction aborted? How often is the step executed? How much back-end processing time is required for the step? What is the level of performance experienced by users of the step? What is the volume of network traffic generated by the consumer and provider during the step? How often did the step experience timeouts? How often is the SOAP application invoked? What is the success rate of calls to the SOAP application? What kind of performance do consumers of the SOAP application experience? What volume of traffic is generated by the SOAP application? What faults are occurring in the SOAP application and what are the details of those faults? What level of performance are consumers of a web service experiencing? What faults are consumers encountering and what are the details of those faults? What volume of traffic is generated by a consumer? What is the success rate a consumer is experiencing? Which operations is a consumer calling and how often? What is the number of SOAP operations that are serviced by the server? What is the success rate of operations on the server? What kind of performance is the server delivering for SOAP requests? What volume of SOAP traffic is the server receiving and transmitting? Which faults are being generated by the server and what are their details? How much server processing time is consumed handling SOAP requests? Are consumers of the SOAP service experiencing faults? What are the most common faults? Which servers are generating the most faults? Which SOAP operations are failing most often? Table 1. Foglight metrics groups You can also create custom groups to suit your unique requirements. Meaningful Presentation of Actionable Information Foglight presents the monitored information through a variety of meaningful views. Below are some examples: 10
The view in Figure 3, which is useful to providers, shows the most active SOAP consumers. Figure 3. Most active SOAP consumers The views presented in Figure 4 and Figure 5 show the slowest SOAP operations and their historic performance respectively. These views are useful to both consumers and providers. Figure 4. Slowest SOAP operations 11
Figure 5. Historic performance of slowest SOAP operations The views presented in Figure 6 and Figure 7 show the busiest SOAP operations and their historic performance respectively. These views are useful to both consumers and providers. Figure 6. Busiest SOAP operations 12
Figure 7. Historic performance of busiest SOAP operations Consumers and providers can leverage the information captured by Foglight for a variety of purposes. Consumers can ensure that providers are meeting SLAs. Providers can audit performance compliance with SLAs, as well as demonstrate compliance to consumers through consumer-specific reports. Consumers and providers can use Foglight as an early warning system to detect problems. In addition, providers can drill down into aggregated data to identify problem causes, speeding problem diagnosis and shrinking mean-time-torepair (MTTR). For example, if a provider has 40 servers supporting SOAP operations, Foglight will show which servers and generating errors and failing to meet SLAs. It s interesting to note that many Quest customers are leveraging the rich data captured by Foglight for purposes outside of SOAP performance monitoring, problem detection, and problem resolution. For example, providers can determine which consumers are consuming specific services, how much they are consuming, and what their peak loads are. Providers can use this information for a range of purposes, such as determining high-value customers or calculating chargeback. (Note that usage counts may have anywhere from 1 to 5% error due to packet loss.) Foglight does not provide specific analytic tools to support these other purposes. However, Foglight does permit you to export data into other tools for analysis. 13
Beyond SOAP Monitoring In addition to monitoring SOAP messages, Foglight can also monitor end-user experience. (See Figure 8.) This gives you a much broader perspective of overall web application performance by enabling you to monitor all activities, not just SOAP transactions. Figure 8. SOAP and end user experience monitoring 14
Conclusion SOAP messages are deeply embedded in the business processes of both web service consumers and web service providers. Any delays or errors in the communication of SOAP messages can have serious business consequences. Therefore, it s crucial that both service consumers and service providers closely monitor SOAP messages for compliance with service level commitments. Effective monitoring presents a challenge, due to the large amount of data that has to be captured and the high volume of SOAP messages. Foglight meets the challenge by providing a clear, comprehensive, and near-real-time view of the performance and reliability of the SOAP web services you provide or consume. This view provides unprecedented insight to help you maintain service uptime and avoid costly downtime. With Foglight, web service consumers can closely monitor the links with their supplier partners to ensure SLA commitments are being met, and providers can closely monitor the links with their manufacturing partners to ensure they are meeting expectations. 15
2011 Quest Software, Inc. ALL RIGHTS RESERVED. This document contains proprietary information protected by copyright. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose without the written permission of Quest Software, Inc. ( Quest ). The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document. If you have any questions regarding your potential use of this material, contact: Quest Software World Headquarters LEGAL Dept 5 Polaris Way Aliso Viejo, CA 92656 www.quest.com email: legal@quest.com Refer to our web site for regional and international office information. Trademarks Quest, Quest Software, the Quest Software logo, AccessManager, ActiveRoles, Aelita, Akonix, AppAssure, Benchmark Factory, Big Brother, BridgeAccess, BridgeAutoEscalate, BridgeSearch, BridgeTrak, BusinessInsight, ChangeAuditor, ChangeManager, Defender, DeployDirector, Desktop Authority, DirectoryAnalyzer, DirectoryTroubleshooter, DS Analyzer, DS Expert, Foglight, GPOADmin, Help Desk Authority, Imceda, IntelliProfile, InTrust, Invirtus, itoken, I/Watch, JClass, Jint, JProbe, LeccoTech, LiteSpeed, LiveReorg, LogADmin, MessageStats, Monosphere, MultSess, NBSpool, NetBase, NetControl, Npulse, NetPro, PassGo, PerformaSure, Point,Click,Done!, PowerGUI, Quest Central, Quest vtoolkit, Quest vworkspace, ReportADmin, RestoreADmin, ScriptLogic, Security Lifecycle Map, SelfServiceADmin, SharePlex, Sitraka, SmartAlarm, Spotlight, SQL Navigator, SQL Watch, SQLab, Stat, StealthCollect, Storage Horizon, Tag and Follow, Toad, T.O.A.D., Toad World, vautomator, vcontrol, vconverter, vfoglight, voptimizer, vranger, Vintela, Virtual DBA, VizionCore, Vizioncore vautomation Suite, Vizioncore vbackup, Vizioncore vessentials, Vizioncore vmigrator, Vizioncore vreplicator, WebDefender, Webthority, Xaffire, and XRT are trademarks and registered trademarks of Quest Software, Inc in the United States of America and other countries. Other trademarks and registered trademarks used in this guide are property of their respective owners. Updated [October, 2011] 16
About Quest Software, Inc. Quest Software (Nasdaq: QSFT) simplifies and reduces the cost of managing IT for more than 100,000 customers worldwide. Our innovative solutions make solving the toughest IT management problems easier, enabling customers to save time and money across physical, virtual and cloud environments. For more information about Quest solutions for application management, database management, Windows management, virtualization management and IT management, go to www.quest.com. Contacting Quest Software PHONE 800.306.9329 (United States and Canada) If you are located outside North America, you can find your local office information on our Web site. EMAIL sales@quest.com MAIL Quest Software, Inc. World Headquarters 5 Polaris Way Aliso Viejo, CA 92656 USA Contacting Quest Support Quest Support is available to customers who have a trial version of a Quest product or who have purchased a commercial version and have a valid maintenance contract. Quest Support provides around-the-clock coverage with SupportLink, our Web self-service. Visit SupportLink at https://support.quest.com. SupportLink gives users of Quest Software products the ability to: Search Quest s online Knowledgebase Download the latest releases, documentation and patches for Quest products Log support cases Manage existing support cases View the Global Support Guide for a detailed explanation of support programs, online services, contact information and policies and procedures. WPD_AnInnovApprToSoap_US_VG 17