Solution Guide: Application Performance Management

Transcription

1 Solution Guide: Application Performance Management Making sense of the APM market jigsaw SWOT analysis and Ovum APM 2012 Rainbow Map Extracts by NetScout Systems, Inc., from Ovum s report Reference Code: IT Publication Date: 25 May 2012 Author: Michael Azoff SUMMARY Catalyst Application performance management (APM) today sits at a juncture that crosses many disciplines and domains: software development, applications in production, IT services, desktops, mainframes, web, mobile, cloud services, virtualization, application testing, network infrastructure, databases, and storage. APM vendors arrive in this space from different historical routes, but they all go under the same label. This report therefore aims to clarify the definition of an APM solution, and how vendors are taking their solutions forward. Ovum takes a representative group across new and established leading vendors, and provides a SWOT (strengths, weaknesses, opportunities, threats) analysis of their solutions. Ovum also conducted a comprehensive features assessment with the results summarized in the Ovum APM Rainbow Map. Ovum view APM tools can be categorized according to their target audiences. So development-oriented APM serves developers with the need to drill down to the code level, and provides the kind of information that programmers need and understand to solve code related problems. An operations-oriented APM tool will provide information about the production environment, and a single consolidated view of information meaningful to sysadmin and operations professionals. QA professionals will want details related to Ovum. Unauthorized reproduction prohibited Page 1

2 performance criteria that can be matched to the application's non-functional requirements. Furthermore, a new generation of APM tools monitor the end-to-end production environment, tracking distinct business transactions that cross multiple services and devices, and providing business transaction management. APM analytics is improving with new tools that can correlate thousands of metrics and identify patterns that provide early warning signs of impending trouble. The appearance of solutions that go beyond real-time monitoring and offer predictive analytics is another new market trend. End-user experience monitoring is an APM category that focuses on how the user is experiencing performance. For example, green lights might be showing in production but the client side might be dead. This feature now needs to encompass applications running on smart mobile devices. Reporting to smart devices like tablets is also a new musthave. While some APM vendors aim for solutions that cover all monitoring and performance categories for the complete range of users, there are smaller players that are purely focused on operations and production. Ovum finds that the APM field is attracting new entrants with innovative solutions, so despite waves of industry rationalizations that occurred some years ago with all-encompassing solutions, the field is diverging again as more distinctly defined users are identified, such as DevOps, with their specific needs such as continuous deployment. APM in relation to cloud computing is a highly active area of solution enhancement by nearly all the vendors reviewed in this report. While some vendors can offer APM as SaaS, others are waiting to see how this market grows or are busy building such features. Taking APM into the cloud for private clouds is less of an issue than doing so on public clouds where there are limits on what can be deployed. Agents that accompany applications to public clouds is a common approach. APM for cloud service providers is another user category and an essential one as these providers look to differentiate themselves. APM solutions therefore need to keep pace not just with the changing nature of the data center, the transition to cloud services, use of server virtualization, messaging, virtual desktops, and so on, but also the categories of users who will use the performance information. Finally, we note in this report that given these changes taking place the applications can no longer be considered static but rather dynamic. Key messages The broad APM functionalities are converging in a single solution set for covering the data center and private cloud environments, but the key user groups are given targeted information. The key APM user groups are systems administrators and operators, application developers, QA, IT service management professionals, and senior managers and executives. Ovum. Unauthorized reproduction prohibited Page 2

3 APM analytics has become a major concern and active area of innovation, due to the sheer growth of data to be processed. Advanced techniques such as predictive analytics can quickly identify potential problems before they escalate and bring down systems. The cloud is impacting APM in a number of ways, from SaaS APM offerings, to solutions that can accompany applications to public cloud platforms, to tools that seamlessly track transactions across all tiers in a virtualized private cloud environment. Mobile devices are increasingly being used in the enterprise, and APM solutions must be able to monitor end-user experiences and app performance. Mobile devices are also becoming popular for displaying dashboards to APM user groups. The APM market is going through revitalization as new vendors emerge and new generation solutions appear from established players. APM MARKET DEVELOPMENTS Market view Application performance management (APM) is currently going through a revitalization following cloud and DevOps technology evolutions (possibly revolutions), as well as innovation in areas such as analytics and real-time monitoring, so it is timely to define exactly what is meant by the term APM. Some four years ago the market appeared to be consolidating but the widespread adoption of virtualization and the recognition of the importance of end user-experience monitoring (EUEM) created new opportunities. The APM market today is seeing further diversification, so the modern, broad definition of APM is an application-centric performance monitoring and management solution that cross-cuts IT tiers including network infrastructure, physical and virtual infrastructure (server and OS), storage and databases, end-user experience with edge devices, and integration into general IT service delivery, with these tiers running in and across traditional data centers, cloud environments (private, public, hybrid), and SaaS providers. The modern APM solution is designed to address different IT cultures, their languages (view of the world), and needs. While movements such as Agile and DevOps attempt to break the walls between traditional silos such as development, QA, operations, network engineering, senior management, and others, there remains the need to address the particular information requirements of these communities. APM tools either provide across the board functionality with interfaces targeting the specific communities or they will be solely focused on one or two communities. APM for pre-deployment development and post-deployment troubleshooting therefore provides developers with source-code-level detail and statistics, whereas operations-oriented APM is concerned with live production where operators are concerned with end-user experiences and end-to-end service delivery, but for troubleshooting they only need details to method call levels. From a process viewpoint, operations also want to be able to have the right information that enables them to identify who the best team members are to solve the issue. Ovum. Unauthorized reproduction prohibited Page 3

4 Furthermore, those in operations care about the resource overheads incurred by deploying APM agents in production, and even low-overhead agents can in sufficient numbers drain production resources and also cause potential perturbations. So the type of solutions these communities want in place can differ markedly. In addition, there are two areas closely related to APM that Ovum believes will see increasing activity within the APM market: application performance testing and application security performance. The former is naturally of interest to QA and testing professionals, and APM tools that target this audience will include synthetic and real-user load-testing. Vendors include CA (which recently acquired ITKO, a pioneer in service simulation technology), HP, and IBM (which recently acquired Green Hat, another pioneer in service stubbing and virtualization). The second area, cyber security, has become an application-level concern as enterprise users increasingly work on devices that sit outside the firewall such as web browsers and mobile devices. This is an opportunity for the APM market to address. Finally, specialized APM solutions that target particular enterprise applications are available where the complexity of the application and the criticality to the business demands targeted tooling. GSX Solutions is one example of a company that provides a single software solution for monitoring Microsoft Exchange, Microsoft SharePoint, BlackBerry services, and Lotus software at the same time, using remote agentless technology. Unlike general APM solutions, GSX knows its targeted application deeply and can provide a level of analysis and reporting that administrators of these business applications will find immediately helpful in their daily maintenance activities. Another example is Knoa Software, which provides user experience monitoring and management for enterprise applications such as SAP and Oracle. How end users perform their daily tasks on these systems is not normally tracked unless the application is misbehaving, but Knoa is able to analyze usability performance in order to help users make optimal use of the often-complex application interfaces. Technology view Application performance management solutions on the market vary according to which technology direction the vendor has historically started from, such as applications or network. The technology categories covered by the solutions can therefore be split as follows (see Figure 1): Development-oriented: Root-cause APM satisfying needs of development teams with programming and testing support, through to lifecycle support in production with instrumentation of virtual machines and application servers for optimal code performance. Operations: A growth area with the emergence of DevOps, the tools are designed to be used by sysadmin and operators, talk their language and solve their production environment problems. Network infrastructure: This is network performance management (covering LAN and WAN) rather than APM, but the need for a unified view (application plus network) is leading to integrated solutions with operations performance management. In addition, unified communications (including Ovum. Unauthorized reproduction prohibited Page 4

5 video and VoIP) lead to increasingly sophisticated traffic (including video and voice-over-ip elements) with specialized performance-management needs. Business transaction management: Tracking a transaction view, as for example initiated by a business user of a service that crosses multiple applications and services. End-user experience monitoring (EUEM): covering client side, from desktop clients to web browsers and mobile devices. Strictly this covers availability, but aspects of EUEM overlap into website/application analytics and optimal behavior monitoring. Physical, virtual, and dynamic infrastructure, on cloud and traditional data center, including SaaS delivery channels. Storage and database: APM for storage and databases are niche specialist areas. While most APM solutions will provide traffic monitoring for data going in and coming out, only a few will instrument storage/database internals. Executive reporting and dashboards. A solution that provides a high-level dashboard for decisionmakers who need easy-to-digest indicators such as traffic lights. Legacy systems, such as mainframes: These are still in use in industries such as financial services, and need to be tied in to ensure performance monitoring continuity. Many solutions simply measure the data going in and data coming out. Innovations in the APM space that cross-cut the above APM categories include: Advanced (Big Data) analytics: This is an increasingly sophisticated sub-topic within APM that aims to deal with the many hundreds of thousands of metrics generating data every second, and making sense of the mountain of data. Techniques tend to be either advanced statistics based, such as those based on correlations, or pattern-matching. Always-on APM suitable for production monitoring in contrast with heavier instrumentation for predeployment testing and after failure event troubleshooting. End-to-end, 360-degree view solution that integrates many of the APM categories listed above. Increasingly this category also includes coverage of web application performance across geographical divides. Dynamic APM: The rise of new generation technology such as cloud services, virtualization, and messaging, means that applications are dynamic rather than static, and places greater demands on APM solutions to make sense of application behavior. Ovum. Unauthorized reproduction prohibited Page 5

6 Figure 1: Ovum APM technology stack Source: Ovum Increasing application dynamism needs multi-tier, end-to-end APM Modern multi-tier applications and services built for SOA environments are more dynamic than ever before, and this dynamism falls into three dimensions. First, the infrastructure on which applications run can switch in an instant across different cloud servers and virtual machines. Second, the applications may comprise mashups pulling in data from diverse sources, and rely on multiple and distributed components and services. Finally, applications are delivered faster and deployed more frequently than before with the rise of Agile methodologies, continuous delivery, and DevOps. For example, continuous delivery establishes a deployment pipeline that can trigger a code change to production use in a matter of minutes. All three dimensions demand more from an APM solution. Solution configuration should be easy to keep up with frequent Agile drops, it must recognize a distinct business transaction that cross-cuts multiple IT layers, it must follow a transaction across physical and virtual boundaries, and it needs to support the needs of different users from developers needing source code line detail to operators concerned with stability and service delivery in the production environment. Ovum. Unauthorized reproduction prohibited Page 6

7 User deployment approaches for APM There are various ways in which APM solutions can be deployed, dependent on the needs and roles of the users. The four main communities are operations, development, QA, and IT services management, with perhaps end users forming a new fifth grouping. Increasingly, a unified view that addresses the needs of all these communities is required by businesses that are heavily reliant on IT. APM solutions therefore need to be end-to-end, operating 24x7, and able to pre-empt problems before they escalate. In many businesses, point solutions are often brought in to deal with particular problems. For example, developers may need to instrument a Java Virtual Machine, or operations is finding that it needs to monitor end users that are suffering issues despite the server side apparently functioning normally. The collection of these point solutions cannot solve issues efficiently that transcend any one IT layer. Especially in a large enterprise environment the lack of a unified view can lead to significant time-wasting in resolving issues, but for any business reliant on mission-critical applications a comprehensive APM solution is essential. Another user category that will find APM essential is the cloud or SaaS provider, whose business model is based on a maximum up time with multiple nines reliability. APM market trends APM analytics has reached big data proportions In a large data center an APM solution can generate thousands of metric data points per second. This data avalanche will contain the health information about the IT environment, and the challenge is to manage the sheer scale of this data. One approach is to store it in a data warehouse and process it with sophisticated analytics tools that are designed to deal rapidly with the large volumes. An alternative approach is to treat the problem as Big Data and sample the data in real time. Both approaches have led to innovation in the APM market place. Advanced analytics tools are available to enhance legacy APM solutions A number of technology initiatives have led to a growth in the data generated by APM solutions. These tools are providing end-to-end application and service monitoring, covering every tier from network to business transactions. Also service-oriented architecture, which is not quite as dead as some might believe, results in an increase in data traffic to be monitored. Furthermore, virtualization and cloud computing are creating a dynamic environment with an increase in the complexity of metrics to be monitored. In order to keep pace with the sophistication of the IT environment the APM solutions started with static thresholdbased monitoring but then progressed to event correlation, dynamic thresholding, and pattern matching. The next step has been to deal with the data overload problem.. There are new tools which are statisticsbased and employ self-learning algorithms that process metric data in real time to make sense of the data Ovum. Unauthorized reproduction prohibited Page 7

8 avalanche and identify issues that need attention. They are based on multivariate correlation and regression and can also provide forecasts of issues that are likely to escalate and bring systems down. Some companies have also created new-generation predictive analytics engines to complement their existing tools, enabling them to scale up their metrics-processing capabilities. The scale of this issue is fast becoming a Big Data problem, and large data center administrators will need to assess their existing APM solutions when handling these volumes of data. APM on and for the cloud Ovum invited the key report vendors to comment on the future of APM, and in particular what effect cloud (IaaS, PaaS, and SaaS) services will have on APM. This section includes perspectives from the industry. There is a general consensus among the APM vendors that XaaS providers will in time be expected by their customers to provide detailed application-performance metrics as part of their services. As one vendor points out, this should ideally be driven by common industry standards, but it could end up being driven by one or two large vendors such as Microsoft or VMware. Ovum sees Amazon AWS as creating a de facto industry standard in cloud API. This would be an opportunity for Amazon to drive a cloud APM API, but there is no sign of this happening. On the question of whether IaaS and PaaS providers should offer an APM solution embedded in their service, customers do not want to deal with multiple APM solutions, but instead prefer one solution that spans cloud, virtual, traditional, and hybrid environments. However, APM as part of a PaaS offering will eventually become a requirement, although there are security considerations to consider, and customers today prefer a single APM solution as mentioned, to span everything. Customers expect APM solutions to be an embedded part of XaaS services, so while the burden is on the service providers to provide proactive APM reporting back to consumers, many customers will self-procure APM solutions because it is ultimately their brand and business opportunity, and SLA penalties from the service provider will not mitigate the impact to the business. The migration of application workloads to cloud platforms via IaaS, PaaS, or SaaS models means that the APM solution must be able to discover application workloads on private and public clouds, deploy the monitoring solutions to those infrastructures, and be able to send data and alerts back to the monitoring solution. This requires flexible data gathering techniques, such as non-agent application data collection, and the cloud dynamic also requires new ways of visualizing APM data. Therefore a new breed of cloud administrators will require a high-level view of application health and cloud resources consumed, and software application stakeholders need to be concerned with the way application workloads are running in the cloud as much as with user experience. Ovum. Unauthorized reproduction prohibited Page 8

9 APM is an integral part of cloud computing, and the cloud model demands a new mindset about how applications are developed, deployed, monitored, and managed. For example, on the cloud you may be trying to monitor resources you may not own, so it is essential to have rigorous monitoring practices to identify the exact root cause of problems that occur. It can also be very difficult for developers to diagnose and fix cloud-based application issues without performance data and enduser information from IT operations monitoring teams. Enterprise adoption of the cloud as a deployment platform will continue to accelerate. Cloud applications are built from assets provided by publicly shared third-party service providers, private clouds, and data center assets, and increasingly powerful end-user clients. In order to monitor the health of these applications, broad visibility across the application delivery chain is critical, spanning publicly shared third-party services, enterprise private clouds, enterprise data centers, and mobile, tablet, and browser-based clients. APM for the data center requires a convergence and integration of the various segments that have traditionally been separate disciplines, namely APM, infrastructure management, and services operations, to yield end-to-end APM with comprehensive integration across multiple tiers. However, the demands of cloud-connected enterprises are pulling in the opposite direction, with service providers owning the infrastructure and back-office applications, and enterprise IT owning the differentiating bespoke application logic and the management of the underlying service providers. APM for mobile devices Smart mobile devices are having an impact on enterprise staff because the consumer-led pattern of adoption is crossing over to the workplace, led by the top C-level executives and by staff on the ground and field. This is affecting the APM market in two ways. First, there is a need for APM solutions to monitor app performance on smart mobile phone and tablet devices, whether through synthetic or real user experience monitoring. It is important here to consider the challenges of monitoring a given app when accessed on different devices at different times. Second, the APM reporting and dashboards need to be geared to deliver performance metrics and results to mobile devices, for both executives and operations administrators. In this review, all the key vendors offer mobile device APM except BlueStripe and Nastel, and all vendors offer a mobile device dashboard, except BlueStripe, Opsview, and Quest, and these vendors are likely to enhance their offerings in these respects in the near future. The extent of coverage of mobile device APM features may be further ahead in some than others. For example, synthetic monitoring may be available, but real-user experience monitoring may not. Ovum. Unauthorized reproduction prohibited Page 9

10 SOLUTION GUIDE FOR APM Competitive landscape Ovum takes a close look at the competitive landscape of relevant technology solutions. The assessment is a quantitative and qualitative representation of Ovum's views and opinions about the competitive APM market environment. The list of included vendors is not intended to be exhaustive, but representative, offering readers an in-depth analysis of the leading vendors in the context of the specific technology area. The final mix of selected vendors provides a good balance between leading established players and newer solutions from smaller vendors. The key vendors included in this solution guide are listed in Figure 2. Of the selected enterprise IT management vendors (BMC, CA, HP, and IBM) only BMC is missing, having pulled out mid-stream. Ovum believes this is because BMC is in the process of launching a new solution and is not yet ready to go public. Ovum asked the reviewed vendors to which segments of the APM coverage areas their solutions were applicable, where the segments were itemized as follows: Business processes (multi-applications/services) Application-level transactions Network infrastructure Performance monitoring and management Performance testing Looking at APM from a TCP/IP layer perspective, the Open Systems Interconnection (OSI) model defines the abstraction layers of communication systems, where a layer serves the layer above it and is supported by the layer below it. The OSI model is briefly defined as: Application, presentation, session layers: L5-L7 Transport layer (flow control): L4 Network layer (data packets): L3 Data link and physical layer (switches and binary transmission): L1-L2 The APM solutions reviewed all covered L3 to L7, with the exception of Nastel, which used a third party for L4. However, with L1 and L2 this is where the main divide between network performance management and APM occurs, so only some vendors offered solutions here. These were CA, ExtraHop, HP, IBM, NetScout, Opsview, and Quest. All the vendors reviewed here offer software APM solutions except ExtraHop, which only provides an appliance (combining software and hardware). NEC and Netuitive provide analytics solutions that are designed to complement APM and other IT management solutions, and are not included in the Ovum APM Rainbow Map, which is designed for full APM solutions. Ovum. Unauthorized reproduction prohibited Page 10

11 Figure 3: Ovum Rainbow Map Of NetScout Source: Ovum Ovum. Unauthorized reproduction prohibited Page 11

12 GENERAL ASSESSMENT Capability assessment The vendor solutions were assessed against a set of broad APM features. Vendors were invited to provide feedback on the structure and content of the features matrix (FM) and were then asked to complete the final version. The weightings that Ovum used in assessing the different sections are revealed in the appendix, and it should be kept in mind that customers will vary with their particular requirements and should not simply select vendors on the basis of the scores achieved in this solution guide. For example, customers may have legacy APM tools in place and may be looking to fill gaps in their coverage. The final scores of the FM were aggregated by section and plotted on a heat-style map, except we refer to it as the Ovum Rainbow Map given that the rainbow sequence of colors are used to rate the scores. The final Rainbow plots are shown in Figure 3. In the following sections the broad APM features assessed are described qualitatively. Application performance optimization This is the core of APM, and covers features such as application performance monitoring, and application availability. Ovum weighted two features in particular highly here: the capability to capture all transactions 24x7 end-to-end from browser through web and application tiers to database and back, and to provide deep visibility into transactions down to code level. Application platform coverage This section assesses the platforms covered, in particular whether the solution is applicable to applications in virtualized and cloud environments, and mobile applications. Topology and change impact analysis The auto-discovery of transactions flow topology is considered important, as are dependency mapping and automated features for transaction visualization and bottleneck detection. End-user experience monitoring End-user experience monitoring from the user's perspective, such as from inside the user's browser, is a hot topic in APM. This set of features can also overlap into website usability, such as tracking visitor satisfaction and web page landing statistics. The deep tracking of users optimal understanding of Ovum. Unauthorized reproduction prohibited Page 12

13 enterprise application GUI interfaces is also covered here. Most highly rated features were: monitoring across all devices, all browsers, capture click paths and Web 2.0 page actions; and trace logic in Web 2.0/Ajax-based end-user clicks into the server side. Application performance testing Performance testing is an important activity in QA, and for organizations with IT development on site this will be a must-have feature. Solutions will need to run performance testing for synthetic users and real users, as well as website testing from diverse locations. Network performance management Network performance management overlaps with broadly defined APM, and here Ovum assesses infrastructure performance optimization and network modeling capabilities. In the former case key features are network performance monitoring, network availability, network traffic management including load management, and being able to identify and pinpoint network bottlenecks between application components. Network modeling includes capabilities such as network scenario and capacity modeling and simulation, and end-user response time modeling. Business transaction management and SLAs BTM is the capability to trace business transactions, looking at services holistically, with all components that make up a service, and assess business impact of issues. It has become a hot area in APM, providing a business-level understanding of the impact of problems arising deeper down in the application and infrastructure layers. The need to manage service-level agreements is also assessed here. Unified communications (UC) performance management Unified communications applications such as for voice, video, video conferencing, desktop video conferencing, and telepresence, are an increasingly important use-case scenario for organizations, and APM solutions need to be able to monitor these systems. Diagnosis and root-cause analysis This is the core analytics functionality, offering event correlation, source code level identification troubleshooting, data packet-level analysis, and with more advanced solutions, automated triage and repair. Ovum. Unauthorized reproduction prohibited Page 13

14 Predictive IT performance analytics Predictive analytics is at the cutting edge of advanced APM, with self-learning modeling and behavior learning engines, automated problem pattern detection, CEP engines, and other techniques in anomaly detection and prediction. Database and middleware support Many APM solutions will cover databases by monitoring delays in signals going in and coming out of the database, but some solutions perform deep dive database optimization. Good coverage of diverse middleware is also a necessity for large data centers. Cyber security Many applications are now accessed via a browser so the traditional firewall is no longer the frontline in protecting the data center from malware. Web applications are allowed to tunnel past the firewall and the applications therefore need to be secure from within. Cyber security is about detecting weak spots in applications and the infrastructure, proactively identifying issues before security breaches occur. Admin, reporting, and alerts APM solutions have in the past been difficult to configure and administer. The aim is near-as-possible zeroconfiguration out-of-the-box deployment, and auto-discovery of applications, devices, transactions, and users. Dashboard metrics and health reporting, both real-time and historical, and reporting to smart mobile devices are assessed here. Alerts cover KPI-based alarming and policy-based alarming. VENDOR ANALYSIS - NETSCOUT NetScout Systems Profile The NetScout ngenius Service Assurance Solution provides a holistic view into the user experience of applications and services by integrating the monitoring of application elements, network traffic, and how their inter-relationships impact one another. NetScout's unified service delivery management (USDM) approach spans APM, network performance management, unified communications management, and cybersecurity management. USDM therefore provides APM in the broadest sense discussed in this report, covering end-to-end performance; services, applications and protocols; physical and virtual infrastructure; voice and video traffic; and detecting cyber threats and forensically investigating cyber attacks. Ovum. Unauthorized reproduction prohibited Page 14

15 NetScout ngenius Service Assurance solution provides a wide range of key service delivery management capabilities that spans service visualization, early warning performance management and analysis, service and policy validation, service and network optimization, forensic analysis, and reporting and trending. The foundation capabilities of the ngenius solution are: visibility using passive, non-intrusive data capture and real-time analysis of IP traffic; correlation and analysis based on an adaptive data model-driven correlation of service flows; and visualization and reporting revealing interrelationships and dependencies and providing actionable intelligence. At the root of these capabilities is NetScout s Adaptive Session Intelligence (ASI) technology, a real-time deep-packet analysis and data mining engine that dynamically tracks, captures, and analyzes complex service delivery and application transactions across multi-domain IP networks for comprehensive real-time analysis of the performance of applications and services across physical and virtual environments. NetScout solutions can instrument a private cloud to track and extract performance metrics for applications executing in the cloud. In order to monitor applications in the public cloud there are two options. If the cloud provider has instrumented its services and makes the information available then this data can be leveraged. Otherwise, ngenius Virtual Agent can be taken into the cloud with the application and deployed on the same virtual server. All the performance meta-data, analytics, and KPIs gathered will then be delivered back to the customer's ngenius infrastructure deployed on-premise. For end-to-end response times NetScout is finding that customers, originally financial traders but increasingly enterprise customers, are seeking granular hop-by-hop latency and performance metrics. This can be achieved by deploying ngenius InfiniStream Appliance for deep-packet capture in each venue or transformation point and ngenius Enterprise Intelligence can provide the hop-by-hop user session transaction analysis with per-hop performance metrics that include granular latency measurements. NetScout's solution is an integrated system providing contextual linkage between each analysis module. These modules share the same common data and metrics and enable a customer to buy what they need with the flexibility to add incremental capabilities in future. The advantage of NetScout's packet-flow approach is that all the data is captured, and a given analytics module then uses the appropriate data it needs. SWOT analysis Strengths NetScout ngenius takes a totally passive, packet-flow approach with the benefit of no agents to deploy or manage (or perturb) the environment, providing a perspective of service delivery that traditional APM tools cannot provide. Ovum scored NetScout well in the application and infrastructure performance management Ovum. Unauthorized reproduction prohibited Page 15

16 segments, unified communications, all aspects of analytics, middleware support, and cyber security. The unified aspect of the solution coupled with advanced analytical and forensic capabilities is a key strength. Weaknesses There are gaps in the solution in end-user deep-dive application-level monitoring, and automated network modeling. There is no application performance testing, and no application dependency mapping. However, NetScout does provide a representation of servers and networks, but it is not automatically created. Elements are discovered, but the system user needs to construct the path, though once constructed, it provides a view of the delivery environment. Opportunities NetScout has clear strengths on the network side of APM, with core functionality for L2-L7, though the most recent tools further extend NetScout s L7 capabilities. Partnering to fill the missing gaps such as end-user monitoring should be an important strategy for NetScout. The vendor is also on a roadmap of enhancing its solution. Threats NetScout has a solid base in the network aspects of APM but faces stiff competition from vendors in other areas of APM. Its strategy is to enhance USDM, organically or through acquisition. Recommendation for enterprise IT customers Ovum's recommendation for prospective customers is to consider NetScout solutions as part of a broad APM deployment (with the benefits of passive and agentless monitoring), especially where you need to focus on the management of a holistic view into service delivery and gain deep-dive visibility into intraserver application transactions. APPENDIX APM Features Matrix and Ovum's Weightings The following sections describe the features matrix that vendor solutions were scored against, with the weightings that Ovum used. The section results were aggregated and rainbow color scheme mapped to produce Figure 3, the Ovum APM 2012 Rainbow Map. Ovum. Unauthorized reproduction prohibited Page 16

17 Application performance optimization Application performance monitoring - 2 Application availability - 2 Application server-side performance and management: application server management - 2 Track individual transaction data packet - 2 Capture all transactions 24x7, end-to-end from browser through web and application tiers to database and back - 4 Provide deep visibility into transactions down to code level - 4 Application transaction management: multi-tier single application view- 2 Interrelationships between application, network, and enablers - 1 Application traffic management: data packet level application routing optimization - 1 Measure real end-user traffic through content delivery networks - 1 Record all environment settings, transactions and application data prior and during a fault for replay by developers - 1 Detection of jitter - 1 Application platform coverage Covers Java platform - 1 Covers.NET platform - 1 Covers applications in virtualized environment - 2 Covers applications in cloud environment - 2 Covers mobile applications - 2 Topology and change impact analysis Auto-discover transaction flow topology for all transactions - 1 Automatically map and visualize the flow of transactions in distributed environments - 1 Automatically detect bottlenecks (specific components introducing latency or causing errors) - 1 Automatically detect flow topology changes (for example, different routes, components added or removed) - 1 Compare environment at two time periods (for example, compare new and current application deployment) - 1 Application dependency mapping - 1 Ovum. Unauthorized reproduction prohibited Page 17

18 End user experience monitoring Monitor end user experience, from the user's perspective inside the browser - 3 Monitoring across all devices, all browsers, capture click paths and Web 2.0 page actions - 3 Track visitor satisfaction (for example, satisfied/tolerating/frustrated) - 1 Compare end-user experience and performance against industry leaders and competitors using dynamic baselines - 1 Automatically detect and monitor the performance of landing pages - 1 Trace logic in Web 2.0/Ajax-based end-user clicks into the server side - 2 Application performance testing Simulated user and transaction (synthetic) load and performance testing for performance analysis - 1 Website testing from diverse locations - 1 Web services testing - 1 Stress testing - 1 Real user performance monitoring for web-based applications - 1 Network performance management: Infrastructure performance optimization Network performance monitoring - 2 Network availability - 2 Network traffic management including load management - 2 Identify and pinpoint network bottlenecks between application components - 2 Capacity management/planning - 1 Server management - 1 Router and switch performance management - 1 Storage performance management - 1 WAN performance management - 1 Cloud performance management - 1 Virtualization network performance management (within hypervisor) - 1 Remote datacenter monitoring - 1 Ovum. Unauthorized reproduction prohibited Page 18

19 Network modeling Network scenario and capacity modeling and simulation - 1 Modeling includes cloud environment - 1 Modeling includes mobile network - 1 End-user response time modeling - 1 Business transaction management and SLAs Trace business transactions (look at services holistically, all components that make up a service) - 1 Business impact management - 1 Service modeling - 1 Alert on business anomalies (for example, incomplete or malformed business processes) - 1 Service delivery management - 1 SLA definition, automatic performance baseline - 1 SLA definition, manually define service-level objectives for key transactions - 1 SLA breach alerting with customizable tolerance rules - 1 Service-level performance visibility - 1 Unified communications (UC) performance management Unified communications applications (voice, video, video conferencing, desktop video conferencing, telepresence) - 1 VoIP performance monitoring - 1 VoIP call playback for historical analysis - 1 Video and video conferencing (at application and network level) - 1 Telepresence (support multi-screen and multi-audio) - 1 Measure all UC services side-by-side - 1 Measure UC services along with the network, other applications and services - 1 User experience management for UC services - 1 Diagnosis and root cause analysis Event correlation: pattern matching and understanding of multiple events for root-cause analysis - 1 Event correlation: identifying event(s) at root of sequence of events - 1 Source code level identification troubleshooting - 1 Ovum. Unauthorized reproduction prohibited Page 19

20 Knowledge base and collaboration: annotating logs and sharing knowledge - 1 Application-level fault management - 1 Network level fault management - 1 Data packet-level analysis - 1 Auto-detect application errors in transactions - 1 Automated triage and repair - 1 Automated memory diagnostics - 1 Application Web 2.0 type UI and distributed objects/components deep-dive analysis - 1 Web application transaction analysis (JavaEE,.NET ) - 1 Historical deep-level analysis - 1 Predictive IT performance analytics Self-learning modeling or behavior learning engines - 2 Predictive analytics and performance benchmarking - 2 Cross-domain (infrastructure, networks, servers, applications, IT business services) analysis - 2 Automated problem pattern detection (for example, too many SQL executions, high CPU executions, hidden exceptions) - 2 KPIs and aggregated metrics for multiple dimensions (for example, OLAP): transactions, users, locations, applications, business processes - 1 Complex event processing (CEP) for advanced analytics. Correlates infrastructure events and business events for smarter rules alerts - 1 Log parsing - 1 Analyze load balancing and show flow anomalies (for example, load is not balanced evenly between cluster nodes) - 1 Highlight excessive chattiness between components - 1 Database and middleware support Database monitoring - 4 Database performance optimization - 4 HTTP/S - 1 SOAP - 1 CICS - 1 Tibco - 1 Websphere MQ - 1 Ovum. Unauthorized reproduction prohibited Page 20

21 Oracle Service Bus - 1 Websphere Message Broker - 1 RMI and RMI-IIoP - 1 T3/S - 1 LDAP - 1 Cyber security Detect potential of cyber security threats - 2 Policy-based security alerting - 1 Security breach forensic analysis - 1 Network-based anomaly detection - 1 Security information and event management (SIEM) integration/data sharing - 1 Deep packet capture and analysis - 1 User session replay capabilities - 1 Admin, reporting, and alerts Methodology Zero-configuration for ease of deployment and use - 2 Auto-discover applications, transactions, and users and auto-adapt to application changes - 2 Real-time reporting -1 Historical trending -1 Integrated reporting and dashboard -1 Mobile device dashboard - 1 Compare transaction performance in two time periods or two different environments - 1 Real-time service dashboard - 1 Real-time reporting - 1 Historical and trending reporting - 1 User-defined reports - 1 KPI-based alarming - 1 Policy-based alarming - 1 Ovum analysts use data collected through primary and secondary research to determine the leading vendors in the APM Solution Guide across all categories. Ovum believes the vendors selected represent Ovum. Unauthorized reproduction prohibited Page 21

22 the main options open to enterprises. The capability analysis required all participating vendors to review an assessment of features available in the solution. Product briefings with vendors were also conducted. Author Michael Azoff, Principal Analyst Disclaimer All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of the publisher, Ovum (an Informa business). The facts of this report are believed to be correct at the time of publication but cannot be guaranteed. Please note that the findings, conclusions, and recommendations that Ovum delivers will be based on information gathered in good faith from both primary and secondary sources, whose accuracy we are not always in a position to guarantee. As such Ovum can accept no liability whatever for actions taken based on any information that may subsequently prove to be incorrect. Ovum. Unauthorized reproduction prohibited Page 22