EMA Radar for Application Performance Management (APM) for Cloud Services: Q1 2012

Transcription

1 EMA Radar for Application Performance Management (APM) for Cloud Services: Q Report Summary and Aternity Profile By Julie Craig, Research Director Enterprise Management Associates (EMA) January 2012

2 Table of Contents Executive Summary...1 Introduction and Methodology...2 Vendors Covered by this Report...4 APM: Evolving with the Cloud...4 The Problem Set: Cloud-specific APM Challenges...5 Management versus Monitoring...6 Criteria...8 Minimum Criteria...8 Ideal Criteria...8 Scoring Cost Efficiency Cost Advantage Product Strength Architecture and Integration Functionality Vendor Strength EMA Radar Maps for Application Performance Management Point Solutions Vendors and Solutions Profiled General Findings Aternity: Value Leader Aternity Profile EMA Award: Most Comprehensive Endpoint Diagnostics Overview Architecture and Integration Functionality Deployment and Administration Cost Advantage Vendor Strength Strengths and Limitations Customer Quotes I

3 Executive Summary Over the past ten to fifteen years, the complexities involved in Application Performance Management (APM) have multiplied exponentially. As applications have moved off monolithic hosting platforms and on to increasingly tiered and integrated systems, monitoring and managing end-to-end execution have become increasingly problematic. Widespread adoption of Cloud computing introduces additional complexity and signals a paradigm shift in enterprise computing that is impacting vendors and customers alike. Cloud is changing the structure of IT organizations and making inroads on traditional on-premise application hosting models. This, in turn, shapes design decisions and product investments by APM vendors. Cloud is changing the structure of IT organizations and making inroads on traditional on-premise application hosting models. This, in turn, shapes design decisions and product investments by APM vendors. Consider the following facts: 1 The majority of medium-sized to large companies have already embraced private Cloud as a viable delivery option for business-critical workloads. Infrastructure as a Service (IaaS), a natural extension of private Cloud, is gaining momentum, with 66% of companies either already using IaaS or planning to do so during the next 12 months. Software as a Service (SaaS) has become a viable alternative as well. While 44% of companies are already utilizing at least one SaaS service, another 33% plan to do so over the next 12 months. Companies are deploying Cloud services in some very sophisticated ways. Almost 50% are running tiered transactions/services that span both Cloud and on-premise. Thirty-five percent have integrated or are in the process of integrating multiple SaaS applications. Cloud readiness is becoming a factor in developing Requests for Proposals (RFPs). Today, over 40% of companies surveyed are re-assessing the Cloud-readiness of their management tools investments for SaaS (35%), Platform as a Service (PaaS- 29%), and IaaS (29%) Clearly, Cloud adoption is challenging traditional application management models and the APM solutions that support them. Widespread adoption of private Cloud technology drives new requirements for visibility to virtual environments. Public Cloud presents unique challenges because, while IT still has responsibility for service delivery, it has virtually no control over the hosting platform. And while IaaS offers almost unlimited capacity on demand, most IT organizations lack management solutions capable of end-to-end management and automated remediation across on-premise and off-premise Cloud environments. Finally, while the purpose of enterprise management solutions is to deliver visibility and control, the bottom line impact of Cloud is to obscure visibility and limit the customer s control. This EMA Radar explores these challenges in more depth. It focuses on the issues IT organizations are encountering in terms of managing Cloud environments public, private, and hybrid and provides in-depth analysis of solutions that can help solve them. 1 EMA Research on APM for Cloud Services 1

4 EMA sees this as an early-stage market with vendors adding Cloud-focused products and features at a furious pace. In developing this Radar Report, EMA engaged eighteen top providers of APM solutions in a detailed analysis of the scope and capabilities of their offerings. These solutions represent a rich cross-section of the IT management tools landscape, ranging from small to very large, from pure software to appliance- and SaaS- based, and from point products to extensive multi-component/ multi-function suites. EMA s intent in producing this report is to give Because nearly thirty vendors were offered the opportunity to participate in this rigorous study, EMA sees the eighteen vendors who did participate as the most Cloudready in the industry. businesses seeking Cloud-ready APM solutions a starting point from which to develop a short list of potential solutions, based on their own specific requirements. Because nearly thirty vendors were offered the opportunity to participate in this rigorous study, EMA sees the eighteen vendors who did participate as the most Cloud-ready in the industry. Although this is still an early market, these vendors all have distinctive capabilities for managing private, public, or hybrid Cloud environments which can deliver significant value to companies of virtually any size or in any industry. Introduction and Methodology Each vendor completed a comprehensive questionnaire consisting of nearly 100 questions and over 500 data points. The survey questions covered the five key functions common to all EMA Radar Reports, which include Architecture, Functionality, Deployment and Administration, Vendor Strength, and Cost Advantage. EMA also conducted lengthy interviews and demos with each vendor to clarify product capabilities and vendor direction. Finally, EMA conducted more than 40 interviews with customers using the products. While all vendors provided some level of access to their customers (from one to three customers), availability of reference customers was viewed favorably in assessing and validating vendor portfolios. The degree to which customers were readily provided and available for dialog was one of the many indicators used for validating APM solutions. Finally, EMA leveraged ongoing industry dialogs and extensive existing knowledge of the APM space to evaluate, consider, and validate each vendor s strengths and limitations. These evaluations focus on providing balanced, consistent insights across all vendors and solutions. EMA has produced a companion piece which is targeted at presenting and explaining Radar Reports in general, entitled How to Use the EMA Radar Report. 2 The goal is to use a combined approach to quantitatively and qualitatively evaluate providers of solutions in a particular IT management functional area and presenting their relative differences in a clear, graphical format. 2 Available at: EMA, April

5 Figure 1: The EMA Radar is optimized to show how vendor solutions cluster in terms of two primary axes: Vendor Strength (architecture, integration, functionality) and Cost Efficiency (ease of administration, deployment, support & services, costs advantage) Quoting from How to Use the EMA Radar Report, No analysis of this type can tell you which vendor is best for you. The data collected for an EMA Radar Report can certainly be used to make that determination, but it must be applied to the specifics of your current environment, level of maturity, and goals and priorities. Since the authors of any given Radar Report do not have your unique specifics, the Radar Report can only be a starting place and a guideline. It can inform you of the market and short-cut your process to developing a short list. Functionality Functionality Architecture & Integration Deployment & Administration Architecture & Integration Deployment & Administration Vendor Strength Cost Advantage Vendor Strength Cost Advantage Strong Product Higher Price Specialized Product Low Cost Figure 2: Radars for each vendor solution are included in the full report and show a fiveaxis contrast between the average profile and the vendor in question. 3

6 Vendors Covered by this Report APM solution providers covered in the report include AppDynamics, AppFirst, Aternity, CA, Compuware, Correlsense, eg Innovations, HP, IBM, INETCO, Nastel, Netuitive, New Relic, OPNET, OpTier, Quest, SolarWinds and Splunk. These solutions are clearly different in multiple respects, and represent a variety of capabilities, technology vantage points, and form factors. While this differentiation makes a strict apples to apples comparison virtually impossible, it is possible to compare and highlight features, costs, and overall value proposition relevant to the APM discipline. In the interest of fairness and for reader clarity, final scoring has been divided into two broad product groups based on general architecture. Both groups answered the same questions and were scored similarly. However, the two groups are charted and averaged separately for this report, depending on whether the solution is built around a single component or around a suite of pre-integrated components. The net impact is to highlight the strengths of both groups while comparing breadth of functionality and value proposition against a more comparable group of solutions. Solution groupings are as follows: Point Solutions : AppDynamics, AppFirst, Aternity, INETCO, Netuitive, New Relic, and Splunk. Multi Component/Suite solutions: CA, Compuware, Correlsense, eg Innovations, HP, IBM, Nastel, OPNET, OpTier, Quest, and SolarWinds. Point Solutions: In general, the point solutions profiled here address a targeted functional APM capability or set of capabilities. Multi-Component/Suite Solutions: These solutions vary considerably in terms of capabilities, with CA, Compuware, HP, IBM, Nastel, OPNET, OpTier, and Quest each delivering distinctive takes on integrated APM toolsets. In contrast, Correlsense and eg Innovations both deliver depth of visibility addressing specific APM problems via a multi-component form factor. Finally, SolarWinds delivers a cost-effective platform for mid-market APM. EMA awards a limited number of citations to vendors with particularly noteworthy capabilities in one or more functional areas. While the vendors participating in this particular study were all strong and the choice was difficult, only six solutions received awards. Four were chosen from the Multi- Component group, and two from the Point Solutions group, commensurate with the number of participants in each. APM: Evolving with the Cloud From multiple perspectives, APM solutions for Cloud services are still evolving. The ultimate answers to many of the questions related to managing Cloud applications are still being addressed. For example, interviews conducted as part of this study indicate that many public Cloud providers do not yet offer monitoring agents or APIs, a factor that hampers the capabilities management vendors can build into APM solutions. Similarly, virtualization vendors have exposed varying levels of insight into private Cloud execution with some stronger than others. VMware, for example, is a poster child for providing exceptional manageability capabilities with vcenter and related APIs. 4

7 While neither public Cloud nor APM vendors have all the pieces in place at present, virtually every vendor is aware of the necessity for developing such capabilities. In fact, many of the vendors included in this study announced new products and upgrades to existing solutions during the short time span required to produce this report. There have also been significant developments with non-participating vendors during this time frame. While BMC declined to participate because product development was in-process, it has since announced enhancements to its APM solutions. In short, although this market is still evolving, it is extremely dynamic and constantly improving. Meanwhile, the management capabilities which are available are already quite impressive. Although this market is still evolving, it is extremely dynamic and constantly improving. Meanwhile, the management capabilities which are available are already quite impressive. The Problem Set: Cloud-specific APM Challenges Today s applications and transactions can be delivered in a wide variety of hosting environments They can be hosted on-premise via traditional tiered hosting or virtualized private Cloud. They can be componentized such that elements of the execution software are hosted on-premise and in the public Cloud. They can access partner and provider Clouds, be delivered via SaaS, and can even encompass SaaS-to SaaS integrations. While this diversity offers a wide variety of deployment options, it also introduces a bewildering array of budgeting, management, and tooling considerations. Some of the most challenging include: New requirements driving APM purchasing: Many companies are finding that Cloud-related APM requirements are beyond the capabilities of incumbent solutions, and are re-examining budgets (and tools strategy) accordingly. In other words, existing APM investments aren t Cloud ready, and cannot support this diversity of hosting environments. From this perspective, Cloud computing is driving new requirements which are, in turn, impacting budget line items and purchasing decisions. End-to-End visibility becomes a must have : Companies that are less operationally mature are still transitioning from monitoring infrastructure to monitoring/managing applications. They lack top down monitoring solutions that deliver insight into end-to-end performance. This can put them at a disadvantage in terms of competitive differentiation. Exponential increase in metrics: Companies leveraging Cloud tell EMA that it introduces new requirements for metrics and big data processing. Outlier measurements such as TCP thresholds, rates, and queue limits have become important elements of the APM equation, and are pushing the processing and analytical limits of existing solutions. As an example, one customer interviewed as part of this research reports that monitoring systems are processing over one billion metrics daily. Real time management becomes the new normal: As application systems become increasingly dynamic, real time analytics capabilities become a must have. This requirement is becoming increasingly pressing as more companies begin to leverage dynamic capabilities such as VMware vmotions and Cloud bursting to IaaS. It permeates every aspect of APM from discovery/ dependency mapping to topology visualization and End User Experience Management. 5

8 Troubleshooting is more complex: This is probably the biggest challenge, as both public and private Cloud deployments increase the number and complexity of moving parts comprising the application ecosystem. To compound the problem, most public Cloud providers are still in the early stages of developing management-level instrumentation that enables them to share key performance information with APM solutions. At the same time, IT s responsibility is still to the user, and it is up to IT to diagnose performance problems regardless of where (and how) the application is hosted. For example, while poor SaaS performance can definitely be on the public Cloud provider side, it can also be impacted by a multiplicity of additional factors such as: Lack of bandwidth on the Wide Area or Corporate network Contention on the user s desktop Slow system calls to internal or external integrations Failure of optimization or load-balancing components Contention with other applications or users Database issues Location-based issues Etc. Without adequate tooling, knowing who to call is a best guess exercise. Without adequate tooling, knowing who to call is a best guess exercise. Management versus Monitoring While the focus of this research is on Application Performance Management, the term means different things to different constituencies. This section clarifies EMA s definitions, which in turn clarifies how vendors participating in this study were assessed. These definitions should also help enable IT organizations to hone in on the products that best meet specific internal monitoring and management requirements. Application Management: The role of Application Management is to deliver visibility and control for transactions, applications, and services. While Application Management covers a very broad range of capabilities required to monitor AND manage applications (see Figure 3), Application Performance Monitoring and Application Performance Management are subsets with varying levels of visibility: Application Performance Monitoring: Assesses performance and availability of a transaction, application or service Application Performance Management (APM): Encompasses the functions of performance monitoring but also supports troubleshooting, problem determination, and root cause analysis. Ideally, and this is an emerging capability, APM also includes the ability to recognize (and potentially resolve) performance and availability problems BEFORE users are impacted. In addition, APM supports and links to other IT functions such as Service Level Management and Configuration Management. 6

9 Such distinctions are particularly important when it comes to Cloud. While multiple solutions may claim to support applications running in public IaaS, SaaS, or PaaS, their levels of granularity can vary radically. For example, while running synthetic transactions every five minutes monitors performance and availability (intermittently), it does not provide the granularity necessary to troubleshoot and diagnose performance problems, particularly in complex multi-tier environments. Vendors have approached these challenges from multiple perspectives. So called top down solutions such as synthetic transactions deliver a point-in-time perspective which is broad versus deep. Bottom up solutions provide deep visibility but lack the cross-technology, end-to-end perspective that is necessary to manage applications versus infrastructure. Clearly, a meld of the two is the holy grail of APM, and APM software vendors have made significant strides in delivering on this vision during the last one to two years. Figure 3: EMA End-to-End Application Management Semantic Model Ultimately, it is essential for Cloud and enterprise management vendors alike to become Cloud-ready. In support of this idea, EMA has added a Cloud Analytics layer to its End-to-End Application Management Semantic Model. This model depicts the management capabilities required to gather and analyze the information necessary to track and manage applications from a 360-degree, end-to-end perspective. It includes metrics gathering and supporting analytics functionality. Although solutions monitoring each of the blue layers in the diagram can be deployed separately (and often are), the collective value of the stack far exceeds that of each separate capability. Solutions capable of gathering and analyzing metrics from multiple layers yield a uniquely comprehensive perspective of the application ecosystem. This is critical to managing, versus simply monitoring, application environments in the Cloud-extended enterprise which is a reason why solutions with breadth of coverage across multiple layers of the model achieved higher scores for this Radar. 7

10 Criteria To clarify vendor placement in this report, it s important to understand the models used to include and evaluate this very heterogeneous group of vendors. This section specifies both the minimum criteria for inclusion and the ideal combination of capabilities used to evaluate all solutions. These criteria were used as the basis for developing and scoring the survey-based portion of this assessment. Minimum Criteria 1. Product fits within at least one of the layers/categories identified in the Semantic Model or is a multi-component solution covering multiple categories 2. Product also supports monitoring/management of at least one of: a) Public Cloud b) Private Cloud c) Hybrid Cloud 3. All elements of the solution were generally available for purchase (GA) at the time the survey was conducted (October-November, 2011) 4. At least one customer available for interview. Ideal Criteria 1. Meets minimal criteria listed above 2. Automated versus manual operation: In general, higher levels of automation and minimal reliance on manual tasks received better scores. Examples include optimal (versus excessive) dependence on scripts or scripting, auto-generated and maintained baselines /thresholds, single agent versus multiple agents, self-learning capabilities, etc. 3. Topology models (or service models): Products supporting auto-generated topology models via application discovery, execution analytics, and dependency mapping were favored over those requiring manual topology modeling. 4. End to End transaction/application visibility spanning on-premise and public Cloud 5. CMDB/CMS support: Although some of the vendors participating in this research contested this requirement, EMA research has found that Cloud computing makes CMDB/CMS systems even more important. Ultimately, keeping track of where application components are hosted requires federated or API-level integration of APM solutions with CMDB/CMS systems. Ensuring that CMS systems are accurate requires automated dependency mapping. For these reasons, products with native integration or API support with CMDB/CMS systems (and with dependency mapping functionality) were favored over those that did not. 6. Open Integration: Products with native support for integration with third party solutions received higher scores than those that do not. Examples include trouble ticket generation, metrics assimilation and correlation, and visibility to a variety of heterogeneous platforms and solutions. 7. Visibility into public Cloud platforms: The ideal APM solution demonstrates broad and deep coverage for AND formalized management partnerships with public Cloud IaaS, SaaS, and/ or PaaS vendors. Vendors with actual versus black box visibility into public Cloud platforms, typically via API or similar integration technology, received higher scores. 8

11 8. Visibility into virtualization platforms: Support for private Cloud and IaaS also requires broad and deep coverage of virtualization platforms. Again, this depth of visibility requires integration with management capabilities delivered by the virtualization vendor. 9. Above-average investments in Research and Development 10. Support for hybrid Cloud, including visibility to integration points 11. Customer Support: Multiple customer feedback options, support options, and interaction methodologies (user groups, online forums, conferences, training, etc.) enabling customers to easily communicate with one another and with the vendor. 12. Fast time to value: While time to value is generally easier to achieve with point solutions versus suite solutions, vendors of all sizes are investing in making this process as easy as possible. Distributing solutions as pre-loaded hard or soft appliances, delivering analytics functions as SaaS, and making application templates and fingerprints available are examples of vendor efforts that improve time to value. 13. Price commensurate with capabilities: While price is certainly a concern, higher value solutions require higher levels of vendor investment and therefore cost more. EMA focuses on Cost Efficiency versus cost alone, a measurement that takes overall value, product and deployment costs, training and support costs, and ongoing administration and consulting costs into account. 14. Proven scalability 15. Customer interviews: While at least one customer interview was a requirement for each vendor, additional customers positively impacted overall scoring. 9

12 Scoring For all Radar Reports, EMA evaluates solutions based on five key areas represented by the five sides of a hexagram. They include Deployment and Administration, Cost Advantage, Architecture and Integration, Functionality, and Vendor Strength. The last category, perhaps the only one that s not self-explanatory, is focused on market and industry presence, vision, and the financial stability of the vendor. Within each of these five evaluation areas, EMA creates a superset of capabilities spanning the known solutions in the marketplace. For each superset, specific Radar Reports add questions about new and emerging areas (in this case virtualization and Cloud), and balance the results with standard comparators used across all EMA Radars. The evaluation model used for this Report is presented as Figure 4. The following section details each evaluation area, along with its scope and rating priorities. Figure 4: Assessment Model for APM/Cloud Radar 10

13 Cost Efficiency This is the first set of measures within the EMA Radar Report framework and one of the two major axes of the Radar distribution diagram. It consists of two major sub-sections: Deployment & Administration and Cost Advantage. Each of these, along with the breakout capabilities assessed for each, is described below. Deployment & Administration Deploying and administering is the first side of the hexagram. These scores include measures that indicate how easy or difficult it is to deploy a given APM solution into the production environment, as well as initial time to value. The sub-areas assessed include: 1. Ease of Deployment: Resources required to initially deploy the product. High marks go to fast time to value, vendor-provided quick start aids (such as templates), minimal consulting requirements, and support for a variety of databases and database vendors. Simplified packaging such as a SaaS form factor also wins points. a) Time to deploy: Vendors were asked to gauge their answers for a hypothetical customer with two multi-tiered enterprise applications. Within this context, specific questions addressed the average time required to deploy, begin data collection, and deliver functional, actionable performance data. Additional questions addressed the professional services requirements and overall costs associated with deployment. b) Packaging requirements and options: Operating environment requirements and available form factors. Scoring was based on whether the solution includes hardware and software as part of the quoted cost, is turnkey or requires add-ons, is available as SaaS (since SaaS-based products are obviously easier to deploy than on-premise solutions). c) Staff training: The vendor s training offerings. Scoring favored breadth of options (i.e., online, partner led, on-premise, etc.) and lower overall cost. It also factored in the average number of training hours necessary for basic proficiency and administrator level efficiency, since training costs for multiple personnel can add significant costs. d) Disruption minimization: The impact to normal IT operations that occurs during product deployment and/or upgrades. 2. Support and Services: Assesses the vendor s customer support, professional services, and product maintenance options. High marks go to vendors with a wide variety of options to support customers of different sizes and with different budgets. a) Customer Support: Vendors are scored on the breadth of support options (online, business hours, 24 X 7, etc.). Also takes into account support levels and vendor-provided vehicles for customer product input (product suggestions and status for public review), as well as availability of user interaction options (online, conferences, etc.). b) Professional Services: Professional services options, whether vendor or third-party delivered. Higher scores go to vendors that include professional services as part of the quoted solution cost (to eliminate unpleasant surprises) and to solutions that require minimal professional services to deploy. c) Version Maintenance: Frequency of full releases and software upgrades. 11

14 3. Ease of Administration: Assesses the product and personnel costs necessary to configure, administer, and maintain the solution on an ongoing basis. a) Ongoing administration: The manpower and time required to manage the solution. b) Update Process: The operational impact and time requirements for upgrades, as well as ease-of-use features that support it. c) Testing/Migration: License fee costs for licenses utilized for product upgrade pre-testing. Cost Advantage Cost Advantage is the second side of the hexagram, and assesses overall cost factors related to Pricing, Licensing and ongoing maintenance. 1. Price: Average license cost for a typical deployment, including whether additional for pay modules are necessary, and whether consulting and/or training hours are included in the quoted cost. Also assesses the time required for the average customer to recoup full ROI. 2. Licensing Models: Breadth of licensing models and form factors. Higher scores go to solutions available via multiple licensing models and form factors, which makes solutions more readily available to a wide variety of company sizes and budgets. 3. Maintenance Costs: Average maintenance fees for highest level of service. Industry average is approximately 19 22%; higher scores went to solutions with lower maintenance fees. Product Strength Product strength is the second major axis of the Radar distribution diagram, and is comprised of two sub-areas, Architecture/Integration and overall Functionality. Architecture and Integration The first of the two major product strength categories is Architecture & Integration. It is the third side of the hexagram, and assesses the enabling technology underlying the APM functionality. 1. Architecture: Assesses design elements contributing to overall functionality and usability. a) Design: Design characteristics such as deployment form factors (SaaS, SaaS with on-premise collector, VM, etc.) and metrics collected. Examples include: i. Data collection methodologies (Synthetic transactions, EUE, RUM, tracking/stitching, instrumentation points, platform coverage) ii. Supplemental data sources monitored (Protocols, network flow, third-party agents, CMDB/CMS, discovery sources, etc.) b) Scalability: Scalability is a key consideration, particularly for rapidly growing companies and large enterprises. Specific questions measured the product s ability to scale across large, geographically distributed environments and the largest deployment to date. 12

15 c) Breadth of Environments Supported i. Support for public, private, and hybrid Cloud environments ii. Support for a variety of tiered transaction environments, i.e. on-premise/public Cloud, on-premise distributed/mainframe (often part of Cloud deployments), SaaS-SaaS integrations, etc. d) Breadth of Applications Supported: i. Application deployment types supported, i.e. Web-based, fat client, Web 2.0. SOA, Mobile, VoIP, etc. ii. Technologies supported, i.e., ESB/EAI, API-based integrations, Middleware, Storage, CDN. iii. SaaS and IaaS partnerships, which indicate the depth of support for public Cloud vendors (SaaS vendors such as Salesforce.com and OpSource and for IaaS vendors such as Amazon and Rackspace). Higher scores went to APM vendors who have built visibility to execution within public Clouds, such as EC2 and Salesforce.com, versus those simply leveraging synthetic transactions as the primary Cloud monitoring technology OR those monitoring public Cloud as a black box. iv. Specific support for and partnerships with virtualization vendors supporting private Cloud. Higher scores went to APM vendors who have built depth of visibility to execution within private Clouds, and those who have incorporated technologies such as vmotions and vcenter integration into end-to-end monitoring and management dashboards and correlation. 2. Integration/Interoperability: Virtually every IT organization has a variety of enterprise management solutions and, sooner or later, will need to integrate. This section assesses the ability of the solution to integrate with other products within the vendor s portfolio, as well as with third party solutions. a) Open Integrations: Integrations to generic events such as SNMP traps as well as to vendordelivered APIs for event, data, and Cloud integration. b) Event Management Integration: The most common fault/event management platforms supported, and whether APIs that support these integrations are provided. c) CMDB/DMS Integration: Several participating vendors questioned whether CMDB/CMS integration is important for APM solutions. EMA s position is based on survey-based data, which indicates that IT organizations are starting to question how Cloud-based services can be managed and tracked. They see CMDB/CMS solutions as the ultimate answer for tracking component sources for increasingly complex and interconnected services. 13

16 Functionality The second product strength category is Functionality. It is the fourth side of the hexagram, and covers product features and ease of use. 1. Features: Features are the heart of any solution, and the overall value proposition ultimately boils down to what a solution can actually do. Features were comprehensively assessed across seven key categories and scoring favored features supporting automation versus manual configuration, modeling, and analytics. Key areas assessed include: a) Application discovery/recognition: One key feature EMA sees as being fundamental to managing complex applications is a visual service topology model, ideally generated in real time and requiring minimal (or no) manual input. Application discovery and transaction tracing are key aspects of generating such a model. Assessment questions covered the breadth of elements leveraged to identify a given application (i.e. port, protocol, IP address, fingerprints or templates, packet analysis, etc.) and available discovery methodologies. b) Metrics and measurements: Metrics and measurements delivered natively and via integrated third party solutions. Examples include performance by tier, performance juxtaposed with infrastructure (topology), performance by user/group, completed versus uncompleted transactions, etc. c) Cross-technology visibility: Functionality related to tracing transactions/applications/ services across heterogeneous tiers and across hybrid on-premise/public Cloud environments. d) Alerting/alarming: Automated alerting/alarming features and triggers. Examples include trouble ticket generation/clearing, dynamic baselines, automatic threshold setting, and dynamic variable thresholds by hour, day of the week, etc. e) Troubleshooting: Features related to improving Mean Time to Resolution/Recovery (MTTR) such as visual triage, probable cause analyses and rankings, time-based metrics comparisons, performance comparisons by users/groups/geographies, etc. Assessment covers general APM, private Cloud, SaaS applications, and IaaS applications/components. f) Security and user management: Security-related features such as single sign-on and LDAP integration, user role support, and access control. g) Analytics/advanced analytics: In this context, advanced analytics generally covers features specifically designed to maximize automation and minimize manual interaction with the tool itself. Specific features assessed include self learning of the application environment, self configuration of the solution based on environmental awareness, rolling baselines that self-update based on ongoing trending, self learning of normal performance across hops, automated real-time service modeling, and visibility across variable transaction execution paths. 2. Ease of Use: Ease of use can make the difference between a product that is fully utilized and one that becomes shelfware. If a product is easy to use, it will require less training and less consulting over time and can be used by personnel with varying levels and types of skills. a) Visualization/reporting: Mapping, grouping, visualization, and reporting capabilities. The ideal solution will include a mix of frequently used canned reports, support for ad hoc reports, and a facility for data sharing (export and import) with tools such as Microsoft Excel. b) Roles supported: Assesses most common IT roles utilizing the solution. 14

17 Vendor Strength This is the fifth and final side of the hexagram and covers the multiplicity of factors underlying overall vendor strength. Collectively, the measurements in this section determine the size of the bubble on the chart. These factors are meant to gauge the overall viability of an APM provider, as well as the quality of their vision, strategy, product execution, and market voice. It consists of the following specific categories: 1. Vision: Vision becomes innovation, and visionary vendors create products that enable customers to more easily assimilate changing technologies and application architectures. For this assessment, vendors were asked to describe their Cloud-related product strategy and vision for APM in 500 words or less. More visionary vendors received higher scores. 2. Strategy: Strategy enables vendors to achieve their vision. Again, vendors were asked to describe their longer-term strategy for advancing APM capabilities. 3. Financial Strength: Components of this score include the vendor s own assessment of financial position, and estimated year over year revenue growth for the APM solution. For public companies, it also incorporates data included in public documents, such as cash available and forward P/E estimates. 4. Research and Development: Vendors were asked to disclose R&D investments via a multiple choice question with five potential answers: 0 5%, 6 10%, 11 15%, 16 20%, and 21 % +. This answer was then ranked against the answers of other vendors in the study and entered as a score on a sliding scale between 1 and Market Credibility: While features and functions are critical, vendor credibility is another key aspect of product selection which goes far beyond marketing. In fact, for assessments of this nature, marketing hype often does more harm than good, as true product capabilities, strengths, and weaknesses can be obscured by market speak geared toward misinformation versus reality. In evaluating credibility, EMA examined a number of measures, including in-depth interviews with customers. In general, vendors providing two or more customer references were viewed more positively than those providing fewer. Other relevant factors include: how focused the vendor is on the APM space, how straightforward the vendor was in terms of the assessment, how transparent the vendor is to the analyst community, and treatment by other credible industry voices. EMA Radar Maps for Application Performance Management As indicated above, final scoring has been divided into two broad groups by basic high-level architecture. The results for the two groups are presented separately, beginning with the Multi Component group. As a review, the vendors in the two groups include: Multi Component/Suite solutions: CA, Compuware, Correlsense, eg Innovations, HP, IBM, Nastel, OPNET, OpTier, Quest, and SolarWinds Point solutions: AppDynamics, AppFirst, Aternity, INETCO, Netuitive, New Relic, and Splunk 15

18 Point Solutions Figure 6: EMA Radar Map for APM: Point Solutions Vendors and Solutions Profiled AppDynamics: AppDynamics Pro AppFirst: AppFirst Aternity: Aternity Frontline Performance Intelligence Platform Version 5.0 INETCO: INETCO Insight Netuitive: Netuitive 5.5 New Relic: New Relic Pro Splunk: Splunk Enterprise The value of a solution is defined by comparing its strength with its cost effectiveness. Figure 6 provides a graphical representation of the industry leaders evaluated, positioned in relation to both critical axes. Scores on the x axis reflect Cost Efficiency, which EMA considers to be a measure of overall value. Products shown towards the right on this axis tend to be less expensive or have strong feature/function in relation to overall cost. The y axis reflects the vendor s coverage of the specific functionality assessed in this case, APM for Cloud services. The size of the bubble represents a relative measure of Vendor Strength. 16

19 General Findings All of the vendors in this category scored as Strong Value or better, which indicates that all exceeded EMA s rigorous basic scoring criteria. As with the Suite solutions profiled above, this does not indicate an overly rosy judgment on EMA s part, but is rather a manifestation of the phenomenon of self-selection. Vendors not prepared to invest the resources required to traverse the Radar process, those daunted by the rigor of the survey and dialogue, and those less certain of product strength declined to participate. From this perspective, those who did participate can thus be considered the APM elite those providers who have invested in delivering complete solutions with substantial value to application management practitioners. Netuitive, AppDynamics, and Aternity all scored in the Value Leader quadrant despite the fact that these are very different solutions. Netuitive is a top-level correlation solution, while AppDynamics is a SaaS-based APM solution focused on code-level analytics. Aternity is an EUE solution focused on the endpoint, with rich capabilities supporting performance management, management of hosted desktop environments, and security compliance. AppFirst, New Relic, Splunk, and INETCO all placed in the Strong Value category. Again, these are very different solutions as well, but all support various elements of APM and related troubleshooting capabilities. AppFirst and New Relic are SaaS-based, one focusing on Application Infrastructure Management (AppFirst) and the other focusing on transaction tracing down to specific lines of code (New Relic). Splunk is a machine data aggregator and analyzer, while INETCO is well established in the payments processing industry and in the process of extending capabilities to address the general APM market. Aternity: Value Leader Aternity: With its Frontline Performance Intelligence Platform, Aternity has developed a distinctive approach to monitoring both Cloud and on-premise service performance, including that of hosted desktop environments. By monitoring the endpoint and the next hop from the endpoint, Aternity delivers a level of visibility to the experience of internal users that is difficult to match with other EUE technologies. This technology can also be applied to a multiplicity of additional monitoring requirements, such as identification of training issues, illicit usage, regulatory compliance, and levels of services delivered by public Cloud vendors. Key differentiators include The Aternity FPI Platform s support for hosted desktop environments and deep visibility to processes, keystrokes, and virtually all other events taking place at the user workstation. 17

20 Aternity Profile EMA Award: Most Comprehensive Endpoint Diagnostics Because of its distinctive approach to End User Experience Management, EMA is pleased to cite Aternity as the award winner for the Most Comprehensive Endpoint Diagnostics. EMA research has repeatedly found that companies leveraging Cloud services see EUE as a primary monitoring methodology for such environments. Monitoring the User experience is a fail safe way to determine the levels of overall service delivered to the desktop. Aternity has developed a distinctive approach to monitoring both Cloud and on-premise service performance, including that of hosted desktop environments. By monitoring the endpoint and the next hop from the endpoint, Aternity delivers a level of visibility to the experience of internal users that is difficult to match with other EUE technologies. This technology can also be applied to a multiplicity of additional monitoring requirements, such as identification of training issues, illicit usage, regulatory compliance, and levels of services delivered by public Cloud vendors. 18

21 Overview Aternity, Inc. is a privately-held software vendor with headquarters in Boston, Massachusetts. The Aternity website lists the software vendor as an Intel Capital portfolio company, and Aternity is led by a management team with more than 100 years of combined experience. For this research, EMA reviewed the Aternity Frontline Performance Intelligence (FPI) Platform. The Aternity FPI Platform is an End User Experience Management solution focused on the desktop. It monitors both Cloud and on-premise service performance from the desktop perspective. Key differentiators include Aternity FPI Platform s support for hosted desktop environments and deep visibility to processes, keystrokes, and virtually all other events taking place at the user workstation. Architecture and Integration Factors comprising this score include overall design, scalability, breadth of support, and integration/ interoperability. Aternity has very good breadth of environment support covering web-based, fat client, machine-to-machine, and Web 2.0 and SOA applications, monitoring performance and availability through a desktop agent. Although Aternity has no native visibility to SaaS vendor networks, it can monitor end-user performance for hosted desktop environments instrumented with desktop agents. Aternity relies on partners for event management integration, and gets high scores for out-of-the-box (OOB) API connectors for both BMC and CA. Additionally, Aternity recently announced a technology partner agreement with CA to provide extensive insight into end-to-end application performance, with CA delivering visibility to back-end and Cloud environments. This score was adversely impacted because of the fact that this survey was heavily geared towards comprehensive data center coverage versus Aternity s more limited focus on the endpoint. Aternity overcomes this limitation by partnering with other vendors as described above, and by delivering a variety of data sharing methodologies for CMDB/CMS systems. Functionality Aternity has good coverage for security and user management, role definition, and alerting/alarming. Virtually every role across IT Operations, service management, finance and executive management is supported by Aternity, the heaviest consumers of the service being application owners, IT Ops, and Service Desk. Aternity s security features are noteworthy, with support for single sign-on and the ability to issue credentials by feature/function and group policy. Aternity can also manage users through both Out of the Box (OOB) and custom role templates. Aternity s alerting/alarming features are strong as well and include functionality for setting thresholds using both static and dynamic rules, as well as rules-based event management. Although the solution natively lacks data center support, Aternity s root cause analysis capabilities focus on the endpoint, enabling Aternity to provide a wealth of execution details which are not visible to data-center focused solutions. 19

22 Deployment and Administration Scores in this area are derived from a combination of deployment-related factors including time to deploy, packaging and training options, and support options. Aternity s simplicity of design and ease of deployment generated high scores in this area. Aternity customers interviewed as part of this research attest to Aternity s strong deployment and training support. Aternity offers boot camp training for both customers and partners, along with a robust combination of online and classroom education. Aternity has also focused on simplifying agent upgrades. Once an agent is deployed, the administrator manages no touch updates from a central console. One area in which Aternity was penalized, again due to the nature of this specific research, was related to the fact that it lacks native support for traditional real-time discovery and topology of components residing on backend servers. However, the solution does discover and build topology maps which supplement traditional data center topology maps with endpoint inventory, device connection, application connection, and application profile data. According to Aternity, this reveals the behavior and interplay of operating system, application and add-on modules comprising a Windows-based application (for both physical and virtual desktops). Cost Advantage Factors comprising this score include product pricing, license model options, and maintenance and support costs. While Aternity cannot be considered a low-cost solution, EMA sees its overall cost as commensurate with value delivered. Aternity can be purchased through a perpetual license or SaaS model. Customers attest that the SaaS model offers flexible pricing structures and lends itself well to smaller POCs that can be easily scaled to larger deployments. With the perpetual license model, Aternity s annual maintenance costs are in line with industry standards. Vendor Strength This score is derived from a combination of factors which include vision, strategy, overall financial strength, R&D investment, partnerships, and market credibility. For smaller private companies such as Aternity, the primary limitations in terms of vendor strength are related to company size and the opaque nature of company financials. While this can introduce an element of uncertainty to prospective buyers in terms of market credibility and financial strength, Aternity has reported very strong year-over-year growth in 2011, one of the higher growth percentages of all the vendors interviewed for this research. Aternity also boasts significant levels of investment in R&D, another key scoring factor for this study. Strengths and Limitations Strengths Strong financial backing from multiple VCs and impressive revenue growth Experienced management team with varying backgrounds and industries Good breadth of environment support, including Web 2.0, SOA, and Virtual Desktop Infrastructure (VDI) 20

23 Agent-based technology easily updatable, remotely and from a single console Technology partnerships with leading Independent Software Vendors (ISVs) and IT Outsourcing (ITO) organizations Good functionality for security and user management, with single sign-on, and role templates Wide variety of training options and delivery methodologies, including a SaaS option. Limitations Product is designed to support internal users (those with managed desktops) versus external users Native visibility limited to end point components (including physical and virtual desktops and associated infrastructure), and the first hop connection to the backend environment. Customer Quotes Prior to purchase, we looked at a competing solution. However, Aternity seemed head and shoulders above the other solution in terms of maturity in company, support capabilities, ability to execute, and software. IT Executive, Government The implementation was as effortless as I could have hoped for. During the POC, support provided thorough and fast response when we needed them. At production, everything was squeaky clean. Network Services and Data Center Project Manager, Financial Institution 21

24 About Enterprise Management Associates, Inc. Founded in 1996, Enterprise Management Associates (EMA) is a leading industry analyst firm that provides deep insight across the full spectrum of IT and data management technologies. EMA analysts leverage a unique combination of practical experience, insight into industry best practices, and in-depth knowledge of current and planned vendor solutions to help its clients achieve their goals. Learn more about EMA research, analysis, and consulting services for enterprise line of business users, IT professionals and IT vendors at or blogs.enterprisemanagement.com. You can also follow EMA on Twitter or Facebook. This report in whole or in part may not be duplicated, reproduced, stored in a retrieval system or retransmitted without prior written permission of Enterprise Management Associates, Inc. All opinions and estimates herein constitute our judgement as of this date and are subject to change without notice. Product names mentioned herein may be trademarks and/or registered trademarks of their respective companies. EMA and Enterprise Management Associates are trademarks of Enterprise Management Associates, Inc. in the United States and other countries. EMA, ENTERPRISE MANAGEMENT ASSOCIATES, and the mobius symbol are registered trademarks or common-law trademarks of Enterprise Management Associates, Inc. Corporate Headquarters: 5777 Central Avenue, Suite 105 Boulder, CO Phone: Fax: Aternity-Summary