First Class Mobile Application Performance Management

Transcription

1 First Class Mobile Application Performance Management August 2012 Jim Rapoza ~ Underwritten, in Part, by ~

2 First Class Mobile Application Performance Management The rise of mobile applications and the devices that they run on has been long expected. However, the rate at which mobile web and native mobile applications are quickly becoming a primary way that critical business applications, content and digital properties are consumed has clearly taken many organizations by surprise. Based on initial findings collected in July 2012 from the Aberdeen group survey The Challenge of Application Performance in a Mobile Application World, the number one pressure that is driving the application performance strategies of organizations is the increased importance of mobile application access (at 69% of responses). However, only 30% of organizations have currently implemented any kind of mobile application performance infrastructure or strategy but 54% plan to do so. In this Aberdeen Group report, we ll look at how tools are evolving to make mobile applications first class APM citizens and we ll also analyze how businesses are working to manage and improve performance for their mobile application infrastructure and look at what the most successful organizations are doing to achieve their results. Best-in-Class Performance Aberdeen used the following four key performance criteria to distinguish Best-in-Class organizations: End user satisfaction: Measured as the year-over-year percent change in the number of end user complaints about mobile application performance and user experience Mobile Application adoption: Measured as the percent of users actually using the organization s mobile applications, on a scale of 0% to 100%. Employee Satisfaction with application performance: Measured on a scale from extremely satisfied to extremely dissatisfied. Customer Satisfaction with application performance: Measured on a scale from extremely satisfied to extremely dissatisfied. The Challenge of Understanding Performance for Mobile Apps One of the biggest challenges when it comes to managing performance and improving end-user experience of mobile applications is the unique deployment and device delivery requirements of mobile apps, which typically require deployment to multiple device types and across different app stores August 2012 Research Brief Aberdeen s Research Briefs provide a detailed exploration of a key finding from a primary research study, including key performance indicators, Bestin-Class insight, and vendor insight. Defining the Best-in-Class Aberdeen s research for, First Class Mobile Application Performance Managements, used four key performance criteria to distinguish the Best-in-Class (top 20% of aggregate performers) from the Industry Average (middle 50%) and the Laggard (bottom 30% ) organizations: Number of end user complaints about mobile application performance Mobile application adoption rate Employee satisfaction with mobile application performance Customer satisfaction with mobile application performance This document is the result of primary research performed by Aberdeen Group. Aberdeen Group's methodologies provide for objective fact-based research and represent the best analysis available at the time of publication. Unless otherwise noted, the entire contents of this publication are copyrighted by Aberdeen Group, Inc. and may not be reproduced, distributed, archived, or transmitted in any form or by any means without prior written consent by Aberdeen Group, Inc.

3 Page 2 or app deployment systems. Also, the difficulty of getting visibility into mobile app performance once the connections leave company networks and travel over external and carrier networks means that traditional monitoring and end-user experience management tools often have gaps in what they can detect. As shown in Figure 1, the approaches that businesses are taking to track performance and user experience for mobile applications differ greatly from how they tend to track traditional applications. Figure 1: Key Indicators Used to Evaluate Application Performance Mobile Applications User ratings on app stores 19% Traditional Applications 88% Fast Fact 25% of users abandon a mobile application after three seconds of delay Social trend analysis Transaction conversion rate Carrier network performance Application availability Calls to help desk 21% 86% 73% 58% 69% 63% 67% 81% 75% 85% Level of SLA achievements Application response times 53% 57% 86% 87% 0% 20% 40% 60% 80% 100% Percentage of respondents, n=63 Source: Aberdeen Group, July 2012 The top two methods for tracking traditional application performance (application response times and SLA achievements) are among the lowest for tracking mobile apps while the main methods used to track mobile app performance (social trend analysis and user ratings on app stores) barely register as ways to track traditional applications.

4 Page 3 This data reveals one of the biggest problems facing mobile applications. The primary ways that the performance and user experience of mobile apps are tracked are based on detecting user problems AFTER the app has been deployed. Ideally, a business should be able to find the problems in their apps before users start sending negative tweets or posting bad reviews on app stores. Proactively Improving Mobile App Performance One key way to find application issues before an application is deployed to end-users is to carry out effective testing during the development stage. As shown in Figure 2, 42% of organizations plan to do performance testing of mobile applications in the development stage and 35% plan to do mobile app functionality testing during development. Figure 2: Top Actions Taken to Address Mobile Performance Issues Mobile Performance Testing During Development Redesign or optimize mobile apps Convert existing applications to mobile apps Mobile Functionality Testing During Development Improve mobile apps based on enduser experience 42% 42% 39% 35% 33% 0% 10% 20% 30% 40% 50% Percentage of respondents, n=69 Source: Aberdeen Group, July 2012 Along with the usage of testing in the development stage, businesses are also planning to take actions to redesign and optimize mobile apps and improve mobile apps based on end-user experience. All of these actions have the benefit of providing valuable information about mobile application performance and user experience before the apps are deployed to the enduser population. Building Out to Mobile App Domination Not too long ago, mobile application access (both mobile web and native) was seen as a nice option to have but not typically vital to business interests. But nowadays more organizations are finding that mobile devices are indeed being used for vital business interests, such as securing sales while in the "Mobile is a disconnected use case with occasional connections." ~ Executive at UK-based social software developer

5 Page 4 field or detecting new opportunities for business growth. And, in many ways, these mobile users require high performance and reliability as much as, if not more then, workers on the company network. While many businesses are still behind in their adoption of tools and technologies to manage, monitor and understand mobile application performance, looking at their planned adoption shows that, within a year, many organizations expect to be deploying dedicated systems to manage mobile application performance. Figure 3: Mobile Application Performance Management Capabilities Currently Implemented Define baselines for acceptable mobile app performance Ability to perform root-cause analysis on mobile application issues Measure mobile application response times Capture of mobile end-user experience data 7% 18% 20% 21% Plan to Implement 79% 62% 61% 61% Fast Fact Top Challenges of Mobile Application Performance 51% - Diversity of mobile devices and operating systems 48% - Lack of tools to measure, monitor and test mobile application performance 43% - Complexity of mobile development platform Use end-user experience to inform mobile application design or changes 27% 55% Test mobile performance of real-world users 20% 55% Software or agent for monitoring mobile enduser experience 20% 51% Simulation and testing of loads and real-world traffic scenarios for mobile applications 19% 46% Software or service for mobile traffic and load simulation 17% 40% Percentage of respondents, n=66 0% 20% 40% 60% 80% 100% Source: Aberdeen Group, July 2012

6 Page 5 The top four capabilities that organizations plan to deploy in the next year include some of what would be considered core elements of traditional performance management systems. These include the ability to define performance baselines (79% plan to adopt), perform root-cause analysis of mobile application performance (62% plan to adopt) and measure mobile application response times (61% plan to adopt). These basic but necessary capabilities make it possible for organizations to gain a better understanding of their overall mobile application performance and user-experience and lay the groundwork for the use of more advanced testing, analytics and optimization technologies. User Satisfaction is Key to Mobile App Performance While the focus of traditional performance management tools has often been centered around network performance and bandwidth utilization, recently, more and more products in the performance management space have realized that the focus needs to be on end-users. No user really cares how the network is running, they only care if the applications that they rely on are performing well and providing good usability. Given this, the most important metric for any application is user satisfaction. In our survey, organizations achieving the best results in mobile application performance were clearly providing a good end-user experience for their mobile applications. Figure 4: User Satisfaction with Mobile App Performance and Usability Best-in-Class Industry Average Extremely Satisfied Very Satisfied Satisfied Somewhat Satisfied 14% 2% 2% 27% 22% 14% 33% 45% "Businesses need to capture feedback from users on a real time basis and innovate to make applications offer critical value to customers" ~ IT Manager, African-based payment processing firm Very Dissatisfied 0% 10% 0% 10% 20% 30% 40% 50% Percentage of respondents, n=71 Source: Aberdeen Group, July 2012 As shown in Figure 4, organizations identified as being Best-in-Class had 86% of users identified as Satisfied or better with mobile app performance and they had 59% very or extremely satisfied. This is compared to Industry Average companies that had 43% of users less than satisfied and 10% of

7 Page 6 users identified as very dissatisfied. In the end, no number of green lights on a performance management dashboard will matter if end-users are dissatisfied with a mobile apps performance and usability. Happy users don't light up social networks with complaints or post negative reviews on app stores. Key Insights More and more users are now turning to mobile devices such as tablets and smartphones as their primary computing platform. And businesses are responding to this trend by delivering more and more of their core web and native applications to these mobile systems. But if an organization is relying on outdated methods and tools to handle application performance management and user experience monitoring, they will find that these older tools can't address mobile application performance in the same way that they handle traditional applications. Organizations should look to these insights in order to be better prepared for the rise of mobile applications: Understand the difference between mobile and traditional applications. It can be easy to assume that there is little difference between managing the performance of a mobile web or native app and managing the performance of traditional web and desktop applications. But these are completely different beasts, as shown in Figure 1 in the radically different ways that organizations are currently tracking mobile application performance. Once businesses understand these differences, they can take the necessary actions to build a mobile ready performance management infrastructure. Don't wait for users to complain about apps. Relying on user responses on social networks and app store rating to find out how mobile apps are performing means that it is already too late for a business to fix problems. 42% of organizations plan to utilize testing during the development stage to head off potential performance and usability problems before the apps end up on user mobile devices. Take action now. Every bit of information and insight that an organization can get on mobile application performance and usability can pay off in avoiding problems. Among the top capabilities that businesses plan to deploy in the next year are core performance management features such as root cause analysis, performance baselines and mobile application response times. Using these capabilities will help these organizations lay the groundwork to utilize all of the capabilities of an end-to-end mobile performance infrastructure. Focus on the user. While every bit of performance and uptime data is useful to understanding mobile application performance, in the end the only metric that matters is end-user satisfaction. Organizations that focus on the user are able to build and optimize mobile applications that provide high levels of user satisfaction.

8 Page 7 While the rise of mobile applications is coming more quickly than some may have expected, organizations that are well prepared are already seeing success when it comes to providing high-quality mobile application experiences. As more of the performance management products turn their focus towards the management, optimization and monitoring of mobile apps, the second class status of mobile applications will end and organizations will expect to have all the performance management and optimization capabilities for mobile apps that they currently enjoy for traditional applications. For more information on this or other research topics, please visit End-user Experience Management: Questions to Ask; July 2012 Convergence of Social, Mobile and Cloud Requires User Focused End-to-end App Approach; February 2012 Related Research Boosting Enterprise Application Performance in Distributed Environments; April 2012 Enterprise Mobility Requires New Strategies for Optimizing Application Performance; January 2012 Author: Jim Rapoza, Senior Research Analyst, Network and Application Performance (jim.rapoza@aberdeen.com) For more than two decades, Aberdeen's research has been helping corporations worldwide become Best-in-Class. Having benchmarked the performance of more than 644,000 companies, Aberdeen is uniquely positioned to provide organizations with the facts that matter the facts that enable companies to get ahead and drive results. That's why our research is relied on by more than 2.5 million readers in over 40 countries, 90% of the Fortune 1,000, and 93% of the Technology 500. As a Harte-Hanks Company, Aberdeen s research provides insight and analysis to the Harte-Hanks community of local, regional, national and international marketing executives. Combined, we help our customers leverage the power of insight to deliver innovative multichannel marketing programs that drive business-changing results. For additional information, visit Aberdeen or call (617) , or to learn more about Harte-Hanks, call (800) or go to This document is the result of primary research performed by Aberdeen Group. Aberdeen Group's methodologies provide for objective fact-based research and represent the best analysis available at the time of publication. Unless otherwise noted, the entire contents of this publication are copyrighted by Aberdeen Group, Inc. and may not be reproduced, distributed, archived, or transmitted in any form or by any means without prior written consent by Aberdeen Group, Inc. (2012a)

9 Page 8 Featured Underwriters This research report was made possible, in part, with the financial support of our underwriters. These individuals and organizations share Aberdeen s vision of bringing fact based research to corporations worldwide at little or no cost. Underwriters have no editorial or research rights, and the facts and analysis of this report remain an exclusive production and product of Aberdeen Group. Solution providers recognized as underwriters were solicited after the fact and had no substantive influence on the direction of this report. Their sponsorship has made it possible for Aberdeen Group to make these findings available to readers at no charge. SOASTA s web and mobile app test automation solution enables developers, QA professionals, and IT Operations teams to test with unprecedented speed, scale and precision. Our innovative solution streamlines test creation, automates provisioning and execution and distils analytics to deliver actionable intelligence faster. With SOASTA, you can have confidence that your app will perform as designed, even in peak traffic. For additional information on SOASTA: SOASTA 444 Castro Street, Fourth Floor Mountain View, CA Telephone: info@soasta.com

10 Page 9 New Relic is the all-in-one web application performance tool that lets you see performance from the end user experience, through servers, and down to the line of application code. New Relic provides visibility to all major web application languages Java,.net, Python, Ruby and PHP. With customers that include Comcast, Intuit, Groupon and AT&T Interactive, New Relic is the largest SaaS-based performance monitoring provider, measuring over 60 Billion metrics for its more than 25,000 customers each and everyday. The product is delivered as a service and available for free at newrelic.com. For additional information on New Relic: New Relic 101 Second Street, 15th Floor San Francisco, CA Telephone: newrelic.com