Managing Mobile Application Performance. APM Experts The Virtualization Practice

Transcription

1 Managing Mobile Application Performance APM Experts The Virtualization Practice

2 Bernd Harzog Analyst, Virtualization and Cloud Performance Management Bernd Harzog is an Analyst for Cloud and Virtualization Performance Management. Bernd is also the CEO of APM Experts, a consulting and analysis firm focusing upon this market, vendor strategies in this market and customer use cases in this market. Bernd was formerly a Gartner Group Research Director focusing upon the Windows Server operating system, CEO of RTO Software, VP of Products of Netuitive and has been involved in vendor and IT strategy since About The Virtualization Practice The Virtualization Practice provides analysis, commentary and resources on current Virtualization and Cloud Computing news, events, and community. We break virtualization into topics, and feature a world class expert in that topic as the analyst for the topic. Topic Analysts are responsible for writing original, objective, analytical posts in their area of expertise, for writing and maintaining a white paper on their market and the vendors that serve that market, and for assisting sponsoring vendors with marketing activities. Executive Summary Smartphones and tablets are the fastest growing end user computing device in the history of the world. 56% of the people in the world now own a smartphone. 50% of mobile phone users use their mobile phone as their primary way to access the Internet. 80% of the time that users spend on their smartphone is spent inside of apps. The above statistics combined with the fact that it is predicted the in 2014 more users will access the Internet via their smartphone than via any other device means that for any business whose revenue, reputation or customer retention is tied to mobile user experience that monitoring and management of that experience is of crucial importance. Managing mobile user experience and application performance is also an extremely unique and difficult problem, unlike managing application performance on servers. It can be best described as a full-stack problem which involves the mobile network, the mobile device, the mobile operating system, the mobile app itself, and everything else (including the serving of ads) that is going on in the device at a point in time. For the above reasons, mobile application performance is a unique problem that must be addressed by vendors focusing exclusively upon this space. This paper discusses this problem, and how a complete mapm solution addresses this problem. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 2

3 About the Virtualization Practice...2 Executive Summary...2 I. APM for Server Based Applications A Well Understood Problem...4 II. Mobile A Different and Rapidly Growing Challenge...4 Variation in User and Customer Relationships...4 Variation in Networks...5 Variation in Devices...5 Operating System Diversity...6 Resource Contention on the Device...7 Multiple Versions of the Mobile Application in Production...7 Service Dependencies with Applications...7 III. Performance Issues Resulting from Mobile Complexity...7 Application Crashes and Exceptions...7 API and Network Errors...7 Unresponsive Applications...8 Transaction Errors...8 Location Based Errors...8 Third Party Services...8 IV. Agile Development and DevOps for Mobile Applications...8 V. Mobile User Experience A Whole Stack Problem...9 VI. Business Cases for Mobile Performance Management...10 Revenue...10 Retention...10 Reputation Resolution Time VII. Requirements for a Mobile APM Solution Built with a Focus upon Mobile Applications and their Environment Built for Mobile Scale Rapid Time to Value Full Stack Solution...12 Coverage of Crashes, Uptime and End User Experience...12 Flexibility to Monitor an Enterprise s Entire Mobile App Portfolio...12 Provide Actionable Diagnostics...12 VIII. Summary and Conclusions...13 April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 3

4 I. APM for Server Based Applications A Well Understood Problem The first Application Performance Management (APM) solution to measure the performance of web applications running on application servers was brought to market nearly 20 years ago. Today managing servers in production and managing the performance of server based applications written in Java,.NET or other newer languages is a well understood and generally easily solvable problem. It is also the case that in most enterprises, the team that supports the application in production is a different team than the one that supports the underlying infrastructure and each team has its own set of tools. One just has to take a look at the tools that are used to manage custom server based applications in production to gain an appreciation for how different the problem of managing server based applications is from managing mobile applications. Server based APM tools deeply instrument the application runtime that supports the language that the application is written in, and generally completely ignore the infrastructure underlying the JVM or other application run time. The problem here is very different that the problem that server based APM solutions solve. Server APM solutions solve ONLY the problem of the application running on the server often in a Java virtual machine. With mobile we have a whole stack problem that spans the application, the application container, the operating system, the hardware and the network. Mobile devices introduce variation at the hardware level and network level that were never present in server-based enterprise applications. This dramatically complicates the problem and dramatically increases the value of the solution. II. Mobile A Different and Rapidly Growing Challenge Mobile devices, networks and applications have completely new and different characteristics from server based applications leading to the following challenges: Variation in User and Customer Relationships Mobile applications get used in a variety of relations between the vendor of the applications and the consumer of the applications: B2C Applications in this case the user of the application is a consumer who owns their own device and the vendor of the application has no control over the device or what else is installed upon the device. B2B Applications In this case the users of the application are usually employees of a business with whom the vendor of the application has a supplier, customer or partner relationship. Here again the vendor of the application has little to no control over the device and what else is running on it as it belongs to a different organization. B2E Applications these applications are typically deployed by a business for use by its employees. In this case the device can be owned and controlled by the business, or they may be owned by the employee but under the management of a mobile device management solution. Mobile APM solutions need to work across the diverse deployment, diverse device ownership, and diverse device management scenarios listed above. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 4

5 Variation in Networks The performance of a device or an application on a device is going to vary tremendously based upon which carrier the device is connecting through, and what the level of service that the carrier provides is at the exact location of the user. While 4GLTE is the current gold standard for high performance data on mobile devices, 4GTLE coverage varies widely across carriers and within the coverage map of a single carrier. Mobile application developers need to be aware of the network characteristics which support their users that they can optimize for each of the common access methods of their particular user base. This is a much more complicated problem that just having to worry about what percentage of the users have broadband Internet connections as is the case with server based applications. Top 20 Mobile Carriers (Subscribers in Millions) China Mobile, Airtel, Vodafone, China Unicom, 276 America Movil,251.8 Telefonica, Axiata, Orange, VimpelCom, 209 China Telecom, 183 MTN Group, Etisalat, 167 Telenor, 166 TeliaSonera, 160 Saudi Telecom, 139 Reliance Communications, Tmobile, Verizon, Idea, AT&T, 107 Variation in Devices Server based web applications don t have to worry much about the differences between vendors of server hardware (a modern Intel server is a modern Intel server) and do have to worry some about the differences between browsers, but these variations are small compared to what exists in the mobile world. Apple, by virtue of the fact that it is the sole manufacturer of hardware for its ios platform does do a good job of keeping its hardware design under tight control. That said there are now enough different version of the iphone and ipads on the market that just the differences in the iphone installed base constitute a somewhat significant source of variation especially when it comes to the local processing power and memory of each phone. The Android platform is completely different. Google only exercises some minimal controls over what hardware vendors can do for devices that run the Android platform. This leads to great variations in design, which leads to great differences in both performance and compatibility across various Android hardware devices. While Microsoft does produce its own hardware, Microsoft also works with third party phone vendors making Microsoft into a hybrid of Apple and Google. The challenges with device diversity are not limited to the differences within one vendor s platform. Most vendors of apps support at the minimum both the Apple ios platform and the Google ios platform. Therefore most vendors of apps have to contend with the level of hardware variation across both of those platforms. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 5

6 Operating System Diversity Since Apple introduced the first iphone in 2008 there have been seven versions of ios released onto the market (ios 1 through ios 7), along with various point releases and updates to each major version. Versions 1 and 2 are considered by Apple to be obsolete. Versions 3 through 6 are considered to be discontinued with Version 7 being the currently supported version. According to data on Apple s App Store, the mix of versions in use as of January 2014 was as shown below. Apple ios Versions in Use Earlier 3% ios 6 17% ios 7 80% This points out that even though Apple aggressively tries to update its installed based to the newest versions of ios as they come out, after four months of availability still 20% of Apple s users were on older versions. This creates a need for application developers to maintain backward compatibility with older versions of ios, and understand performance and user experience across the versions in use. The first version of Android was also released in Since then there have been 18 releases culminating most recently in the release of KitKat on in September of The chart below shows the distribution of usage across the various Android releases. Android Versions in Use (Jelly Bean) 54% 2.3 (Gingerbread) 17% 4.0x (ICS) 12% 4.4 (KitKat) 9% (Eclair) 4% 2.2 (Froyo) 3% (Honeycomb) 1% Source: April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 6

7 The point here is obvious. There is considerably more variation in the installed base of the version of Android than is the case with ios. This is probably because different carriers have different policies for how frequently and how aggressively they push out updates to different devices combined with the fact that the device manufacturers themselves create updates to the newest releases of Android at different rates. Mobile application developers therefore have to understand the performance of their applications across the versions of both ios and Android that are in use. That means for someone supporting an app on these platforms they would have to support that app and understand the performance of that app across two ios versions and 5 Android versions if they wanted to cover more than 95% of the users of those platforms. Resource Contention on the Device While an end user is using any one app on a device, that is not all that is going on the device. At the same time that device may be scanning for WiFi networks and receiving GPS updates. The mail app may be running in the background and downloading new s. If the app is offered via an advertising supported model, the process displaying the add will be using compute, memory and network resources in competition with the app itself. Multiple Versions of the Mobile Application in Production Vendors or enterprises with mobile applications do not generally have the ability to force the users of the end user application to upgrade to the latest version. Therefore teams supporting mobile applications generally have to support the most recent version as well as several older versions in production. This is an fact an area where mobile APM is very different than server based APM, as the team that supports a server based application can generally force the upgrade of all of their servers whenever they want to. Service Dependencies with Applications Mobile applications are unique in that most of them have direct dependencies in the mobile application itself upon multiple third party cloud and application services. These services include Facebook, Twitter, Google+ and Linkedin for authentication, multiple different ad serving services, various analytics services, various content delivery networks, YouTube for video, and various internal commerce systems or customer relationship management systems. Research shows that the average mobile application is dependent upon 6 or more of these cloud based services, all of whom have to perform well in order for the user to have an acceptable end user experience. III. Performance Issues Resulting from Mobile Complexity The complexities outlined directly above create a variety of performance and reliability issues for users of mobile applications. Application Crashes and Exceptions Application crashes produce downtime. Downtime directly affects the revenue derived from the mobile application which is covered in detail in The Business Case for Mobile Application Performance below. In order to minimize the downtime visibility into what caused the crash is needed and this must be visibility into whatever layer of the entire stack is at fault, not just visibility into the application code. API and Network Errors An unresponsive mobile network can make a mobile application slow, or it can even make the application appear to be frozen. This can easily result in the user killing the application which creates the same downtime problem that is created by a crash. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 7

8 Unresponsive Applications Applications can easily become unresponsive themselves or they can become unresponsive because they are dependent upon other servers that are themselves unresponsive. This again can result in the user killing the application which again results in the same problem as downtime created by a crash. Transaction Errors Problems in the logic of an application can result in errors within the application itself which can directly affect the online shopping experience, and therefore directly affect the revenue generated by the application. Location Based Errors The uneven rollout of 4GLTE within the US and the disparity that exists across global networks can make an application that is running perfectly fine at one moment slow down or appear to freeze as the use roams to a different cell with a different level of service. Only find grained monitoring of transactions correlated with the type of the network and the performance of that network can identify and help diagnose these kinds of problems. Third Party Services Most mobile applications rely upon a combination of third party services including ad-serving, payment processing, and various social media services like Twitter, Facebook, and Linkedin. The responsiveness of each of these services and their errors rates and crashes contribute to the end user experience of the application. Therefore the latency, error rate, and crashes of all third party services that the app relies upon must be monitored as well. IV. Agile Development and DevOps for Mobile Applications While the complexity of the mobile environment discussed above certainly contributes to the challenges associated with monitoring the performance and reliability of mobile applications, the manner in which mobile applications are themselves developed and supported contributes to these challenges as well. Since most mobile applications have been developed fairly recently they tend to have been developed with some of the newer languages and newer methodologies. In particular Agile Development techniques are frequently used to build new version of a mobile application, and DevOps techniques are frequently used to support mobile apps in production. In general there are sound business reasons for using these processes with mobile applications as competitive pressures require frequent updates and therefore short release cycles. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 8

9 The Agile Development and DevOps processes for rapidly developing new releases and supporting rapidly changing applications in production are shown in the diagram below. The important point is that there is rapid iteration in each step of the process. Android Versions in Use PLAN CODE BUILD TEST RELEASE DEPLOY OPERATE DevOps Value Continuous Integration Continuous Delivery Agile Development Collaboration It is the rapid iteration in the deployment and operating cycles combined with the permutations and combinations in the entire mobile stack that make mobile end user experience management and application performance management into such a unique and difficult problem. Basically it all boils down to having to support many versions of a rapidly changing application on a diverse stack consisting of many networks, devices, operating systems and environmental factors. These factors all combine to make mobile application performance into a unique whole stack problem as discussed in the next chapter. This problem is completely different than the performance management problems that have preceded the current situation with business critical mobile applications. V. Mobile User Experience A Whole Stack Problem For the reasons listed above, managing the end user experience and application performance of an application on a mobile device is not just an application performance management problem. It is a whole stack problem requiring a holistic approach that understands the network, the hardware, the operating systems, and the environment in which each application is running. This is a completely different challenge that which is addressed by traditional APM vendors in the server based Application Performance Management space. These vendors focus upon the code running in the Java Virtual Machine (or its equivalent for other languages) and generally have only a weak ability to understand what is happening in the rest of the server and network stack that supports these applications. Therefore it is the case that mobile application performance is a problem best tackled with a clean sheet of paper one that includes the entire problem affecting mobile end user experience and application performance and not just the operation of some code in a JVM. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 9

10 VI. Business Cases for Mobile Performance Management Revenue For many types of mobile applications like mobile banking, other forms of online financial services (trading stocks), online retail, and any application with an advertising supported revenue stream (which includes all of the popular games in the mobile environment), the end user experience of the user using the mobile application ties directly back to revenue. Users know that for many applications an alternative is only a few swipes away at the app store, so bad end user experience can translate in revenue losses extremely quickly. The graphic below shows how crashes translate into downtime for the application and how that downtime translates directly into lost revenue for a company generating with $1B in annual mobile revenue. It is important to point out that most unmanaged mobile applications have a availability of less than 99% per month, which means that crashes and downtime cost that $1B company at least $821,918 in lost revenue per year. Uptime (%) Downtime per Month 97% 98% 99% 99.9% 99.99% 21h 14h 7h 43m 4m Lost Mobile Revenue ($) -500,000-1,000,000-1,500,000-2,000,000-2,500,000 -$2,465,753 -$1,643,836 -$821,918 -$82,192 -$8,219 Retention Every business knows that it is more expensive to get a new customer than it is to keep an existing customer. A bad user experience (even if not caused by the application itself, but which shows up while that app in use) will cause uses to look for alternatives which are in many cases a click away on the app store icon. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 10

11 Reputation For many users the mobile app is the most frequently accessed incarnation of a company s presence online. Furthermore app rating systems make it extremely easy for users to voice their pleasure or displeasure with an app or even the latest version of an app. Therefore online reputation even for those companies that do not generate direct revenue from their mobile applications is critically influenced by the end user experience delivered by each application. Resolution Time Mobile application teams supported by a fully functional mobile APM solution can typically resolve problems with mobile applications 5 to 10 times as quickly as can teams without such a tool. Speedy resolution of problems directly affects Revenue, Retention, and Reputation. VII. Requirements for a Mobile APM Solution Due to the unique nature of the environment in which mobile applications run (the combination of networks, devices, operating systems, and application dependencies), an effective mobile APM solution must meet a unique set of requirements. Those requirements are: Built with a Focus upon Mobile Applications and their Environment Ensuring a great end user experience for users of mobile applications is requires a level of monitoring and instrumentation across the mobile environment that makes mobile APM into a whole company problem, and not a problem that a vendor can address with a team that produces something that ends up being one line item in a list of products. Mobile APM is sufficiently different from APM for server based solutions so that any vendor that tries to do both is going to have to reconcile differences between customer requirements and business requirements that will distract them from doing a great job at either server based APM or mobile APM. Built for Mobile Scale Whether the application of interest is for your employees, your customers and prospects, or for consumers in general, it is highly likely that your mobile applications will have more users then any other applications in your company. Monitoring this many devices running potentially more than one application of interest, with each application having dependencies upon multiple services makes mobile application monitoring into a big data problem. Rapid Time to Value The need to operate a big data back end in support of a Mobile APM solution is outside of the scope of interest and expertise for most organizations especially the Mobile APM teams that have to develop and support their applications in production. Therefore in order for the Mobile APM solution to be easily deployable and supportable, it should be offered as a SaaS service to the organization supporting the April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 11

12 applications. A SaaS based solution has the advantages allowing the customer to realize value from the Mobile APM solution more quickly, and allowing the customer to benefit from a long term reduction in the total cost of ownership of the Mobile APM solution. Full Stack Solution Server based APM solutions tend to focus just upon the container for the application (the Java virtual machine or its equivalent for other languages) and not upon the entire environment supporting that container. This approach is woefully inadequate for mobile applications as the focus needs to be upon the applications (and their dependencies upon other applications), the variety of operating systems and their versions, the variety of devices, and the variety of networks (including 3G, 4G and WiFi) through which devices connect as they roam from one network to another. Coverage of Crashes, Service Responsiveness, and End User Experience A mobile application that crashes frequently, has poor uptime or that delivers a less than compelling (fast) end user experience is going to create reputation, revenue and retention issues for the vendor of that application. Therefore an effective Mobile APM solution must monitor all of the crashes, the uptime and the user experience within the application, the operating systems, the devices, dependent cloud services, and the networks. Flexibility to Monitor an Enterprise s Entire Mobile App Portfolio Enterprises today often have both B2C and B2E mobile apps that might have been developed internally, outsourced, or be third party off-the-shelf apps. All these apps are critical to a company s bottom line and should be instrumented for performance monitoring. In the case a company has access to the app s source code the mapm tool should enable direct instrumentation of the app by having developers add in the necessary code and SDK. But in cases the company doesn t have access to the app s source code (as is the case with outsourced or third party apps), the mapm should be able to use app wrapping to auto instrument the app for performance monitoring. This means a company can monitor these third-party apps with no code changes, preventing costly rollbacks and additional customer support as a result of underperforming applications. Therefore, a flexible and effective mapm must be able to instrument apps by both adding an SDK to the source code or using zero-touch App Wrapping capabilities. Provide Actionable Diagnostics This is another area where mobile APM differs tremendously from APM for server based applications. With server based applications the focus is upon the code that runs in the Java virtual machine (or its equivalent for other languages) and the diagnostics are usually confined to a call stack that slows the slow or failing call within the code. The problem in the mobile device is much more complex as Full Stack Solution section directly above. This makes diagnostics into a much more difficult and important problem, and these diagnostics need provide detailed and actionable insights across a wide variety of issues that simply do not affect server based applications. For example of a bad user experience is cause by a combination of a particular network, a particular release of the mobile operating system and a particular release of the application itself, a traditional approach to APM will never find these dependencies across the stack, and never be able to provide actionable diagnostics across this stack. April The Virtualization Practice. All Rights Reserved. All other marks are property of their respective owners. 12

13 VIII. Summary and Conclusions Managing the end user experience of mobile applications is a business critical issue that directly affects the revenue, customer retention, and reputation of vendors and enterprises with mobile applications. Managing mobile end user experience is also a unique problem requiring a unique vendor who is dedicated to solving this problem across the entire set of environments that affect end user experience. The combination of the number of networks, the number of devices, the number and versions of operating systems and the dependencies of applications upon each other an upon cloud services makes this into a problem that is separate and distinct from the problem of managing applications on server platforms. For the above reasons, enterprises and vendors of mobile applications and services are urged to choose a vendor that specializes in solving these problems as opposed to someone who has simply migrated their server based solution over to mobile operating systems. About Crittercism Crittercism is the world s first mobile application performance management (mapm) solution, offering both error monitoring and service monitoring solutions. Crittercism products monitor every aspect of mobile app performance, allowing Developers and IT Operations to deliver high performing, highly reliable, highly available mobile apps. Monitoring over 1 billion monthly active users, Crittercism offers a real-time global view of app diagnostics and app errors across ios, Android, Windows Phone 8, Hybrid and HTML5 apps. The company s investors include Scale Venture Partners, Shasta Ventures, InterWest Partners, Opus Capital, Google Ventures, Accenture, Kleiner Perkins, and AOL Ventures. Crittercism is an AngelPad company Crittercism, Inc. facebook.com/getcrittercism linkedin.com/company/crittercism