App user analytics and performance monitoring for the business, development, and operations teams CA Mobile App Analytics for endto-end visibility CA Mobile App Analytics WWW.OVUM.COM
Written by: Michael Azoff Published June 2015, Ovum MANAGEMENT SUMMARY INTRODUCTION The rush risks are high and will not only affect the usability of a company s mobile app but can greatly influence the reputation of the business itself. In today s application economy smart mobile devices and their apps have completely transformed the way consumers and businesses interact. Mobile app downloads today number in the hundreds of billions, and new apps are continually being created in this booming economy. For example, the Apple App Store attracts tens of thousands of apps each month and the mobile app economy is worth billions of dollars, and growing. Against this increasing use and acceptance of mobile apps within society there is pressure on businesses to feed into this app economy or be left out altogether. There is a danger that businesses rush into producing mobile apps that are below par in user experience and performance. This rush risks not just use of the mobile apps in question but also the reputation of the business itself. Research has shown that app users today have high expectations, comparing their experience with the best apps on the market, and if their experience is poor, such as due to a slow start up screen, then they will not only stop using the app but they will not return to the brand and vendor. Therefore, businesses must realize that their mobile app is today s equivalent of the shop window and the user s first impression of their mobile app is paramount to their success. The mobile app end-user experience is naturally a reflection of the app s look and feel, but most critical is the overall app usability. App performance is a major factor in determining the usability of a mobile app. Most mobile apps depend on communication with back-end systems. Any communications issue, at any point in the process can have serious repercussions for the entire business. Therefore businesses need to address mobile app performance across all phases of the app lifecycle: development, production, and operations. CA Technologies has a solution to enable customers to develop and maintain 5-Star mobile applications: CA Mobile App Analytics (CA MAA). Firstly, it provides the business owner with analytics on app usage, user experience, and other design aspects of the app that are relevant to the app s business owner. Secondly, the operations team has app performance analytics, helping them maintain the application to ensure the application is responsive at the level required for the best user experience. Additionally, when combined with CA APM, it can provide a complete end-to-end mobile app performance management (APM) solution designed to cover every interface in today s complex transaction environments from the mobile app all the way to any back-end system, including mainframe systems. Finally, to meet the needs of the developers, CA MAA provides detailed crash diagnostics, including source code level detail, together with crash analytics enabling the developers to understand the trends and patterns that are associated with app failures. KEY MESSAGES Businesses are risking their reputation with poorly performing mobile apps, and therefore need to invest in tools that provide visibility into app performance. Ovum. This White Paper is a licensed product and is not to be photocopied Page 2
Mobile app performance management is essential to ensure that the user has a 5-Star rated mobile experience. Businesses need to ensure revenue generating apps are optimal by monitoring mobile app transaction response times, and behaviour of end users in the apps. Developers need to have detailed crash metrics that can pinpoint, down to the problematic line of source code, the cause of individual app crashes, as well as revealing the broader patterns of failure such as common device type, OS version, carrier, and more. CA MAA is a solution that addresses the needs of the key mobile app stakeholders: developers, IT operations teams, and the business. Using CA MAA with CA APM lets enterprises gain end-to-end visibility of transactions spanning mobile front-end to back-end systems that include servers, networks, and mainframes. MOBILITY IS CHANGING HOW ENTERPRISES DRIVE BUSINESS CONSUMER AND BUSINESS MOBILE END-USERS Mobile app usage, targeting consumers in the early mobile period with games and simple mashups, for example that combined mapping and location sensing with useful information on nearby restaurants and other retail outlets, has progressed today to embrace every conceivable business and public service. It would be surprising to find any large enterprise without a customer facing mobile app, and often small to medium sized enterprises (SMEs) are first to grasp a mobile app opportunity. Over 50% of today s workforce leverage more than one device during a typical work day (e.g. Laptop/Desktop + Mobile Device). This defines mobile apps as a disruptive technology (see Figure 1). Figure 1: How many different devices do you use during a typical working day? Source: Ovum, based on 2014 survey. For example the financial sector has recognized mobile apps as an essential channel given the shift towards more time spent on mobile devices by consumers. Banks and insurers have apps that allow bank accounts to be accessed on the move and for insurance claims to be filed on the spot of an incident. Ovum. This White Paper is a licensed product and is not to be photocopied Page 3
A multi-national bank needed end-to-end transaction visibility from mobile to mainframe. CA Technologies has an example of a customer, a multi-national bank, which needed end-to-end transaction visibility from mobile to mainframe in order to achieve and maintain a 5-Star application rating. This top-20 bank with over 100 million customers wanted to become the leader in digital banking, supporting a mobile payment system for millions of mobile banking customers. It relied on another software vendor to provide insight into its banking app performance but this gave insufficient depth of analysis, for example, there was a lack of visibility into business metrics such as retention, and similar lack of visibility into developer and operator oriented crash and performance metrics. This customer wrapped CA MAA around its Android app for a rapid proof of concept of the capabilities of the solution. It subsequently integrated the product SDK into the source code of its apps. This enabled the client to capture metrics such as user retention and stack traces, gaining end-to-end visibility from mobile to back-end mainframe (in combination with CA APM). This in turn enabled the bank to deliver better end-user experience and therefore improved customer retention. ENSURING AN EXCELLENT END USER EXPERIENCE The business of mobile apps is increasing, with estimates running into billions of dollars spent in recent years, and this annual expenditure is expected to expand in the years ahead. With such a growth in the business the competition for space on a user s mobile app screen is intense: the average user has just 26 apps on their device (out of over 2 million available) and to become one of the 26 apps a user uses on their mobile device, an app has to first load fast and then respond fast in any given transaction. For example CA Technologies quotes research that shows 57% of users give up on an app (whether loading or in use) after 3 seconds of inactivity. The end-user experience therefore is not just about the look and feel but as much about the app performance. For most business and many consumer apps there is a need to connect the device-side with a back-end system to complete transactions: such as to pay a bill, add an item to a shopping cart, submit an insurance claim, etc. The performance of the app therefore depends on the total round trip taken by the transaction, and therefore understanding issues around mobile apps requires an end-to-end perspective, with monitoring and analytics implemented at every stage of the transaction round trip. ANALYTICS ACROSS MULTIPLE MOBILE APP PROCESSES Fundamental mobile app development practices involve a series of processes. In Figure 2, the inner process typically leverages agile iterative development that is common in mobile app development (discussed below). A security process should be wrapped around app development insuring that it is baked-in as part of the overall lifecycle at each phase of development and deployment. Finally, an overarching DevOps process provides a continuous delivery model through orchestration and automation of the project build, testing, and deployment processes. Throughout these lifecycles analytics and performance metrics provide visibility into the app s behavior across the key domains of business, development and operations. Ovum. This White Paper is a licensed product and is not to be photocopied Page 4
Figure 2: The multiple mobile app development and deployment processes Source: Ovum Agile development is ideal for mobile apps Mobile apps in the consumer space typically have short application lifecycles, with frequent updates, whether to do with the adding of new features, device and mobile OS manufacturer updates, or for reasons to do with increasing user engagement. Agile working methods are ideal in this scenario. Often a business owner will spot a market opportunity and, using agile methodology, is able to update or release a new mobile app in a matter of weeks. However the typical App Store publishing process does not readily facilitate an excessive frequency of app updates, and moreover consumers find excessive app updates inconvenient. Therefore it is desirable to optimize the effectiveness of each new app release, and minimize the number of times an app must be pushed through a public App Store. CA MAA can help in the optimization process by providing insights into the users app experience, thereby enabling an enterprise to better understand what is working well or less well. In order to allow for even greater agility, many enterprises are using hybrid app development techniques, whereby significant parts of the app s user interface are rendered under control of RESTful APIs together with typical browser application development techniques such as HTML5, CSS, and JavaScript delivered from back-end servers. Agile development, supported by DevOps technologies for build, test, and release automation, enables continuous delivery for these back-end components supporting the enterprise s mobile apps. CA MAA adds value to this process by providing real-time usage metrics to gain feedback on each new production release as soon as possible. The app s supporting components can then be revised, perhaps on a weekly basis, with Ovum. This White Paper is a licensed product and is not to be photocopied Page 5
each iteration refining the user experience. The weight of end user behaviour then drives the direction development will take. MOBILE ANALYTICS AS A STRATEGIC BUSINESS INVESTMENT CA MAA AND CA APM FOR END-TO-END VISIBILITY CA MAA in combination with CA APM provides end-toend visibility of business transactions that originate in the mobile app and get fulfilled by back-end systems. Agile and DevOps methodologies are helping drive innovation throughout the mobile app lifecycle at a faster pace, while the operations team manages the production environment s stability and ensures risk levels are acceptable. Once in production there is a need for insight and visibility into mobile app usage. CA Technologies recommends an analytics implementation strategy as follows: Mobile end-to-end performance monitoring from the mobile device to back-end systems, including mainframes. Metrics at every stage of the application lifecycle, from development to operations to business analysis. Deep visibility with in-depth performance metrics and analytics for incident troubleshooting and monitoring end-user behavior. CA MAA in combination with CA APM provides end-to-end visibility of business transactions that originate on the mobile app and get fulfilled by back-end systems, including mobile-to-mainframe visibility. Business transactions typically span multiple views, web services, networks, and other infrastructure components. CA MAA has a tag and follow technology that allows monitoring of critical data as it moves from mobile device to back-end systems and returns. This technology is able to inform all the stakeholders with insightful analytics: app usage and retention data for the app owner; detailed performance data for the operations and support groups; and app session trace data and crash analytics for the app developers. The net result is that app interactions that lead to specific business outcomes such as View Product, Add to Cart, and Checkout, can be traced and analysed. For example, the transaction "View Product" is shown in Figure 3: the mobile transaction is tagged with CA MAA and followed through using CA APM. Ovum. This White Paper is a licensed product and is not to be photocopied Page 6
Figure 3: Tag-and-follow technology allows end-to-end transaction visibility Source: Ovum CA MAA S VALUE TO INVESTMENT RATIO CA MAA is available as a cloud based Software as a Service (SaaS) or on-premise offering (see Figure 4). There are two ways in which mobile app monitoring can be enabled: one is through the inclusion of a simple Software Development Kit (SDK), another is using application wrapping. Available on Android only, application wrapping enables the operations team to quickly wrap their app with CA MAA using what CA Technologies calls Smart Containerization. The wrapping approach does not need the involvement of developers or any additional coding and once wrapped, the application will instantly provide analytics to CA MAA. On the other hand, the CA MAA SDK can be installed for both ios and Android apps with a few easy Ovum. This White Paper is a licensed product and is not to be photocopied Page 7
CA MAA SDK can be installed for both ios and Android apps with a few easy steps, in most cases taking only minutes to implement. steps, in most cases taking only minutes to implement. CA MAA can be customized to meet the needs of any customer; a few examples of analytics customization include: Logical transaction grouping allowing sequences of related network requests to be grouped together and instrumented as a single, logical unit of work. Post-crash user feedback reporting allowing the developer to request extra information from the user when an app crashes. Custom event data reporting allowing site or app-specific custom data to be reported to the CA MAA console. As shown in Figure 4, the CA MAA analytics metrics, whether for a cloud or on-premise deployment of the solution, are collected by a multi-tenant system via a secure API gateway. CA MAA provides a single console through which the whole range of analytics metrics can be visually inspected and drilled into for finer detail. All the data collected by CA MAA is accessible for custom analysis through the CA MAA console or by leveraging secure reporting Application Programming Interfaces (RESTful APIs). Figure 4: CA Mobile App Analytics Source: CA Technologies ADDRESSING STAKEHOLDER NEEDS BUSINESS NEEDS: UNDERSTANDING THE USERS The product owner wants to deliver value to both the business and the end user. Without the analytics on user behavior and hence to drive the business function, as well as to help guide the app design, it becomes a matter of guesswork and trial and error. This can increase the time to successful adoption of an app, and it may also kill adoption as early versions of an app are too far off the mark. Mobile app analytics are therefore essential to ensure the business is achieving its aims with its apps and also ensuring apps are not risking the reputation of the business with poor performance. Ovum. This White Paper is a licensed product and is not to be photocopied Page 8
Mobile app analytics are also helpful in mining the data that an app generates. For example, consumer facing apps contain valuable insights on consumer product or service preferences, app usage patterns, responsiveness to offers and incentives, which geographies users originate from, and so forth see Figure 5. Figure 5: Usage analytics provide business insights Source: Ovum Using CA MAA, business analysts can gain deeper visibility into all the stages of app usage, from initial user engagement to daily user activity, as well as helping maintain high user retention rates. Using CA MAA, business analysts can gain deeper visibility into all the stages of app usage, from initial user engagement to daily user activity, as well as helping maintain high user retention rates. Amongst other things, the business oriented features available in CA MAA offer organizations the ability to: Gain visibility into performance of key business services. Identify popular business flows in the app; capture user flow through apps. Gain insight into mobile app user engagement, activity, and retention. Segment and analyze usage data by app version, network (e.g. carrier/wifi), location, platform, and OS. Track and report stickiness based on users revisits. Tag a business event (e.g. Gold Customer, shopping cart with more than 100 items) and look at impact on the top line. DEVELOPER NEEDS: DEEP DIAGNOSTIC INFORMATION Developers need to understand the users app experience, and when issues do occur on a mobile app they need to receive timely and relevant diagnostic information to troubleshoot and correct defects. For example in the case of an app crash they require detailed crash metrics that can pinpoint, down to the problematic line of source code, the root cause of any issues. In addition they need analytics to uncover the trends that affect app failures, such as answering the questions: Is this new version of the app more prone to crash than older versions? Ovum. This White Paper is a licensed product and is not to be photocopied Page 9
Is my app less stable on this new version of Android / ios? Does my app crash more frequently on certain manufacturers device models? Does my app only fail in certain network configurations, for example only when the device is on WiFi or only with certain carriers etc.? These are not questions that can be answered by analysing crash reports in isolation. OPERATIONS NEEDS: STABILITY AND FAST APP PERFORMANCE Operations teams need to identify and isolate network or data center issues when they occur. They need alerts when problems such as app crashes occur and need to understand which back-end services are affecting app performance. The operations team is concerned with ensuring mobile apps are performing optimally and without side effects that reduce the capability of the IT environment. The end-to-end approach taken by the CA MAA and CA APM solution gives operators the information to satisfy these needs. Increasingly, organizations that are agile in development are also adopting DevOps ways of working in operations. This leads to the creation of a continuous deployment pipeline, so that a change in source code can travel the pipeline, possibly with a manual testing stage as part of the process, and go live into deployed apps. CA MAA allows operations to measure performance metrics and ensure that the automated continuous deployment is performing as required. Gaining deeper visibility is critical to managing app performance, and using CA MAA, operations professionals can segment performance. Gaining deeper visibility is critical to managing app performance, and using CA MAA, operations professionals can segment performance by: app version, carrier, geography, operating system, network, device type, and custom events. CA MAA features of interest to operational professionals include the ability to: Segment app performance by version, carrier, geography, OS, and device. Gain visibility into back-end services affecting the app performance. Analyze performance of networks and locations to correlate with app performance. Isolate and rule out domain, network, and data center issues. Identify mobile app hotspots: usage, performance, coverage, and issues. And provide: Real-time alerts on threshold violations impacting SLAs, user experience, or revenue and the ability to prioritize alerts. CONCLUSIONS It is essential in today s highly competitive application economy to be able to implement systems that can help develop and maintain 5-Star mobile applications. CA MAA offers advanced and in-depth performance analytics that can support the range of stakeholder roles involved in a mobile app, from the line of business to development and operations. When combined with CA APM additional value is obtained through the capability to track and monitor transactions from end-to-end. Many enterprises today operate a mobile first Ovum. This White Paper is a licensed product and is not to be photocopied Page 10
strategy, with back-end data sources exposed through web services. The combination of CA MAA and CA APM provides the end-to-end analytics which are essential in understanding the health of mobile app transactions as they cross multiple tiers. As mobile apps increase in importance during daily work, businesses will need to ensure they reduce risk and provide their business owners and DevOps teams with the visibility to ensure optimal app performance. Most mobile apps are not standalone; consumer facing enterprise mobile apps are typically providing an alternate channel for business services that were primarily targeting the desktop/laptop browser. Examples include banking apps and insurance claim apps; these apps make calls to back-end systems to fulfil business transactions, and an end-to-end monitoring capability is necessary to gain visibility across business transactions, to not only see the impact on app health but also on revenue. A trial of CA MAA may be arranged via the following link: http://trials.ca.com/us/ca-mobile-app-analytics.aspx. THE APPENDIX AUTHOR Michael Azoff, Principal Analyst, IT Infrastructure Solutions michael.azoff@ovum.com OVUM CONSULTING We hope that this analysis will help you make informed and imaginative business decisions. If you have further requirements, Ovum s consulting team may be able to help you. For more information about Ovum s consulting capabilities, please contact us directly at consulting@ovum.com. DISCLAIMER All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of the publisher, Ovum (an Informa business). The facts of this report are believed to be correct at the time of publication but cannot be guaranteed. Please note that the findings, conclusions, and recommendations that Ovum delivers will be based on information gathered in good faith from both primary and secondary sources, whose accuracy we are not always in a position to guarantee. As such Ovum can accept no liability whatever for actions taken based on any information that may subsequently prove to be incorrect. Ovum. This White Paper is a licensed product and is not to be photocopied Page 11