The Challenges Facing Mobile App Testing Why 44% of defects are found by users Why read this paper? Testing is perceived as expensive, time consuming, and often considered a chore. However, it s vital that you actively do test to ensure your users have a positive experience when they use your mobile apps. Mobile app testing helps identify navigation and presentation errors, controls usage problems, and provides feedback about the app s general appeal and engagement level. Yet many apps are launched without sufficient prelaunch testing, so users not testers wind up as guinea pigs and discover the issues that arise. To help you avoid the potential disastrous fallout that will diminish, or worse, terminate user engagement and tarnish your brand, this whitepaper covers what things to look for in testing your mobile apps, and why it s worth the time and effort. If users don t like an app on the first try, they often never come back. Bad reviews in the app stores tend to linger, making it very hard for a tainted app to get attention. 1
Unfortunately, a lot of marketers launching mobile apps take shortcuts and skimp on testing or use a reactive approach and only worry about what users think once the app hits the market. Or they put far too much emphasis on time-to-market, citing competitive reasons and the rapid technological pace of mobile as justification to get their mobile app out there fast. Mobile apps should be tested proactively, not reactively. Consider this: a Localytics study shows that 24% of users try out a downloaded app only once. 1 That s because when users find fault with your mobile apps, they simply don t use them again. The goal of your testing efforts should not be just to find errors. Your rationale for testing should be to understand the quality of your app. Does it work? Does it function as it was designed? Will it meet the needs of your users so that they come back, again and again? Survey respondents said that in order to reduce mobile app defects, the following critical measures are needed: More functional testing 6 More device/os coverage 7 More performance testing 8 Failure to test before release is not an option: Perfecto Mobile, a leader in mobile application quality, released a benchmark survey this year, Why Mobile Apps Fail, which examined the challenges faced by testers and users, and how issues can be addressed to minimize defects in mobile apps before launching. The survey found that users and not testers identify almost half of mobile application issues. 2 In essence, that means users are the final QA step. Obviously this is not the desired course of action. That means you must be proactive in your testing, so you catch issues before the user experiences them. The report also identified the top three challenges in mobile testing: 1. Insufficient coverage (63%) 3 2. Lack of reliable automation (48%) 4 3. Needing more coverage to execute test scenarios than time permits (47%) 5 Source: Perfecto Mobile 2014 Benchmark Survey, Why Mobile Apps Fail, 2014 www.perfectomobile.com/whymobileappsfail Commenting on the ramifications of the user finding app errors, Perfecto Mobile CEO Eran Yaniv said, Enterprises spend months investing, planning and building an app to coordinate with the company s mobile strategy, and when it launches they want it to be successful and have the same quality experience across all mobile platforms for every unique user. Having users discover bugs and then abandon the app not only tarnishes the brand reputation, it diminishes the investment and value that went into building the app, and negatively impacts the business. 9 Testing before as opposed to after release: Usability testing is not just a backend activity; it should occur throughout the development process from initial design to just before an app is released to allow you to address issues. 2
By testing an app prior to its release and during development, it discovered in the early stages of development, you will not only save the higher costs of fixing them post-release but also avoid potential damage to your brand as a result of the bad reviews you will likely receive. You should also test at every phase of development and record user interaction. Monitoring mobile apps is essential to examining their performance; it s vital to monitor continuously, especially during performance testing, so you can isolate trends and differentiate between systems and root causes. Asking the right questions in testing can provide better insights: Opinion polls and their accompanying results are only as good as the questions asked. The same holds true for mobile app testing. You need to compile a comprehensive list of focused questions to ask users to get usable data that will help direct your app development. Here s a list of the types of questions you should be addressing when testing: On what platforms does this app work? On what OS and device should this app work? What is this app supposed to do? What specifications exist? What user documentation exists? Is there a log of existing bugs? What kind of data is processed by the app? Does the app integrate with external apps such as APIs or data sources? Does the app work with certain mobile browsers? What do existing customers say/think about the app? How are releases/updates made? User testing vs. beta testing: While any testing is both necessary and positive, beta testers get too familiar with your mobile app, so they develop a kind of white line fever or fixation on a single point and lack fresh eyes, making them less likely to spot usability problems. User testing, on the other hand, gives you feedback from typical users who aren t familiar with your app and have no bias or emotional attachment to it. These user tests help you understand the needs and reactions of normal users, the types of people who write reviews in the app store. Your user testing shouldn t be fixed on one type of user either. As a mobile tester, you need to adopt the mindset of a wide range of users and put yourself in their shoes. Some of the categories of users to consider are: Novice users Experienced users Fans Hackers Competitors The types of mobile app testing you should do: There are two parts of mobile app user testing you should undertake: Usability testing or a systematic observation under controlled conditions to determine how well users interact with the app on the mobile device Concept testing to assess the appeal/engagement level of the users before the app is released User testing isn t just about usability testing; you also need to consider the main functions of the app and ensure they re intuitive. Of course, you need to address whatever concerns users may have about the app. Guerrilla usability user testing when little or no budget is available Lab usability design phase testing to identify needs and user patterns Contextual usability testing to observe user involvement in the right environment Remote usability research with the user and researcher connected remotely in different locations A/B testing offer alternative versions of the app to different user groups and compare the results to determine which one performs better Feedback surveys 3
Here are other criteria you need to test: The buttons, icons, and controls; many users cite these as potential areas of confusion Emotional engagement; in other words, how do users feel about the app, can they quickly accomplish what they want to do without frustration, distraction or impatience, and will they use it again? Mobile apps are not cookie-cutter and your testing shouldn t be either: Mobile apps differ from Web applications in their specific purpose. As such, when an app is created and deployed, there needs to be more upfront planning specific to the utilization of the app not simply a scaled-down version of the website. Users will be using your mobile app under much different conditions than when they use your website. The mobile app must facilitate this usage with ease. The challenge you are faced with mobile apps is to first understand the logic, emulate the user, and then build logic into the script. Not only do you have to create realistic app testing scenarios that differ from each other but also take into account how user activity changes within the app itself. On the other hand, mobile apps communicate less over the network than Web apps, so while testing scripts is more complex, they tend to be shorter. Because apps are extranet, they re easier to test since you can use the cloud and emulate: Different geographic locations Various networks The bandwidth It s important to remember that mobile consists of two devices: smartphones and tablets. If your app is designed to run on both, you should run tests on both of them. The reason? A screen layout may look good on a smartphone but cumbersome or unappealing on a tablet, or vice versa. Additionally, user behavior on the two devices has subtle, yet distinct, differences. So many devices, so little time to test: One of the biggest challenges to testing your mobile apps is the wide variety of devices that are used to access your app; this is an important consideration you need to make when testing. While you can arbitrarily limit the number of device types you test on, you risk that the app might not work on a particular device and alienate those users. The best way to address this issue is to choose from one of three options: test exclusively using real devices, test exclusively with emulated devices, or use a combination of the two. The ABCs of A/B testing for ios apps: Testing apps for ios mobile couldn t be simpler. By using a Visual Website Optimizer (VMO) A/B testing solution, you can gain a better understanding of how your mobile traffic can be quickly optimized to impact your bottom line. It s easy and efficient because you don t need to have an understanding of HTML or coding, its point-andclick interface eliminates any need for IT resources and training, and streamlines the time it takes to initiate a test (in some cases as little as hours). A/B TESTING FOR ios NATIVE APPS To learn more about the Visual Website Optimizer for ios apps, view this video: http://www.youtube.com watch?v=lscj2w8qsrs 4
Testing updates in apps: Because mobile test devices often receive new software updates, are overhauled as a component part of a new model device release, or feature a new mobile OS release or update patch, mobile testing must account for these changes. As such, you need to be mindful of whether: The latest device/os change needs to be tested in the context of the latest app Older, existing apps will function properly in the context of the new changes Is your gauge of success calibrated properly? Another issue marketers need to be aware of is that the measure of how successful your app is, isn t a factor of quantity; it s quality and value. As such, app engagement is a much better metric than the number of downloads; it gives developers a much better sense of how much value the app provides users. A big component of user engagement with your mobile app relates to the user experience (UX). That s why it s very important to gauge how your users perceive the UX. Here are some important elements to consider in testing your app as it relates to UX: 1. Is your app icon appealing enough to persuade the user to open the app? Your app name should explain what it does. 2. Will the user understand what problem the app solves or what need it fulfills quickly? 3. Does your app provide value in a few minutes/ seconds? 4. Does the app perform tasks it was designed to do or tasks it wasn t? 5. Does the app have an interface that is clear and engaging? 6. Do users go back when they aren t supposed to? 7. Are users impatient and hit keys multiple times? 8. How does the app perform when being used consistently; does it crash or is it sluggish? 9. Is it possible to break or hack the app? 10. Where does the app redirect the user? 11. Is communication/marketing consistent with the app s function, design and content? 12. Is the sign-up process easy and does it inte grate with social media channels? 13. Have you created a reason for the user to return to the app? To ensure a better user experience and improve overall app quality, you should: Employ a cloud-based device-as-a-service to help your teams focus on testing, not managing devices Use an automated tool that can support continuous, unattended testing, so you can do more testing with fewer resources and greater consistency Deploy a monitoring system that provides early detection of performance issues before they impact users Mobile app testing is an investment that pays off: Testing is often considered logical, planned and predictable, full of processes, test scripts, passes and fails. But this is not the full extent of what it encompasses. You need to do more than just test cases and find bugs; you need to find the issues that matter to users, to provide information of value that determines when it is okay to release an app, and that mitigates any issues beforehand. By testing apps within your target audience during development, you are more likely to create an app that is not only wanted by the user but also one that is engaging. Testing helps you avoid navel-gazing and looking at your own needs only; it gives you the insights into what your users want and the type of app that will enhance your brand. SOURCES: 1 App engagement is a much better metric than downloads, Mobtest; 2, 3, 4, 5, 6, 7, 8, 9 Perfecto Mobile Launches 2014 Benchmark Survey That Finds 44 Percent of Mobile App Errors Are Detected by Users, Business Wire; Testing mobile apps: Strategies for monitoring and user emulation, Caroline de Lacvivier, TechTarget; Testing Strategies and Tactics for Mobile Applications, Keynote Systems; Testing Mobile Apps, Empower Internet Services; Expert Q+A Mobile App Testing, UserTesting; A Field Guide To Mobile App Testing, Rosie Sherry, Smashing Magazine; Is your mobile app ready for launch?, Mobtest; Mobile App User Testing, DMI. 5
About the author Gregg Guzman Gregg focuses on driving campaign testing strategy and consumer research for all of Ansira s clients. For six years, he has developed a reputation for being a data-driven marketer, and has increased conversions for multiple clients, including Direct Energy, Jiffy Lube, Michaels and Wholly Guacamole. Gregg s experience in paid media, HTML development, site usability, campaign strategy and e-commerce optimization has paved the way for robust results at Ansira. He graduated from Oklahoma State University with a degree in marketing and management. Ansira: The Data-Driven Results Marketing Agency Ansira continually engages customers along the entire consumer life cycle and purchase funnel, using conversion optimization to drive higher sales at a lower cost. Our customer-centric approach applies data and analytics to every interaction, helping clients maximize customer profitability and optimize customer experiences. Designed to adapt with our clients ever-changing consumer base, we take the guesswork out of marketing. Our repeatable programs integrate all channels and media to steer consumers along the path to purchase pre and post leading to predictive ROI modeling. While traditional conversion optimization measures only one phase of the conversion process, we measure each and every consumer touchpoint to determine the best way to maximize marketing spend. Every single program is viewed as an opportunity to test, learn and optimize. Using multivariate and A/B testing, Ansira continually scrutinizes a variety of factors, including landing page effectiveness, messaging and targeting methodologies. We take full advantage of the latest in technology, message delivery and measurement. Our Optimization Science engineers design our programs, focusing on predictive modeling and ROI, and monitor results in real time, making changes to campaigns whenever necessary all while the campaigns are live. We are a privately held corporation with offices in Dallas, Houston and Portland. Contact Ansira at www.ansira.com 6