White Paper Bridging the Essential Gap between Continuous Quality and Crowd Based Testing 1
Contents Introduction... 3 Testing Lifecycle... 4 Testing typologies... 4 Functional Testing... 4 Manual vs. Automated... 5 Inhouse/Cloud vs. Crowd... 5 Usability Testing... 6 Testing Internally... 7 Testing with the Crowd... 8 Testing Tips: Interoperability... 8 Conclusion... 9 About Perfecto Mobile... 9 About Testbirds... 9 2
Introduction According to Gartner, more than 2.4 billion tablets and mobile phones will be shipped by the end of this year 1. The Apple App store and Google Play store each offer more than one million apps for users to download to their devices. So what keeps users from sticking to a specific app rather than switching to the next one? The basic functionality of an application needs to run flawlessly and solve the consumers need in an intuitive fashion. http://opensignal.com/reports/2014/androidfragmentation/ Consumers engage with mobile applications in a completely different way than they engage with websites. Given the evergrowing diversity of mobile devices, operating systems and form factors (e.g. smartphone, phablet, and tablet) in the market, mobile testing has never been more important to ensure that mobile application users have a positive experience. This whitepaper discusses the fundamental building blocks of efficient functional and usability testing for mobile, across the entire mobile application life cycle. These include, among others, the importance of developing an inhouse testing plan, the early incorporation of consumers into the process and some key tips for interoperability and using automated testing across platforms. This whitepaper should serve as a tool to guide your mobile development and testing teams with the most important aspects to create a holistic mobile testing strategy. 1 http://techcrunch.com/2014/07/06/gartnerdeviceshipmentsbreak24bunitsin2014tabletstoovertake- pcsalesin2015/ 3
Testing Lifecycle Testing typologies 1. Prototyping: Mockups, Click Dummies 2. Development a. Unit Testing b. Sanity/Smoke Testing 3. QA a. Regression: After updates/changes b. Interoperability: Voice / SMS interrupts, Notifications, Battery / Cable Removal c. Performance: CPU Usage Testing, Network Usage, Page Render time or Activity Render time d. Stress Testing: Robustness, Availability, Error Handling e. Compatibility: Device and OS diversity f. Usability/User Experience: Expert Review, Studies, Competitive Analysis 4. Production a. Monitoring: Engagement, Activity, Retention Rate Testing can, and should be, implemented during different stages of development as well as after the release of an application. This whitepaper will highlight the important aspects of the above typology, grouping them into two main categories: Functional Testing and Usability Testing. Functional Testing Never before has functional testing been as challenging as it is in the mobile app industry. An ever- growing landscape of devices and OS versions requires a complex matrix of testing factors in order to 4
perform functional testing sufficiently. It is key for your testing team to identify the device matrix relevant for your user base. Choosing the sufficient and right devices for your testing matrix relies on several aspects: 1) Your market vertical (Banking, Healthcare, digital media, etc.) 2) Your own production analytics realize which devices are mostly used by your endusers to access your servers and your apps and on which OS versions 3) Market dynamics do not lose sight of the market evolvement and innovations new smartphones, tablets and even wearables pop up periodically and need to be considered in your overall device matrix. Manual vs. Automated Testing Ideally, manual and automated testing go hand in hand throughout development to maximize efficiency and to cover the various use cases of the application under test. In mobile, the need for manual interoperability testing (IOP) is an essential part of testing. Mobile app vendors must know how the application will react when incoming events such as messages, calls etc. occur while the app is running, as well as when the network condition changes. At the early stages, automated unit and sanity testing should be used to test the basic quality of your code. Six to eight musthave devices are usually enough at this early stage 2. Later on, regression, performance and stress testing should follow with the major devices of the respective operating system, comprising around 1215 devices. The above number of devices should be aligned with the relevancy to your market segment and geography (retail, banking, travel, etc.) as well as meet some form of analytics to get insights into what devices are mostly used by your customers. Because the mobile market is so dynamic, the OS versions and device mixes available for testing need to relate to the trends and changes in the market at a specific time. Inhouse/Cloud vs. Crowd Device diversity poses a major challenge for testing mobile applications. The combination of devices and operating systems can be endless. Inhouse testing is best suited to cover the major devices used in a specific market or region. Compatibility testing, however, requires the availability of the most recent models as well as older ones. This is why, after the inhouse testing team feels confident about the quality of its application, an external crowd should test for compatibility and general functionality under reallife conditions. CROWDTESTING Crowdtesting is software testing combined with the principle of crowdsourcing. Crowdtesting therefore uses the collective intelligence of the global internet community to test websites, mobile apps, games, and enterprise software in order to get rid of bugs and optimize usability. The end customers, who are based on a desired target group, test under real life conditions on their own devices even before the release. This step is a complementary test method to inhouse cloud testing and addresses the need to cover complex and maybe unforeseeable user behavior and network conditions as part of quality assurance. Apple retracted its ios 8.0.1 shortly after users reported that their new iphone 6 failed to establish network connections after the update 3. A perfect example of critical issues that only occurred after releasing the software. Through crowdtesting, virtually all devices available on the market become accessible for testing. This comes very close to the actual usage of the application after its release. While your inhouse 2 3M methodology for choosing the right mobile devices: Perfecto Mobile White Paper 3 http://support.apple.com/kb/ht6487 5
team tests the application against expected behavior, the crowd does the unexpected. They are not familiar with the functions of the app, have a fresh eye and are motivated by a payperbug system. A typical bug test could look like this: Android App, 30 Testers (20 consumers, 10 professionals), explorative Testing guided by typical use cases A mix of consumers and experts has proven to yield the best results. If necessary, each participating tester crosschecks defects on his own device. This is called Bug Approval and offers additional insights into defects that need to be addressed immediately while others might only occur on very specific setups. In order for a crowdtesting project to be successful, extensive project and test management during the testing phase is crucial. Testers can work remotely from their homes or anywhere else. They document their work through written documentation, screenshots or screencasts. This is important for two reasons: 1. Since the testers are not inhouse, documentation is the only way to ensure that there is sufficient test coverage. Was everything that was part of the initial test really tested? 2. If issues are found, they have to be traceable and reproducible. How did the user get to this point and exactly what is the issue? Depending on your internal resources, you can either manage crowdtesting projects yourself or outsource it as a whole. It makes sense to start out with at least one managed crowdtesting project. It is then possible to compare the effort and cost with a test done in selfservice. Usability Testing As user experience becomes key to business success and app adoption, UX testing ought to be part of any mobile app test plan. 6
Usability Performance Failed channel shift Achieving high app usability can be achieved through Single User Performance testing early in the development cycle, which includes vitals testing (CPU, Battery, memory leaks etc.) as well as network condition testing (latency, packet losses, behavior in various network types like 2G, 3G, LTE, Wifi). UX testing can also be done with proper adherence to the platform vendor s design guides (Apple, Google). The important thing to keep in mind that the mobile app needs to: Function well on various devices, OS versions Provide good user experience from a performance perspective Be intuitive to the users (easy navigation, meaningful error messages) Solve a problem or meet the need of its users Adhere to the platform vendor guide lines (do not reinvent the wheel) endusers expect UI that they know from other apps, provide them similar experience Testing Internally When going to test usability with an internal cloud vendor, the cloud platform should be flexible enough to be hosted either on premise, remotely or through a shared public cloud, the values returned to the application team are as follows: Device management and governance headache removed the app team can focus on its tasks without needing to worry about devices, upgrades, security. Clean room environment for variation testing like network virtualization testing to assess the impact on the enduser experience 7
Ease of debugging and collecting logs/data/screenshots and videos to reproduce usability defects App test team has access to the same set of devices under test for further enhancements and defect fix validation Ability to implement agile continuous integration practices for ongoing testing With this platform and capabilities, development/test teams can really focus on their app developmenttestingdeployment activities while getting all of their infrastructure as a service in an assured, predictable and managed way. Testing with the Crowd Crowdtesting enables companies to gather insights and feedback from their prospective users at any time during the development process. This is relevant in order to tailor an application to the specific needs of your target group. A wellperforming and functioning app could still get negative reviews if it doesn t address the user s needs. The added values of crowdtesting returned to the application team are as follows: Selection of specific demographics through more than 50 criteria (education, hobbies, online behavior etc.), offering access to any target group Access to specific users at any time, which reduces testing times Applicable in all stages of the development process in order to ensure development close to the user s needs: o Prototyping o Competitive analysis o Beta Testing o Usability study Setup according to your requirements (Questionnaires, Think aloud videos etc.) Opening up your application to a select crowd of users before launch increases chances of success after release tremendously, given the test setup is reliable and their feedback is incorporated. Confidentiality agreements as well as high security standards prevent a leak of information before launch. Testing Tips: Interoperability Does your app react to network events, device functionalities or other applications? The following tips are helpful when testing inhouse or with the crowd. ü Minimize app to use other disruptive apps, then return back to original app ü Change internet connection while testing: Wireless, LTE, 3G ü Make/receive calls while using the app ü Play music/use camera while using the app ü Switch flight mode/gps/bluetooth on & off at various times ü Change location (if GPSfunction is used) ü Change screen orientations (Landscape/Portrait) ü Install, uninstall, upgrades test all of these with and without memory card on the device (some devices requires apps to be installed only when an SD card is in the device). Perform these operations on various mobile OS s. 8
ü Language For localized apps, change device language and see whether it impacts your application. Conclusion Releasing a mobile app or several apps is always a great challenge, a challenge that continues to become more complex as mobile devices and OS platforms progress with innovative features. Mobile app vendors need to keep in mind that enduser expectations are also rising you need to be on top of the market and react quickly to changes. A version release to the market is not a oneanddone release; it requires ongoing support, maintenance and attention to remain relevant and retain your users. Therefore trying to leverage both the cloud which enables real enduser devices elasticity in a secured, managed and collaborative manner to your business & crowdtesting which complement the cloud with enduser exploratory as well as ondemand functional and usability testing as part of your testing strategy is an efficient method to keep up with the above challenges. It allows you to scale as both your app and its user base grow, while having complete visibility into your enduser experience. In addition, an evergrowing device landscape can be addressed with a combination of cloud monitoring solution and crowdtesting (Pre and Post production). About Perfecto Mobile Perfecto Mobile is transforming the way enterprise organizations go mobile, enabling them to develop, test, deploy and monitor their mobile applications and services and gotomarket with confidence. Perfecto Mobile s cloudbased MobileCloud Platform and endtoend mobile quality product suite enables users to remotely access a large selection of real mobile devices connected to local cellular networks around the world and leverage them throughout the mobile application delivery lifecycle from development, functional and performance testing to monitoring and support. The MobileCloud is available either as an enterprise private cloud or a sharable public cloud. More than 1,500 customers, including the top Fortune 100/500 across the banking, insurance, retail, telecommunications and media industries rely on Perfecto Mobile to optimize mobile timetomarket, customer engagement, risk mitigation and costs and continuously serve their mobile users with confidence. For further information about Perfecto Mobile s testing services, please visit http://www.perfectomobile.com. Follow us on Twitter @perfectomobile, Facebook and our blog to get realtime updates. About Testbirds Testbirds is a crowdtesting service provider headquartered in Munich, with offices in the UK, the Netherlands and Hungary. The company specializes in the testing of mobile apps, websites and other software for the entire device landscape (mobile, desktop, smart TV, wearables etc.) on all major operating systems. Testbirds uses crowdtesting to facilitate software testing in today s fastgrowing world of technology. They hold a worldwide network of testers (crowd) consisting of experts as well as consumers to test software under real world conditions. The crowd finds bugs and gives critical usability feedback, ensuring the highestpossible user experience before launch. 9
Extensive device ecosystems as well as increased user expectations challenge conventional testing methods. Testbirds offers indepth and costeffective quality assurance for web applications and mobiles apps through the crowd on any device, with any target group, at any time. Today, crowdtesting is fast becoming a necessary tool for developers to help improve functionality and overall usability of any application. For further information about Testbirds crowdtesting, please go to http://www.testbirds.com. Follow us on Twitter @Testbirds, Facebook and our blog. 10