White Paper Bridging the essential gap between Mobile Cloud and crowd based testing 1. Introduction 2. Testing Lifecycle a. Testing typologies (Functional, Usability) b. Functional Testing: Manual <-> Automated In-house <-> Crowd c. Usability Testing Testing internally Testing with the Crowd 3. Testing Tips 4. Conclusion 5. About Perfecto Mobile and Testbirds
Bridging the essential gap between Mobile Cloud and Crowd based testing 02 1. Introduction According to Gartner, more than 2.4 billion tablets and mobile phones will be shipped by the end of this year.¹ The Apple App Store and Google Play Store each offer more than one million apps for users to download to their devices. So what keeps users from sticking to a specific app rather than switching to the next one? The basic functionality of an application needs to run flawlessly and solve the consumers need in an intuitive fashion. Consumers engage with mobile applications in a completely different way than they engage with websites. Given the ever-growing diversity of mobile devices, operating systems and form factors (e.g. smartphone, phablet, and tablet) in the market, mobile testing has never been more important to ensure that mobile application users have a positive experience. This whitepaper discusses the fundamental building blocks of efficient mobile functional and usability testing across the entire mobile application life cycle. These include, among others, the importance of developing an in-house testing plan, the early incorporation of consumers into the process and some key tips for interoperability and using automated testing across platforms. This whitepaper should serve as a tool to guide your mobile development and testing teams with the most important aspects to create a holistic mobile testing strategy. http://opensignal.com/reports/2014/android-fragmentation/ ¹ http://techcrunch.com/2014/07/06/gartner-device-shipments-break-2-4b-units-in-2014-tablets-to-overtake-pc-sales-in-2015/
Bridging the essential gap between Mobile Cloud and Crowd based testing 03 2. Testing Lifecycle Testing can and should be implemented during different stages of development as well as after the release of an application. This whitepaper will highlight the important aspects of the above typology, grouping them into two main categories: Functional Testing and Usability Testing. a. Testing Typologies Prototyping Development Quality Assurance Production Mockups Unit Testing Regression Monitoring Click Dummies Sanity/Smoke Testing After updates/changes Interoperability Engagement Activity Retention Rate Voice/SMS interrupts Notifications Battery/Cable Removal b. Functional Testing Never before has functional testing been as challenging as in the mobile app industry. An ever-growing landscape of devices and OS versions requires a complex matrix of testing factors in order to perform functional testing sufficiently. It is key for your testing team to identify the device matrix relevant for your user base. Choosing the sufficient and right devices for your testing matrix relies on several aspects: Performance CPU Usage Testing Network Usage Page Render time Activity Render time Stress Testing Robustness Availability Error Handling Compatibility Device and OS deversity 1. Your market vertical (Banking, Healthcare, digital media, etc.) 2. Your own production analytics realize which devices are mostly used by your end-users to access your servers and your apps and on which OS versions Usability User Experience Expert Review Studies Competitive Analysis 3. Market dynamics do not lose sight of the market evolvement and innovations new smartphones, tablets and even wearables pop up periodically and needs to be considered in your overall device matrix Manual <-> Automated Ideally, manual and automated testing go hand in hand throughout development to maximize efficiency and to cover the various use cases of the application under test. In mobile, the need for manual interoperability testing (IOP) is an essential part of testing. Mobile app vendors must know how the application will react when incoming events such as messages, calls etc. occur while the app is running, as well as when the network condition changes. At the early stages, automated unit and sanity testing should be used to test the basic quality of your code. Six to eight must-have devices are usually enough at this early stage. ² ² 3M methodology for choosing the right mobile devices: Perfecto Mobile White Paper
Bridging the essential gap between Mobile Cloud and Crowd based testing 04 Later on, regression, performance and stress testing should follow with the major devices of the respective operating system, comprising around 12-15 devices. The above number of devices should be aligned with the relevancy to your market segment and geography (retail, banking, travel, etc.). They should also meet some form of analytics to get insights into what devices are mostly used by your customers. Because the mobile market is so dynamic, the OS versions and device mixes available for testing need to relate to the trends and changes in the market at a specific time. In-house/Cloud <-> Crowd Device diversity poses a major challenge for testing mobile applications. The combination of devices and operating systems can be endless. In-house testing is best suited to cover the major devices used in a specific market or region. Compatibility testing, however, requires the availability of the most recent models as well as older ones. This is why, after the in-house testing team feels confident about the quality of its application, an external crowd should test for compatibility and general functionality under real-life conditions. This step is a complementary test method to in-house cloud testing and addresses the need to cover complex and maybe unforeseeable user behavior and network conditions as part of quality assurance. Apple retracted its ios 8.0.1 shortly after users reported that their new iphone 6 failed to establish network connections after the update.³ A perfect example of critical issues that only occurred after releasing the software. Through crowdtesting, virtually all devices available on the market become accessible for testing. This comes very close to the actual usage of the application after its release. While your in-house team tests the application against expected behavior, the crowd will attend to the unexpected. They are not familiar with the functions of the app, have a fresh eye and are motivated by a pay-per-bug system. A typical bug Crowdtesting test could look like this: Crowdtesting is software testing combined with the principle of crowdsourcing. Crowdtesting therefore uses the collective intelligence of the global internet community to test websites, mobile apps, games, and enterprise software in order to get rid of bugs and optimize usability. The end customers, who are based on a desired target group, test under real-life conditions on their own devices even before the release. Android App 30 Testers > 20 consumers > 10 professionals Explorative testing guided by typical use cases A mix of consumers and experts has proven to yield the best results. If necessary, each participating tester crosschecks defects on his own device. This is called Bug Approval and offers additional insights into defects that need to be addressed immediately while others might only occur on very specific setups. In order for a crowdtesting project to be successful, extensive project and test management during the testing phase is crucial. Testers can work remotely from their homes or anywhere else. They document their work through written documentation, screenshots or screencasts. This is important for two reasons: 1. Since the testers are not in-house, documentation is the only way to ensure that there is sufficient test coverage. Was everything that was part of the initial test really tested? 2. If issues are found, they have to be traceable and reproducible. How did the user get to this point and what exactly is the issue? Depending on your internal resources, you can either manage crowdtesting projects yourself or outsource it as a whole. It makes sense to start out with at least one managed crowdtesting project. It is then possible to compare the effort and cost with a test done in self-service. ³ http://support.apple.com/kb/ht6487
Bridging the essential gap between Mobile Cloud and Crowd based testing 05 c. Usability Testing As user experience becomes key to business success and app adoption, UX testing should be a part of any mobile app test plan. High app usability can be achieved through Single User Performance testing early in the development cycle, which includes vitals testing (CPU, Battery, memory leaks etc.) as well as network condition testing (latency, packet losses, behavior in various network types like 2G, 3G, LTE, Wifi). UX testing can also be done with proper adherence to the platform vendor s design guides (Apple, Google). The important thing to keep in mind that the mobile app needs to: Usability Performance Function well on various devices, OS version Provide good user experience from a performance perspective Be intuitive to the users (easy navigation, meaningful error messages) Solve a problem or meet the need of its users 9 9 Adhere to the platform vendor guide lines (do not re-invent the wheel) - end-users expect UI that they know from other apps, provide them with a similar experience Failed channel shift Testing Internally When going to test usability with an internal cloud vendor, the cloud platform should be flexible enough to be hosted either on premise, remotely or through a shared public cloud, the values returned to the application team are as follows: Device management and governance headache removed the app team can focus on its tasks without needing to worry about devices, upgrades, security Clean room environment for variation testing like network virtualization testing to assess the impact on the end-user experience Ease of debugging and collecting logs/data/screenshots and videos to reproduce usability defects App test team has access to the same set of devices under test for further enhancements and defect fix validation Ability to implement agile continuous integration practices for ongoing testing With this platform and capabilities, development/test teams can really focus on their app development/testing-deployment while getting all of their infrastructure as a service in an assured, predictable and managed way. Testing with the Crowd Crowdtesting enables companies to gather insights and feedback from their prospective users at any time during the development process. This is relevant in order to tailor an application to the specific needs of your target group. A well-performing and functioning app could still get negative reviews if it doesn t address the user s needs.
Bridging the essential gap between Mobile Cloud and Crowd based testing 06 The added values of crowdtesting returned to the application team are as follows: Selection of specific demographics through more than 50 criteria (education, hobbies, online behavior etc.), offering access to any target group Access to specific users at any time, which reduces testing times Applicable in all stages of the development process in order to ensure development close to the user s needs: Prototyping Competitive analysis Beta Testing Usability study Setup according to your requirements (Questionnaires, Think aloud videos etc.) Opening up your application to a select crowd of users before launch tremendously increases the chances of success after release, given the test setup is reliable and their feedback is incorporated. Confidentiality agreements as well as high security standards prevent a leak of information before launch. 3. Testing Tips: Interoperability Does your app react to network events, device functionalities or other applications? The following pointers are helpful when testing in-house or with the crowd. Minimize app to use other disruptive apps, then return back to original app Change internet connection while testing: Wireless, LTE, 3G... Make/receive calls while using the app Play music/use camera while using the app Switch flight mode/gps/bluetooth on & off at various times Change location (if GPS-function is used) Change screen orientations (landscape/portrait) Install, uninstall, upgrades - test all of these with and without memory card on the device (some devices requires apps to be installed only when an SD card is in the device). Perform these operations on various mobile OS s. Language - For localized apps, change device language and see whether it impacts your application
Bridging the essential gap between Mobile Cloud and Crowd based testing 07 4. Conclusion Releasing a mobile app or several apps is always a great challenge, a challenge that continues to become more complex as mobile devices and OS platforms progress with innovative features. Mobile app vendors need to keep in mind that end-user expectations are also rising - you need to be on top of the market and react quickly to changes. A version release to the market is not a one-and-done release; it requires ongoing support, maintenance and attention to remain relevant and retain your users. The cloud provides real end-user devices elasticity to your business, in a secured, managed and collaborative manner. Crowdtesting complements the cloud with end-user exploratory and makes on-demand functional and usability testing a part of your testing strategy. Trying to leverage both is an efficient method to keep up with the above mentioned challenges. It allows you to scale as both your app and its user base grow, while having complete visibility into your end-user experience. In addition, an ever-growing device landscape can be addressed with a combination of cloud monitoring solution and crowdtesting (pre- and post-production). 5. About About Perfecto Mobile Perfecto Mobile is transforming the way enterprise organizations go mobile, enabling them to develop, test, deploy and monitor their mobile applications and services and go-to-market with confidence. Perfecto Mobile s cloud-based MobileCloud Platform and end-to-end mobile quality product suite enables users to remotely access a large selection of real mobile devices connected to local cellular networks around the world and leverage them throughout the mobile application delivery lifecycle from development, functional and performance testing to monitoring and support. The MobileCloud is available either in an enterprise private cloud or a sharable public cloud environment. More than 1,500 customers, including the top Fortune 100/500 across the banking, insurance, retail, telecommunications and media industries rely on Perfecto Mobile to optimize mobile time-to-market, customer engagement, risk mitigation and costs and continuously serve their mobile users with confidence. For further information about Perfecto Mobile remote testing services, please go to http://www.perfectomobile.com. Follow us on Twitter @perfectomobile, Facebook and our blog to get real-time updates. About Testbirds Testbirds is a crowdtesting service provider headquartered in Munich, with offices in the UK, the Netherlands and Hungary. The company specializes in the testing of mobile apps, websites and other software for the entire device landscape (mobile, desktop, smart TV, wearables etc.) on all major operating systems. Testbirds uses crowdtesting to facilitate software testing in today s fast-growing world of technology. They hold a worldwide network of testers (crowd) consisting of experts as well as consumers to test software under real world conditions. The crowd finds bugs and gives critical usability feedback, ensuring the highest-possible user experience before launch. Extensive device ecosystems as well as increased user expectations challenge conventional testing methods. Testbirds offers in-depth and cost-effective quality assurance for web applications and mobiles apps through the crowd on any device, with any target group, at any time. Today, crowdtesting is fast becoming a necessary tool for developers to help improve functionality and overall usability of any application. For further information about Testbirds crowdtesting, please go to http://www.testbirds.com. Follow us on Twitter @Testbirds, Facebook and our blog.