Whitepaper The New Mandate in Mobile Testing Deliver Continuous Quality with Expertise, Automation and Real Devices Mobile Performance Testing Mobile Test Automation
Table of Contents Executive Summary... 3 Mobile App Quality: Is Your Organization At Risk?... 4 Why This Is Important - The User Experience... 5 Pillar 1: The SOASTA continuous integration testing framework... 6 Pillar 2: Putting the mobile in mobile testing... 7 The Key Advantages of SOASTA Touchtest Mobile Labs... 8 Pillar 3: World-class testing expertise... 9 Conclusion: Hands-Free Testing In A Hands-On Lab... 9 About SOASTA, Inc.... 9 SOASTA, Inc. MOBILE TESTING LAB WHITEPAPER 2
Executive Summary This white paper discusses the three pillars of mobile app testing: Continuous Integration methodology Hands-on access to on-site devices in the lab Experts to support mobile testing initiatives In a market where users see new updates pushed out every 48 hours, the landscape for mobile application development and testing is seeing profound change. Today, developers need a infrastructure to accelerate development, improve testing coverage, and reduce costs. SOASTA responds to this new mandate in mobile testing with a comprehensive solution encompassing support for a continuous integration development framework; on-site mobile app testing labs with a complete range of testable smartphones, tablets, and other devices; and consulting services from mobile testing experts. The result: developers create higher quality mobile apps in less time, with less risk, and at less cost. SOASTA, Inc. MOBILE TESTING LAB WHITEPAPER 3
The road to mobile app quality requires addressing major risks. Mobile Test Automation Real Device Performance Mobile Testing Skills Full System Load Testing Access to latest devices Pushing apps to devices 28% 41% 41% 37% Mobile Testing Risks 52% 94 percent of respondents in multiple gaps in their mobile testing initiatives. 57% Mobile App Quality: Is Your Organization At Risk? For software developers and testers, the mobile application platform has ripped up the rule book. Almost overnight, the proven methodologies and infrastructures that the software professionals have relied on for years to deliver high quality applications now seem hopelessly antiquated. In margins are sometimes measured in a handful of pennies, and where device types endlessly proliferate, the pressures on software development and testing are profoundly disruptive, calling for a radical rethinking on several levels. Traditionally, developers created code over weeks or months. They created builds in version control systems and passed the builds over the wall to testing professionals for functional and performance tests that could take weeks and involve multiple iterations and months of work. In no longer feasible particularly when there are dozens of devices to validate. Unfortunately, in the face of these relentless time and cost pressures, many organizations are foregoing the testing cycle and putting quality at risk. According to the 2012-13 World Quality Report. Only 31 percent of survey respondents currently test mobile applications and those surveyed readily admit to being ill-equipped for mobile testing... [T]wo-thirds of organizations (65 percent) do not have the right tools to test mobile applications and equally concerning is the 52 percent who do not have access to the required devices. 1 Naturally, these companies prefer to adopt and implement ways that will preserve and enhance mobile app quality. Unfortunately, that s not as simple as it seems. Most companies surveyed in the World Quality Report disclosed that they lack the tools and devices to properly test mobile applications. And many also report that they don t have the processes or expertise to support the mobile testing function. What s more, a 2012 SOASTA survey of more than 500 IT professionals From SOASTA s perspective, these factors are all inextricably integrated: a development/testing delivered by experts who understand global best practices for software testing and the intricacies and nuances of mobile technology. 1 World Quality Report, Fourth Ed., 2012-13. Primary authors: Murat Aksu, vice president of global alliances for Capgemini, and Charlie Li, vice president of global testing services for Capgemini. Pp. 24-29. Additional contributions from Stefan Gerstner, vice president of global testing services for Sogeti; Alain Mey, global alliance director for HP; and Govindarajan Muthukrishnan, vice president and testing practice lead for Capgemini. SOASTA, Inc. MOBILE TESTING LAB WHITEPAPER 4
Why Developers Can t Test. World Quality Report data tools, processes, people and in-house device access. DO NOT HAVE THE RIGHT TOOLS TO TEST DO NOT HAVE THE DEVICES READILY AVAILABLE DO NOT HAVE THE RIGHT TESTING PROCESS/METHOD NO MOBILE TESTING EXPERTS AVAILABLE DO NOT HAVE IN-HOUSE TESTING ENVIRONMENT 65% 52% 34% 29% 19% 18% NOT ENOUGH TIME TO TEST 485 respondents. Source: World Quality Report, 4th Ed. 2012-13. Why This Is Important - The User Experience The mobile user experience is critical to the success of today s mobile applications. According to BI Intelligence in January 2013, 29 percent of mobile users have made a purchase with their smartphones. Walmart estimated that 40 percent of all visits to their Internet shopping site in December 2012 were from mobile devices. Bank of America predicts European and U.S. shoppers will make US$67.1 billion in purchases from mobile devices in 2015. Today, there are dozens even hundreds of similar or nearly identical mobile applications that provide many of the same services, from e-commerce and maps to news. If one mobile app fails to perform well and meet user expectations, the user can simply delete it and move on to numerous alternatives. SOASTA, Inc. MOBILE TESTING LAB WHITEPAPER 5
Pillar 1: The SOASTA continuous integration testing framework The right methodology to accelerate mobile software development Today, an increasing number of software developers are adopting continuous integration (CI), an agile software development methodology that brings together each team member s output as often as multiple times per day to reduce the headaches of episodic integration and testing. In traditional methods, the longer a unit of code is checked out, the greater the risk and greater early and often approach, along with automation, to reduce the cost and time required to develop applications. Developers perform simple unit tests before adding their code back to the code base. SOASTA is an enthusiastic supporter of the CI methodology and has developed a CI testing framework for mobile test automation. This CI testing framework is mobile optimized to enable and enhance continuous testing and feedback, which can support the new speed of development required for mobile applications by compressing the development lifecycle. The key components of the CI testing framework are: Builds SOASTA s micro-footprint TouchTest library integrates automatically within the mobile app during the continuous integration process. Regression Testing SOASTA executes tests directly from any CI server. The CI server polls the version control systems for changes. Testing results from TouchTest can be displayed in the CI dashboard so that when developers are alerted to tests that fail, they can identify and rectify problems before updating their code in the source control system. Innovative Test Creation and Recording - Patented visual technology captures a test by recording the steps that might otherwise be completed by a manual tester. Validations, waits, seed data and other test elements are added to the test. This unique visual paradigm allows developers and functional testers to quickly automate test scenarios. Object-based validations allow access to all app internals such as variables and methods. Real Tests, Real Devices SOASTA s recording and automated test execution is performed on real mobile devices that do not need to be jailbroken or tethered. Recording can be performed over WiFi or carrier network and can be done from multiple locations by multiple users. The TouchTest library inside the app records all gestures, actions, and events with perfect precision. execute tests with repeats, waits and pauses. Multiple tests can be setup to run against the same (against the backend infrastructure) can be mixed and matched on the same test composition to understand the impact of transaction volume on the mobile application. Analysis and Feedback SOASTA provides real-time analysis of test results using an easy to consumption, data and battery usage, and more during test execution for all devices to identify potential bottlenecks within the mobile app. Automated testing has reduced functional testing time by more than 70 percent for some clients who were previously testing mobile applications manually. Cycle testing time has been reduced from weeks to days and has given developers immediate feedback on issues with their mobile applications. SOASTA, Inc. MOBILE TESTING LAB WHITEPAPER 6
Mobile device labs need to be implemented quickly, always accessible and contain the latest models & operating systems. Pillar 2: Putting the mobile in mobile testing Every week, it seems like there s yet another mobile device entering the market a new tablet, a new smartphone all with new screen sizes or user interfaces. Implementing a mobile lab that can automatically and reliably test a single application on dozens of devices can be a daunting challenge, given the costs and complexities. Even the basic process of simply wiping the devices time-consuming. Not surprisingly, it s tempting for many development shops to explore remote testing sites, virtualized test suites, or simulators. But, as any savvy mobile developer can tell you, there s simply no substitute for actually seeing the app s performance on real devices. Unlike PC or Webbased applications, mobile apps have a much higher threshold for usability and unique functions gestures like swipe, pinch, pan, rotate and long press. Developers don t want to virtualize located with the development team provides immediate feedback on the usability, functionality and completeness. This process compression allows for faster turn-around times along with higher acceptance among application owners who can see the application run on an array of devices. SOASTA understands this fundamental requirement for mobile testing with devices that are accessible and controllable. The TouchTest Mobile Lab is the industry s only turnkey mobile device spectrum of mobile devices that can be used to perform comprehensive testing. The SOASTA TouchTest Mobile Lab is powered by the company s acclaimed TouchTest software that enables automated and simultaneous testing of native and mobile Web apps on any number of ios and Android devices, models, versions, and form factors. We can source and bring the devices to your facilities or you can leverage your own devices so you have direct, physical control. And they connect to a CI server that automatically distributes the testable mobile apps to the devices. That means hands-free, automated testing that completes in minutes, not days. coordinated, comprehensive testing coverage around the world. SOASTA, Inc. MOBILE TESTING LAB WHITEPAPER 7
TouchTest Mobile Lab TouchTest Precision in app mobile test automation Continuous Integration - Software for automated test app deployment On-Site Devices A complete range of dedicated, current, and unaltered mobile devices The Key Advantages of SOASTA Touchtest Mobile Labs Rapid Deployment We move at the speed of mobile. We get your lab up and running in just a Local Presence Seeing is believing. With the TouchTest Mobile Lab, all of the devices can be right there in your lab, under your direct control. Complete Test Coverage Every Day With a TouchTest Mobile Lab, you enjoy comprehensive test coverage that aligns with agile CI development principles. needed for your application. Higher Quality Mobile Apps Script automation and continuous feedback allow for increased test SOASTA, Inc. MOBILE TESTING LAB WHITEPAPER 8
Pillar 3: World-class testing expertise testing increasingly becomes a non-negotiable mandate there s little time to waste with lengthy learning curves and easily avoidable pitfalls. SOASTA can give app developers and testing teams a you how the software and the methodology work. First and foremost, we understand the need for agility and speed. We get the importance of continuous integration. No other commercial vendor has fully embraced CI platforms such as the open source Jenkins or Atlassian s Bamboo by pulling code from a repository, compiling it and enabling it for testing, pushing the testable application to one or more mobile devices, and when developers are alerted to errors or failures, they can quickly display and analyze results in the CI dashboard. If there are errors, from within Jenkins the developer or tester can quickly navigate directly to where the error occurred. SOASTA knows mobile testing. We can show your team how to convert manual test cases into those tests an essential skill as your builds evolve and proliferate. Since TouchTest uses a highly visual interface, your QA team doesn t need programming expertise, which means they can be productive in this new environment very quickly. We can help train, augment, or outsource your resources for test planning, design, creation, execution, and analysis. Of course, once your functional testing is complete, you ll want to ensure that your application scales with appropriate levels of performance. SOASTA CloudTest is the complement to your mobile app testing process, ensuring your app infrastructure can properly support the anticipated load of simultaneous users. For more information on CloudTest, visit http://www.soasta.com/ products/cloudtest/. knowledge base, a thriving support forum, and tech doc and release notes, as well as our CloudLink Premier support service. Wherever you are in your mobile testing process, we have you covered. Conclusion: Hands-Free Testing In A Hands-On Lab The testing processes and infrastructure for mobile applications embody a certain contradiction. Companies want and need the hands-free automation that performs functional tests in minutes, not days. But they also want hands-on access to devices, enabling them to physically build and execute tests on a full range of mobile devices. The continuous integration framework provides the foundation for agile, rapid software development. SOASTA has fully embraced this methodology in its TouchTest Mobile Lab that provides facilities, equipment, and expertise to mobile application developers. For more information, visit the SOASTA website at http://www.soasta.com/products/touchtest/ About SOASTA, Inc. SOASTA is the leader in cloud testing. Its web and mobile test automation solutions, CloudTest and TouchTest, enables developers, QA professionals and IT operations teams to test with unprecedented speed, scale and precision. The innovative product set streamlines test creation, automates provisioning and execution, and distills analytics to deliver actionable intelligence faster. and headquartered in Mountain View, Calif. For more information about SOASTA, please visit www. soasta.com. Headquarters 650-210-4950 444 Castro St, Fourth Floor Mountain View, CA 94041 FAX: 650-210-4957 EMEA +44 (0)1753 752375 Thames Court 1 Victoria Street Windsor, Berkshire SL4 1YB To learn more visit: soasta.com or email us at info@soasta.com 2012 SOASTA. All rights reserved. SOASTA, the SOASTA logo, and SOASTA CloudTest are trademarks of SOASTA. All other product or company names may be trademarks and/ or registered trademarks of their respective owners. Connect with us: