White Paper Testing in a Mobile World April 2014 Share this White Paper
Contents Introduction 3 1. Testing in agile projects 4 2. Testing tools 6 3. Testing practices 10 4. Testing as a service 14 Conclusion 2
Introduction Mobile apps have exploded since the introduction of the Apple iphone: today there are literally millions of apps for mobile users to choose from. Mobile devices have also rapidly replaced the PC as the way most people access an organisation s digital presence and as a result you re not really doing business without a mobile app. Today s best apps are created using agile development techniques: software is built in stages, with regular releases, and communication maintained between developers and users throughout. However, these processes alone don t guarantee quality. The guarantee of quality comes from rigorous testing, which you ll start from day one and continue through into maintenance and enhancement. This paper provides you with an introduction to the tools and practices that come together to ensure the quality of your app and includes: 1. An overview of testing within an agile project, which will provide you with context for the content that follows. 2. A look at the testing tools that you ll want to consider deploying for your project. 3. A description of each type of testing you ll want to subject your app to. 4. An examination of how a testing specialist can assist you in implementing your testing. Mobile apps have exploded since the introduction of the Apple iphone: today there re literally millions of apps for mobile users to choose from. 3
1. Testing in agile projects Agile approaches to software development emerged in the 1990s and have rapidly gained acceptance, because these methods quickly deliver effective and engaging software. This success is achieved by maintaining a high level of interaction between the user or customer and the software developers. This is facilitated through regular incremental releases of software, called sprints, to review progress and gather feedback. This process enables the ideas behind the app to be validated quickly and improvements made throughout the development: effectively a rapid process of build, test, and correct and improve. As shown below, a project starts with the creation of a set of test cases. These test cases are used to confirm that the app is meeting its requirements. As development starts, unit tests are used to confirm that each piece of software works as required. Individual software items (which are usually stored in a Version Control System or VCS) are then built into the application (often by a Continuous Integration Server). The built app is then subjected to a set of tests to ensure that the pieces work together (integration testing), that any earlier work hasn t been broken by more recent work (regression testing), and that non-functional requirements, such as performance or security, are met. These tests may be conducted manually or automatically by a test robot. As a result of this testing the code may be further refined and retested, before a sprint is complete and the app passed to the customer for testing. This process continues until the final software is delivered. 4 This is where a partner like Penrillian can help, but the key message here is this: no one partner in this ecosystem can deliver a successful mobile application on their own.
1. Testing in agile projects In many cases while the interfaces between these providers may be available in theory, in practice much work is needed to make them communicate effectively. For example, one issue Penrillian has faced is that Application Programming Interfaces (APIs) have often been constructed with the desktop web in mind. The differences between the desktop environment and mobile in terms of browser capabilities, connection speeds and reliability means these are usually unsuitable for the mobile environment. When you can t demand a complete rebuild of a partner s infrastructure, you need to find ways to negotiate between different standards with the requisite level of performance and security. This is where a partner like Penrillian can help, but the key message here is this: no one partner in this ecosystem can deliver a successful mobile application on their own. All parties must collaborate effectively to meet the customer s expectations. Testing is therefore central to the success of an agile development project. It s a continual day-by-day, if not hour-by-hour activity; seeking to squash bugs as soon as they arise. Developers Test cases Customer Development Unit tests Source code VCS/Repository Application Continuous Integration Server Test execution robot Test reports Integration tests Regression tests Non-functional tests Sprint development User acceptance tests 5
2. Testing tools You can perform testing with a minimal infrastructure: just the software and some enthusiastic testers. There are, however, a number of tools and techniques that ll help improve your ability to efficiently execute tests. This section looks at four of these: 1. Automated testing 2. Continuous integration 3. Bug tracking 4. Test cases 6
2. Testing tools 2.1 Automated testing Testing can occupy a large portion of a project s time and resources. It s also repetitive, which can lead to oversights and omissions. These factors make testing an ideal candidate for automation. Automation also makes it easier to extend coverage, as once specified and automated additional tests don t need additional people to execute them. Test automation can be achieved at every level of testing in a sprint: Unit Integration Regression Non-functional Security testing User acceptance testing can be automated also, however, care should be taken not to rely entirely on automated testing as it cannot test for human factors such as usability and visual appeal. Another benefit of automated testing is that it enables tests to be repeated across multiple environments with little incremental effort, once the appropriate setup has been done. Something as apparently simple as delivering an Android app might involve testing on multiple releases of the Android operating system and on devices from several manufacturers. Automated testing makes achieving high levels of coverage cost effective for a typical mobile app project, automated testing might deliver around 1000 tests across 10 environments and be able to complete those tests in a few hours. There are a number of tools that can be used to implement automated testing and they fall broadly into two groups, those for: Browser based testing, such as Selenium. Native app testing, such as Appium, Cucumber with Calabash, or (for Android only) Monkey Runner. To run automated tests, devices need to be available and harnessed to the test automation software in a test lab. Where the required number and range of devices cannot be sourced locally, cloud-based services, such as Sauce Labs, will rent access to devices as needed. 7 Another benefit of automated testing is that it enables tests to be repeated across multiple environments with little incremental effort, once the appropriate setup has been done.
2. Testing tools 2.2 Continuous integration Traditionally, developers spent days or weeks creating individual app components then, at some arbitrary point, all the components were brought together and tested to see if they worked as a whole. This process often resulted in a protracted integration phase. Agile methods avoid this issue by delivering software frequently, enabling integration issues to be found and resolved early. However, best practice takes this regular integration a stage further with what is known as continuous integration. The goal of continuous integration is to test and validate immediately, as each software change is made to the app. This has two principle benefits: Changes are limited in scope, so if a new failure is detected there are a limited number of components that could have caused the problem. This process is not without its challenges as each software component is committed unit, integration, regression, and non-functional tests need to be executed. Even on a small app project the number of tests is far from trivial. Two tools assist with making continuous integration practical: An integration server, such as TeamCity. These tools handle the receipt of changed software components; building the components into a working, executable application; deploying the software for testing; triggering tests; and providing centralised reporting on the outcome of each integration. Automated testing. Developers are familiar with their recent changes; so don t have to refresh their memories when it comes to finding coding errors. 8 This process is not without its challenges as each software component is committed unit, integration, regression, and non-functional tests need to be executed.
2. Testing tools 2.3 Bug tracking Testing will inevitably uncover bugs. These can be everything from issues with the ideas the software is based upon, through to coding defects. In an agile development project each iteration becomes a mix of activities: addressing defects found and adding new features and functions to the app. Every project must therefore have an effective way to deal with bugs: determine their severity, prioritise when they should be addressed, allocate them to project members, and track whether they have been resolved. Tools such as Mantis make the recording and management of bugs easier in all but the simplest projects. In addition, many of these tools will work with test case management tools, enabling the loop to be closed between testing and the defects it detects. 2.4 Test cases Test cases, definitions of the app s expected behaviour and how to confirm that behaviour has been achieved, drive the agile development process. Test cases need to be accessible to all members of a project. This can be achieved using simple, plain English given-when-then definitions of tests: (Given) some context (When) some action is carried out (Then) a particular set of observable consequences should be obtained 1 Simple, clear test cases not only ensure that all project members can understand what is being tested and judge whether the desired outcome is achieved, they re also easier to execute and form an excellent basis for test automation. Individual test cases are usually built up into test suites, normally tailored to the goals that have been set for the app development in each sprint. While no special tools are required to write test cases, there are tools such as TestRail that can help in defining, managing, executing, and reporting on test cases. Simple, clear test cases not only ensure that all project members can understand what is being tested and judge whether the desired outcome is achieved... 1 http://guide.agilealliance.org/guide/gwt.html 9
3. Testing practices As testing is central to an effective agile development project, understanding the testing methods and techniques used is important for everyone involved. This section describes some of the key testing activities and looks at what s involved in executing them. 10
3. Testing practices 3.1 Requirements testing Testing your app doesn t require a line of code to be written, testing should start from the moment you have an initial set of requirements. Indeed testing at this stage is a tool to refine the requirements: restating the requirements into a set of test cases can identify gaps, inconsistencies, or contradictions. 3.2 Unit testing Unit tests define what each software component should do and how it will communicate with other parts of the overall application across their interfaces. App development begins with individual software components, often in isolation from one another. For example, one group of developers may start building the app s web server while another group starts on the mobile app. Initially the interfaces between these elements of the final application may not include the code to exchange data. Early testing therefore commonly uses dummy code to represent the interfaces or components not yet developed. 3.3 Integration testing Unit testing can only provide assurance that individual software components work as expected, they don t help assess what happens when these components are brought together to create the complete app. The next level of testing is therefore on integrated collections of the app s components. The goal of agile development is to deliver a complete version of the app as soon as possible, albeit with minimal functionality at the start. As a result the typical agile project is integration testing the entire system within the first few sprints (if not the first). Integration tests should be run whenever a new version of a collection of integrated components or the whole app is built. This can be as regularly as each new version of a component is made available by a developer, but a good practice would be to run the tests no less than daily. If the project has implemented a continuous integration process, integration tests become ideal candidates for automation. As a component is coded the developer will run the unit tests often, at least as often as the components are checked in to the overall project. These tests will continue to be exercised throughout the life of the project. If the project has implemented a continuous integration process, these tests become ideal candidates for automation. The goal of agile development is to deliver a complete version of the app as soon as possible, albeit with minimal functionality at the start. 11
3. Testing practices 3.4 Regression testing Each sprint will consist of a mix of old and new code. Ideally features developed and delivered during an earlier sprint should continue working as they did before and only new features should require testing. However, the addition of new features and bug fixes often means that code created in earlier sprints has been changed and is therefore a potential source of new issues. Regression testing addresses this by rerunning most, if not all, previous tests to ensure that the app still passes these tests (or where bugs were found, now passes tests that failed earlier.) It should be performed at least once for each sprint, but ideally will be performed with each and every software change to enable rapid and effective detection and correction of bugs. Again, if the project has implemented a continuous integration process these tests become ideal candidates for automation. 3.5 Non-functional testing Simply because an app executes all its defined tasks correctly doesn t mean it will meet user expectations: users will also want the app to be responsive. Various tests can be used to confirm this aspect of the app, such as: Performance testing to ensure that specific tasks can be completed within a certain time. Load testing to ensure that the app, in particular any backend systems, can cope with the expected volumes of transactions. These tests are often interrelated, for example, can the required performance be maintained at peak loads. These tests are particularly suited to automation, and in many cases automation is the only practical way to run them. For example, automation is usually required to simulate load on a backend. Ideally features developed and delivered during an earlier sprint should continue working as they did before and only new features should require testing 12
3. Testing practices 3.6 Security testing Depending on the nature of the app, specific testing may be required to ensure that appropriate security is in place. While some aspects of security can be covered in other tests; interception of data, intrusion, and other threats may require specialist testing. In most cases use of an independent tester is advisable, as they will have a detailed and up to date knowledge of known vulnerabilities and the methods used to take advantage of them. 3.7 User acceptance testing User acceptance testing is arguably the most important part of the testing process, as it s the testing stage that ultimately delivers the app for live implementation. However, within an agile project user acceptance testing isn t a single end of project activity. Each sprint is subject to user acceptance testing, with the key goals of validating the app s requirements, confirming that those requirements are correctly implemented, and that the non-functional aspects of the app are measuring up. User acceptance testing isn t and never should be simply about hard testing, it needs to consider factors such as usability and visual appeal as well. As a result user acceptance testing isn t a good candidate for test automation. And even with testing completed for each sprint, there should still be final user acceptance testing, usually involving a wider group of stakeholders, before the application is accepted. 13 User acceptance testing is arguably the most important part of the testing process, as it s the testing stage that ultimately delivers the app for live implementation.
4. Testing as a service The goal of agile development is to bake quality into the app through a process of iterative development where testing is an integral part of that process. It might therefore seem problematic to seek support from an external vendor to assist with testing, however, there are a number of ways in which external assistance can benefit your project. 14
4. Testing as a service 4.1 Kick-starting testing Testing has rapidly developed into a specialist activity within software development, with a vast array of tools and techniques at its disposal. On boarding a testing specialist can assist with project start up by identifying and implementing the appropriate tools and techniques rapidly. Once a project has commenced, whether a test specialist acts as a mentor or gets their hands dirty in creating and executing tests, they can simplify processes and rapidly dismantle roadblocks that might otherwise cause delays. 4.2 Test automation Test automation is a complex activity, involving the translation of manual tests into automated scripts, then scheduling those automated tests appropriately to test the functionality delivered in each sprint. Test automation is a development task in its own right. Obtaining the assistance of a specialist can free your development team to focus on applying their core skills and not get diverted by the need to upskill in new tools and techniques. 4.3 Implementing continuous integration If you don t have a continuous integration environment, there is a range of tools available you can use and a myriad of configurations that can be implemented. A specialist knowledge can reduce selection and implementation time and bring the appropriate technology into your project quickly. 4.4 Management of user acceptance testing If there is one area of testing where independence is required, it s user acceptance testing. The two most common challenges with user acceptance testing are: That it s usually conducted by people who are unfamiliar with testing processes. Finding a representative sample of users, given that for mobile apps they ll often be external to your organisation. A testing specialist can help with both of these situations. Your test specialist could mentor your user acceptance testers, to ensure they are able to define and execute appropriate tests. At the other extreme they could take complete management responsibility for the user acceptance testing: recruiting and training testers, scheduling and executing tests, and helping draw up test reports and recommendations. In the case where locating a complete and representative set of users to perform the tests is problematic, a test specialist could act in place of some or all of those users. 15... whether a test specialist acts as a mentor or gets their hands dirty in creating and executing tests, they can simplify processes and rapidly dismantle roadblocks...
4. Testing as a service 4.5 What to look for in a testing specialist When selecting a test specialist you should be looking for the following: Track record in delivering quality apps. Understanding of agile software development techniques. 4.6 Bringing a testing specialist on-board If you decide to bring specialist testing into your project, regardless of the scope of that involvement, the iterative nature of an agile project means a critical success factor is starting early ideally bringing the vendor on-board from day one of the project. Extensive knowledge of test techniques and tools. Staff with the ISTBQ qualification. People you can trust. the iterative nature of an agile project means a critical success factor is starting early... 16
Conclusion Crowded app stores mean that getting your app noticed can be a challenge you ll be leveraging a range of traditional and new media to get the word out and gain those all-important downloads. And once you have a user s attention you ll want to keep it, as they have no shortage of alternatives if your app doesn t measure up to their expectations. A great app starts with a great idea, but today a great idea alone isn t enough. The user experience has to be top notch. Your app has to be easy to learn, easy to use, and always do what the user expects and wants. Testing is key to achieving this. A successful agile development project integrates testing into every aspect of the project, from unit testing on individual app components through to user testing and validation of the app at each and every sprint. Implementing a best practice testing regime involves bringing together a diverse toolset: developers and customers/users, bestpractices, and tools. For new or large projects, the involvement of a testing specialist can simplify the process. A test specialist may be particularly helpful in executing tasks such as test automation or security testing, enabling a project to start-up quickly and focus on the goal delivering a quality app rather than mastering processes and tools. Among your potential testing partners, Penrillian stands out with their track record of successful mobile app projects. This success is built on a depth and breadth of skills and expertise in the processes of testing and the tools to achieve efficiency and automation. 17 For more information on how Penrillian can help with your next project, visit the testing services page or fill in the contact form on the Penrillian website.
Penrillian Clint Mill Cornmarket Penrith CA11 7HW United Kingdom Email: enquiries@penrillian.com Tel: 44 (0) 1768 214400 Web: www.penrillian.com 2014 Penrillian mobile phone software developers - All rights reserved.