Whitepaper Why effective Test Automation drives successful and qualitydriven mobile payments sqs.com Answers on how to improve cost effectiveness and reduce time to market Introduction The adoption of m-payments is increasing, as is the number and diversity of mobile devices, and software, upon which these payment systems operate for Android alone, over 420 devices and 29 versions of the operating system have been released since 2007 1. For organisations developing m-payments products and services, testing the many permutations of software and hardware is a logistical and quality assurance nightmare. While the device and software landscape increases in complexity, commercial pressure is growing to reduce time-to-market as retailers, mobile network operators (MNOs), banks and financial institutions alike chase market share. The effective use of test automation is critical for organisations aspiring to release stable m-payments solutions. For most organisations, frequent, reliable and cost-effective testing through test automation is the only effective way to solve the quality assurance problem for m-payments. Authors: Faisal Hussain (Senior Consultant) faisal.hussain@sqs.com Andrew Brown (Senior Consultant) andrew.brown@sqs.com Darryl Kennedy (Principal Consultant) darryl.kennedy@sqs.com SQS Group Limited, United Kingdom Published: March 2014 1) As of 4Q 2013 SQS the world s leading specialist in software quality
A recent report on automated testing found that: The important conclusion is that there exists a large, untapped pool of economic benefits that are yet to be claimed through highly automated testing Companies will increasingly realise economic benefits in cost savings, improved business agility, efficiency and quality. Harvesting economic benefits will continue to drive the shift toward highly automated testing and away from manual approaches, as companies continue to push for higher quality execution and greater business agility at lower cost. 2 This paper explores the challenges of automating m-payments system testing, and considers the key elements of an effective solution that will enable the confident delivery of working m-payments technologies. Test Automation Demand Mobile Payments Solution Complexity 2) Worksoft Market Research Report, 2013 Trends in Automated Testing, http://www.worksoft.com/files/resources/worksoft-research-report-2013-trends-in-automated-testing.pdf Page 2
1. Testing more for less, test automation ROI It has long been recognised in traditional software development that test automation is a proven technique for running large numbers of tests more accurately, reliably and more rapidly than can be achieved through manual testing. Additionally it enables testing outside of core hours, as the tests are executed by a computer rather than by a person. Automation is also important for regression testing running large numbers of tests against a code base to ensure that changes have not introduced new bugs to existing code. Software test automation is no longer a luxury with a questionable return on investment (ROI). Instead, automated functional testing and automated regression testing have crossed over to a definite necessity, as mobile apps have grown ever more important and complex. In short, manual approaches simply cannot keep up with the growing complexity and demands of mobile application testing. 3 Figure 1 illustrates the benefits of one of our mobile application development projects, comparing the savings between manual and automated test execution. In this case, compared to the cost of manual testing, project break-even point was at the end of year one and with savings of 93 % in year two and 300 % in year three. The impact of test automation is not felt solely in the savings made on testing resource. Automation allows broader and more frequent testing of the payments system than manual methods, leading to more bugs being detected earlier and ultimately, a better quality product being released to production. Extensive testing often involves the allocation of subject matter experts from within the business and by automating tests, the demands on these experts can be reduced, minimising the impact on day-to-day business operations. Cumulative cost Live Example 60,000 50,000 SQS test automation for SEPA and new product platforms reduces cost by 80 % for a major international banking organisation 40,000 30,000 20,000 10,000 0 1 11 21 31 41 51 61 71 Manual Testing Figure 1: Mobile test automation ROI Test cycles executed Automated Testing 3) The ROI of automated mobile application testing, Mobile Labs, June 2013, http://mobilelabsinc.com/the-roi-of-automated-mobile-application-testing/ SQS was asked to conduct automated and functional testing by a leading multi-national bank. The bank already had Android and ios retail banking applications and was planning to enhance these with SEPA and new product features.sqs developed an offshore-enabled test automation infrastructure that provided consistency of testing across the mobile platforms and operating systems. The testing automation framework was designed for reuse enabling the bank to apply these test automation methods to future m-payment projects. A significant reduction in the testing lifecycle was achieved through automation and as a result, the project was able to increase the rate at which new versions of the software were released. The project s 2,000 test scripts were executed in just 30 man-days, reducing the cost by 80 % compared to the equivalent manual testing. Page 3
2. M-payments testing challenges For all its benefits, test automation requires investment and it is important to understand the business case and ROI, nonetheless, for m-payments, test automation is almost always necessary because of the: be secure otherwise any weakness (or perceived weakness) will lead to concerns about security. Proven weaknesses in m-payments security for any given provider could have significant impacts upon the success of its business plan. Significant number of risks to be mitigated Large matrix of diverse test combinations Many test data permutations to be exercised 2.1. There are a significant number of risks to be mitigated in m-payments M-payments services present a wide variety of risks, which can only by mitigated by an effective testing solution with the capacity to achieve high levels of coverage. Trying to mitigate all of these risks effectively without the use of automation would be a very complex, time consuming and precarious endeavour. Risks associated with m-payments include: User experience/ confidence Security Maintainability Interoperability Performance 2.1.1. User experience / confidence An m-payments system that consumers do not like, or are unwilling to adopt, is doomed to failure. The user base for m-payments systems is enormous, and growing daily, as is the IT literacy of this user base so any weaknesses in the user experience will be very apparent and communicated widely. 2.1.2. Security Security risks are significant due to the nature of the transactions involved and the relative immaturity of the technical solutions being implemented. The way a payment system is perceived by its user base is crucial, simply being secure for m-payments is not enough, the system must be seen to For more information about security for m-payments, see the SQS whitepaper: How will Security Testing help to reduce risks and build customer confidence in mobile payments 2.1.3. Interoperability M-payments systems need to support a wide range of technologies, interfaces and contributing systems. They need to operate seamlessly in a varied and evolving technical environment. As the Internet of Things grows, the types of interaction the payment system must support will change and so too will the need for comprehensive assurance of m-payments systems. These systems operate in the kind of varied and diverse environment that has never been seen before, which presents significant ongoing risks. 2.1.4. Maintainability As a technology, m-payments is still quite young. New m-payments systems need to be as maintainable as possible, in order to remain flexible enough to meet future demand; in development terms, this means minimising technical debt. Testing for maintainability is a significant challenge, and one that is well served by automation. 2.1.5. Performance As with user experience there is a growing level of expectation on the part of the consumer that m-payments systems will not only work, but that they will work quickly and smoothly regardless of how heavily loaded the overall system is. Web site development has shown that any drop in site performance will see users switch to an alternative very quickly the same risks exist for m-payments solutions. Page 4
2.2. Large matrix of diverse hardware and software test combinations Looking at the market leading ios and Android compatible devices, it is apparent that testing device and operating system combinations alone is becoming a very large and complex task. Consider the following statistics: 29 versions of Android released since 2007 420 models of Android mobile phone 18 versions of ios released since 2007 6 models of the iphone have been released since 2007 Disparities between devices, variances in operating systems, versions and additional manufacturer customisations, together with differences in internal architecture, screen size and screen resolution lead to a huge matrix of potential test combinations. Fully testing 420 device models over 29 Android versions requires that 12,180 combinations be tested for each test. Adding the various combinations of ios devices and operating system versions introduces a new layer of complexity and the increasing popularity of Windows Phone adds yet another. This presents significant challenges to the quality assurance programme that only automation can resolve. 2.3. Test data permutations The successful testing of m-payments applications requires many permutations of the same (or similar) business process to be executed with different data sets. Test data sets can be used to check facets of the system such as: Acceptable boundary values Upper and lower limits Customer types Localised international payment formats Internal, domestic and cross-border payments Business processes requiring a high volume of different data sets for testing, are ideal candidates for test automation, because the technical script maintenance remains low due to a relatively small test set. In these situations, effective test data management can yield an extremely high number of test cases. The business case for the test automation of m-payments will demonstrate a very high return given the number of data permutations, which would otherwise need to be executed across a large set of device and operating system combinations manually. Page 5
3. Test automation solutions for m-payments Without automated testing it would be nearly impossible to mitigate the risks inherent in m-payments systems and benefit realisation would be significantly challenged. So what test automation techniques and tactics should be considered? 3.1. Test automation strategy and management A well thought out test automation strategy is crucial: clear requirements, rationalisation of devices and limiting the number of combinations of operating system are imperative for the intended market. Such prioritisation typically results in a test strategy that focuses on testing the most recent versions of important operating systems in combination with popular mobile hardware. The strategy should consider all risks and identify a cohesive automated approach to mitigating these risks. The table below illustrates how many different factors need to be taken into account during the creation of the mobile test automation strategy. The SQS whitepaper Efficient Test Management can help you to launch mobile payments faster, smarter and cheaper discusses approaches and strategies for test management in m-payments system development. 3.2. Test environment management Test environments that enable representative testing of real-world conditions are essential for all but the simplest mobile software. However, for Android alone there are more than 12,000 permutations of device and operating system versions. The effort required to test these versions is likely to be uneconomical for most organisations and so, prioritisation of environments and risk-based testing is required in order to create a pragmatic testing programme. High overall complexity Wide variety of platforms, devices, operating systems and versions Multiple configuration options on mobile devices Multiple network and access options (cable, WiFi, UMTS, LTE, Bluetooth, NFC) Use of different programming languages, debuggers and emulators Integration into payment systems Changing requirements for deployment management Increased need for Device Management processes and decision-based functionality Need to comply with roll-out plans Release frequency and regression testing effort The many combinations of operating systems, versions and updates increases the number of releases Regression testing effort increases due to the steady increase offered by mobile application functionality Table 1: Considerations for developing a mobile test automation strategy Page 6
Devices Market ios, MS, BB Android V2.3 V2.3.3 V3 V2.3.4 V3.1 V2.3.5 V3.2 V2.3.7 V2.3.6 V4 Product Version Jan July Dec 1 year - 2011 Figure 2: An example of the number of device and operating system releases in just one year (source: perfecto mobile) Must Major Market Figure 3: Prioritising devices for testing (source: perfecto mobile) Page 7
3.3. Behaviour driven development and test data driven automation Enhancing automation in mobile device testing through best practices such as behaviour-driven development (BDD) 4 can lead to further improvements in both quality and efficiency. BDD helps establish a common understanding about testing between developers, testers, business analysts and other business stakeholders. BDD closes the gap between the people who implement automated testing and the people who best understand the business requirements. BDD puts control of test conditions into the hands of business-facing analysts and subject matter experts (SMEs), reducing the likelihood of misunderstandings and enabling developers to validate their code more easily. Example of shared test data specifications used by both business analysts and test automation specialists. 4) For the purposes of this paper, behaviour-driven development and test-driven development are synonymous. Page 8
3.4. Test environment considerations for automation In order to conduct automated testing, the test tools employed must have programmatic access to the device and software to be tested. The main techniques for establishing tool access to devices for testing include: Jail-breaking - removing the Operating System (OS) security control by unlocking/rooting/jail-breaking the mobile device. The benefit of this approach is that complete control of the mobile device is then possible. However, the downside is that this approach is viewed as a nonstandard OS, so any testing on a jail broken device may be invalid, since it is technically on a different OS version than a majority of end users. This method is also reliant on an Internet community to provide these unlocking options. Emulators having the mobile device software running in a simulation of the mobile OS. This removes the requirement for real devices and the need for remote access. However, the downside is that it does not fully match an end users experience. Cloud some vendors are now hosting a suite of mobile devices that they claim are native (i.e. not unlocked/ rooted/jail-broken) and provide full object recognition and remote device access. Although this method offers fantastic coverage for minimal maintenance, subscriptions can be costly and dedicated access to the latest handsets can prove expensive. Security of data within the cloud also needs to be considered and appropriate risk mitigation put in place. Object recognition many mobile test automation tools use either native object recognition or image recognition to identify each of the controls on the screen. The native approach depends on the application being written to specifically support object recognition. Image recognition is typically slower and often multiple scripts are required for the different operating systems as the look and feel of Apple and Android is very different. Further work may also be required to revise the tests when a new operating system is released. Further, the need to ensure backwards capability can lead to the number of regression suites rising very quickly. Instrumented applications each of the test automation tool providers integrates with native applications via instrumentation, which requires changes to the application source code. However, for mobile devices, no operating system can currently be instrumented; so testing is solely on the application s functionality and not the behaviour of the OS when the application is accessed or running. Some types of instrumentation may also cause a dramatic increase in execution time. Page 9
4. Conclusion Test automation can bring significant cost savings and market advantage to organisations launching m-payments systems to their customer base. A recent SQS client in the finance sector realised the following benefits after introducing test automation into its mobile development programme: Reduced mobile regression testing time from three weeks to two days. As m-payments technologies spread far and wide across the consumer market the need to adopt automated testing solutions will only increase. Organisations that do it well and implement it first will be those who reap the biggest benefits from it and establish themselves as leaders automated testing solutions are key to achieving this. Improved test coverage over 100 end-to-end test cases developed across twenty handsets. Improved debugging support defects reported to R&D included video, screenshot, operating system logs and device vitals to facilitate understanding of issues across teams. Improved device coverage ability to support new devices (over 40) for automated testing. Improved quality automated end-to-end testing of transactions across mobile and web systems. Improved development intelligence via comprehensive reports automated mobile test results available in central location. SQS Software Quality Systems AG, Cologne 2014. All rights, in particular the rights to distribution, duplication, translation, reprint and reproduction by photomechanical or similar means, by photocopy, microfilm or other electronic processes, as well as the storage in data processing systems, even in the form of extracts, are reserved to SQS Software Quality Systems AG. Irrespective of the care taken in preparing the text, graphics and programming sequences, no responsibility is taken for the correctness of the information in this publication. All liability of the contributors, the editors, the editorial office or the publisher for any possible inaccuracies and their consequences is expressly excluded. The common names, trade names, goods descriptions etc. mentioned in this publication may be registered brands or trademarks, even if this is not specifically stated, and as such may be subject to statutory provisions. SQS Software Quality Systems AG Phone: +49 2203 9154-0 Fax: +49 2203 9154-55 info@sqs.com www.sqs.com Page 10