Whitepaper Effective Test Management can help you to launch mobile payments faster, smarter and cheaper sqs.com Ensuring Quality Assurance is implemented in complex systems Introduction The primary role of test management is to ensure that a quality assurance and testing programme is conducted efficiently and effectively. In principle, the role of test management for m-payment systems is the same as that for traditional systems, in practice however, there are important differences that can mean the difference between success and failure. The many combinations of mobile devices, operating systems and infrastructure create a logistical nightmare and not all organisations will have capacity to test all possible permutations. A 2013 global survey of 1,500 executives 1 found that over 56 % of respondents cited the lack of specialised methods and processes as the biggest barrier to mobile testing in their organisation. By balancing the variables to be tested against risk, an m-payment system test strategy can ensure that sufficient testing will be conducted within reasonable timescales and budget levels. Further, a testing programme for mobile succeeds by focussing on the unique characteristics of mobile devices. Considerations such as battery consumption, network hand-offs between mobile and Wi-Fi networks, required levels of security as well as user interface and App Store delivery need to be considered within the overall test strategy. This paper presents a pragmatic, risk-based approach to testing m-payments systems including issues such as organisation structure, development methodology and project risk. Author: Sven Euteneuer Global Head of Technical Quality sven.euteneuer@sqs.com SQS Software Quality Systems AG, Germany Published: March 2014 SQS the world s leading specialist in software quality
1. The mobile quality challenge There are three primary considerations to be addressed in mobile application testing programmes: Complexity: the complexity of the software, hardware and infrastructure involved Quantity: the number and variety of system configurations to be tested Characteristics: the characteristics of the system that need to be tested (SQS uses the term quality fingerprint to describe this) 1.1. Complexity of m-payment systems Studies have shown that there is often a strong correlation between the complexity of software and the number of defects in that software 2. Despite their apparent simplicity, mobile applications are often more complex than their desktop or webbased counterparts while offering fewer features and functions to end users. To understand the complexity of m-payments systems it is necessary to look at the system as a whole, not just the app. Outside the device, links to back-end systems and network connectivity add a layer of complexity, while inside the device, hardware, software and operating system variations result in mobile systems that are typically much more complex than traditional software development projects. This complexity makes effective test management and testing practices in m-payments crucial. The issue of complexity and, by implication, quality, applies throughout the whole life of the system, not just the initial release. Research by the Rochester Institute of Technology 3, which used lines of code (LOC) as a measure of complexity, highlighted how mobile applications are not only more complex than traditional equivalents, but also become more complex more rapidly when new versions are released. The research showed that mobile applications demonstrated a rapidly accelerating growth in complexity as new versions of the applications were released, while desktop equivalents tended to towards a steady growth in complexity. 1.2. Quantity and variety of testing configurations Complexity of the payment system itself is not the only issue, the sheer number and variety of mobile platforms upon which mobile applications must operate presents a major quality management challenge. Test strategies for m-payment systems need to support testing on multiple mobile devices across multiple platforms, but they must do so pragmatically. Consider: fully testing 420 device models over 29 Android versions would require 12,180 combinations to be tested for each test case. Even with test automation, which can dramatically reduce the effort required, is testing all 12,180 combinations just for Android worthwhile? Ensuring a secure and reliable customer experience against the backdrop of a technically complex and diverse mobile environment is one of the key challenges when planning quality assurance activity. 1.3. Characteristics of m-payment system quality the quality fingerprint To many the notion of quality is somewhat ethereal, yet to define a test concept for m-payments, it needs to be operationalised, i.e. there must be a way to measure it and determine when the desired degree of quality is achieved. Quality models are used to accomplish this and the ISO/IEC 25010 model is used as the de-facto standard in the IT industry. This model uses quality attributes to subdivide the general notion of quality into more tangible sub-characteristics (Figure 1, next page). As for every system, the specific combination and prioritisation of these characteristics varies and can be thought of as a kind of fingerprint. Just as fingerprints differ between virtually any two people, so too does, the specific combination of quality characteristics for an IT system the quality fingerprint. Page 2
Software product quality Functional suitability Reliability Performance efficiency Operability Security Compatibility Maintainability Transferability Appropriateness Accuracy Availability Fault tolerance Recoverability Timebehaviour Resourceutilisation Appropriateness Recognisability Learnability Ease of use Helpfulness Attractiveness Technical accessibility Confidentiality Integrity Nonrepudiation Accountability Authenticity Replaceability Co-existence Interoperability Modularity Reusability Analyzability Changeability Modification stability Testability Portability Adaptability Installability Figure 2: ISO 25010 Software product quality model Often the sub-characteristic of functionality alone is targeted by testing activities, but for m-payments systems, this is simply not enough. The quality fingerprint for m-payments stresses the importance of non-functional attributes such as security, reliability, portability and functional stability. To address the mobile quality fingerprint, a test strategy should also consider additional test objects such as networking, user interfaces, mobility and software delivery mechanisms as highlighted below. Network constraints User interface Mobility Software delivery Network throughput Fidelity Network hopping, handover Differing screen sizes and resolutions Touch based No mouse Soft-key only Finite battery life Need for frugal use of resources such as CPU Interruptions (Closed) app store Install via central download Mobile software product lifecycle Table 1: Additional test objects for mobile systems Page 3
In traditional software development, risk-based planning can be applied to determine the degree to which functionality should be tested which features and functions to prioritise and which aspects of the system can be simply smoke tested. However, when applied to mobile systems, the risk-based approach needs to be broadened to include platforms, operating systems and versions for each device the application will run on. Additionally, it needs to consider all test tasks that arise from the broader scope of the mobile quality fingerprint. This broader approach to risk-based testing facilitates the pragmatic allocation of people and infrastructure, helping to deliver a product that has been sufficiently quality assured while at the same time keeping testing costs from skyrocketing. Multiple Distribution Channels Dozens of Target Platforms Hundreds of Devices Vendor Figure 3: Diversity of platforms, versions and devices in m-payments systems Live Example SQS strategic solution reduces testing effort by 80 % for a major mobile payments platform SQS proven experience in m-payments gave a leading financial services organisation the confidence to appoint them as a strategic partner for its complex multi-device retail mobile banking services. Developing a test strategy and environments was challenging as testing needed to include six major operating system versions across both ios and Android devices. SQS built a test infrastructure to support an offshore testing model, enabling the team to execute tests remotely without the need for on-shore resource. SQS implemented test automation to increase the pace and quality of testing: the designed environments and infrastructure enabled the client to execute each test cycle of 2,000 test scripts in just 30 person-days, a saving of 80 % compared to equivalent manual testing. Page 4
2. Test organisation and development methodology Risk can help determine where resources are focused, but risk-based testing of itself is not enough. Speed, agility and efficiency in test management and development practices are key to meeting the challenge of delivering systems in the dynamic and complex mobile marketplace. Achieving the requisite speed, agility and efficiency, means underpinning testing practices with a systematic, processbased testing approach. With such an approach in place, test cases and other assets are created efficiently and the rate at which those assets are produced also rises. Re-using test assets also makes automation simpler an important benefit when dealing with regression testing across mobile devices and platforms. Adopting a systematic or disciplined approach to test management does not remove flexibility from the quality assurance process. Regardless of whether Agile or Waterfall methods are employed, test management needs to be tailored so that it provides the flexibility required to fit in with the development methodology adopted. Best practices such as early error detection and the shift left of testing (i.e. testing tasks being performed earlier in the development lifecycle) are typically, though not exclusively, seen in Agile development projects. Bringing these practices into an m-payments development project will increase effectiveness and ultimately reduce cost. Page 5
3. Test effectiveness and efficiency Project and programme owners sometimes regard testing as a burden that slows delivery. However efficient and effective testing not only supports product delivery, but also accelerates the delivery of a quality product. The testing team s ability to deliver effective and efficient testing that will increase the speed of product delivery is largely determined by the tools, processes and strategy adopted in short, the test management approach. 3.1. Doing the right thing: testing effectively Test managers need to think differently about mobile testing compared to traditional software testing projects: methods and processes need to be modified to make testing more effective without huge increases in budgets. But what needs to change in test management for m-payments? Risk needs to be used much more comprehensively in prioritising testing effort. By expanding the concept of risk-based testing, quality assurance effort can be laser-focused to mitigate the biggest risks and so maximise the testing budget. Minimising product risk within the available testing budget means prioritising effectively. How? By extending risk-based testing to include the unique characteristics of m-payment systems. Applying risk management at all levels enables test and development teams to allocate resources effectively. The chart below illustrates how risk and priority drive the effort allocated to testing, for example, while critical functional test cases will be executed on every relevant device at least once, test cases with a lower than moderate risk and priority might only be executed once on a single device. Less frequently used device configurations can be risk assessed by conducting smoke tests through crowd-based testing services while higher priority requirements, such as security, can be addressed by in-depth security testing. Risk Each test case is executed on every relevant device at least once. A B C 1 All All All 2 All All Once Each test case is executed at least once on one device. Priority 3 All Once Once Figure 4: Prioritising test effort by risk and priority Page 6
3.2. Testing more by testing efficiently Prioritising testing tasks helps to reduce the overall volume of testing work. Nevertheless, there is still a clear need to conduct testing efficiently and make best use of available resource while still achieving the desired quality assurance outcomes. There are several strategies to improve testing efficiency - two of the most effective approaches for m-payments are testing earlier in the development cycle and using both test tools and automation frameworks. 3.2.1. Testing earlier: shifting test activities left reduces cost and effort Shifting testing left introduces test activities earlier in the development lifecycle. Opportunity cost, re-test effort and correction effort are all typically much higher the closer a project or an agile iteration is to go-live. Identifying and eliminating a defect early is almost always cheaper and faster than addressing that same defect later. The benefit of shifting left can be realised across all types of testing, from functional requirements that describe what the product does to non-functional requirements that define qualities such as security, performance and scalability. An example of the value of shifting test activity left can be found when addressing one of the top considerations in m-payments for both end-users and providers security. Tasks such as threat modelling, static analysis and dynamic analysis can often be conducted early in the project and help weed out conceptual as well as technical flaws that would have turned into real-life vulnerabilities. Carrying out security-testing tasks earlier in the lifecycle helps to minimise the risk of unexpected vulnerabilities being discovered late in the project and incurring the corresponding cost of rework and/or delay. Conducting testing activity earlier in the lifecycle does not necessarily mean that testers need to execute test cases against software delivered by the development team. Techniques such as document review, developer testing or using analysis tools are all capable of identifying errors early, well before the start of classic testing activity. 3.2.2. Tools and automation increase speed and reliability A key target for test automation is testing the functionality of the user interface (the GUI) where the speed and reliability of automated testing can yield significant benefits, particularly when testing multiple devices and configurations. Other tasks that can be automated include: Test environment provisioning Test data provisioning Security testing Reliability testing However, the return on investment when implementing automation for these tasks should be examined carefully before proceeding. Test environments that support multiple device and operating systems can require a significant investment. It is important to assess the benefits of mobile test tools and mobile test automation products against the total cost of ownership. Page 7
4. Conclusion SQS 2012 iqnite survey found that testing accounts for a significant portion of project budget 4, while work at the University of Cambridge found that on average, developers spend 50 % of their time finding and fixing bugs 5. Ensuring quality can be a costly undertaking; the key is to conduct quality assurance and testing effectively and efficiently. Why do we spend so much time and effort on testing and quality? Not just for the sake of creating a quality product per se, but because creating a quality product makes sound financial sense. For m-payments, where the risk of financial and reputation loss is much higher than with most other systems, creating high quality software saves money: it reduces risk by ensuring that the end product is mature, reliable, cheaper to maintain and fit for purpose. No organisation wants to be subjected to the public and customer relations disaster of a failed m-payments system and effective test management will help to minimise the risk of such an event ever occurring. SQS has been providing test management and test execution services to corporates across the globe for over 30 years. With a track record of innovation in quality assurance, SQS consultants are leading the drive towards more efficient and effective testing of new technologies such as m-payments through techniques such as better risk management, test automation, environment management and security testing. Page 8
5. References 1) World Quality Report 1013-14, http://www.capgemini.com/resources/world-quality-report-2013-14 2) Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects, Subramanyam. R, Krishnan, M.S., 2003 3) The evolution of mobile apps: an exploratory study by Zhang, Sagar & Shihab, 2013. 4) Up to 30 % of project budget in 69 % of new projects: SQS iqnite survey 2012 5) http://www.prweb.com/releases/2013/1/prweb10298185.htm SQS Software Quality Systems AG, Cologne 2014. All rights, in particular the rights to distribution, duplication, translation, reprint and reproduction by photomechanical or similar means, by photocopy, microfilm or other electronic processes, as well as the storage in data processing systems, even in the form of extracts, are reserved to SQS Software Quality Systems AG. Irrespective of the care taken in preparing the text, graphics and programming sequences, no responsibility is taken for the correctness of the information in this publication. All liability of the contributors, the editors, the editorial office or the publisher for any possible inaccuracies and their consequences is expressly excluded. The common names, trade names, goods descriptions etc. mentioned in this publication may be registered brands or trademarks, even if this is not specifically stated, and as such may be subject to statutory provisions. SQS Software Quality Systems AG Phone: +49 2203 9154-0 Fax: +49 2203 9154-55 info@sqs.com www.sqs.com Page 9