Building a Successful Testing Partnership for Outsourced Agile Delivery

Whitepaper Building a Successful Testing Partnership for Outsourced Agile Delivery sqs.com Introduction The growing drive for an acceleration in the pace and quality of IT change is fuelling a global increase in demand for agile delivery. However, given the wide variance in the quality of the output delivered by outsourcing organisations, clients face a serious risk of selecting the wrong outsourcing partner and failing to realise the benefits associated with agile delivery as a result. There is a real need for organisations, working with suppliers who profess agility, to be confident that they are getting the right level of service delivery. A key way of achieving this is by assessing the suppliers approach to agile testing and quality assurance. Testing is so central to agile delivery that it acts as a very effective barometer of agile delivery standards. SQS agile testing specialists have experienced many of the challenges faced by organisations making a shift towards agile delivery methods. This paper will consider some of the key signs that your supplier may not be well structured to deliver high quality agile development as indicated by their approach to testing and quality assurance. We will help you to: Understand whether your agile service partner is a good fit for your needs Clarify the risks presented by their approach to agile delivery Implement a good, working agile model to ensure that your goals are met Authors: Ivan Ericsson (Head of Service Governance) ivan.ericsson@sqs.com Paul Wilford (Principal Consultant) paul.wilford@sqs.com Francis Balfe (Principal Consultant) francis.balfe@sqs.com SQS Group Limited, United Kingdom Published: October 2014 SQS the world s leading specialist in software quality

1. Agile methods have hit the mainstream Agile methods continue to grow in popularity 1. As a result of this growth, clients are increasingly asking outsourcers and Systems Integrators (SIs) to operate within agile delivery models. The speed to market, cost savings and risk lowering benefits offered by agile methods are creating a further wave of peer pressure as organisations perceive they will be left behind competitively if they cannot keep up. A recent article from Gartner (David Norton, Gartner, 2012) asserts that, while an increasing number of outsourcing companies offer agile delivery services to meet this growing market demand, there is a wide variation in the quality of these suppliers and their ability to deliver true agility. This in itself is no great surprise, but the supporting statistics for this are telling. 60 % of these outsourcers do not have any clear internal strategy for agile delivery, with a third of those (one in five of all suppliers) merely being a marketing response to the emerging agile market. A further 30 % of the total supplier pool have made a strategic choice to offer agile outsourcing, but may be at some stage of transformation to it themselves, sometimes with it being a veneer on top of a waterfall culture. So this means that up to 90 % of agile suppliers represent a gamble in terms of supporting agile adoption, and over 60 % of agile suppliers represent a gamble in delivering project success. If this seems troubling, the figures for success quoted 2 are more disturbing, with only one-third of all suppliers being structured to deliver a likely successful outcome. The conclusion of the Gartner paper is a call for greater end user due diligence in supplier selection. Sound advice, but if the client s agile experience is limited, they may not see through a veneer of agile experience if they perform the due diligence themselves. If the client does not perform the due diligence themselves, then they still have to find a suitable and impartial agile entity to help make the decision, which is a similar dilemma. The absence of any formal recognised agile capability or maturity model only compounds this challenge. % of outsourcers 40 30 20 10 0 Agile Marketing Tactical Project Focus Increasing likelihood to support successful agile adoption Figure 1: How agile are agile outsourcers Strategic Adoption but in transformation Agile Core Values Additionally, the inclusion of agile may be only one part of a wider and larger service delivery agreement which involves many other considerations. A recent report from the UK National Audit Office investigates the fact that a Department for Work & Pensions project, with an IT spend in excess of 300m so far, has been largely unsuccessful using agile methods with outsourced providers. The blame, however, is placed firmly with the Department for Work & Pensions for its poor governance and weak supplier management. One of the conclusions states: Given the tight timetable, unfamiliar programme management approach and lack of a detailed operating model, it was critical that the Department should have good progress information and effective controls. In practice the Department did not have any adequate measures of progress. 1. In Ambysoft s 2009 DDJ State of the IT Union Survey, 55 % of respondents stated that their organisation had applied agile processes. In the same survey in 2012, 70 % of respondents said their organisation had successfully applied agile processes. 2. Note that success is not defined, whether it be individual project success or a successful transformational adoption of established agile practices and culture. And in the case of project success, whether this is in terms of maximising business value through flexible change as opposed to just being on time and to budget with some acceptable amount of scope. Page 2

This paper identifies some common indicators that testing (and therefore agile delivery) may not be as strong as it could be, and provides some guidance around selecting the right agile outsourcing partner. Have you found the right outsourcing partner? Given that the principles behind the Agile Manifesto state that Working software is the primary measure of progress, it seems logical that one of the key ways in which suppliers can be measured is by the regular delivery of working software with increased value (complete new features) each time. So, if you are using one or several outsourced vendors across many projects, attempting to deliver using agile methods and practices, how can you tell if you are on the path to failure? Identifying failure patterns can provide useful learning as to the path an organisation is on. Both clients and outsource suppliers can fail to deliver on key agile principles. Here are some common ones. Do any ring true of testing in your organisation? 1.1. Supplier failure patterns Supplier failure patterns Agile principles broken Testing is not an integrated discipline Testers carry out their tasks as a separate function to the rest of the delivery team Testers are rushing to complete testing in the last few days/hours of a sprint Test team is testing one sprint behind the development team Hardening sprints are used to address the growing backlog of quality issues Non-Functional testing and Regression phases outside of the standard team iterations Working software is the primary measure of progress Continuous attention to technical excellence and good design enhances agility Lack of collaborative communication Testers become aware of changes to the details of requirements only when the feature is delivered to test Developers and product owners will discuss details without including a tester in the conversation Testers and product owners discussing issues without developers The most efficient and effective method of conveying information to and within a development team is face-to-face conversation Business people and developers must work together daily throughout the project Quality is not visible Test results and assets are the sole responsibility of the test team Testers cannot explain and demonstrate, in business-relevant terms, what they have tested and how they have tested it Reporting is based on test execution progress rather than test efficacy Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done Page 3

Supplier failure patterns Agile principles broken Conflict of interest Suppliers are reluctant to experiment with innovative new approaches Suppliers are reluctant to share knowledge, of an automation framework that they have developed, with other suppliers Our highest priority is to satisfy the customer through early and continuous delivery of valuable software People are not empowered Suppliers testers have no voice Roles, responsibilities and toolsets are explicitly defined in supplier contracts Staff are frequently working overtime to deliver on commitments Agile processes promote sustainable develop-ment. The sponsors, developers and users should be able to maintain a constant pace indefinitely Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done Automation is not an integral part of the process Developers are unable to assess the impact of changes to the system The number of regression tests is growing beyond the capacity of the team to execute them Automated test suites are hard to maintain and break frequently Acceptance tests are written by the testers after development has started Working software is the primary measure of progress Deliver working software frequently, from a couple of weeks to a couple of months, with a preference for the shorter timescale Supplier is unable to deliver incremental features Suppliers deliver frameworks and infrastructure in the first 3-4 iterations Suppliers teams are structured based on technology areas Suppliers focus on technology slices instead of business slices Working software is the primary measure of progress Deliver working software frequently, from a couple of weeks to a couple of months, with a preference for the shorter timescale Missing business engagement and leadership The supplier is delivering to specification but missing the business requirement Specification is an unprioritised set of requirements Clients do not have the time to attend stand-ups, planning sessions or demonstrations The most efficient and effective method of conveying information to and within a development team is face-to-face conversation Business people and developers must work together daily throughout the project Page 4

Supplier failure patterns Agile principles broken Up-front solutions Simpler solutions that deliver the value are turned down as they do not meet the signed specification Developers are over-engineering for all the just-in-case scenarios Technology solutions constrain the business s ability to change The best architectures, requirements and designs emerge from self-organising teams Simplicity the art of maximising the amount of work not done is essential Lack of commitment from client management Suppliers are hitting delays due to environment issues Blockers are not resolved in time Resources are moved around to fire-fight BAU issues Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done Inflexible contracts driving wrong behaviour Contracts defining, in detail, the scope of work to be delivered Suppliers are delivering to SLAs and contractual agreements but are still not adding value Suppliers charge changes to scope at a premium Welcome changing requirements, even late in development. Agile processes harness change for the customer s competitive advantage Project Governance function is not aligned with agile delivery principles Project Governance measuring the wrong thing Reporting is in depth but does not help the reader to understand reasons for good/bad progress Timesheets and utilisation reports are the primary measure of progress Working software is the primary measure of progress Stagnant Culture Clients unwilling to inspect, retrospect and futurespect how they interact with IT Clients force suppliers to use existing toolsets even if they are unsuitable for iterative development Projects encounter the same failure patterns with no impetus for resolving the root causes At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly Page 5

2. Solutions How can you tell if your outsource partner is working effectively with you to deliver business value with agility? This is a question of measurable agility and determining success criteria. One of the key things here is to address the right things first. If, as is likely, you establish the need for change for both you and your suppliers, then it is wise to prioritise your approach to change in a coordinated manner. In other words, it is vital to ensure that all parties are working towards the same ultimate outcome, and all understand what success looks like. While working with a systems integrator (SI) that also uses agile methods is important, more prosaic details about the contract with the SI are also important. Contracts need to reward progress and value rather than task completion and deliverables. Additionally, teams need to treat SIs as genuine partners, demanding transparent reporting and providing easy access to internal resources and systems and even coaching and training. Forrester report January 2010 2.1. Implementing a solution By targeting process changes in the right areas, there is often the potential to effect fast feedback and improvement. Agile principles provide an effective framework allowing us to: Ask the right questions in order to identify the problem(s) Define the right things to measure in order to understand the problem(s) Define and implement an improvement plan with prioritised changes Measure the effectiveness of that improvement plan Bear in mind that measuring will drive behaviour Focus on quality and value delivered Continue to monitor the situation, with the ability to repeat the cycle above as things change 2.2. Multiple Suppliers Given the growth of increasingly complex application architectures organisations may well find themselves using multiple suppliers to deliver a range of applications in support of their key business functions. Given that increasing numbers of these organisations are offering to support agile delivery, it is critical that their true level of agility is measured and the risks understood at the commencement of the engagement. This involves not only the supplier organisation but also the customer, and any other groups who play a part in delivery. Tying multiple suppliers together using a consistent framework, vocabulary and outcome-based approach may sound like a monumental task, but this is not something that organisations should shy away from. In fact, taking the same uniform approach with all suppliers should serve to reduce the size of the task. 2.3. Picking the right supplier(s) Some examples of expectations that a business should have of its agile delivery suppliers might be: Test automation should be integral to the development process; a supplier should be expected to work with the team to provide a framework for automation. That framework should provide tests that can be understood by the business stakeholders, are easy to maintain, are executed at the right level and are quick to run. The framework should allow regular scheduled execution (all tests daily is the target) and failing tests should be addressed immediately. This should not just involve existing automation assets but experience of building new, bespoke automated testing solutions to meet specific demands. Page 6

Suppliers should be involved in a team-wide effort to ensure that all reporting around status and progress is completely open and honest. When issues are impacting the pace of progress, it is the removal of these issues that should receive the most focus. By measuring the right things, teams can be empowered to get the best possible result for the business. For example, if you target testers on how many tests they execute, and the developers on how many bugs they fix, then lots of tests will be run, and lots of bugs fixed, but does this really add to quality? Possibly not, so measuring things like defects found in production and value delivered (and ensuring the backlog is prioritised by value at a story level) should encourage the team to focus on delivering a minimal number of defects and maximum value. Other metrics could include things like coverage of code by the automation framework and the number of automated tests failing more than x times in a set period. All organisations involved should be positioned to openly track progress against an agreed framework and this should be built into the contract. Any example of protectionism above collaboration should be stamped out. Suppliers should be expected to provide experienced test resources with the expertise and knowledge to help test drive development, assisting with the definition of acceptance criteria and working with development to automate these in order to provide a quality target for the code. At SQS we find that the most effective approach involves tailored services which blend the appropriate amount of training, coaching and mentoring for the whole team, including business stakeholders. It is not unreasonable for a supplier to also expect that the business representatives involved in the process are familiar with and committed to the agile way of doing things. Outsource providers are ideally placed to constantly improve their people and should be committed to a culture of continuous improvement in conjunction with all of their clients. This empowers the people provided by these suppliers to continually drive innovation and supports them in building bespoke, efficient solutions. The effect of all this should be to encourage a more productive and longer-lasting relationship. This is likely to be one of the key differentiators that set 10 % of outsourcers apart from the rest. 2.4. You are equally responsible for success A business must also be prepared to implement the appropriate internal changes to support delivery. Some examples of this are outlined below: In agile software development the role of customer is crucial. In Scrum, the most popular agile process, the Product Owner is the customer representative with the responsibility to make decisions on priority, risk and scope. They act as a knowledgeable conduit of customer need to the delivery team. The Product Owner needs to have access to all the required Subject Matter Experts to make informed decisions. This access needs to be enabled by management for the full duration of the project; from inception through to completion. With this access the project can deliver what the business needs rather than what the supplier believes the business needs. Note: business needs not business wants. To ensure that the business gets the highest-value items first, the business needs to be able to prioritise its list of requirements, regardless of how many stakeholders are involved. The priority needs to be a linear, absolute prioritisation based on the value of each feature, not just a MoSCoW or Low/Medium/High grouping. The business also needs to be able to compromise on each and every feature. If the team are unable to deliver the required feature with the resources they have, the client should be able to decide to accept a reduced feature or provide the additional resources required. The business and project team should not compromise on the iteration length or on the quality of the feature. The contracts for an agile outsourced project need to reflect this flexibility in scope with the understanding that changes in scope will affect the length of time to deliver the project or the scope of work that can be delivered in a fixed-length project. A good contract is likely to be based on output of value and cost of ownership of delivered solutions. This approach will reward suppliers who improve their efficiency and effectiveness. Page 7

3. Principles for Successful Agile Testing Obey the principles without being bound by them Bruce Lee ((source unknown)) Rules are for the obedience of fools and the guidance of wise men Douglas Bader Traditionally, test management has tended to make the same fixed processes, practices and deliverables apply to every team, project and environment in the same way with little regard for context. Test strategies are written, with weight in kilograms seemingly the main objective over contextual substance and often by managers who will not be responsible for delivering the work. At best this is a recipe for abandoned shelf-ware and at worst you have an inflexible testing philosophy that leads to inefficiency and ineffectiveness, which does not meet the needs of a team looking to improve, grow and adapt. Processes do not guide you in decision making, particularly around areas of uncertainty, and trying to bend the context to your approach in this way is surely the wrong way round; just ask King Cnut. Instead of providing a strategy that makes testing tick, you can constrain a team from improving and deprive them of the power and trust to make change, often demotivating people in the process. 3.1. Proven principles At SQS, our agile teams prefer to apply principles over prescriptive processes when establishing team policies for testing. In support of this we have developed the following mnemonic framework of principles to guide teams. This does not tell teams precisely what processes and tools to implement; rather it provides guidance on what to consider and how to apply the core values of the Agile Manifesto. Done includes Testing Explicit Working Practices Living Documentation Improve Continuously Visible Testing Empirical Measurements Review Regularly Business Involvement Risk Driven Apply the Hive Mind mentality Value Focused Exploratory and Context Driven Testing Feedback Early & Continuously Automate Appropriately Collective Test Ownership Team Approach Share Knowledge Figure 2: Mnemonic Framework of principles Page 8

Done includes Testing Before a story or feature can be considered done it must have been subject to all forms of testing required to meet stakeholder acceptance (including functional and non- or cross-functional) Note that truly done means live, being used and serving its business need. Not just tested and accepted This definition of done also needs to include other agreed standards for testing, e.g. automation, reporting, test quality, storage and so on Explicit Working Practices High visibility of key activities and practices makes for smoother communication, adoption (e.g. by new team members) and whole-team decision making Explicit working practices allow the team to operate independently without management intervention and regulation Living Documentation By writing acceptance tests in advance of development we set a target for developers, which reduces the potential for waste in terms of re-work Tests written in the business domain language are an executable documentation library that describes what the system does In order to do this they must be kept running and passing. This can help reduce the total cost of ownership of a software product over time Improve Continuously We are constantly looking at ways to improve the way we work We set specific goals and actions to achieve them and monitor the effect of our actions We attack impediments that stand in the way of our efficiency and effectiveness Visible Testing Tests must be clear to those that need to act on the information that they provide. Those people must be able to read, understand, analyse failures and maintain them The output from testing should visibly highlight failures, risk and importance Test process policies must be explicit, disseminated and understood by the whole team There should be a clear mechanism for storing, maintaining and organising test assets Empirical Measurements We want to measure team (and testing) efficiency and effectiveness The most valuable measurement is running features in live (tested and in use) We select metrics and measurements that we feel drive the correct behaviour We need to be able to measure the impact of changes we make to both process and product Review Regularly We review work regularly and collectively The closer to the event we review, the cheaper and easier to address any issues We allocate specific time (slack) to reviews, to implementing actions and to innovation Business Involvement The closer to the real end user of the product, the more accurate the feedback Regular business involvement catches errors and changes earlier Business people and testers can learn from each other in pairwise testing The closer to something happening we action an improvement, the better Page 9

Risk Driven We want to focus more of our efforts towards areas of greater business and technical risk The context of risk will vary between projects, teams, products and people We value defect prevention over detection Apply the Hive Mind When issues and impediments arise, swarming as a group will resolve them more quickly Specify collaboratively, get perspectives on testing from other disciplines Harness the power of rapid group test design, scenario and bug bashes Value Focused Work should be prioritised by business value, and so should testing effort Keep in mind the value of information to relevant stakeholders Awareness of business goals and value aids better decision making in testing Exploratory and Context Driven Testing Try to optimise the time spent in test execution, improve through simultaneous design and execution Learn more about the product you are testing from other people (pairing) and find issues that machines cannot find Structuring our approach to exploratory testing (missions, time-boxes, repeatability, recording) will give more focused and valuable information Use project and feature context to inform the testing performed Feedback Early & Continuously We value feedback from tests alongside development to help inform and guide it Testing is a pervasive technique that is fundamental to the development of working software, not a phase following development The faster we get feedback that we have broken something, the easier it is to locate and fix Appropriate Automation Automated tests enable faster release cycles by reducing manual (especially regression) testing Automate at an appropriate level; lower and isolated is normally quicker and more reliable Make the tool fit the needs of those interacting with your tests and your culture; never the reverse Consider the cost-benefit for automating tests, particularly the harder to automate and broader ones Collective Test Ownership Anyone who reads and interacts with tests should be able to interpret and maintain them Everyone should be able to work on, add to and update any area of testing Everyone has a responsibility to hold the line when tests are broken, find the problem and fix it Team approach Testing and quality are a collective responsibility; everyone should feel responsible for them The ownership for creating and maintaining tests is a collective team responsibility Working closely together when building and testing a product builds better problem-solving capability Share Knowledge Share knowledge frequently; when we find something new and interesting, we share it to help others We build a knowledge repository for testing and maintain it visibly (e.g. online, intranet, wiki) We promote a community around testing such that we can share and amplify learning amongst the team Page 10

3.2. Adherence to test design principles Tests should be designed to meet the needs of the situation. As is illustrated in Figure 3, a foundation based on strong Environment & Build Management, Data & Dependency Management, and Flexible Test Automation frameworks is essential. From that, the six pillars provide a means to better understand your approach. Rapid feedback, a proven track record and clear test intentions will provide confidence for the courageous. But whilst courage may be enough for some situations, those requiring more safety will look more towards clear test intentions, visible results and high test coverage. In the majority of scenarios, however, a balance of all five features is likely to be required, and the agile approach to test design should consider all of these things. Confidence Courage Safety Rapid Feedback Credible & Regular Track Record Clear Test Intentions Visible Results High Test Coverage Flexible Test Automation Frameworks Testability, Data & Dependency Management Environment, Build & Configuration Management Figure 3: The pillars of testing 4. Conclusion As more organisations are looking to use outsource models to deliver agile projects, the abilities of the outsource providers are being put to the test. Gartner has found that many are unfortunately falling short as they lack the maturity to provide effective agile services. Organisations need to have high expectations of their suppliers in the testing arena if they are to succeed, and this article highlights a few areas where suppliers should be looking to deliver high value in agile teams. If you are about to start on the journey of agile software development with outsourced suppliers, then you should take the time to understand what the role of your own people will be, and also ensure that the contracts are correctly defined to measure and encourage the correct behaviours. If you are already on this journey, and find yourself working with one of the 90 % of suppliers that are falling short of expectations in agile delivery, then hope is certainly not lost. If you work out what good looks like, and make the right changes, then success can still be an option for you and your suppliers. Page 11

5. References / Bibliography http://www.ambysoft.com/surveys/stateofitunion200911.html http://www.ambysoft.com/surveys/stateofitunion201209.html Gartner report Mainstream Outsourcers Offering Agile Development Will Require Greater End-User Diligence David Norton, 7th June 2012 UK National Audit Office report - Universal Credit: early progress - http://www.nao.org.uk/wp-content/uploads/2014/09/ Full-Report.pdf - 5th September 2013 Forrester report - Agile Development: Mainstream Adoption Has Changed Agility - Dave West and Tom Grant, Ph.D., 20th January 2010 Bruce Lee - http://en.wikiquote.org/wiki/talk:bruce_lee (source unknown) SQS Software Quality Systems AG, Cologne 2014. All rights, in particular the rights to distribution, duplication, translation, reprint and reproduction by photomechanical or similar means, by photocopy, microfilm or other electronic processes, as well as the storage in data processing systems, even in the form of extracts, are reserved to SQS Software Quality Systems AG. Irrespective of the care taken in preparing the text, graphics and programming sequences, no responsibility is taken for the correctness of the information in this publication. All liability of the contributors, the editors, the editorial office or the publisher for any possible inaccuracies and their consequences is expressly excluded. The common names, trade names, goods descriptions etc. mentioned in this publication may be registered brands or trademarks, even if this is not specifically stated, and as such may be subject to statutory provisions. SQS Software Quality Systems AG Phone: +49 2203 9154-0 Fax: +49 2203 9154-55 info@sqs.com www.sqs.com Page 12