Distributed Engineered Test Efficiency Improvement

Transcription

1 Whitepaper Distributed Engineered Test Efficiency Improvement sqs.com Boosting Test Automation with a Managed Testing Service Approach Authors: Heinrich Grefe, Commercial Director Bert Kleinikel, Junior Delivery Manager Christian Marx, Head of Test Centre Görlitz SQS Software Quality Systems AG Germany Published: August 2013 SQS the world s leading specialist in software quality

2 Distributed Engineered Test Efficiency Improvement 2 HEINRICH GREFE Commercial Director heinrich.grefe@sqs.com Heinrich Grefe studied economy and media computer science. He works at SQS Software Quality Systems AG as a Commercial Director. He is an accredited Principal Assessor and Appraiser in the environment of SPICE and CMMI, and a Quality Assurance Management Professional. Since January 2003 Heinrich Grefe has had complete responsibility for selected customer accounts and is responsible from the stage of acquisition to the provision of, and acceptance of, services. He joined SQS in Bert Kleinikel Junior Delivery Manager bert.kleinikel@sqs.com Bert Kleinikel has been with SQS since In this time he has worked on various client projects in the telecommunications, insurance, networking service provider/social networking and civil service sectors. His role involved test preparation and design, and controlling and coordination of small teams at the Görlitz Test Centre. In January 2012 Bert Kleinikel took up the position of Junior Delivery Manager. In this function he supports delivery/project management onsite and is responsible for delivery/project management offsite. Christian Marx Head of Test Centre Görlitz christian.marx@sqs.com Christian Marx studied mathematics and started working in consultancy in Thus he has a long experience mainly in software engineering, project management, quality- and test management. Due to his manifold project assignments he has special expertise in different business domains, especially in the public sector and industrial manufacturing (automotive and steel production). He joined SQS in 2007 and since October 2011 he has been Head of Test Centre Görlitz, Germany.

3 Distributed Engineered Test Efficiency Improvement 3 Contents 1. Management Summary Introduction Managed Testing Services Professional Testing and Process Maturity Market Current Status and Outlook Optimisation of Testing Test Automation as MTS Starting Position Challenge and Solution Reasons for the Improvement Project Set-Up of the MTS Test Automation Project Project Phases Project Organisation Managing the Transformation Project Achievements and Lessons Learned Conclusion and Outlook Bibliographical References...19

4 Distributed Engineered Test Efficiency Improvement 4 1. Management Summary For over 30 years now, SQS AG has been a recognised partner for software testing and quality management to its clients. As of 2012, the number of successful projects was in the 5,000s and this number is steadily growing. The IT market as a whole has seen major changes in terms of service orientation and outsourcing in recent years. Accordingly, the testing market has been evolving, and testing services have emerged in SQS portfolio in order to meet the clients need to transfer more responsibility for testing tasks to SQS so that clients can focus on their core businesses. In particular, SQS offers Managed Testing Services to leverage mutual benefits in terms of pricing, scalability and efficiency of testing. In a managed service setting, SQS takes over full responsibility for testing tasks from its clients and typically delivers results on the basis of a transactional pricing model at competitive rates. Consequently, as a service provider of Managed Testing Services SQS can leverage further advantages e.g. effective and efficient resource and capability management, or the optimisation and automation of testing to increase efficiency and reduce the duration of test cycles. The service provider s benefits can be shared with its clients and provide them with a stable, reliable testing service based on a clearly defined pricing model that can be deployed flexibly. Clearly, Managed Testing Services may not be suitable in each and every context, and they do come with prerequisites but experience shows that many of the popular concerns can be mitigated and Managed Testing Services can be applied in a host of different environments with great success. The transformation of time & material testing projects into transaction-based Managed Testing Services is one of the most essential steps towards Managed Testing Services and requires a clearly defined scope. The present whitepaper reports on a Managed Testing Services project that was started in order to increase test efficiency with uncompromised test coverage for a client who formerly worked on a time & material model. It shows how improvement projects can be implemented as Managed Testing Services and achieve tremendous results with regard to time to market for business software releases while keeping the highest level of quality and coverage of the test assets, and consequently boosting efficiency in testing.

5 Distributed Engineered Test Efficiency Improvement 5 2. Introduction 2.1. Managed Testing Services According to SQS and the market (Simon & Simon, 2012), Managed Testing Services are typically defined as follows: Managed Testing Services (MTS) are managed services for testing-related tasks across one or more projects delivering one or more applications and systems, spanning the life cycle of software and system development and system operation. The resources for testing (staff, testing infrastructure, system under test) are managed by and under the responsibility of the service provider, to support the customer s business processes. Based on transaction- or business value-based prices, the service provider ensures scalability of the MTS and takes care of the resource management according to the utilisation scheme required to support the tasks. On the provider s side, delivery of the testing activities can be managed with full flexibility and thus leverage the commercial advantages of e.g. resource pooling and task relocation across different regions, which ensures competitive pricing and the highest possible scalability for the clients at the same time. The established standard process for implementing the MTS decision is depicted in Figure 1. The process typically starts with a Pre-Engagement phase to clarify the constraints and the set-up of the MTS project. Next, the MTS Framework Design is developed to build the platform for the target state of MTS Operation. While the first and the second steps of the process are described in detail in Simon & Simon (2012) and primarily deal with political decision-making and commercial negotiations, the scope of the present whitepaper provides a more detailed look into the MTS Operation phase (see Figure 2). Definition 1: Managed Testing Services (MTS) 1. Pre-Engagement 2. MTS Framework Design 3. MTS Operation Figure 1: Transformation process for MTS

6 Distributed Engineered Test Efficiency Improvement 6 3. MTS Operation Continuous Improvement Readiness Assurance per Application Knowledge Transfer & Collaborative Delivery Service Operation Service Completion Figure 2: MTS Operation takes MTS from idea to implementation 2.2. Professional Testing and Process Maturity There are three different types of prerequisites that must be fulfilled before the benefits of MTS can successfully be leveraged on the client s side: Industrialised Process Chain: The organisation and processes on the customer s side must be mature enough to support the division of labour across organisational units and different locations. This typical state of industrialisation has at least four prerequisites: 1. Modularisation: The overall business process chain must be defined to an appropriate level and comprise activities, responsibilities, dependencies, and results. 2. Standardisation: Based on the modularisation, the process landscape as well as the interfaces must utilise standards to guarantee a specific level of quality. 3. Automation: For each activity within each process step, the highest possible level of automation has to be considered. 4. Focus on Core Competencies: The fundamental decision about whether it makes sense to utilise MTS can only be taken after a conscious management assessment of the core competencies of the business. MTS Volume: The degree to which MTS deliver commercial value depends on the size of the testing effort and the number of applications subject to MTS. Typically, the benefits of MTS are realised in three- to five-year engagements. Openness for Change: Setting up MTS in an organisation implies the shift of labour from internal (or body leasing or outtasking) to external resources having their own responsibilities. Ideally, the customer already has experience in the field of supplier management and can build upon this experience to collaborate with the service provider.

7 Distributed Engineered Test Efficiency Improvement 7 The project we are reporting about here had already successfully mastered the prerequisites. The customer operated with a business model driven by business processes, and was ready for MTS in terms of his industrialisation level. In fact, the customer had already outsourced most of his testing activities to SQS, and the collaboration was well established. The project size and complexity was significant, therefore the volume of MTS-delivered testing tasks per release of the business software had enough room for commercial benefits. 3. Market Current Status and Outlook Conservative estimations anticipate that by 2015 about 20 % to 25 % of all testing activities provided by test service specialists will be realised as MTS. For independent test organisations, this amounts to an overall business size of up to 121 million of revenue for MTS (i.e. 25 % of the total market size as listed in Figure 3). In addition to independent test organisations, established full-service providers will try to move into the market of MTS. However, the more mature a client s organisation is (characterised by greater process awareness and insight into its core business competencies as opposed to tasks to be outsourced), the more important he will consider attributes like provider exchangeability and dependency reduction, in order to avoid provider lock-ins as well as output-based pricing. These considerations strengthen explicit testing awareness on the client s side and will naturally lead to the application of MTS provided by test service specialists. Further evidence of the benefits of independent testing providers (Wikipedia contributors, 2011) has been quantified in recent empirical research (Simon & Simon, 2013). From the point of view of SQS, as of 2014 the proportion of MTS in the market will be in the range of 40 %. As depicted in Figure 4, revenue in MTS is expected to grow from 10 % of overall revenue in 2011 to over 40 % of overall revenue in 2014, to a total of 120 million.

8 Distributed Engineered Test Efficiency Improvement 8 m SQS Tescom DNV-IT Tesnet Plan IT Thinksoft Imbus FHG/IESE Amsphere MTP Acial ps-testware Quality House Maverick QA Infotech RTTS Genilogix Squerist Experior Immune Assurity KJR Figure 3: Overall revenue of independent test organisations (NelsonHall Report, 2010) Total revenue in m MTS revenue in m Figure 4: Revenues of the market leader in MTS

9 Distributed Engineered Test Efficiency Improvement 9 4. Optimisation of Testing Test Automation as MTS 4.1. Starting Position The customer is a major insurance company with subsidiaries all over Germany. It is leading in its domain, and its business processes depend on efficient and reliable IT applications. Therefore, the customer operates and develops a tailored business-critical Java application involving several business departments. The customer s core competencies had clearly been identified outside the IT and consequently, outsourced testing tasks have been delivered by SQS since The business application is a core insurance application consisting of four different modules matching four different business departments in the value chain of the customer. Each business department is responsible for exactly one of the modules, as depicted in Figure 5. For testing purposes, a homogeneous approach to the definition of testware had been applied, and the use of test tools was harmonised by means of SQS-TEST/Professional for test case and test data definition. For a small number of test cases, SQS-TEST/Professional also served as a tool for the management of test automation scripts. The execution of the automated test scripts was implemented by a specialised tool for running business process tests at the GUI level. A typical test cycle took ten weeks per release to execute the test cases required for regression testing and the testing of new or changed functionality. In total, the testware comprised approximately 3,000 business process test cases. Up until 2009, the customer and SQS had successfully collaborated on a time & material model, and most testing tasks had been delivered onsite at the customer s premises. Business Department A Business Department B Business Department C Business Department D Module A Module B Module C Module D Core Insurance Application Figure 5: Structure of the project s IT applications and business organisation

10 Distributed Engineered Test Efficiency Improvement Challenge and Solution In the course of 2009, the project parameters for the evolution of the customer s core insurance application changed, and the pressure to reduce software release cycles without compromising functional quality significantly challenged the overall release planning and development and testing processes. For the quality assurance part of the customer s release plan, SQS considered various options of how to optimise the test assets to meet the challenges, and identified test automation as the means of choice. This measure was aimed at the following: Reducing test execution times by means of automated execution of regression tests Maintaining or improving the level of coverage of business processes by test cases (i.e. improving test coverage) by reusing the existing testware as the basis for automation Consequently, identifying deficiencies faster and shortening feedback loops for communicating software errors to development units Thereby reducing testing costs and induced-error costs In the light of the anticipated lifespan and numerous future releases of the core insurance application, it was technically and commercially desirable to develop robust and sustainable automated test cases so as to limit maintenance effort for future software changes. The original test suite of the business application consisted of approximately 3,000 test cases to be executed per software release. According to an initial estimate, out of these 3,000 test cases about 2,300 qualified for a regression test suite to be executed repeatedly, and as such were in the focus of test automation Reasons for the Improvement Project Due to the overall release schedule, it was decided to implement the test automation effort in parallel to the further development and testing of the core insurance application. Further, in order to achieve the ROI with regard to the test automation cost and realise the benefits of automation for software releases as soon as possible, the timeframe of the Test Automation Project was limited to five months. SQS proposed to the customer to run the test automation as a managed service project for a number of reasons: The transformation process towards MTS (as described in Section 2.1) was considered ready regarding the Pre-Engagement and the MTS Framework Design phase, due to the previous long-term engagements of the customer and SQS. From the perspective of the MTS Operation process (see Figure 2), the readiness of the application was given and the knowledge transfer had already taken place, with a continuous improvement procedure being well established. Because of the amount of work involved and the extremely tight deadline for the automation of all tests, execution of the project onsite seemed unwise. On the other hand, SQS Test Centres were in an excellent position to fulfil resource demands within a short timeframe. The customer s business organisation was distributed across Germany, and so were its IT interfaces. Consequently, the change towards a distributed testing approach with SQS Test Centre delivering the major part of work was an easy step to take.

11 Distributed Engineered Test Efficiency Improvement 11 The project s scope was clearly focused on, and restricted to, the business process automation. From the very beginning, it was possible to provide solid estimates for the efforts since SQS had a solid testware at hand as a basis for automation and was well familiar with the processes and particularities of the customer s application. The total price for the service was above 1 million, and as such significant in size. From a commercial point of view, the project was set up MTS-based on average prices to automate test activities per test case, if the test cases could be delivered as promised according to schedule. In case SQS did not deliver according to schedule, penalty fees were agreed per missing test case. The total duration of the MTS Test Automation Project was 14 months, with the transformation of manual test cases to automated test cases taking the first five months in four intermediate milestones, specifying the number of automated test cases per milestone. In line with the business goals, the project goals focused on the following: Reducing overall test execution times per release Increasing test coverage per test cycle and per release without compromising on effectiveness of testing Identifying issues in business process testing faster Reducing test cost per test execution Reducing impact cost for errors 4.4. Set-Up of the MTS Test Automation Project As a critical success factor for the project, the focus lay on business process automation and high-level test cases rather than low-level technical automation of individual test steps. The full test suite for the customer consisted of about 3,000 test cases. Of those, a total of 2,300 business process test cases had been identified as having the potential to be automated, and thus were in scope of the automation effort Project Phases The Test Automation Project was planned in five phases to achieve the overall goals (see Figure 6): 1. Revise and Refactor: In this initial phase, the test cases had to be analysed for up-to-dateness with respect to business specifications to ensure a solid basis for automation. In cases where gaps between test case and business specification were identified, the gaps were closed in liaison with the business experts of the client. This collaborative task was carried out by SQS onsite teams. At the same time, it was assured that the test cases fully complied with standard structures to facilitate later automation. 2. Establish Standard Structures: After refactoring and aligning them with business specifications, the test cases were then reworked to follow standard naming conventions in a second phase. Next, for the modularisation of test cases, the test steps were adopted and building blocks for test cases were created. Finally, the test cases were linked with the necessary test data. The entire project was delivered from SQS Test Centres with bridgeheads at the customer s premises in two cities.

12 Distributed Engineered Test Efficiency Improvement 12 Revise and Refactor Establish Standard Structures Automate Execute and Fine-Tune Deliver Release Tests Figure 6: Project phases for MTS Test Automation 3. Automate: The third phase tackled the automation of business process test cases itself by the test automation teams located in the Test Centres. The teams established the preconditions for the test cases and restructured test cases to separate GUI components from business logic steps for ease of maintenance in future releases. The test cases were evaluated and the automated execution was set up per test case. 4. Execute and Fine-Tune: In the fourth phase of the transformation, the tests were executed under tight control and quantification of test execution efforts and timeframes. This included the preparation per automated test case, its automated execution, as well as the effort required for the analysis of the test results. In this phase, the overall success of the transformation was evaluated. Finally, through the experience gained, a precise estimate on the maintenance effort for the automated test cases could be derived. 5. Deliver Release Tests: The final phase represented the return to the steady state of MTS and aimed at delivering two iterations of test executions of the automated test cases, with benefits for two designated business software releases Project Organisation The project organisation s set-up was driven by the business organisation s structure of the customer and its corresponding core insurance application structure. From SQS perspective, the organisational set-up of the client was mirrored in both test automation and test execution. For the initial refactoring efforts, a small onsite team was deployed in a bridgehead role, taking care of the interface to the customer s business experts. The onsite refactoring team was backed up by a larger offsite refactoring team in the Test Centre which provided sufficient manpower to carry out the refactoring tasks. In line with the size of the business software modules (and the corresponding initial testware), four teams of test automation staff were set up in the Test Centre. The organisation chart is depicted in Figure 7: the team set-up of the improvement project is aligned to the client s business organisation, and the average head count of the team sizes reflects the complexity of the tasks with regard to the individual business modules. In order to facilitate interaction between onsite and offsite staff and to provide an adequate escalation mechanism for resolving automation issues, communication channels were established between the offsite teams and designated responsibles onsite. The communication chart is depicted in Figure 8.

13 Distributed Engineered Test Efficiency Improvement 13 Project Management (PM Lead) Refactoring Team Onsite Refactoring Team Test Centre Module A TPA Test Centre Module B TPA Test Centre Module C TPA Test Centre Module D TPA Test Centre Team Lead Refactoring Team Lead Alpha Team Lead Bravo Team Lead Charly Team Lead Delta Figure 7: Organisation chart: Team set-up of the improvement project Test Management (Onsite) TM Lead Module A Onsite Team Module B Onsite Team Module C Onsite Team Module D Onsite Team Test Manager Alpha Test Manager Bravo Test Manager Charly Test Manager Delta Module A TPA Team GR Module B TPA Team GR Module C TPA Team GR Module D TPA Team GR Team Lead Alpha Team Lead Bravo Team Lead Charly Team Lead Delta Project Management (Offsite) PM Lead Figure 8: Set-up of communication lines for the teams for improved project delivery (onsite-offsite split)

14 Distributed Engineered Test Efficiency Improvement 14 To capture the required business knowledge and to gain access to business input relevant for testing, SQS utilised the bridgeheads at the client s site. For refactoring and test automation tasks, technical test experts were deployed in SQS Test Centre in Görlitz. In the course of the project, even the resources in the German Test Centre approached maximum utilisation, and action had to be taken in order to keep the promised delivery schedule. In this situation, the idea of MTS showed its strength since it was possible to augment the staff of one Test Centre (in Görlitz) with staff from another: as the legal and regulatory context of the Test Automation Project allowed for the transfer of individual tasks to other legal regions, SQS made use of additional resources in its Test Centre in Cairo, Egypt, to account for peak workload and keep the timelines. This increase of staff was not transparent to the customer and did not have commercial side effects other than informing peers of the transfer for formal reasons, no changes in the process were necessary nor was there any change in the level of quality Managing the Transformation Project The management and implementation of the distributed Test Automation Project started with a challenging time horizon. Within less than five months, a vast amount of test cases had to be reviewed, reworked, and automated. Consequently, SQS project management team followed an engineering approach towards project management and applied different strategies to handle and manage the tasks. Some of these strategies were the following: Rolling wave planning with detailed planning for the current week and the next, and only a high-level plan for the following weeks Per-head planning of resource availability Daily status reporting per sub-team, and consolidation onto a project dashboard Daily status meetings to recap project progress and manage project risks Generating forecasts for subsequent activities As the minimum level of documentation required for managing the project, the documents shown in Figure 9 were produced and maintained.

15 Distributed Engineered Test Efficiency Improvement 15 Initiating Documents from Pre-Initiating Project Charter Planning Master Test Plan Work Breakdown Structure Scope Baseline Schedule Baseline Budget & Cost Baseline HR Plan Executing, Monitoring & Controlling Adapted Scope Adapted Schedule Adapted Budget & Cost Adapted HR Plan Performance Report Closing Invoice Acceptance Lessons Learned Figure 9: Minimum set of project management documents required for successful management

16 Distributed Engineered Test Efficiency Improvement 16 Refactoring Module A Status CW 01 (04/01/10 08/01/10) CW 02 (11/01/10 15/01/10) CW 03 (18/01/10 22/01/10) CW 04 (25/01/10 29/01/10) CW 05 (01/02/10 05/02/10) CW 06 (08/02/10 12/02/10) CW 07 (15/02/10 19/02/10) CW 08 (22/02/10 26/02/10) Refactoring Module A plan Refactoring Module A revised plan Refactoring Module A forecast Refactoring Module A achieved Automation Module A Status CW 01 (04/01/10 08/01/10) CW 02 (11/01/10 15/01/10) CW 03 ( /01/10) CW 04 (25/01/10 29/01/10) CW 05 (01/02/10 05/02/10) CW 06 (08/02/10 12/02/10) CW 07 (15/02/10 19/02/10) CW 08 ( ) Automation Module A plan Automation Module A revised plan Automation Module A forecast Automation Module A achieved Figure 10: Performance status dashboard of refactoring and automation for Module A

17 Distributed Engineered Test Efficiency Improvement 17 An example of performance reporting of the refactoring phase of Module A is depicted in Figure 10. Thanks to daily progress reporting, the overall reporting and management was aggregated into a management view that allowed for fast mitigation action in case there was a deviation from plan Achievements and Lessons Learned The goals set out at the beginning of the project were fully achieved. In particular, the achievements are the following: Test automation of 2,300 test cases based on business processes was completed according to schedule (no penalty fees were incurred during the project). Efficiency improvement in terms of test cycle times of 50 % was reached without compromising the test coverage level of the business processes. The first benefits were generated when the first new releases of the software had to be tested even while the test automation was still on its way. Consequently, a faster identification of errors in business process tests is now a reality. Due to shorter test cycle times, the overall release process is more efficient. In retrospect, the lessons learned from the implementation of the project regarding methodology are as follows: The transformation of manual test cases to automated test cases requires solid and wellstructured test assets. The maintenance of automated test cases is facilitated by the structured approach. The clear separation of GUI components from business logic additionally increases the reuse of the automated test cases, and if changes occur, test automation can be adapted quickly as only the GUI parts need to be adapted. The multi-release testing capabilities of SQS- TEST /Professional contribute to easily maintainable regression test suites and enable the application of already automated test cases while the automation project is still underway. The existing test specification (i.e. the basis for the initial steps in the transformation) must be mature and of excellent quality low quality of test assets poses a high risk for test automation projects as a managed service. The availability of test environments is crucial. A solid, enterprise-level test environment management is on the critical path of MTS and should receive the highest level of attention. In SQS Test Centres, test environments are provided on enterprise level, and can be scaled up or down quickly however, this process must be thoroughly tested and customised to the transformation project s specific needs. Ramping up specific business knowledge to an adequate level within the test automation teams posed a particular challenge. It turned out, however, that for test automation only little business knowledge was required to refactor and structure existing test cases. The technical skills were far more important when dealing with test automation issues the few business-related questions were sorted out by the onsite teams, and the client never had to interface with the test automation experts directly.

18 Distributed Engineered Test Efficiency Improvement 18 From the managerial perspective, the engineering approach to project management showed its value. The most important factors are: Tight project controlling: the challenging constraints of the improvement project such as fixed deadlines and penalty payments for nondelivery provided a strong argument in favour of rigorous controlling of the project so that effective and fact-based management could be realised. Splitting the teams into different groups in line with the client s organisation resulted in a competitive set-up and inspired the Test Centre teams to deliver an extraordinary quality and pace. Following the project engineering approach, daily status meetings were not used for surveillance but rather as a forum for demonstrating progress and fixing issues early. The organisational set-up of the project with many staff in the Test Centre and bridgeheads onsite with the client contributed to clean and lean communication channels. Responsible contact persons were always known, and interfaces to onsite teams were clearly defined. In summary, the conclusion is that distributed test automation is possible! At least, that is, with a reasonable bridgehead onsite and clear interfaces to the customer. SQS teams were easily ramped up or down via resource pools in Test Centres at different locations without the customer noticing a difference. Internally, it is important to have the right mix in the teams getting one senior team member (or team lead) on board who is used to collaborating across long distances, is a must. As in all projects things change over time. However, as the payment model is very precise, strict change management procedures will be required if tasks are extended or if the scope of the managed service shifts. For example, business process changes induced by the review steps at the beginning of the transformation will entail efforts to adopt business process test cases accordingly. Since the project was paid on a per-test-case basis, there was little room for tolerance, and changes had to be tracked rigorously to bring transparency into the additional efforts.

19 Distributed Engineered Test Efficiency Improvement Conclusion and Outlook Delivering standard testing tasks as managed services is an established delivery model for mature organisations. In this whitepaper, we reported on a more challenging task: implementing a test optimisation project as MTS by means of test automation. If the prerequisites are met, test automation can be delivered as MTS with great success in MTS as discussed above, test cycle times were reduced by 50 % without compromising test coverage, at the same time reducing overall cost for future regressions. The payment model for MTS even in the case of test automation can simply be transaction-based. In our case, the test cases were the key driver for cost in the transformation and execution of the service. 6. Bibliographical References NelsonHall Report. Independent Testing Providers vs. System Integrators. Cologne: SQS AG, April 2010 Simon, F., & Simon, D. Managed Testing Services. In: Thought Leadership Cologne: SQS AG, 2012 Simon, F., & Simon, D. QS im Wandel: Test-Unabhängigkeit. In: OBJEKTspektrum, To appear. Wikipedia contributors. Independent Test Organization. [Cited: 27/07/2011]