Information Commissioner's Office

Transcription

1 Information Commissioner's Office Review of the Agile process used to develop the ICE system Ian Falconer Partner T: E: Last updated 15 March 2013 Will Simpson Senior Manager T: E: Simon Edwards Associate Director T: E: Distribution Timetable For action Fieldwork completed 14 December 2012 Daniel Benjamin Director of Corporate Draft report issued 16 January 2013 Services Simon Entwistle Director of Operations Management comments 1 February February 2013 For information Christopher Graham Information Commissioner Audit Committee Final report issued 22 February 2013

2 Contents Sections 1 Executive Summary 1 2 Detailed Findings 4 A Internal audit approach 10 B Definition of internal audit ratings 13 C Overview of Methodologies 14 This report is confidential and is intended for use by the Management and Directors of The Information Commissioner's Office only. It forms part of our continuing dialogue with you. It should not be made available, in whole or in part, to any third party without our prior written consent. We do not accept responsibility for any reliance that third parties may place upon this report. Any third party relying on this report does so entirely at its own risk. We accept no liability to any third party for any loss or damage suffered or costs incurred, arising out of or in connection with the use of this report, however such loss or damage is caused. It is the responsibility solely of The Information Commissioner s Office management to ensure that there are adequate arrangements in place in relation to risk management, governance and control.

3 Glossary General terms Capita ICO ICE IT Project Board Agile Waterfall ICO's current IT service provider Information Commissioner's Office ICO Customer Engagement (replacement for Data Protection Notification System) Information Technology Body responsible for representing ICO, monitoring progress and providing authorisation to project on decisions Software development methodology based on iterative and incremental development, where requirements and solutions evolve through collaboration between selforganising, cross-functional teams (see Appendix C) Software development methodology based on a sequential process of requirements analysis, design, implementation, testing (validation), integration, and maintenance. Often referred to as "Traditional" (see Appendix C) Waterfall development terms FRS NFRS SRS Functional Requirements Specification Non Functional Requirements Specification System Requirements Specification UAT RC User Acceptance Testing - final stage before release Release Candidate potential live version of software Agile development terms Epic Requirement for a piece of software typically covering a number of business processes and made up of a large number of Stories eg web site to enable customers to maintain their registration Story A new or upgraded capability of the software able to be delivered within 1-2 weeks by a small number of team members. For each Story an estimate is made of the number of units of work, eg days or hours. Sprint A group of Stories to be delivered together within a 2-6 week period. Scrum Velocity Backlog Done Daily team meeting also known as stand-up meeting The number of units of work completed in a certain interval e.g one week. Outstanding Stories Acceptance criteria for a completed Story

4 1 Executive Summary 1.1 Background The ICE project is designed to replace the ICO's main customer facing system, including the registration of organisations and individuals under the Data Protection Act, and to create a website for self service. Currently the systems used to manage complaints and investigations are excluded from the scope of the project, although it is anticipated that the ICE system will be extended to cover these functions at a later date. The initial business objective was to replace the system in The project began in 2010 and Capita was appointed as the main contractor, but with some specialist work such as the website subcontracted to other companies. The project was developed using a methodology known as 'Waterfall' (see Appendix C) and a detailed Functional Requirement Specification was agreed that is still being used as the basis for the system requirements. In mid-2012 the project began to slip against its original schedule and estimates for testing the software underpinning the new and untested system became unaffordable. In order to address the emerging issues, and improve the speed and cost of managing the systems changes and improvements required, the Project Board decided to adopt a new development methodology, "Agile" (see Appendix C). It formed an Agile team comprising members of management, users and IT Development and based them in the Operations area close to the Registration team who will be early users of the new system. 1.2 Scope The objective of our review was to identify any significant risks associated with the use of the Agile approach to develop a new system. The key subrisks we considered were that: the revised scope and approach to the project may not meet all of the ICO's original business objectives, and/or its use of an Agile methodology may lead to a loss of focus on those business objectives; the ICO may not be ready to adopt an Agile development methodology, leading to development delays or disruption to its business as usual operations; and the ICO may not apply sufficient rigour to the testing phase to ensure a usable system is available to users and/or excessive bugs remain in the system resulting in additional rework and expense to resolve the operational issues. Further details of our scope and approach can be found in Appendix A. 1

5 1.3 Audit Opinion Our focus during this phase of the audit has been on the revised scope of the project, whether the adoption of Agile will affect future success of the project, and the robustness of the approach to testing etc adopted by the ICO to date. Design effectiveness Overall, we have concluded that, except for the specific weaknesses identified by our audit, in the areas examined, the risk management activities and controls are suitably designed to achieve the risk management objectives required by management. The Agile approach has only been partially adopted and the controls within the Agile methodology are not yet fully in place. ICO is still trying to work in a sequential "Waterfall" methodology. This could delay progress on the ICE project. Operating effectiveness Except for the controls listed below those activities and controls that we examined were operating with sufficient effectiveness to provide reasonable assurance that the related risk management objectives were achieved during the period under review. Management needs to ensure that ICO fully adopts the Agile approach across all functions affected by the ICE system, to ensure that the initial release and subsequent enhancements to the system are delivered, implemented and supported successfully. Further details of our findings and recommendations are provided in Section 2. Amber Amber 1.4 Key findings The following table details the key findings from our review. Further details of our findings and recommendations are provided in Section 2. Risk / Process High Medium Low Improve't Meeting original objectives ICO not ready to adopt Agile Sufficiently robust testing approach Total Refer to Appendix B for definitions of internal audit recommendation ratings. We have made no high priority recommendations and four medium priority recommendations. Each aims to help the ICO to establish an appropriate approach using the Agile methodology in the future. Our four medium rated findings are as follows: The Project team is responsible for determining priorities for the Backlog (outstanding work) but has no clear business priorities to support its decision making. The Project team has not documented the benefits, tests and acceptance criteria against each requirement. The ICO has only partially implemented the Agile methodology, with only limited testing of work in progress, no planning to adapt its operational IT support arrangements once ICE has gone live, nor to alter IT processes to accommodate the more frequent releases of software that the new methodology will generate. Some ICO teams who are key to the ultimate success of the ICE project are currently not directly represented on the Agile team, nor are they operating within the Agile methodology. 1 2

6 1.5 Basis of opinion The ICO has adopted an Agile development methodology and established a multi-disciplined team to apply it. The team is working cohesively to develop software in relatively short periods of time, i.e within two to six weeks (these short development phases are known as Sprints). A appropriate team working dynamic is in place within an environment that allows collaborative working. Staff are able to operate as a virtual team and are being allowed to concentrate on Agile matters which will enable an effective operating model to develop. The key areas that have yet to be established are arrangements for complete testing within each development phase and preparations to allow a wider community within the ICO to operate with the Agile methodology once the project has gone live. 1.6 Acknowledgement We would like to take this opportunity to thank the staff involved in for their co-operation during this internal audit. 3

7 2 Detailed Findings 2.1 Meeting original objectives 1. Medium Business priorities not clear to the project team Finding and Implication Proposed action Agreed action (Date / Ownership) The project's current objectives are related to the need to replace the Management to establish clear business Existing backlog reviewed to ensure current system that will become unsupported in the future and to reflect priorities for the ICE project future business benefits clearly prioritised. the potential changes to Data Protection requirements of the European development, and the criteria to be used by Union. the Agile team when deciding which stories to work on next. Complete - December 2012 To deliver this, the ICO needs to establish a set of criteria to allow the Agile team to determine priorities for development. Therefore, when determining what is next for development, it is unclear how this will be done when management delegates the authority to the Agile team to determine priorities based upon a common agreed criteria, in line with Agile methodology. Other business priorities exist but are not clearly stated and used when the team prioritises the Backlog (outstanding functionality) of Stories (discrete software functionality to be delivered) e.g. Longer term, set up an agile steering group to support our agile team by identifying priority areas for system development and articulating business benefits to be realised Taken forward by Simon Entwisle February 2013 Customer service Cost Time Saving Reduction in postage and paper The implication is that future system enhancements will not necessarily lead to an improvement in business performance, as described by the benefits, as business priorities will not be formally considered when the Agile team decides what to do next. 4

8 2. Low Existing ICO project reporting ineffective on the ICE project Finding and Implication Proposed action Agreed action (Date / Ownership) There is a disconnect between Agile project controls and existing ICO project reporting which is more geared to the Waterfall methodology. The gap is being filled by project team management trying to control in one way and report in another. Management to resolve the problems with the Agile reporting and use this to control the project going forward. Project reporting/demonstrations now utilise further agile sprint information from Team Foundation Server dashboards to add further project controls. This problem is compounded by the uncertainty of completeness at the end of each sprint, and delays to testing. This results in difficulties in controlling time and cost Complete - January 2013 Longer term as part of the steering group set-up the ICO is planning to review its project management approach with a view to implementing an agile approach. Review to begin in May Steering group 5

9 2.2 ICO not ready to adopt Agile 3. Medium Functional Requirements have not been defined in full Story format Finding and Implication Proposed action Agreed action (Date / Ownership) As a consequence of the change in methodology mid-project, the Management to ensure that all new Review of user stories carried out and redefined to include full story format including existing Functional Requirements Specification continues to be used as requirements are fully documented in Story the reference for many of the requirements. However, the FRS does format and contain benefits, tests and acceptance criteria. All future requirements not contain all of the information required to follow the Agile acceptance criteria. to be articulated in full user story format. methodology, particularly; Benefits Tests Acceptance Criteria (Done) The consequence is that the absence of test and acceptance criteria makes it time consuming to specify system test requirements and has delayed testing until later Sprints. Complete December

10 4. Medium Agile team skills and working methods Finding and Implication Proposed action Agreed action (Date / Ownership) The current team includes all the skills required to develop and test each Sprint. The team is strongly linked to the business function and gives regular demonstrations, building trust and effective sharing of information. Team working and communication within the team appears good. Management to ensure that all functions necessary for the success of the project are directly represented on the team and operating in Agile methodology. However, the team does not include the skills to be able to migrate data and go-live. These need to become part of the team as soon as possible to encourage a one-team approach, essential for Agile. Two key individuals, the Scrum Master and one of the testers, are leaving the team shortly. This will leave a gap, particularly the Scrum Master who is the person most able to communicate across all areas of the project, technical, business, and management. Test resource returned to the project full time; scrum master retained part-time. The data migration resource had been managed at arms-length by the scrum master; however this resource is now working as part of the agile project team. Going forward agile teams will be adequately resourced including long-term assignment of scrum master and product owner; additional resources such as data migration expertise will be bought in for specific sprints/activities. Complete January Low Agile process not fully integrated into business Finding and Implication Proposed action Agreed action (Date / Ownership) The current team is strongly linked to the business function it is directly supporting, but business functions not on the project team are not yet fully aware of, or following the Agile methodology. It is difficult to work in both Agile and non-agile ways at the same time. Management to educate all areas of the business on the new way of working with IT, and what people can expect if they are involved in an Agile project. Working in a matrix management style is also not fully understood and appreciated by all functions impacted by the project. This is the normal way of working with an Agile methodology where the project team changes over time as the project looks at new functions and processes. It is felt that matrix management and cross office working is a well-established working practice at the ICO. However, longer term as part of the steering group set-up the ICO is planning to review its project management approach with a view to implementing an agile approach which will include education on the agile approach and lessons learned from ICE. There is potential for creating barriers between functions working in different ways, and some resistance to new ways of working Review to begin in May Steering group 7

11 2.3 Sufficiently robust testing approach 6. Medium Main Agile controls and methods not fully implemented Finding and Implication Proposed action Agreed action (Date / Ownership) Under the Agile methodology it is assumed that the completion of a 1/As per finding 2 above, management to 1/All new requirements now fully Sprint means all Stories in that Sprint have been completed, are ready ensure that all new requirements are fully documented in Story format and contain a for use, subject only to final checks and data migration, and that a documented in Story format and contain definition of done. release to production will be undertaken at the end of every Sprint. benefits, tests and acceptance criteria. However, in the ICE project: there is no agreed definition of "Done" for the project, and test and acceptance criteria are missing from the initial FRS documentation and also from new Stories too much testing remains at the end of each Sprint a full system test of all completed Stories has not yet been undertaken whereas Agile would require this to be done frequently full testing is not possible except in the production environment which was only available late in the project, and once live can no longer be used for testing product demonstrations show the systems operating but the level of completeness of testing and readiness for release is misleading no planning for Agile support post go-live has been undertaken the current release process is not designed to support regular releases of new Stories The implication is that the main Agile measure of progress, "Velocity", does not fully reflect actual progress on the project and therefore end date estimates are not reliable. 2/Management to provide an environment that allows full end to end testing including interfaces at any time. This will be required on a permanent basis to support testing of both maintenance Stories and new Stories 3/Management to plan post go-live support under Agile. 4/Management to review the calculation of Velocity and assumptions made, and agree a reliable measure of progress. Complete December 2012 Accepted. We have production, staging and development environments for both the CRM and web components. These will carry over as the service goes live and be permanently available. The level of testing that can be carried out in each environment is different and subject to some restrictions. The staging environment is a very close replica of live with the exception that it does not interface automatically to the external components. Where full end to end testing is considered necessary from the staging environment to any external component then manual processes/steps and controls will be put in place to transfer data. It is considered that the small number of changes likely to require full end to end testing are low, the external components are loosely coupled and do not require immediacy of data transfer and can be handled with manual intervention without impact the testing schedule or the confidence in testing Closed January /Post go live support has been agreed 8

12 6. Medium Main Agile controls and methods not fully implemented Finding and Implication Proposed action Agreed action (Date / Ownership) with Capita through to July 2013 for CRM, support for the servers and SQL databases. Support of the ICE configurations will lie with the Agile development team through a post go live support period. More detailed planning will take place through March and April to agree how the CRM and Agile team handle both support and new developments. Owner :- Head of Corporate services and Operations. April /As part of the steering group set-up the ICO is planning to review its project management approach with a view to implementing an agile approach which will include a review of the calculation of velocity and agreement on a reliable measure of progress. Review to begin in May Steering group 9

13 A Internal audit approach Approach and Scope The objective of the review was to identify significant risks when adopting Agile approach to developing a new system, for future phases and projects. We focussed on the following sub risks: the revised scope and approach to the project may not meet all of the ICO's original business objectives, and/or the use of an Agile methodology may lead to a loss of focus on those business objectives; the ICO is not ready to adopt an Agile development methodology, leading to development delays or disruption to business as usual operational; and the ICO may not apply sufficient rigour to the testing phase to ensure a usable system is available to users and/or excessive bugs remain in the system resulting in additional rework and expense to resolve the operational issues. Our internal audit approach is based upon the underlying principles of the UK Corporate Governance Code (2010) guidelines on internal control that require management to identify, assess and manage the risks that are significant to the achievement of the organisation s overall business objectives. Our role as internal auditor is to provide objective and independent assurance to the Audit Committee and management that it is doing this successfully for each of the areas being audited. Our audit was carried out in accordance with the guidance contained within the Government s Internal Audit Standards (2011) and the Auditing Practices Board s Guidance for Internal Auditors. We also have regard to the Institute of Internal Auditors guidance on risk based internal auditing (2005). In accordance with our agreed internal audit plan, we agreed to undertake a review of the agile process applied to the Information Commissioner's Office's ICE project to replace key systems within ICO. This review will further inform our on-going understanding of ICO's governance and risk management activities. We achieved our audit objectives by: meeting with key staff to gain an understanding of the arrangements in place, building upon the information we have already gained through our audit planning process; assessing the effectiveness of the control mechanisms in place to mitigate identified risks; discussion of key findings with management and preparation of a draft report. The findings and conclusions from this review will support our annual opinion to the Audit Committee on the adequacy and effectiveness of internal control arrangements. 10

14 Responsibilities It is the responsibility of management to ensure that there are adequate controls and activities in place to ensure that the ICO's business objectives can be met and that the risks to the ICO are minimised. Based on the work we have carried out, we provide an objective assessment of the adequacy and effectiveness of controls and activities established by management to manage the identified risks to the ICO. During the course of our review we have conducted interviews and, where necessary, testing/verification work to support our assessment of the adequacy and effectiveness of current arrangements. It is our reporting protocol to balance our reporting of positive practice with areas for attention. This enables the ICO to build upon its strengths, whilst focusing upon key findings and associated recommendations, which if acted upon, should enhance the control environment and improve the management of key risks. Please refer to our letter of engagement for full details of responsibilities and other terms and conditions. Additional information Client staff The following people were consulted as part of this review: Daniel Benjamin, Director of Corporate Services Simon Entwistle Director of Operations David Wells, Head of IT Paul Arnold, Head of Customer Contact Agile Team Members Emma Dean Uno Smith Traci Shirley Andrew Jarvis Julia Harrison Ian Campbell Richard Bennett Vijay Deshpande Brian McCone Simon Mackel Locations The Wilmslow Office was visited during the course of this review: 11

15 Documents The following documents were reviewed during the course of this engagement: Changes - user journey.docx Deploying Microsoft Dynamics CRM Solutions from Development through Test and Production Environments.pdf Development progress tracking spreadsheet xlsx ICE - Operational readiness backlog v0.1.xlsx ICE Backlog pre TFS.xlsx Non Functional Requirements - V1.0.docx ICE project weekly update meeting - agenda and actions.msg Indigo ICE project roles.docx Indigo_CRM_Functional design specification_v61 - ICO.docx IRCICERegParticularsPaperJuly2012v1 0.docx Microsoft Word - ICE Reqirements Final v1.0.pdf MicrosoftDynamicsCRM2011-ImplementationGuide-Planning.pdf Ops readiness plan Jan-Apr 2013.pdf Ops readiness timeline Jan-Apr 2013.pdf P ICE - Plan - Budget.xls P ICE - Plan - Project Initiation Document.docx PB update msg Registration Application - user journey.docx Renewals - user journey.docx Service transformation summary Sept12.docx Simplified NoW requirement.docx Sprint 6 and 7 outstanding.xlsx Sprint five backlog.docx Sprint four backlog.docx Sprint one backlog - pre TFS.xlsx Sprint one backlog.docx Sprint summary - sprint five.docx Sprint summary - sprint four.docx Sprint summary - sprint one and two.docx Sprint summary - sprint three.docx Sprint summary - sprint zero further update to docx Sprint summary - sprint zero interim to docx Sprint three backlog.docx Sprint two backlog.docx 12

16 B Definition of internal audit ratings Internal audit opinion Design effectiveness Opinion Operating effectiveness Rating We have not been able to form an opinion on whether the internal controls examined have been designed to achieve the risk management objectives required by management Overall, we have concluded that, in the areas examined, the risk management activities and controls are suitably designed to achieve the risk management objectives required by management Overall, we have concluded that, except for the specific weaknesses identified by our audit, in the areas examined, the risk management activities and controls are suitably designed to achieve the risk management objectives required by management. Overall, we have concluded that, in the areas examined, the risk management activities and controls are not suitably designed to achieve the risk management objectives required by management. No opinion can be given Green Amber Red We have not been able to form an opinion on whether the internal controls examined were operating to provide reasonable assurance that the related risk management objectives were achieved during the period under review Those activities and controls were operating with sufficient effectiveness to provide reasonable assurance that the related risk management objectives were achieved during the period under review Except for the controls listed below those activities and controls that we examined were operating with sufficient effectiveness to provide reasonable assurance that the related risk management objectives were achieved during the period under review. Those activities and controls that we examined were not operating with sufficient effectiveness to provide reasonable assurance that the related risk management objectives were achieved during the period under review No opinion can be given Green Amber Red Audit issue rating Within each report, every audit issue is given a rating. The ratings are summarised in the table below. Rating Description Features High Medium Low Improvement Findings that are fundamental to the management of risk in the business area, representing a weakness in control that requires the immediate attention of management Important findings that are to be resolved by line management. Findings that identify non-compliance with established procedures. Items requiring no action but which may be of interest to management or best practice advice Key control not designed or operating effectively Potential for fraud identified Non compliance with key procedures / standards Non compliance with regulation Impact is contained within the department and compensating controls would detect errors Possibility for fraud exists Control failures identified but not in key controls Non compliance with procedures / standards (but not resulting in key control failure) Minor control weakness Minor non compliance with procedures / standards Information for department management Control operating but not necessarily in accordance with best practice 13

17 C Overview of Methodologies Waterfall (traditional) systems development methodology Waterfall development methodology generally start with a business analyst designing the new or revised business process, followed by production of a Functional Requirements Specification (FRS) detailing all functional capabilities of the new or revised system to support the business process. This FRS is signed off by the user departments as the start of the development. General system features such as performance, user access, security and other non-functional requirements are documented and approved in a Non-Functional Requirements Specification (NFRS). Following selection of the technology to be used, a development team produces a System Design and takes the FRS and NFRS documents and turns them into System Requirements Specifications (SRS). Each part of the system is then built and tested individually by the development team against these SRS documents. The parts of the system are then assembled into one release for testing by the user departments. Following initial user testing, bugs and change requests are fed back to the development team which either reworks the system, or moves change requests into a later release. Once an acceptable level of functionality has been achieved, and migration of live data has been tested, User Acceptance Testing (UAT) of a release candidate (RC) is undertaken. Once the RC has successfully completed a final (UAT), the system is handed over to production, users are trained, migration of live data is undertaken and the new or upgraded system goes live. The process then repeats for further releases. The key features of this process are: extensive formal documentation of exactly what is required, produced well in advance, and signed off by the recipients. a series of handovers between several different departments and business functions. testing and acceptance generally done at the end of the process releases to production done infrequently typically every few months missing or incomplete requirements identified during development often have to wait until later releases. Agile development methodology Software development methodology based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organising, cross-functional teams. The team (often located into one area of an office) is created with all key skills required to deliver and implement a completed system including business analysis, design, end-user experience and knowledge, end-user management, project management, development, testing, training, migration and IT operations. 14

18 Initial meetings clarify the overall scope of the new or revised system and processes, documented as one or more Epics. In parallel, User Journeys are documented, the typical routes through a system by end users. For each Epic, workshops create Stories of the following form: As a...role I want.capability so that... BENEFIT. how will I know...test. I am satisfied if.done.. Most stories are driven by user roles and relate to business or functional requirements. However, Stories can also relate to non-functional requirements and support for the work of the team members. A Story should be as self-contained as possible and at a level of complexity which can be fully developed and tested with a couple of weeks. The Stories are added to a Backlog which becomes a prioritised list of outstanding Stories, and records progress on completing Stories. The key features of this process are: documentation produced on an as required basis no handovers as all key skills are in the same team testing and acceptance done all the way through the process rather than at the end control of scope and end dates down to the skill of the Scrum Master who must have technology, business and project skills missing or incomplete requirements identified at every stage and simply added to the Backlog no distinction between a development project and maintenance; maintenance requests are simply more Stories added to the Backlog The team is led by a Scrum Master who acts in the role of team facilitator and project manager. The team meets each day for a short meeting, called the daily Scrum, often undertaken standing up to reinforce the need for a short meeting. Anyone in the organisation can attend a Scrum as an observer. The Scrum Master prioritises the Backlog and agrees at a Scrum a number of Stories to work on over the next period, known as a Sprint. A Sprint will last for between 2 and 6 weeks, and at end of the Sprint the outcome is normally a group of new or changed system capabilities, tested and ready for release. Team members take responsibility for a specific Stories and can talk to other team members and request support at any time. The Scrum reviews new Stories, progress and issues each day. The Scrum Master maintains an updated Backlog, and uses Velocity, a measure of the number of Stories and effort expended per period, and the size of the Backlog to estimate time and cost for the project. 15

19 Grant Thornton UK LLP. All rights reserved. "Grant Thornton" means Grant Thornton UK LLP, a limited liability partnership. Grant Thornton UK LLP is a member firm within Grant Thornton International Ltd ('Grant Thornton International'). Grant Thornton International and the member firms are not a worldwide partnership. Services are delivered by the member firms independently. This publication has been prepared only as a guide. No responsibility can be accepted by us for loss occasioned to any person acting or refraining from acting as a result of any material in this publication.