Disaster Recovery and Contingency Planning

Transcription

1 ITEM: 7(ii) AUDIT COMMITTEE 2 NOVEMBER Nottingham City Homes Disaster Recovery and Contingency Planning June Final Report Executive Summary & Action Plan Assurance Level: Partly meets expectations Audit Sponsor George Pashley Staff Interviewed Ian Rabett, Robert Allen, Robert Barton, Dave Kelly, Kate Watret, Glenn Langham, Jas Padam Audit Team - Nicola Higginbottom, Ashley Homburg

2 1 Introduction 1 Executive Summary 1.1 We have carried out the audit in accordance with the programme agreed with management and the Audit Committee. Based on the audit work carried out we have concluded that the level of control over disaster recovery and business continuity planning is: Partly Meets Expectations. 1.2 The Organisation is currently progressing the development of its continuity arrangements, and a coordinating role for this process has been allocated to the And Safety. Some areas of the business have well developed incident response plans, and the process in place to develop an overarching business continuity plan is currently gathering information on key business processes and their criticality. 1.3 From our experience at other housing providers we have observed that the current arrangements at NCH are largely aligned with those commonly seen throughout the industry sector, with some areas where these exceed the usual arrangements seen at other social housing providers. Our conclusion (partially meets expectations) is based on our concerns that, at the time of our audit, the arrangements that were in place would not fully meet the recovery needs of NCH. 1.4 NCH is fully reliant on Nottingham City Council for delivery of its ICT service, covering network and core IT servers. Service arrangements are currently defined at a high level, and do not include details of system recovery processes, continuity arrangements or service levels. The Interim Director of ICT is currently negotiating with the Council for a detailed service description and service level agreement. Further details of the Council s infrastructure and resilience arrangements are also being requested. There are several known areas of the ICT infrastructure where reliance is placed on a single piece of ICT hardware or network connection, increasing the likelihood of prolonged service loss after hardware failure. The Finance, Human Resources and Payroll departments have their own contracts with the Council, which include IT systems provision but are out of scope of the ICT service contract. 1.5 We noted a number of areas where the current business continuity plan development process could be enhanced: The current plans have been based on recovering from issues affecting specific departments, for example snow affecting staff access to the customer call centre; or loss of the Nottingham on call building. The overarching business continuity plan may be improved by aligning it to disaster event scenarios, driven by risks identified in the NCH risk register, and supporting coordinated organisationwide recovery activities. The continuity plan development is being progressed by one staff member. Good practice suggests that a business continuity team, including representatives of the key business functions, should work together to develop a Business Continuity Management System (BCMS). 1.6 During the course of our audit the electrical fire at the Lenton Court occurred. We observed the approach adopted in response to this incident by NCH and noted that the recovery effort was carried out in a logical and common sense manner. The 78 tenants were not put in unnecessary danger and were found temporary accommodation while the issue was resolved. Other than the list of emergency contacts there was no business continuity plan or event card covering this scenario. 1.7 Finally, we wish to thank all members of staff for their availability, co-operation and assistance during the course of our review. PKF (UK) LLP June June Executive Summary 1

3 2 Action Plan R1 There is currently no BCMS board to ensure that development of the BCMS is aligned to cross-business needs. Section of ISO:27000 outlines the good practice for Business continuity programme management and governance. A more project-based approach to the development of the BCMS should be adopted. This should include: Formation of a BCMS development board to coordinate activities between departments; Creation of a project plan identifying the tasks that need to be completed, such as continuity plan / event card development, procurement of supporting infrastructure, contracts, kit, etc; Budgeting for resource requirements; and Development of a programme of plan testing. Recommendation requires consideration by the Executive Management Team. George Pashley Director of OD End of August R2 The Business continuity planning process does not currently link with the NCH risk management processes, and does not include all business risk scenarios (that require a Business Continuity mitigation response). The current corporate risk register is being revised, and should be available from mid-july. The BCMS process should be linked to the NCH risk management process and cross-referenced to the revised risk register when available. Recovery scenarios should be based on recognised business risks, and recovery options should consider all (key) departments affected by the event. The latest version of the risk register will be considered and BC risks incorporated into the BC plan. Our BC procedures to be reviewed to ensure that this process becomes a matter of course. End of September June Action Plan 2

4 R3 Many of the current business continuity documents pre-date the current BCMS development process, and vary in structure, version control, storage, etc. All existing BCP documentation should be migrated to the current, userfocussed format and they should be stored in an appropriate repository (e.g. on the Intranet with off-site paper copies) and version managed. Agreed. This requirement had already been identified. Priority has been afforded to assessing the BC needs in critical services where no current plan exists. End of Nov R4 There is a text alert system in place to alert staff when an incident has occurred, but no defined core incident response team to identify which of the alerted staff are required to support the recovery process, or when their input is required. An incident response team should be nominated. Their primary role should be to attend incidents (where safe to do so) and identify, mobilise and coordinate other NCH resources and responses as appropriate. The text alert system should be used to place staff on stand-by, but (other than the nominated response team), their presence on site should be by specific request only. The text alert system will remain in place as this works well for us. The recommendation to develop an incident response team will be examined in the first instance as part of the Lenton Court debrief process. End of August June Action Plan 3

5 R5 The processes for providing critical information early after the incident are not fully documented. There is an opportunity to improve the speed at which critical information is provided to the emergency services. We understand that a debrief has been planned which will include input from NCH staff, the fire and rescue team and the Nottingham City Council emergency planning team to identify (information) improvements. Following the planned debrief, NCH should document the information requirements of all the interested parties (i.e. emergency services, NCH and Council) The processes for providing this information in a timely, efficient and user-friendly manner should be documented and supporting infrastructure put in place. The technology for this recommendation was recently introduced at NCH, and was used effectively during the incident at Lenton Court. It is agreed that this could have been utilised earlier, and needs to become part of a documented process. The scope will be examined in the first instance as part of the Lenton Court debrief process. End of August R6 Tenant communication was performed by door-to-door visits by NCH staff after the Lenton Court fire was confirmed as extinguished. Other communication channels could have been deployed earlier (e.g. via the CSC) to inform tenants of the risks and what was required of them. NCH should consider how emergency tenant communication procedures can be made more efficient. This could include using CSC and/or Nottingham on Call to contact tenants and where required their associated support workers / family members. This recommendation will be discussed in the first instance with call centre management as the resources available vary according to the time of day and day of the week. End of August June Action Plan 4

6 R7 There was no emergency response kit, meaning the response team had to spend time locating basic equipment in support of their recovery response. A number of emergency response kits should be prepared and distributed to sites / members of the emergency response team as appropriate. These should be small / portable and include equipment such as: Torches High visibility jackets First aid kit Megaphone Pads and pens The business continuity plan Agreed. Number and content of kits to be identified. End of September. R8 A contract with the Council for ICT service is being renegotiated to include service level targets, service descriptions, etc. As finance, HR and Payroll have their own contracts (which cover systems), they are not included in this negotiation process. All service level agreements, including those covering Finance, HR and Payroll systems, should include clearly defined scopes, service levels for systems and infrastructure availability, resilience controls (e.g. backup) and the expected response times in case of a disaster. As we understand that service delivery for Finance, HR and Payroll are being migrating to shared service models, the revised contracts should also include these prerequisites. Draft ICT SLA has been authored with NCC. Final version shall include details of how the ORACLE system shall be supported by NCC (and EMSS through NCC) if affected by an IT failure. As ORACLE is used by NCC support and contingency arrangements shall be required by NCC in any case, for their own purposes. Robert Allen - Head of ICT End of October June Action Plan 5

7 R9 There are a number of single points of failure in the ICT infrastructure that may increase the likelihood of service disruption. There are a number of contingency arrangements in place at the Council, but the appropriateness of these for NCH purposes is not known. a) NCH should perform a risk assessment of the single points of failure, system fail-over arrangements and disaster recovery capacity and validate where greater levels of resilience or service levels are required. b) We understand that the Council s Internal Audit function has performed a review of this area recently, and NCH should request access to the findings and action planning arising from this review. Risk assessment to be carried out to establish where greater levels of resilience or service levels are required pending outcome of ongoing accommodation review and SLA negotiations with NCC. Information requested from NCC 03/07/ / Robert Barton - Interim Director of ICT End of October R10 There were areas where the Hounds Gate server room fell below best practice for physical and environmental security. Should the room cease to operate, all telephone and network communications to Hounds Gate (and CSC) would be terminated and none of the key IT systems would be available to office-based staff. The physical and environmental controls in the Hounds Gate server room should be reviewed and improved where deemed appropriate. Risk assessment to be carried out to establish whether or not interim controls are required pending outcome of ongoing accommodation review / Robert Barton - Interim Director of ICT End of September June Action Plan 6

8 3 Definitions Assurance Level Fully meets expectations Substantially meets expectations Partly meets expectations Does not meet expectations Definition Our audit work provides assurance that the arrangements should deliver the objectives and risk management aims of the organisation in the area under review and meet or exceed relevant external requirements. There is only a small risk of failure or non-compliance. Our audit work provides assurance that the arrangements should deliver the key objectives and risk management aims of the organisation in the area under review and meet most relevant external requirements. There is some risk of failure or non-compliance. Our audit work provides assurance that the arrangements will deliver only some of the key objectives and risk management aims of the organisation in the area under review or may not meet relevant external requirements. There is a significant risk of failure or non-compliance. Our audit work provides little assurance. The arrangements will not deliver the key objectives and risk management aims of the organisation in the area under review or will not meet relevant external requirements. There is an almost certain risk of failure or non-compliance. Recommendation priority High priority recommendations priority recommendations Low priority recommendations Definition Those that failure to address would result in a significant and unacceptable risk to the organisation arising or continuing. Those that failure to address would result in a moderate risk to the organisation arising or continuing or relate to significant best practice improvements. Those that failure to address would result in a minor risk to the organisation arising or continuing or relate to moderate best practice improvements. June Definitions 7