NOTTINGHAM CITY HOMES IT REVIEW OF THE DISASTER RECOVERY ARRANGEMENTS Report issued: February 2011 Audit Plan: The matters raised in this report are only those that came to the attention of the auditor during the course of the internal audit review and are not necessarily a comprehensive statement of all the weaknesses that exist or all the improvements that might be made. This report has been prepared solely for management's use and must not be recited or referred to in whole or in part to third parties without our prior written consent. No responsibility to any third party is accepted as the report has not been prepared, and is not intended, for any other purpose. TIAA neither owes nor accepts any duty of care to any other party who may receive this report and specifically disclaims any liability for loss, damage or expense of whatsoever nature, which is caused by their reliance on our report.
INTRODUCTION - EXECUTIVE SUMMARY - 1. We have reviewed the Disaster Recovery arrangements at Nottingham City Homes. The review was carried out in July 2010 as part of the planned internal audit work for. SUMMARY 2. One Key Risk Control Objective was identified and based on the findings from this work an overall evaluation of the overall adequacy of the internal controls was established (figure 1). Figure 1 - Evaluation of the Effectiveness of the Internal Controls Evaluation Limited Assurance KEY FINDINGS 3. The key control and operational practice findings that need to be addressed in order to strengthen the control environment are set out in the Management and Operational Effectiveness Action Plans. The prioritisation of the recommendations are summarised below (figure 2). Figure 2 - Summary of Priorities of Recommendations Urgent Important Routine Operational 1 3 1 - MANAGEMENT RESPONSES 4. Recommendations for improvements should be assessed by the Company for their full impact before they are implemented. RELEASE OF REPORT 5. The table below sets out the history of this report. Date draft report issued: 1 st September 2010 Date management responses recd: 23 rd February 2011 Date final report issued: 23 rd February 2011 Page 1
MANAGEMENT ACTION PLAN PRIORITY 1, 2 AND 3 RECOMMENDATIONS Risk Finding Recommendation Priority Management Comments Implementation Timetable Responsible Officer Failure to direct the process through approved policy & procedures. It was ascertained that key IT systems and services had been identified and prioritised for recovery in a disaster situation, however, no evidence was provided to substantiate this. Recommendation 2: The prioritisation of IT systems and services be undertaken to identify the critical recovery path should such a disaster occur. 1 The information on key systems and services will be consolidated, allowing a critical recovery path to be identified and agreed with the NCH Business Continuity lead officer End of May 2011 Robert Allen Head of ICT Failure to direct the process through approved policy & procedures. There is no evidence to support the identification of risks associated with IT systems. Significant risks to systems must first be identified before a comprehensive recovery plan can be developed, tested and implemented. Recommendation 1: A risk assessment be undertaken to identify significant risks relating to the loss of IT systems and services. 2 A risk assessment will be carried out for all key systems/services identified in the response to Recommendation 2 to identify risks to system/service availability End of May 2011 Robert Allen Head of ICT PRIORITY GRADINGS 1 URGENT Fundamental control issue on which action should be taken immediately. 2 IMPORTANT Control issue on which action should be taken at the earliest opportunity. 3 ROUTINE Control issue on which action should be taken. Page 2
Risk Finding Recommendation Priority Management Comments Implementation Timetable Responsible Officer There is no procedure for restoring critical business systems following an incident. Backup and recovery of IT systems is undertaken by the Local Authority, who provide IT services to NCH under contract. It is understood that a comprehensive SLA has been sought with the Local Authority for some time and that a recent draft has been written. No evidence of the draft SLA was provided during the review. Recommendation 3: A comprehensive SLA with the Local Authority be sought to ensure that NCH is receiving acceptable levels of service and that value for money from the service provided can be demonstrated. 2 Formalisation of the backup and recovery arrangements for our systems is one of the reasons that NCH has been attempting to develop a comprehensive ICT SLA with NCC and progress against this recommendation is tied to the progress on the SLA. A deadline of March 2011 has been set to agree with NCC a new ICT SLA Action Plan. Once this is in place, timings for progress against this recommendation may be available. TBC Robert Allen Head of ICT PRIORITY GRADINGS 1 URGENT Fundamental control issue on which action should be taken immediately. 2 IMPORTANT Control issue on which action should be taken at the earliest opportunity. 3 ROUTINE Control issue on which action should be taken. Page 3
Risk Finding Recommendation Priority Management Comments Implementation Timetable Responsible Officer There is no procedure for restoring critical business systems following an incident. A backup schedule was provided to Internal Audit. A review of the schedule identified several servers that were not backed up, although it should be noted that there were some that had a legitimate reason for not being backed up. However, there were still servers that were not backed up and therefore information may not be recoverable should a disaster occur Recommendation 4: The current backup arrangements be reviewed to ensure that critical systems are effectively backed up and the schedule is sufficiently documented to reflect the actual arrangements. 2 COMPLETE All of NCH s critical business systems are backed up appropriately. Further checks have ensured that the missing information leading to the audit finding has been added to the backup schedule document. N/A Robert Allen Head of ICT PRIORITY GRADINGS 1 URGENT Fundamental control issue on which action should be taken immediately. 2 IMPORTANT Control issue on which action should be taken at the earliest opportunity. 3 ROUTINE Control issue on which action should be taken. Page 4
Risk Finding Recommendation Priority Management Comments Implementation Timetable Responsible Officer There is no procedure for restoring critical business systems following an incident. There is no formal documented Disaster Recovery plan in place with reliance placed upon the Nottingham City Council (NCC) to provide a recovery service. Whilst this process is appropriate for NCH, there was no evidence to suggest that NCC has a detailed disaster recovery plan which has been tested to ensure that NCH s IT systems can be fully and accurately recovered should a disaster occur. Recommendation 5: Confirmation be sought from NCC that they have a fully tested and detailed Disaster Recovery plan that identifies NCH s critical systems and that these can be effectively recovered should a disaster occur. 3 Confirmation has been sought and NCH are awaiting a response from NCC. End of March 2011 Robert Allen Head of ICT PRIORITY GRADINGS 1 URGENT Fundamental control issue on which action should be taken immediately. 2 IMPORTANT Control issue on which action should be taken at the earliest opportunity. 3 ROUTINE Control issue on which action should be taken. Page 5
OPERATIONAL EFFECTIVENESS MATTERS Item Management Comments No Operational Effectiveness Matters were identified. ADVISORY NOTE Operational Effectiveness Matters need to be considered as part of management review of the procedures, rather than on a one-by-one basis Page 6
SCOPE AND LIMITATIONS OF THE REVIEW 6. The review considered the extent to which the organisation has put into place arrangements which provides reasonable but not absolute assurance that the impact on the organisation of any major incident will be minimised. The scope of the review did not include providing assurance that the actual testing of hardware/software etc has been carried out effectively. 7. The limitations and the responsibilities of management in regard to this review are set out in the Annual Plan. ASSESSMENTS OF THE KEY RISK CONTROL OBJECTIVES 8. This review identified and tested the controls that are being operated by the Organisation and an assessment of the combined effectiveness of the controls in mitigating the key probity risks is provided. The assessments are: Substantial Assurance robust series of internal controls in place which should ensure continuous and effective achievement of the control objective. Reasonable Assurance reasonable number of internal controls in place, however may not be operated all the time. Limited Assurance the controls in place are not sufficient to ensure the continuous and effective achievement of the control objective. No Assurance fundamental breakdown or absence of core internal controls. MATERIALITY 9. NCH places reliance of the Local Authority to provide ICT services. These services included the recovery of IT systems in the event of a disaster scenario. Page 7
Risk Failure to direct the process through approved policy & procedures. Risk Control Objective Arrangements in place provide for compliance with established policies, procedures, laws and regulations. Evaluation Limited Assurance 10. The following matters were identified in reviewing the Key Risk Control Objective: Risk: Critical business systems are not identified and as a consequence are not considered a priority for restore and recovery. 10.1 There is no evidence to support the identification of risks associated with IT systems. Significant risks to systems must first be identified before a comprehensive recovery plan can be developed, tested and implemented. Recommendation 1: A risk assessment be undertaken to identify significant risks relating to the loss of IT systems and services. 10.2 It was ascertained that key IT systems and services had been identified and prioritised for recovery in a disaster situation, however, no evidence was provided to substantiate this. Recommendation 2: The prioritisation of IT systems and services be undertaken to identify the critical recovery path should such a disaster occur. 10.3 It was demonstrated that, for new systems and projects, the requirements for resilience and recovery were addressed at the time of inception. Risk: There is no procedure for restoring critical business systems following an incident. 10.4 Backup and recovery of IT systems is undertaken by the Local Authority, who provide IT services to NCH under contract. It is understood that a comprehensive SLA has been sought with the Local Authority for some time and that a recent draft has been written. No evidence of the draft SLA was provided during the review. Recommendation 3: A comprehensive SLA with the Local Authority be sought to ensure that NCH is receiving acceptable levels of service and that value for money from the service provided can be demonstrated. 10.5 A backup schedule was provided to Internal Audit. A review of the schedule identified several servers that were not backed up, although it should be noted that there were some that had a legitimate reason for not being backed up. However, there were still servers that were not backed up and therefore information may not be recoverable should a disaster occur. Recommendation 4: The current backup arrangements be reviewed to ensure that critical systems are effectively backed up and the schedule is sufficiently documented to reflect the actual arrangements. Page 9
10.6 There is no formal documented Disaster Recovery plan in place with reliance placed upon the Nottingham City Council (NCC) to provide a recovery service. Whilst this process is appropriate for NCH, there was no evidence to suggest that NCC has a detailed disaster recovery plan which has been tested to ensure that NCH s IT systems can be fully and accurately recovered should a disaster occur. Recommendation 5: Confirmation be sought from NCC that they have a fully tested and detailed Disaster Recovery plan that identifies NCH s critical systems and that these can be effectively recovered should a disaster occur. Risk: Data is lost and/or is irrecoverable. 10.7 The current data centre contains a mixture of physical and virtual servers. Plans are underway to move the data centre from its current location with a separate recovery location being available. During this migration it is understood that more systems will be virtualised, where possible, and a storage area network will also be implemented. It is anticipated that data will be replicated across sites and therefore provide online resilience for network systems and services. Regular backups should still be taken for archive purposes. --------------- Page 9