Appendix 6c. Final Internal Audit Report Disaster Recovery Planning. June Report 6c Page 1 of 15

Transcription

1 Appendix 6c Final Internal Audit Report Disaster Recovery Planning June 2007 Report 6c Page 1 of 15

2 Contents Page Executive Summary 3 Observations and Recommendations 8 Appendix 1 - Audit Framework 13 Appendix 2 - Staff Interviewed 14 Statement of Responsibility 15 Disaster Recovery Planning 6c 2006/2007 Audit Ref: 723 p 2 of 15

3 Executive Summary Introduction & Background 1. This audit forms part of the 2006/2007 Internal Audit Plan, which has been approved by the Mayor and the Audit Panel. The plan entails a review of the Authority s Disaster Recovery Planning arrangements. 2. In light of events, including the 7/7 terrorist attacks and the Bunsfield Oil depot explosion, both occurring in 2005, the need for effective disaster recovery arrangements has been forcibly demonstrated. To some extent both of these events served to test the adequacy of the Authority s disaster recovery plans, which showed that they were generally effective, although some lessons have been learnt. 3. The aim of Disaster Recovery Planning (DRP), is to identify potential impacts that threaten an organisation and provides a framework for building resilience with the capability for an effective response that safeguards the interests of its key stakeholders, reputation and brand. In essence, the GLA needs to maintain Disaster Recovery Plans to ensure the continuation of business activities in the event of a disaster or any incident or circumstances that could potentially cause disruption to normal business activities, and to ensure that the Mayor and Assembly are given proper support to carry out their roles in the event of any major incident or London-wide emergency. 4. The Technology Group is responsible for the management, maintenance and execution of the Disaster Recovery Plan. They are accountable for preparing for, responding to and the recovery of technology systems for any major City Hall incident. 5. Overall, Disaster Recovery is well managed and the Disaster Recovery site is very well equipped to ensure that in the event of a disaster, the Authority s critical business systems would be recoverable within agreed and acceptable timeframes. The Disaster Recovery Plan is also tested on a bi-annual basis and further ad-hoc tests, for example, when new systems are implemented, are also undertaken as and when required. However, some areas for further potential improvement in the control framework were identified during the course of the audit, and five recommendations were raised and agreed with management and detailed in the body of the report and highlighted below: Updating Disaster Plan Documentation; Updating Major Incident Plan Documentation; Contact List Record Maintenance; Off Site Recovery and Reconfiguration Data; and Completeness of Test Log Records. 6. Key staff was interviewed and prime source documents pertaining to the financial year 2006/2007 were selected and reviewed to evaluate the adequacy of the Disaster Recovery Plan framework and plans currently in operation. Our audit work was carried out during the period of February and March 2007 and a summary of the audit findings is given in the following paragraphs. Report 6c page 3 of 15

4 Critical Business Functions 7. The Authority has identified critical business locations and critical systems within the Disaster Recovery Plan (entitled Technology Group Business Continuity and Disaster Recovery Plan v 2.2 dated February 2006). However, on examination it was noted that not all current systems are included in the Plan and we were advised that the Technology Group are currently updating the Plan to include all new systems. The Plan also categorises type I and II systems in accordance with their criticality, i.e. Category I systems are to be recovered within 24 hours and Category II systems within 5 days of a major incident. However, it was noted that not all systems listed have been included in the corresponding Appendices that contain details of the hardware and software required, restore and recovery procedures and resources. Furthermore, the Plan does not give details of the 3 rd Party NDR Disaster Recovery fallback site, nor does the Plan make any reference to the 3 rd party supplier responsible for the offsite data backup tape storage. In addition, despite the Plan making specific reference to the fact that the Technology Group are responsible for the full deployment of Category I, II and III systems, there is no list of Category III systems included. One recommendation has been raised following our review of this area: Updating Disaster Plan Documentation Disaster Risk Assessment 9. A detailed formal risk assessment has been undertaken covering environmental, organised and deliberate disruption, loss of utilities and services; equipment or systems failure, serious information security incidents, other emergency situations including acts of violence in the workplace and disruption to Public transport. In addition, a Component Failure Impact Analysis was also undertaken and a Business Impact Analysis was conducted for all services and critical business processes for all Departments, including the Mayor s office, Assembly & Secretariat, Chief Executive, Corporate Services, Finance & Performance, Media & Marketing and Policy and Partnership. No recommendations have been made following our review of this area. Contact List Maintenance 10. The Technology Group Disaster Recovery Pack 2006 is a critical pack of documents that contains information and items that may be required in the event of an incident at City Hall. It includes a Technology Group Contact List. It was noted that the contact list was not up to date or complete. One recommendation has been made following our review of this area: Contact List Record Maintenance. Disaster Escalation Procedures 11. It was confirmed that adequate disaster escalation procedures have been created, discussed, agreed and communicated to all relevant staff. The procedures also include relevant sections for invocation of the Major Incident Plan. In addition, specific Invoke and Standby procedures have also in place and have been distributed to all relevant staff. Report 6c page 4 of 15

5 No recommendations have been made following our review of this area. Emergency Action Procedures 12. Communication is key to effective implementation of Disaster Recovery Plans. The Authority has introduced a staff emergency number that contains a recorded message, which all staff are encouraged to ring if they hear of any incident within the vicinity of City Hall whilst on their way to work. The standard default message is that there are no incidents reported and staff should come into work as normal, but this will be updated in the event of any incident to advise staff as to what action they should take. 13. In addition, the Major Incident Plan and the Invoke and Standby procedures, which form part of the Disaster Recovery documentation suite, have also been distributed to all relevant staff. It has been noted at 8 above, that the Major Incident Plan requires updating. 14. Each Directorate has a telephone list of key contacts that in turn are able to cascade messages on to their own groups (i.e. Heads of Service to team managers to staff). Each of the plans maintained centrally by Corporate Services, were reviewed and it was noted that many of them are supported by a confidential contact list for the respective directorates. However, further testing showed that whilst some lists are held centrally, each directorate maintains their own master telephone list and arrangements to allow effective communication to their staff during times of emergency, in accordance with the approved procedures. 15. A distribution list of all blackberry users (managers and team leaders) is maintained on the Outlook public folders of the Authority s network. This record is maintained to allow quick and effective communication. 16. All messages to the public in City Hall are conveyed via tannoy announcement and by security staff. Otherwise the public website or news bulletins/press releases will be used to get messages out to the wider public. It was confirmed that the tannoy is tested on a weekly basis during the fire alarm test. 17. In the event of a major incident, the Mayor has a direct link to Gold Control at Scotland Yard from his briefing room in City Hall. Furthermore, there is equipment in place to enable broadcasting form City Hall, so that the Mayor is able to make public statements. 18. The Technology Group Incident Log Book that is maintained once a disaster scenario has been invoked, has two contact numbers in operation; one for personnel to contact the Duty Disaster Recovery Person to telephone and second number that connects to an up to date recorded message giving details of disaster recovery status report and announcement during the emergency situation. No recommendations have been made following our review of this area. Salvage Procedures 19. Adequate salvage procedures are included in the Disaster Recovery Plan. Salvage procedures are the responsibility of the Operational Recovery Team, whom in turn report to the Strategic Recovery Team, on the extent of the damage, and make recommendations regarding possible reactivation and relocation of data centre and Report 6c page 5 of 15

6 user operations. In the event of a disaster, the Salvage exercise is headed by the Operations Manager and activated during the initial stage of an emergency. No recommendations have been raised following our review of this area. Disaster Recovery Procedures 20. A Disaster Recovery documentation suite has been produced in conjunction with business Departments and distributed to all relevant staff. The procedures are adequate for there purpose, however, see Recommendations with regard to updating the Plan, the Major Incident Plan and the Contact Lists, detailed above. 21. There are two main suppliers of Disaster Recovery services to the GLA. The first is NDR who supply the actual Disaster Recovery site and equipment. It is a hot site but only has provision for 30 staff initially, although there is an option to increase to a maximum space of 200 desks. The site has all attendant technical support and infrastructure as required. 22. The second major supplier of Disaster Recovery services is Data Protect UK who manages the offsite storage of data back up tapes. 23. Site visits to both companies ascertained that facilities, services and related procedures are of a high standard and would be adequate in the event of a disaster scenario. 24. However, it was evidenced that at the Disaster Recovery site the Black Box which contains critical Disaster Recovery documentation, information, hardware and software and as detailed in the Plan was not complete. Whilst there are incomplete systems lists, recovery procedures and incomplete items in the Black Box in the event of a disaster, timely recovery of critical systems may not be achieved. One recommendation has been raised following our review of this area: Off Site Recovery and Reconfiguration Data. Temporary Arrangements 25. The Major Incident Plan refers to the activation of temporary facilities in the event of a disaster. The facility is GLA owned and is Telstar House, London W2 6LG. Telstar House is a facility provided by Facilities Management and is part of the overall GLA plan offering. However, this facility would be for use by a very limited number of essential Disaster Recovery staff. 26. However, the majority of staff are configured for remote access and home working, so this could be an option if City hall were not available. No recommendations have been raised following our review of this area. Finance 27. Up to date and adequate copies of insurance certificates are available in the Plan as detailed under the contents of the Black Box. Insurance in place covers the entire computer suite both at City Hall and the Disaster Recovery sites and virus attacks and malicious code, however, please see item 23 above, for details concerning the lack of up to date information and equipment in the Black Box.. Report 6c page 6 of 15

7 28. Additional financial funding in the event of a disaster is also available and is governed by the Major Incident Teams. No recommendations have been raised following our review of this area. Disaster Recovery Testing 29. The Disaster Recovery facility is tested on a bi-annual basis. A planning meeting is undertaken approximately one month prior to test date and test requirements are documented and circulated to all relevant staff. The last test undertaken was October 2006 and was deemed to be overall successful. However, there were two key systems that were not successfully restored, due to problems encountered during testing. However, the systems were subject to individual testing in January In addition to the bi-annual test, for new systems implementation, various ad-hoc testing of components and systems are tested. 31. During a Disaster Recovery test, test results are maintained, via Technical Log and an Incident Log Book. However, neither of these had been fully completed for the October 2006 test. 32. Following testing, a post-test meeting is held and all relevant personnel attend. Lessons learnt are noted and the Disaster Recovery suite of documentation and procedures are updated accordingly. 33. The next Disaster Recovery test is scheduled for April One recommendation has been made following our review of this area: Completeness of Test Log Records. Audit Opinion Substantial Assurance Evaluation Opinion: While there is a basically sound system, there are areas of weakness which put some of the system objectives at risk, Testing Opinion: and/or there is evidence that the level of non-compliance with some of the controls may put some of the system objectives at risk. Report 6c page 7 of 15

8 Observations and Recommendations In order to assist management in using our reports: We categorise our opinions according to our assessment of the controls in place and the level of compliance with these controls Full Assurance Substantial Assurance Limited Assurance No Assurance There is a sound system of control designed to achieve the system objectives and the controls are being consistently applied. While there is a basically sound system, there are areas of weakness which put some of the system objectives at risk, and/or there is evidence that the level of non-compliance with some of the controls may put some of the system objectives at risk. Weaknesses in the system of controls are such as to put the system objectives at risk, and /or the level of non-compliance puts the system objectives at risk. Control is generally weak, leaving the system open to significant error or abuse, and/or significant non-compliance with basic controls leaves the system open to error or abuse. b) We categorise our recommendations according to their level of priority. Priority 1 Priority 2 Priority 3 Major issues for the attention of senior management. Other recommendations for local management action. Minor matters. Report 6c page 8 of 15

9 Risk Assessment & Impact 1. Updating Disaster Plan Documentation (Priority 2) Recommendation It is recommended that the Technology Group Business Continuity and Disaster Recovery Plan v 2.2 should: be updated to ensure that all systems are listed and categorised in accordance with their criticality. This should include updating the Disaster Recovery site information; be reviewed to ensure that all Category I, II and III systems have been identified and cross referenced to corresponding Appendices; include reference to 3 rd Party resource responsible for the safe offsite storage of all data and archive tapes. Once the above actions have been implemented, the Plan should be circulated to all relevant personnel forthwith; and include definitive systems lists that are agreed with the 3 rd party supplier. Rationale A formally produced, tested and up to date Disaster Recovery Plan suite of documentation ensures that a business s systems and processes are recoverable within an acceptable and agreed timescale. It also ensures that in the event of a disaster, there is minimum disruption and loss of services to both users and customers alike. At present, the Disaster Recovery suite of documentation is out of date and incomplete. Whist the Disaster Recovery suite of documentation is out of date and incomplete, there is a risk that GLA the Organisations systems as deemed critical in the event of a disaster, would not be available within an agreed and acceptable timeframe. This could lead to financial and reputational damage and statutory compliance failures e.g. meeting the time requirements of Freedom of Information Act requests. Management response: Technology Group Operations Manager Implemented - At the time of the audit approval to update the contract with our Disaster Recovery partner to include additional equipment was being sought. This has now been approved and the Disaster Recovery plan has now been updated to include this equipment. The plan has now been updated to categorise all systems and will agree the list of systems to be covered with the third party supplier. A reference has been made to the 3 rd party responsible for safe offsite storage data and archive tapes. Report 6c page 9 of 15

10 Contact Details 2. Contact List Record Maintenance (Priority 2) Recommendation It is recommended that contact lists for all corresponding Disaster Recovery documentation, including the Business Continuity and Disaster Recovery Plan v2.2 the Major Incident Plan and the TG DR Pack 2006 contact list, is verified, updated and distributed to all relevant personnel forthwith. Rationale Up to date contact information is essential to the success of a disaster recovery scenario being invoked. It was noted that contact information within the Disaster Recovery Suite of information, was in some cases, incomplete and out of date. There is a risk that in the event of a disaster occurring, key contact personnel would not be contactable. This could jeopardise the Disaster Recovery exercise and may lead to systems unavailability within an acceptable timeframe to the business and customers. This may lead to reputational and financial loss and possibly legal action (in the event of information being unavailable within agreed timeframes to customers.) Management response: Technology Group Operations Manager Agreed The Technology Group regularly updates and circulates its contact list (monthly or as soon as change is made). This has already been implemented. Report 6c page 10 of 15

11 Disaster Recovery Procedures 3. Off Site Recovery and Reconfiguration Data (Priority 2) Recommendation It is recommended that the NDR Disaster Recovery site, all documentation, information, software and hardware as deemed critical in the Plan and named Black Box, is complete and up to date. Rationale A complete and adequate set of recovery and reconfiguration procedures is a key component of a Disaster Recovery Plan and test. It was noted that the 3 rd Party NDR Contract and the Systems List, did not correspond in V2.2. of the GLA Disaster Recovery suite documentation. An inspection of the offsite Black Box held at the NDR Disaster Recovery site found that, only ten of the thirty-three items listed as critical, were present and recovery and reconfiguration data were not in place for all systems. Whilst there are incomplete systems lists, recovery procedures and incomplete items from the critically deemed Black Box, there is a risk that Disaster Recovery would not be successful, or that the systems were not recovered within an acceptable timeframe to both the business and customers alike. Management response: Technology Group Operations Manager Agreed When audited some contract information was missing from the Black box. The Black box now has a complete set of relevant documentation. This has already been implemented Report 6c page 11 of 15

12 Disaster Recovery Testing 4. Completeness of Test Log Records (Priority 2) Recommendation It is recommended that the Technical Log and the Incident Log Book that is maintained during testing and in the event of a disaster, should be completed for all actions noted. Rationale The Technical Log and the Incident Log Book are an important management information and analysis tool that ensure that problems identified during testing are timely resolved and disaster recovery documentation is updated accurately. During a Disaster Recovery test, test results are maintained, via the Technical and the Incident Log Books. However, examination of the for the October 2006 test records noted that neither of these logs were fully completed. There is a risk that testing results will not be properly analysed and that lessons will not be learnt, or reported which could lead to disaster recovery documentation being out of date and inaccurate and hinder future testing of the disaster recovery scenario. This could lead to unavailability of systems and or data. Management response: Technology Group Operations Manager Agreed A person will now be designated as the owner of the incident and technical log. A Disaster Recovery test exercise will take place during April and the test and incident logs will be examined following this. This has been implemented Report 6c page 12 of 15

13 Appendix 1 Audit Framework Audit Objectives The audit was designed to ensure that management has formulated adequate and effective plans and contingency arrangements to be instigated in the event of any incident that has an impact on business continuity. Audit Approach and Methodology The audit approach was developed with reference to an assessment of risks and management controls operating within each area of the scope. The following procedures were adopted: identification of the role and objectives of each area; identification of risks within the systems, and controls in existence to allow the control objectives to be achieved; and evaluation and testing of controls within the systems. From these procedures we have identified weaknesses in the systems of control, produced specific proposals to improve the control environment and have drawn an overall conclusion on the design and operation of the system. Areas Covered Audit work was undertaken to cover controls in the following areas: Critical Business Functions; Disaster Risk Assessment; Contact Details; Disaster Escalation Procedures; Emergency Action Plan; Salvage Procedures; Disaster Recovery Procedures; Temporary Arrangements; Finance; and Disaster Recovery Test Plan. Report 6c page 13 of 15

14 Appendix 2 - Staff Interviewed We would like to thank all staff that provided assistance during the course of this audit, and in particular: Operations Manager, Technology Group Internal Systems Manager, Technology Group NDR Representative Data Protect UK Representative Report 6c page 14 of 15

15 Statement of Responsibility We take responsibility for this report, which is prepared on the basis of the limitations set out below. The matters raised in this report are only those, which came to our attention during the course of our internal audit work and are not necessarily a comprehensive statement of all the weaknesses that exist or all improvements that might be made. Recommendations for improvements should be assessed by you for their full impact before they are implemented. The performance of internal audit work is not and should not be taken as a substitute for management s responsibilities for the application of sound management practices. We emphasise that the responsibility for a sound system of internal controls and the prevention and detection of fraud and other irregularities rests with management and work performed by internal audit should not be relied upon to identify all strengths and weaknesses in internal controls, nor relied upon to identify all circumstances of fraud or irregularity. Auditors, in conducting their work, are required to have regards to the possibility of fraud or irregularities. Even sound systems of internal control can only provide reasonable and not absolute assurance and may not be proof against collusive fraud. Internal audit procedures are designed to focus on areas as identified by management as being of greatest risk and significance and as such we rely on management to provide us full access to their accounting records and transactions for the purposes of our audit work and to ensure the authenticity of these documents. Effective and timely implementation of our recommendations by management is important for the maintenance of a reliable internal control system. Deloitte & Touche Public Sector Internal Audit Limited St Albans June 2007 In this document references to Deloitte are references to Deloitte & Touche Public Sector Internal Audit Limited. Deloitte & Touche Public Sector Internal Audit Limited is a subsidiary of Deloitte & Touche LLP, which is the United Kingdom member firm of Deloitte Touche Tohmatsu ( DTT ), a Swiss Verein whose member firms are separate and independent legal entities. Neither DTT nor any of its member firms has any liability for each other s acts or omissions. Services are provided by member firms or their subsidiaries and not by DTT Deloitte & Touche Public Sector Internal Audit Limited. All rights reserved. Deloitte & Touche Public Sector Internal Audit Limited is registered in England and Wales with registered number Registered office: Stonecutter Court, 1 Stonecutter Street, London EC4A 4TR, United Kingdom. Report 6c page 15 of 15