Columbus City Schools Office of Internal Audit

Transcription

1 Information Technology Disaster Recovery Plan Review Report Date: March 24, 2011 Internal Audit Mission Statement To support the overall mission of the Columbus City Schools by providing quality management advisory and business process auditing services to the District. Internal Audit is an extension of management. Through our reviews we assist the Board of Education, Superintendent, Treasurer, and all Columbus City Schools personnel in carrying out their responsibilities.

2 Table of Contents Title Page Executive Summary 3 Results of DR Plan Testing Documentation Review 3 Background 4 Objectives 4 Scope and Methodology 5 Observations and Recommendations 5 Management Responses and Action Plans 7 2

3 Executive Summary On February 8, 2011, The (IA) completed an initial review of the testing documentation provided in support of the Information Technology Department Disaster Recovery Plan (DR Plan), last revised July The IT Operations Manager is responsible for executing DR Plan testing, documenting test results and completing the semi-annual update to the DR Plan, as described in the July 2010 plan. The following report summarizes our observations and the results of our testing documentation review. The report contains several recommendations designed to strengthen the testing procedures and the testing documentation included in the DR Plan. Results of DR Plan Testing Documentation Review The initial review of the July 2010, Information Technology DR Plan testing documentation indicated the testing documentation should be enhanced to support evidence of the execution of an effective, proactive DR Plan testing methodology. The DR Plan testing plan should establish the required minimum testing activity to be accomplished annually. The IT Operations Manager stores the core DR Plan Book, in hard copy and electronic copy, separate from the annual DR Plan testing support documentation. The DR Plan book (hard copy and electronic copy) should be expanded to include a reference to the location(s) where the evidentiary support for annual DRP testing is maintained. The current DR Plan document contains descriptions of IT Operation s reactions to various situations that present during daily operations, such as equipment malfunction and requests for data restorations, etc.. The documentation of successful recovery from the unanticipated event has been a mainstay of the DR Plan testing methodology and support documentation. The DR Plan testing methodology should be enhanced to include an effective forward looking annual testing plan. The plan should include documentation of the results of actual testing scenarios, lessons learned from the testing scenarios and documentation describing subsequent DR Plan changes which will be incorporated into the annual update of the DR Plan Book, hard copy and electronic copy. 3

4 Background Disaster recovery planning is the process, policy, and procedures related to preparing for recovery continuation of technology infrastructure critical to an organization after a natural or human-induced disaster. Business continuity involves planning for keeping all aspects of a business functioning in the midst of disruptive events, disaster recovery, a subset of continuity planning focuses on the IT or technology systems that support business functions. The IT Operations Manager described the evolution of the Information Technology Disaster Recovery Plan for Columbus City Schools. Prior to the development of the District s disaster recovery plan the Information Technology (IT) Department maintained a book of system documentation/restore procedures. In the Information Technology Department identified the mission critical applications and built out the three data centers. The disaster recovery plan was formalized by critical application in accordance with the disaster recovery capabilities of the three generator backed up facilities. The IT Department supports 148 district buildings and over 250 departments. The IT department supports three data centers which house over 200 physical servers, approximately 29,000 traditional computer/thin client work stations, 4,000 laptop computers and over 1,000 unique software applications. Objectives The objectives of the review were the following: Determine District management continues to develop, test and refine the disaster recovery plan to respond to data processing demands and environmental changes within the district. Determine the plan includes documentation describing plan revisions including the sections modified, when modifications were completed, the individual(s) responsible for the modifications and documentation supporting change approval by senior management. Determine the plan includes documentation of the testing cycle and recent test results. Determine that critical personnel have a current copy of the plan and are aware of their roles and responsibilities during a disaster recovery. Determine that the applicable June 30, 2010 Auditor of State management letter recommendations have been addressed and documented in the disaster recovery plan. 4

5 Scope and Methodology The scope of the review is limited to the IT Department and the review of their DR Plan. During the review, IA interviewed the Chief Information Officer (CIO) and IT Operations Manager. IA referenced the Control Objectives for Information and related Technology (COBIT) and other guidance; researched reviews performed on IT DR Plans, and used the internet to research DR Plans & governance. COBIT is an IT governance framework and supporting toolset that allows managers to bridge the gap between control requirements, technical issues, and business risks. COBIT enables clear policy development and good practice for IT controls throughout organizations. Observations and Recommendations Observation Number 1 - The Information Technology (IT) Department DR Plan is not regularly tested. Disaster Recovery Plan testing should be a multi-step process. The preparation and execution of disaster recovery plan testing involves pre-planning, scenario building, scheduling of personnel and facilities, pre-test reviews, finalization of the actual disaster recovery plan and lastly, making sure all the resources you need are available when testing the plan. Effective disaster recovery plan testing helps to determine the readiness of all systems, people, and processes, helps identify gaps in the DR Plan, and allows IT Operations management to constantly improve the plan to ensure it is a living document. The last formal, pre-planned, IT Department table top exercise was performed in December The lack of scheduling and performing planned testing could lead to the District failing to protect intellectual property or sensitive information and files from loss or significant damage. Recommendation Nos. 1, 2 1. IA recommends IT Operations management perform scheduled testing of the DR Plan, conducted on a regular basis, at least annually. The testing at a minimum should include tabletop exercises. 5

6 2. IA recommends IT Operations management generate and retain supporting documentation as evidence of completion of the test objectives, the individuals involved, test results, lessons learned, and any procedural changes to be incorporated into the DR plan, so the plan evolves over time. Observation Number 2 - Lack of Consistent Reviews and Updates of the Disaster Recovery Plan. The current DR Plan, Administration Section 4.0, states: The Disaster Recovery Plan and Emergency Operation Manual will be reviewed by Information Technology personnel in June and December of each year. The review will be initiated by the IT Operations Manager and will consider hardware and operating system changes, application software changes, staffing changes and organizational changes. All modification documentation will be submitted to the IT Operations Manager. The DR Plan has been revised and updated twice, once in January 2009 and again in July Lack of a system to test and ensure controls are in place to properly update the disaster recovery plan could lead to outdated information being contained in the plan. Recommendation Nos. 3, 4, 5, & 6 3. IA recommends IT Operations management review and update the DR Plan at least annually. 4. IA recommends IT Operations management develop a mechanism to identify sections changed with date, and what was revised and/or changed during the review cycle. 5. IA recommends IT Operations management ensure that all individuals identified as required to have DR Plan have the most up-to-date copy. 6. IA recommends IT Operations management conduct an annual meeting with key personnel who would be involved in the disaster recovery, to ensure they know their roles and responsibilities during an event of a disaster. February 8, 2011 Status Update Received from IT Operations The IT Operations Manager revealed a significant effort is underway to update the July 2010 version of the DR Plan. The end of February 2011 is the target completion date for the latest DR Plan revision. Also stated was the intent to eliminate semi-annual DR Plan updates, replacing them with a single, annual DR Plan update to be scheduled for the month of February each year. The months of March and May of each year are to be dedicated for testing and documenting the results of DR Plan testing scenarios. 6

7 The IT Operations Manager indicated the electronic and hard copy versions of the DR Plan will be switched out during the first week of March The updated hard copies of the DR Plan will be provided to individuals identified in DR Plan Section 4.1. A 90 Day Review will be performed. Management Responses and Action Plans Recommendation No. 1 Annually we will be doing a tape restore test in March of each year and will be doing a walk through test of one of the critical system in May of each year. This is outlined in the DR Plan revised February 2011 in the Introduction, Section 5.2 Testing schedule, on page 11. Target Implementation date: March 2011 for the tape restore test and May 2011 for the walk through of the critical system. Recommendation No. 2 Ultimately the IT Operations Manager is responsible for ensuring the tests are completed and documented. The IT technical and applications teams will conduct the tests. Individuals will change depending on the system tested and the names of those involved will be noted in the test documentation. Target Implementation date: March 2011 for the tape restore test and May 2011 for the walk through of the critical system. Recommendation No. 3 The plan was updated in February of 2011 and will be done annually in February going forward. If there is a significant change to any of the recovery plans within the year, the plan will be updated accordingly. 7

8 Ultimately the IT Operations Manager is responsible for insuring the plan updated are completed and documented. The IT technical and applications teams will make the necessary changes. Individuals will change depending on the system documented. The names of those involved will be noted in the documentation. Target Implementation date: Annually in February Recommendation No. 4 For the update done in February 2011, the changes are noted in an excel spreadsheet. The spreadsheet includes section updated, the lead for the changes and the team members who made the changes. Going forward the change tracker will be turned on when the changes are made. Target Implementation date: Annually in February. Recommendation No. 5 & 6 For the individuals that are receiving a paper copy, the copies are delivered and signed for by the recipient once the old copy is turned over and the recipient understands their role in the plan. For those who have access to the electronic copy, a meeting is held to let the team members know the updates are complete, where the file is located. Once they have verified their access and understand their role in the plan, they sign a document confirming this. Target Implementation date: The first week of March annually. 8