Palomar Community College District Data Center Disaster Recovery Plan

Palomar Community College District Data Center Disaster Recovery Plan Revised December 2009 Page 1

Table of Contents Sections 1.0 Introduction 1.1 Authority 1.2 Distribution 2.0 Basic Recovery Plan Requirements 2.1 Disaster Recovery Team 2.2 Offsite Storage of System Backup Tapes 2.3 Backup Facility 3.0 Recovery Plans 3.1 Disaster Preparation 3.2 Emergency Response 3.3 Recovery Procedures 3.3.1 Recovery Time Table 4.0 Disaster Recovery Plan Review Appendixes A Disaster Recovery Team B Data Backup Tape Procedure C Backup Data Center D Recovery Team and Vendor Contact Information E Data Center Applications F Emergency Telephone List The Information Services Department gratefully acknowledges the model and selected text from the Disaster Recovery Plan of Glenville State College and the Administrative Information Systems Disaster Recovery Plan of Syracuse University, Southern West Virginia Community, and Macalester College. Revised December 2009 Page 2

1.0 Introduction The personnel, equipment, software systems, and databases, which comprise the District s Data Center, are necessary in order for the District to function in an effective manner. The Data Center is located on the main campus at 1140 West Mission Road, San Marcos, CA. The purpose of this plan is to provide guidance in recovering from a disaster which might befall Data Center services to minimize downtime and to assist users in accommodating their critical processing requirements. 1.1 Authority The decision to implement disaster recovery operational procedures is the responsibility of the Director of Information Services or his/her designee in consultation with the Vice President, Finance and Administrative Services. The Disaster Recovery Team will convene as soon as possible after a disaster has occurred to assess damages and make recommendations to the Director. It is the responsibility of the Director to report this information and the decision to the Vice President, Finance and Administrative Services. 1.2 Distribution This plan will be distributed to and used by those persons (identified in Appendix A) responsible for its implementation and operation. This document will be maintained and updated by the Information Services Department whenever significant changes occur. 2.0 Basic Recovery Plan Requirements The basic requirements for the Recovery plan are as follows: Disaster recovery team Disaster recovery documentation Backup computer facilities 2.1 Disaster Recovery Team The Disaster Recovery Team (see Appendix A) has been established and organized to assess the damage to the computer facility, to control and coordinate recovery/backup actions, and to make recommendations to the Director of Information Services. The team will consist of a cross section of persons responsible for one or more of the following functions: Recovery administration (insurance notification, supplies, organization) Systems software Application software Communications Operations Facilities Hardware Revised December 2009 Page 3

The Director of Information Services serves as the Chair and in the Director s absence, the Manager of Systems and Programming is to serve as the Chair. If both are absent, the chair assignment will be made by the Vice President, Finance and Administrative Services. The disaster recovery team shall meet annually to discuss the recovery plan and make recommendations for changes in order to keep it current. All changes must be agreed upon by the majority of the team before they become part of the plan. It will be the responsibility of the Chair to make the necessary changes and distribute the new document to the appropriate individuals. In the event of a disaster or major failure, the team should convene with as many team members as available. All members of the team should assess damages and report to the Chair at the earliest opportunity. The Vice President, Finance and Administrative Services will be notified of the steps to be taken and subsequent outcomes, who will then report to the President. 2.2 Offsite storage of system backup tapes In the event that disaster befalls the current Data Center, having backup tapes stored off-site is critical. Daily differential and weekly full backups of system data are stored in the Data Center and the most current full system backup, as defined in Appendix B, is stored off-site for safety in case of fire or flood. The District s system data backup tape procedures ensure data which may be lost during a disaster can be recovered. Off-site storage of system data backup tapes is provided by Cor-o-van Records Management and their storage facility is located in Poway, CA. 2.3 Backup facility In the case of fire or natural disaster it may become necessary to move the Data Center to a backup location. The District s backup facility (see Appendix B) is in Room 814 of the Escondido Center located at: 1951 East Valley Parkway, Escondido, CA. This site was selected for the following reasons: Property: The Escondido Center is owned by the District. Technology: The Escondido Center is the largest remote site connected to the District network. As a result it has a dedicated 20 MB fiber connection to the Data Center and its own CENIC connection to the Internet. Facility: Room 814 which serves as the IDF room for the center s 800-wing was enlarged in October 2009 and designed to be the District s backup Data Center. 3.0 Recovery Plans Several recovery plans have been identified for use, depending upon the circumstances. Disaster Preparation: Being ready and planning ahead is the easiest way to be sure the District can quickly and fully recover from a disaster. As a result, this Disaster Recovery Plan and the accompanying System Continuity Manual have been prepared. Revised December 2009 Page 4

Emergency Response: These are the first actions taken in an emergency situation, designed to bring the computer systems back to operation, even if not at full capacity or in a degraded state. Appendix D provides key personnel and vendor contact information. Recovery Procedures: These are the procedures designed to return the computer systems to a fully operational state, or degraded state, including bringing up the backup facility. Appendix E provides the priority in which the Data Center applications should be brought back online and the System Continuity Manual provides the detailed information needed in order to recover. 3.1 Disaster Preparation This section outlines the minimum steps needed in order to ensure the District can fully recover from a disaster. The disaster plan must be kept current and all personnel on the recovery team must be made aware of any plan changes and fully trained to perform their assigned tasks. The offsite storage area should be inspected annually to ensure it is clean, organized and that the correct backups are in storage. The fire fighting system in the computer room should be inspected annually. Department heads should be aware of the consequences of a disaster and develop alternate data processing procedures while recovery is in progress. The Facilities department maintains and distributes the District s Emergency Telephone Numbers List of pertinent personnel to contact in case of an emergency. Procedures and lead times for replacement equipment and communications should be established. All computing personnel should be informed of the proper emergency and evacuation procedures. The Communications department is responsible for informing the campus community of the disaster and the steps being taken to remedy the situation. In the event that there is warning of an impending disaster, e.g. potential flood situations, earthquake activity in the immediate area, fire or potential building damage, the following steps should be taken: Notice should be given to as many recovery team members as possible. The Director of Information Services or designee should be briefed and a decision should be made whether to shut down the systems. Revised December 2009 Page 5

The recovery team should convene and review whatever actions may be necessary. 3.2 Emergency Response This section details the basic actions that need to be taken in the event of a disaster situation. The Director of Information Services or designee should be notified as soon as possible. The disaster recovery team should be notified and assembled as soon as reasonable under the circumstances. Team members should assess damages to their individual areas of expertise. The recovery procedures should include an estimated timeline for restoration of specific services based on the service priority described in Appendix E. The estimate can include alternative interim solutions for specific services during the reconstruction of permanent solutions. Team members should advise the Director of Information Services as to the extent of damage and recovery procedures necessary so that the decision to move the Data Center can be made after the assessment of the damage to the current facility has been determined. Pertinent vendors should be contacted and negotiations should be made for the delivery of equipment, delivery time should be noted. (See Appendix D) All department heads should be informed of the decision and given an estimated time for the return to either full or degraded services. Each member of the disaster recovery team should supervise his or her own area of expertise. The computer facility should be secured. 3.3 Recovery Procedures Recovery from a complete failure to a degraded mode of services may be necessary. In this case it may be possible to bring up individual departments on a priority basis. The decision to operate in a degraded mode and the order in which departments are to be brought back into service should be made by the Director of Information Services in consultation with team members and the Vice President, Finance and Administrative Services. With Information Services support, the Payroll Department developed a Payroll Recovery Procedure that is not dependent on Data Center processing and the materials are stored off-site with the system backup tapes. As a result, it is likely that Payroll would be able to issue Revised December 2009 Page 6

paychecks to faculty, staff and students on the scheduled pay dates without interruption. A required cash fund transfer to the San Diego County Office of Education could also be handled. If it is decided to transfer Data Center operations to the backup facility located at Escondido Center it is assumed that the basic emergency procedures have been followed as detailed in section 3.2. An inventory of the status of existing equipment and files should be compiled. The Director of Information Services should coordinate the move of facility. Vendors (see Appendix C) should be contacted to initiate delivery of replacement equipment to the Escondido Center backup facility. The estimated time of delivery should be noted. A new offsite (backup) storage facility should be located and used immediately, if necessary. All facility systems should be verified operational at this time. Systems should be tested and loaded as soon as the vendors release them to the District. Communications, networking, operations and applications software personnel should be prepared to install and or setup their individual function in the appropriate order. All department heads should be made aware of progress and or setbacks on a regular basis. Existing safety and emergency procedures at the backup facility should be examined for their adequacy as a computer room. 3.3.1 Recovery Timetable The following timetable is estimated and does not take into account the amount of time required to input data held on hardcopy during the recovery period, or re-inputting data which may have been lost during recovery. Day 1 Day 2-4 Convene the disaster recovery team, ascertain the extent of the damage and evaluate potential consequences, notify department heads, contact vendors, discuss options. Release formal communication to the campus community. At the disaster site, carry-out a safety inspection, inventory status of existing equipment and files, make a full evaluation of the damage, provide detailed accounting for insurance claims, and retrieve vital documents and reusable equipment. At the Escondido Center, prep the backup facility, set up Revised December 2009 Page 7

administrative support services, make arrangements for staff work space, compile an inventory of surviving equipment and assess communications (voice/data) and network capabilities. Define priorities for restoring minimal network services and Data Center operations, order new equipment and estimate the emergency mode processing time frame. Test the Payroll Recovery Procedure, expedite authorization of recovery expenditures, record the costs, and notify regulatory authorities of the work place change. Day 5-12 Day 13-14 Obtain system data backup tapes and ensure all relevant documentation is retrieved from the offsite storage facility for restoration of network services and information systems. Take delivery and install new equipment. Restore programs and data, and test integrity of programs and data. Reconfigure communications network equipment and ensure that the reinstated communications environment is operable and tested. Make priority determination of data processing and systems, restore partial operation to priority departments, enforce current computer and information systems security standards, Before undertaking any processing, transfer security copies of all files and programs to offsite storage location. Restore full communications and networking capabilities. Work with departments to verify data and operation of application. Start processing in accordance with prepared production schedules. 4.0 Disaster Recovery Plan Review The following steps will be taken to ensure that the Disaster Recovery Plan is current, feasible and effective: Every year the Disaster Recovery Team will be convened to review the Plan and Appendices. Updates or revisions will be made at this time. The Director of Information Services or designee will subject the contents of the off-site disaster backup tape storage to unannounced periodic audits. Results of the audit will be documented and filed with the Disaster Recovery Plan addenda materials. Revised December 2009 Page 8

Appendix A Disaster Recovery Team Vice President, Finance and Administrative Services Director, Information Services Director, Facilities Director, Fiscal Services Director, Business and Contract Services Information Services: Manager, Systems & Programming Information Services: Manager, Network &Technical Services Information Services: Database Administrator Information Services: Senior Network Specialist Revised December 2009 Page 9

Appendix B Data Backup Tape Procedure 1) Backup strategy The Information Services tape library contains all the required tape sets for the weekly Differential Backups. There are 5 backup sets, WEEK1, WEEK2, WEEK3, WEEK4, and LAST with 12 tapes per set. This device is a Dell LTO4 Tape Library that is downward compatible to the LTO2 tapes Offsite Backup tapes are rotated on a four (4) week cycle. All Offsite Backup tapes are stored at the Cor-o-van data storage facility. 2) Backup tape sets: Differential Backups - Contains system data files changed since the last Full Backup Full Backups - Contains all system data files maintained in the Data Center Offsite Backups Full Backup for Offsite storage Semi-Annual Backups - Another Full Backup twice a year in June and December 3) Backup tape schedule: Differential Backups - Scheduled Sunday Thursday nights Full Backups - Scheduled Friday nights Offsite Backups - Scheduled Saturday nights Semi-Annual Backup 1- December Semi-Annual Backup 2 - June 4) Backup software and media rotation: Symantec Backup Exec version 12 is used to backup the servers. The software has been very effective in backing up data and restoring lost data and it is upgraded by the Information Services staff when new updates are made available. Sunday starts the new backup week and the tapes will be rotated automatically using job policies for each server within the Backup Exec software. The Offsite containers are delivered every Thursday to the Information Services Helpdesk and inserted into the tape library for the upcoming cycle. The tapes are removed every Monday morning and retrieved by Cor-o-van that same afternoon for offsite storage. Any changes to the schedule due to holidays are made in advance with Cor-o-van for both deliveries and pickup dates. EZ-BACK is used to backup hard drive information (Desktop, My Documents, Favorites) on faculty and staff desktop computers to the SAN backup share drive. This drive is part of the nightly Backup Exec backups, but the user must run the EZ- BACK application because it does not run automatically. Revised December 2009 Page 10

Appendix C Backup Data Center (Room 814) Revised December 2009 Page 11

Appendix D Recovery Team and Vendor Contact Information Recovery Team Home Phone Cell Phone Bonnie Ann Dowd 858-487-4450 858-382-2284 Don Sullins 858-679-0956 858-735-0056 Mike Ellis 760-751-9936 760-415-0140 Phyllis Laderman 760-533-3858 Ken Jay 858-485-5086 Scott McClure 760-724-8137 760-805-2982 Jose Vargas 760-802-4090 Leonora Carandang 760-470-0150 Mike Dimmick 760-802-4087 Vendors Name Office Phone Cell Phone Abtech Systems Robby Wright 760-827-5141 877-671-2596 (pager) AT&T Robert DeGuire 619-237-4088 619-246-7381 Cor-o-van Tracey Reber 858-748-1100 ext.1228 Cisco Systems Emily Diaz 619-300-3840 858-526-1203 CPT Systems Sean Murphy 760-745-8562 Dell Bob Monagan 740-496-8978 Nexus Chad Williams 858-427-2623 858-752-2678 Oracle Susan Medici 619-890-2638 Veritas Software Tracy lee 714-840-7554 Revised December 2009 Page 12

Appendix E Data Center Applications Information Services reviewed the business functions supported by the applications running in the District s Data Center and recommend the following priorities: BUSINESS FUNCTION APPLICATION PRIORITY Telecommunications /Voicemail Cisco Unified Communications Cisco Unity Critical Email & Email Archiving Microsoft Exchange & SQL Server EMC SAN & EmailXtender Critical Internet CENIC/ Microsoft Windows Critical District Websites & Departmental Folders Microsoft Windows/ EMC SAN Critical Registration and Student Records PeopleSoft SA/HR & eservices Essential Financial Aid PeopleSoft SA/HR & eservices Essential Payroll & Human Resources PeopleSoft SA/HR & eservices Essential Courses and Term Schedule PeopleSoft SA/HR & eservices Essential Fiscal Accounting, Purchasing, Budgeting PeopleSoft Financials & Accounts Payable Essential Online Courses & Content Blackboard Essential Counseling SARS GRID Important Assessment Testing COMPASS Important Earth Science ESRI ArcGIS Important Graphics FTP Server Important Helpdesk Remedy Convenient Asset Management Vertere Convenient Document Imaging Hershey Convenient Library Database SirsiDynix Convenient Health Services MedPro Convenient Revised December 2009 Page 13

Appendix F Emergency Telephone List (Attach most current list here) Revised December 2009 Page 14