RCCD Disaster Recovery Plan for Technology Services DRAFT 9/15/12

Transcription

1 Introduction RCCD Disaster Recovery Plan for Technology Services This plan provides the operational priorities and procedures for restoration of communication and information technology systems to support district business and briefly reviews the hazards across the District service area. It provides a detailed summary of the technical aspects of recovery from a major disaster event. The District needs to maintain business functions during an event if possible, or to promptly restore service afterwards. Recovery actions are facilitated by a pre-tested action plan and documented resource information. The system restoration actions support local and wide-area network and internet connectivity, integrated communications systems, and essential computer systems Mission & Objective - The IT Disaster Recovery Plan establishes defined responsibilities, actions, and procedures to recover the communication, and network and computer environment in the event of major interruption. The plan is established from assessment and prioritization of client and business objectives. Provide a written and tested plan Recovery of the physical network within the Critical Time Frames Recovery of the applications and communications systems Minimize the impact on business operations Scope Summary of existing procedures and systems, documentation Priority of actions and restoration, procedures, responsibilities Resources and contacts for district, community, vendors Framework for ongoing review and improvement, recommendations Focused on critical business systems communications and connectivity (VoIP, LAN/WAN network, messaging, wireless), system applications (Colleague, security, file servers) Factors The area of northern California served by Redwoods Community College District is subject to natural disasters and to outages of critical infrastructure. Substantial earthquakes occur routinely (Cape Mendocino - 9/1/1993 M7.0, 4/25/92 M7.3; Eureka - 11/8/1980 M7.2, 1/10/1910 M6.5, 6/15/1905 M7.2, 1/31/1922 M7.3). Even those with lower magnitude 1

2 are capable of and have caused localized damage. The Eureka Campus sits directly atop the Little Salmon fault zone. Tsunamis are created locally by the 700 mile-long Cascadia subduction zone (01/26/1700) and have impacted from distant areas (Alaska 3/28/1962, Japan 3/11/2011). Localized flooding has occurred resulting from extreme weather or tide conditions (12/18/1964 1/7/1965). Power outages are occasionally days long in duration. The area is heavily forested and falling trees have taken down many sections of electrical transmission lines. The area has limited internet connectivity, with several outages having occurred when the fiber optic lines were severed or burned. A fire occurred within the Student Union in 2010 which destroyed the network switch used for meal plan terminals and point of sale registers The District is very exposed to the aftereffects of an earthquake and / or tsunami event collapsed bridges, blocked roadways, sites and highways at or near sea level elevation, interrupted communication circuits. This puts local storage and access to backup data archives at risk. The Ellucian ERP platform is on unique and expensive hardware, and runs on a legacy internal database software (UniData) such that backup platform sharing with another college is difficult. Contagious pandemic or major flu outbreaks (avian flu 2009) would cause a closure of the campus sites. The District web site is a crucial link for communications during these events. It needs to remain available to Students and the overall community, and staff must be able to be make updates. The integrated network provides data, voice and video communications and is essential to district operations. The overall Emergency Messaging system utilizes video, cell phones, and messaging to provide alerts and updates, and it must remain operational or be restored promptly. Summary The functional plan is provided in Appendix A, and will support activities during response to an incident. Appendix B provides the integrated planning elements required to assess, develop, and improve the overall IT Disaster Plan. The IT plan provides a framework for restoration of communication services and applications support. Planning by other departments will support their specific operation requirements and response. The overall district Emergency Response Plan provides an overview of all support functions, team definitions, policies, and procedures. This plan is based on templates and elements from the Palomar CCD Data Center Recovery Plan and the Glendale State College Administrative Information Systems Disaster Recovery Plan. 2

3 Appendix A Operations and Procedures Preventative Measures Annual Review and System Testing Recovery Timetable Hardware / Software Restoration Priority List Recovery Procedure Checklists Disaster Recovery Team: responsibilities, authorizations Technology Services and District Contact List cell phones and alternative Vendor Contact List Organizations, Hardware, Software CR Resource Documents - web links Server Inventory Backup strategies, documentation, recovery Network Inventory Appendix B - Integrated Planning processes Assessment, summary, results Planning, other related documents Recommendations, improvements 3

4 Appendix A Operations and Procedures Preventative Measures These operational and preventative measures will reduce the chance of a computer disaster occurring and minimize its impact. The measures need to be performed routinely to maintain their status and to provide current information for systems and procedures. Annual testing of plan, including drills and restoration from backup data Restrict access to the computer facility to authorized personnel only Ensure there are no excessive combustible materials located in the systems rooms. Conduct regularly scheduled service on support systems, including air conditioning, fire retardant, and UPS devices Check for overloaded circuits or worn/damaged electrical and power cables Perform regularly scheduled backups and store at off-site facility Store copies of vital documentation off-site - Disaster Recovery Plan, Configuration Schematics, Maintenance and Service Contracts, etc. Annual review and improvement utilizing integrated planning processes 4

5 Annual Review and System Testing The annual testing of data archive restoration will provide confirmation of system functionality and illustrate actions to improve the processes. The measures need to be performed routinely to maintain their status and to provide current information for systems and procedures. Annual testing of plan, including drills and restoration from backup data. The disaster plan must be kept current and all personnel on the recovery team must be made aware of any plan changes and fully trained to perform their assigned tasks. The offsite storage area should be inspected annually to ensure it is clean, organized and that the correct backups are in storage. The firefighting system in the computer room should be inspected annually Department heads should be aware of the consequences of a disaster and develop alternate data processing procedures while recovery is in progress. The Facilities department maintains and distributes the District s Emergency Telephone Numbers List of pertinent personnel to contact in case of an emergency. Procedures and lead times for replacement equipment and communications should be established. All computing personnel should be informed of the proper emergency and evacuation procedures 5

6 Recovery Timetable RCCD Disaster Recovery Plan for Technology Services The following timetable is estimated and does not take into account the amount of time required to input data held on hardcopy during the recovery period, or re-inputting data which may have been lost during recovery. Day 1 Day 2-4 Day 5-12 Day Convene the disaster recovery team, ascertain the extent of the damage and evaluate potential consequences, notify department heads, contact vendors, discuss options. Release formal communication to the campus community. At the disaster site, carry-out a safety inspection, inventory status of existing equipment and files, make a full evaluation of the damage, provide detailed accounting for insurance claims, and retrieve vital documents and reusable equipment. At the Escondido Center, prep the backup facility, set up administrative support services, make arrangements for staff work space, compile an inventory of surviving equipment and assess communications (voice/data) and network capabilities. Define priorities for restoring minimal network services and Data Center operations, order new equipment and estimate the emergency mode processing time frame. Test the Payroll Recovery Procedure, expedite authorization of recovery expenditures, record the costs, and notify regulatory authorities of the work place change. Obtain system data backup tapes and ensure all relevant documentation is retrieved from the offsite storage facility for restoration of network services and information systems. Take delivery and install new equipment. Restore programs and data, and test integrity of programs and data. Reconfigure communications network equipment and ensure that the reinstated communications environment is operable and tested. Make priority determination of data processing and systems, restore partial operation to priority departments, enforce current computer and information systems security standards. Before undertaking any processing, transfer security copies of all files and programs to offsite storage location. Restore full communications and networking capabilities. Work with departments to verify data and operation of application. Start processing in accordance with prepared production schedules. 6

7 Hardware / Software Restoration Priority List These actions occur after safety, site access, and power conditions are restored. Additional applications and priority information to be determined from user survey analysis. System Function Priority Network Infrastructure all 1 Local area networks Local data exchange 1 Wide area networks District-wide data 1 Communication systems exchange Voice, , messaging Colleague ERP 2 1 Telephone, Internet, Radios, Wireless internal 3 Wireless internet 4 Meal Plan 8 Residence Halls use SARS-Grid 9 SARS-TRAK 9 Easy Badge 9 7

8 Recovery Procedure Checklists The objectives of the emergency procedures are to: Minimize injury to personnel, students and visitors Minimize damage to equipment and facilities Achieve a report of injury to personnel and damage assessment within four hours of the interruption Recover the system and LAN capabilities and functionality The safety of every RCCD employee, student or visitor is top priority. In an emergency situation where with danger of physical harm, personnel must immediately leave the facility. The Systems, Applications and Network Software Team members are responsible for the recovery and activation of essential systems, application and network hardware and software systems. Pre Disaster Understands the role and responsibilities within the Disaster Recovery Plan Works closely with recovery management team to ensure physical security of equipment Participates in Disaster Recovery Plan tests as required Post Disaster Arranges for and receives delivery of off-site data archive and equipment Restores operating system, applications and network software from backup media Tests and verifies the operating system, applications and network software are up and operational Modifies communication systems and configuration to meet site configuration Returns backup media in storage locally and off-site 8

9 The Emergency Procedures Checklists are grouped as: General Procedures Recovery Management Damage Assessment and Salvage Physical Security Administration Hardware Installation Systems, Applications, Network Software Communications Operations General Procedures Mission: To report a potential or actual disaster for action to minimize injury to RCCD personnel and damage to facilities and equipment. IN A HAZARDOUS OR LIFE THREATENING SITUATION - STOP HERE IMMEDIATELY LEAVE THE FACILITY 1) To report an emergency situation dial 9 (to obtain an outside line) and then 911. Report the type of emergency and your name and address. Eureka Campus Tompkins Hill Road, Eureka, CA (707) , fax (707) Eureka Downtown Site 333 Sixth Street, Eureka CA, 99 (707) , fax (707) xxx-xxxx Mendocino Campus 1211 Del Mar Drive, Fort Bragg, CA (707) , fax (707) Southern Humboldt Instructional Sites South Fork High School, 6831 Avenue of the Giants, Miranda, CA Redway School, 344 Humboldt Ave, Redway, CA Garberville School Site, xxx, Garberville, CA, 95xxx Del Norte Campus Washington Blvd., Crescent City, CA (707) , fax (707) Klamath-Trinity Instructional Site xxx, Hoopa CA (530)

10 2) Immediately notify your supervisor as to the type of emergency. If not available, notify Administrative Services or other administration office. 3) Notify the Recovery Management Team of the potential or actual disaster. Name Office Phone Cell Phone Home phone Alternative Steve Stratton Lee Lindsey George Kapitan CR Emergency ) Evacuate the building as instructed by emergency personnel or facilities management. Under no circumstances should the Damage Assessment and Salvage Team make any public statements regarding the disaster, its cause or its effect on the operation at RCCD. Recovery Management Mission: To decide which escalation plan segments to be implemented, oversee and coordinate the technology disaster recovery operation, notify district staff of estimated time of outage and assistance required to resolve problems. 1) Upon notification of a potential or actual disaster, immediately notify the available Management Team members and the Damage Assessment and Salvage Team to conduct a survey and damage assessment of the data center facilities. In the case of a total loss of the primary facility or the loss of personnel due to injury or death, the highest ranking employee (or contractor if so designated by the DRP) who is in a position to direct the DRP process must assume the primary management role until relieved by someone senior from the enterprise. 2) Make an outage assessment based upon the verbal report from the Damage Assessment and Salvage Team. 3) Senior Recovery Manager determines where the recovery will be conducted; at the RCCD locations or an alternative site. 10

11 4) Gain approval for activation of the necessary Recovery Teams and relocation to alternative site, if required. 5) Notify other Recovery Team members of the situation and request they assemble at a designated location for a briefing on the damage assessment and selected escalation plan. The designated location will either be the RCCD offices or the alternative site, depending upon the severity of the disaster. 6) Notify RCCD department, division, and administrative heads on the severity of the disaster and the estimated recovery time. 7) Conduct a briefing with all Recovery team members and apprise of the severity of disaster and determine: Equipment repairs or replacement acquisitions Estimated timelines 8) Assist the Recovery Teams as needed with procurement or any other problems which may require management involvement. 9) The Recovery Manager, reporting to the Senior Recovery Manager provides the coordination and assistance to the Recovery Teams in performing their recovery functions. 10) Coordinate with the Public Information Office to provide information regarding the disaster as it relates to RCCD. Damage Assessment and Salvage Mission: To assess the damage to the systems and data center within four hours, notify the Management Team of assessment, and coordinate salvage of equipment where possible. 1) Assess the requirement for physical security; minimize possible injury, restrict unauthorized persons from entry to the facility, potential for theft or vandalism to assets. 11

12 2) Utilizing the following checklist as a guideline, survey the systems and data center facilities to assess damage upon notification from the Management Team. I. Building A. Exterior B. Interior 1. Data Center - walls, ceiling, floor II. Environmental/Control A. Electrical 1. UPS 2. Transformers 3. Emergency/Building B. HVAC 1. Air Handling 2. Air Conditioning 3. Water C. Fire Suppression 1. CO 2 2. Water III. Computer Room Contents A. Equipment 1. Servers 2. External Disk Drives 3. Tape Backup 4. Network Cabling 5. Communications 6. Computers 7. Equipment B. Other 1. Magnetic Tape Media 2. Spare Parts 3. Documentation IV. Technology Services offices A. computers, workstations B. network components C. support equipment The checklist provides a guide in the review and assessment of damage to the network and/or the data center facilities. In using the checklist, the Damage Assessment and Salvage Team must consider: Is the area safe for employees or vendors to work in? Can the equipment function and at what percent of normal capacity? What must be done to recover damaged equipment? Timeline to repair or replace the damaged equipment? 12

13 3) Estimated time to recover specific system and service components. Level I Minimal damage; 72 hours Level II Moderate damage; 3 to 7 business days. Level III Extensive damage; greater than 7 business days. 4) Identify equipment, documentation or spare parts which are immediately salvageable or need repair. 5) Notify the Management Team of survey results, assessment of damage, estimated time to recover systems and services, and potentially salvageable equipment. 6) Document findings from the survey and damage assessment. 7) Attend the recovery briefings as scheduled by the Senior Recovery Manager to apprise Recovery Team members of current system and service status and restoration timelines. 8) Log salvageable equipment, items for disposition, and locations. 9) Coordinate with the Administrative Team, vendors and suppliers in restoring or replacing salvageable equipment. 10) Assist in the cleanup of the disaster area in regard to the technical system facilities to permit eventual renovation and/or reconstruction. Physical Security Mission: To ensure the physical security of the disaster site for files, reports and equipment while in transit and act as liaison with emergency personnel. 1) Upon notification of a disaster by the Management Team assemble at the designated site for a briefing on the extent of damages, escalation plan implemented and support required. 13

14 2) Establish physical security at the RCCD facilities, restricting access to the damaged area unless tasks require a presence in the area, (Damage Assessment and Salvage Team, insurance, RCCD investigators, RCCD vendors, and building engineers). Considerations in the level of security required are: Is entry into the damaged area safe? Is there damage to the entire building? Has access been restricted by emergency, facilities, or administrative personnel? Are guards required to restrict access for safety or security? 3) Depending upon the extent of the damage to the physical building, coordinate with emergency personnel and facility operations for access to the location. office Cell home Garry Patrick Tim Flanagan George Kapitan CR Emergency ) Schedule security for all files, reports, and equipment in transit as requested by the Management Team. 5) Assist in any way possible the authorized investigation of the damaged site. Administrative Support Mission: To provide administrative support to all Disaster Recovery Teams, including procurement of equipment and supplies, telephones (acquisition and installation), and other administrative functions not provided by other team members. 1) Upon notification of a disaster by the Management Team assemble at the designated site for a briefing on the extent of damages, escalation plan implemented and support required. 2) Coordinate, prepare and submit for authorization to the Management Team procurement requests for equipment, supplies and services required to support the recovery process as requested by the Recovery Team members. 14

15 3) Maintain log of all procurements in process and scheduled delivery dates. Notify Recovery Team members of scheduled delivery dates; coordinate with vendors to ensure deliveries or service requests are completed as required. 4) Complete the acquisition and documentation of communications equipment and services. 5) Supply required clerical, filing and other administrative support as required by Recovery Team Hardware Installation Mission: To plan, design, schedule, install, and verify computing hardware required to provide computer capabilities within the time frame specified. Coordinate with the vendors. 1) Upon notification of a disaster by the Management Team assemble at the designated site for a briefing on the extent of damages, escalation plan implemented and support required. 2) Retrieve the equipment, system and LAN configuration from the storage containers or when delivered by the off-site storage vendor. 3) Review the Hardware and Software Inventory lists to determine the equipment required. 4) Coordinate with the Damage Assessment and Salvage Team to inventory usable and salvageable equipment. 5) Coordinate with the Administration Team in the procurement of any additional equipment required. 6) Coordinate with the alternative site for installation and connection of temporary systems to provide communications and services access. This includes support equipment (desk, chair, files, printer, scanners, etc.). 7) Coordinate with the alternative site and the Disaster Recovery Team for installation and connection of workstations and servers on the network to support the applications, if required. 15

16 Systems, Applications & Network Software Mission: To obtain off-site tape backups, restore and test the operating systems, applications and network software needed to provide the capabilities required within the Critical Time Frames specified. 1) Upon notification of a disaster by the Management Team assemble at the designated site for a briefing on the extent of damages, escalation plan implemented and support required. 2) Contact the off-site storage facility and request the off-site storage backup tapes, equipment, manuals and documentation. You will need to provide them with the following: RCCD Account Number: xxxx Sherlock Records Management - info@sherlockrm.com 1749 Alamar Way Fortuna, CA (707) ; fax (707) ) Receive delivery of backup tapes, manuals and documentation at or from the recovery site. 4) Restore the operating system, applications, network software, and production data from the backup tapes. 5) Test and verify that the restore completed successfully. 6) Modify configuration of operating and network software to meet configuration. 7) Return backup medium in storage containers to off-site storage. Communications Mission: To design, install and verify the communications equipment and network cabling. These roles may be provided at facilities other than the primary recovery facility. 1) Upon notification of a disaster by the Management Team assemble at the designated site for a briefing on the extent of damages, escalation plan implemented and support required. 16

17 2) Review the Hardware/Software Inventory list found in the appendix to determine the communications and network equipment required. 3) The Communications Team coordinates with the Damage Assessment and Salvage Team on equipment to obtain an inventory of usable and salvageable communications equipment. 4) Coordinate with the Administration Team in procuring network cabling and installation, communications equipment, and telephone lines. Operations Mission: To provide operating support for the production systems at the backup data center. 1) Upon notification of a disaster by the Management Team assemble at the designated site for a briefing on the extent of damages, escalation plan implemented and support required. 2) Schedule new pickup point with off-site storage vendor. 3) Initialize new tapes and backup devices. 4) Complete daily backups of all systems and coordinate with off-site storage vendor to ensure tapes and backup devices are sent to secure storage daily and off-site per schedule. 5) Set-up and operate a sign-in, sign-out procedure for all materials sent to and from the offsite storage location. 6) Monitor security of the network. 7) Provide production support to users as required. 17

18 CR Technology and Support Staff Contact List Confidential Information, not for public distribution Name Title Office Home Cell Radio Alternative Technology Services Steve Stratton Dir, Tech Ops (707) Don Bryant retired (707) Paul Chown Applications Analyst (707) pchown2001@yahoo.com Tom Cossey Applications Analyst (707) tomits@sbcglobal.net Mark Cummings Technology Specialist II (707) mark-cummings@sbcglobal.net Kirt Frischknecht Electronic Systems (707) Specialist II Ray Kingsbury Network Administrator (707) Tim Kirschbaum Electronic Systems (707) Specialist II Jose Ramirez Electronic Systems (707) Specialist II Brian Van Pelt Web Specialist (707) Mark Bernards Technology Specialist I (707) mb.eureka@gmail.com Jason Brewer Technology Specialist I (707) maverick2812@netscape.net Mendocino Geisce Ly Dean Linda Turner Administrative Office Manager Josh Edelman Instructional Support Specialist III Del Norte Anita Janis Dean Jim Graham Technology Specialist II Geoff Cain Director, Distance Ed (707) Garry Patrick Director, Maint & Ops (707) Tim Flanagan Director, Facilities & Plan (707) Debbie Williams Admin Office Assistant II (707) maint dwmsy99@gmail.com Tami Engman Account Clerk II (707) maint engman1975@att.net Max Capps Maint Spec IV Electrician (707) Matt Hanson Purchasing Specialist (707) Lee Lindsey VP Admin Services (707) Lorie Walsh Administrative Assistant (707) Carla Spalding Controller Kathy Lehner President/Superintendent (707) Michelle Anderson (707) Utpal Goswami VP - Instruction (707) Crislyn Parker Exec Assist to President Keith Snow-Flamer VP Student Development (707) Paul DeMark Public Information Officer (707) RCCD Master Employee List 18

19 19

20 20

21 21

22 Contact Information - Organizations work in progress Organization Service Name Title Phone Humboldt County Office of Potential backup Gary Eagles Superintendent Education HCOE network Gabor Sziladi Director, IT Humboldt State University Potential backup network County of Humboldt Financial Chancellor s Office Potential backup network CENIC network Access Humboldt Network Sean McLaughlin (707) (707) City of Eureka Potential backup network Academy of the Alert system Redwoods/FUSD Suddenlink, Charter, Comcast network AT&T (and others) sean@accesshumboldt.net California Highway Patrol Sheriff s Department - Humboldt Sheriff s Department Mendocino Sheriff s Department Del Norte Sheriff s Department - Trinity 22

23 Contact Information Hardware vendors work in progress Company Function Contact Phone/fax Dell Servers Tangent Computers AT&T Phone / internet SuddenLink Internet Dev Group Cisco gear ConnectX Cisco/wireless/VoIP Steven Roper CDWG misc Kevin Cooper kevicoo@cdw.com STA HP hardware CCS - Eureka cabling Dan Brown (707) ccseureka@aol.com Xerox - local Copiers, scanners RWS radios Redwood Supplies, Electronics 711 Summer St, Eureka Campton Electric supplies 23

24 Contact Information Software vendors work in progress Company, address, site Ellucian 4375 Fair Lakes Court Fairfax, VA Function Contact Phone/fax Tech support details Colleague ERP, Sharon Keith o: (951) WebAdvisor, Orchestrator Business Advisor c: (951) Tom O Rourke Account Manager, o: c: tom.orourke@ellucian.com Western Region f: SARS Grid Grid, Trak CI Solutions ID cards National Student Clearing House XAP CCCApply, Credentials Solutions Blackboard Transact Meal plan Nebraska Bookstore System Capital Business Machines GE Alliance GoPrint HigherOne KOHA RackSpace rsmart Tegrity bookstore Cafeteria POS SSAT security Steve Watt Regional Account Executive c f Steve.Watt@blackboard.com Behind.blackboard.com Microsoft ComputerLand of Silicon Valley Parature Softdocs CCCO Sherlock Outlook, Active Directory AskCR ticketing Document imaging, eforms MIS reporting 24

25 CR Resource Documents Technology Services Recovery Plan Emergency Preparedness & Safety Committee - Emergency Preparedness Plan - 2b.pdf Emergency Action & Fire Prevention Plan - Board Policy District Response to Emergency Situations - Safety Procedures