Wiley College Information Technology (IT) Disaster Recovery Plan Dr. Haywood L. Strickland President and CEO Submitted by: Nathaniel E. Hewitt, III Vice President for Information Systems and Technology June 14, 2012
DISASTER RECOVERY PLAN TABLE OF CONTENTS 1.0 Purpose... 3 2.0 College Mission... 3 3.0 Objectives and Constraints... 4 4.0 Assumptions... 4 5.0 Incidents Requiring Action... 5 6.0 Contingencies... 5 7.0 Physical Safeguards... 6 7.1 McLeod Hall... 6 8.0 Insurance Consideration... 6 9.0 Recovery Team... 6 9.1 Disaster Recovery Coordinator... 6 9.2 Disaster Recovery Team Headquarters... 7 9.3 Academic Systems Recovery Team Leader Responsibilities... 7 9.4 Administrative Systems Recovery Team Leader Responsibilities... 8 10.0 Preparing for a Disaster... 8 10.1 General Procedures... 8 10.2 Software Safeguards... 9 11.0 Recovery Procedures... 10 11.1 Central Facilities Recovery Plan... 10 11.2 Systems & Operations... 10 1
11.3 Degraded Operations at Central Communications Site... 12 11.4 Use of Alternate Sites... 12 11.5 Networks... 13 11.6 Personal Computer Recovery Plan... 14 11.7 Computer Lab Recovery Plan... 14 11.8 Emergency Procedures... 15 11.9 Alternate Computing Services Facility... 15 11.10 Off-Site Storage... 15 2
1.0 Purpose The purpose of the Information Technology (IT) Disaster Recovery Plan is to provide a structured approach for contingency planning that defines procedures for responding to incidents that threaten or disrupt business continuity that is supported by computing systems, telephony services, network infrastructures and other associated technology applications and processes. The Plan is to provide for an orderly recovery and resumption of critical operations of the College including support for academic departments that rely on information technology, within reasonable time and cost constraints. The information technology resources at Wiley College include, but are not limited to, computers, printers, local/wide/wireless area networks (LAN/WAN/WLAN networks), software, electronic mail (e-mail), web sites, video, telephones, and voice mail. Over the past several years, Wiley College (WC) has set up a highly computerized operational environment. This includes the use of microcomputers in offices and laboratories and servers in the computer center that provide much of the operational support for the administrative and academic units. A campus-wide network ties these various systems together and provides communications to other computer networks, colleges and universities, and the Internet. In recent years, computers and computer-based systems have become more reliable, and failures that occur can often be diagnosed automatically and repaired promptly using both local and remote diagnostic facilities. Many computer systems are designed with various levels of redundancy, which improve their reliability and provide continual operation when certain failures occur. However, some more catastrophic disruptions and disasters may result in extended down time, and require operating critical information systems at another physical location. Almost any disaster will require special funding from the College in order to allow the affected systems to be repaired or replaced. This plan assumes that these funds will be made available as needed. Proper approval will be obtained before any funds are committed for disaster recovery operations. 2.0 College Mission Wiley College, founded in 1873 in Marshall, Texas, is a historically black, primarily liberal arts, residential, co-educational, baccalaureate degree-granting institution affiliated with The United Methodist Church. Committed to the principle of educational access, the College serves traditional and nontraditional students from diverse backgrounds that have expressed a desire and potential for learning in a Christian environment. The College, in fulfilling its basic purpose of providing a liberal arts education with a global focus, endeavors to provide an intellectually stimulating environment, promoting student competencies in communication, as well as, critical and analytical thinking. The College also supports spiritual, ethical, moral, and leadership 3
development. To achieve these superordinate goals, the College promotes an atmosphere of academic freedom and employs a faculty committed to excellence and innovation in teaching, advising, and scholarship. The faculty provides a rigorous curriculum for preparing graduates for professional or graduate studies and/or productive careers in traditional and emerging career fields. Wiley College is committed to shared governance and exemplary stewardship of its resources. The College employs innovative techniques and strategic planning in all its administrative processes, using cutting-edge technology in the delivery of services to its clientele. Acknowledging its covenant relationship with The United Methodist Church, the College affirms the ideal of social responsibility and seeks to contribute to the welfare and revitalization of its community. (Approved by the Wiley College Board of Trustees July 15, 2011.) 3.0 Objectives and Constraints The objective of this Plan is limited to the information technology support necessary to maintain the most critical administrative and academic systems that provide business continuity for the College. Units and departments should develop their own plans in conjunction with the College s Information Systems and Technology Department (ISTD) to address manual operations within their offices should computer and/or network services be disrupted. Due to costs, the College developed a business relationship with Pronet Solutions to deploy offsite information systems backup services to meet the College's current requirements for restoration of crucial digital information managed by ISTD. Reciprocal disaster recovery arrangements are also currently being made with other regional institutions (East Texas Baptist University, LeTourneau University, Jarvis Christian College and Texas College in a planned effort to restore disrupted information technology services. 4.0 Assumptions This section contains some general assumptions, but does not include all special situations that may occur. Any special decisions for situations not covered in this plan, and needed at the time of an incident classified under the Disaster Recovery Plan, will be made by senior information technology staff members on site and communicated to President s Cabinet as appropriate. This plan will be invoked upon the occurrence of an incident. The senior staff member on site at the time of the incident or the first one on site following an incident will contact the Vice President for Information Systems and Technology for a determination of the need to declare an incident. The President and the Executive Vice President will also be notified. The senior information technology staff member on site at the time of the incident will assume immediate responsibility. The first responsibility will be to see that people are evacuated as needed. If injuries have occurred as a result of the incident, immediate attention will be given to those persons injured. The Vice President for Business and Finance, Director of Administrative Services, and the Security Department will be notified. If the situation allows, attention 4
will be focused on shutting down systems, turning off power, etc.; however, evacuation of personnel is the highest priority. Once an incident, covered by this plan has been declared, the plan, duties, and responsibilities will remain in effect until the incident is resolved and proper College authorities are notified. Invoking this plan implies that a disaster recovery operation has begun and will continue with top priority until business continuity computing systems and necessary telephone support to the College has been re-established. 5.0 Incidents Requiring Action This Disaster Recovery Plan for Wiley College will be invoked under one of the following circumstances: An incident which has disabled or will disable, partially or completely, the central computing facilities, telephony and other communications network resources for a period of 24 hours or more. An incident that has significantly impaired the use of computers and networks due to circumstances that fall beyond the normal processing of day-to-day operations. An incident that is considered to be malicious in intent that can or will disable, partially or completely, the computing facilities or resources, telephony and other communications network resources for a period of 24 hours (major security breaches (hacking) or viral attacks) or more. An incident that was caused by problems with computers and/or networks that has resulted in the injury of one or more persons at Wiley College. 6.0 Contingencies General situations that can destroy or interrupt computer and telephony services usually occur under the following major categories: Power/Air Conditioning Interruption Fire Water Weather and Natural Phenomenon Sabotage and Interdiction There are different levels of severity of these contingencies necessitating different strategies and different types and levels of recovery. This plan covers strategies for: 5
Partial recovery - operating at an alternate site on or off campus. Full recovery - operating at the current central site and client areas, possibly with a degraded level of service for a period of time. 7.0 Physical Safeguards 7.1 McLeod Hall Manual door locks on the exterior entrances protect McLeod Hall. Internal offices and computer room contain dead bolt locks. Employees and students have access to the building through the exterior doors after Safety and Security personnel have opened them. Keys to exterior doors are restricted to selected personnel. 8.0 Insurance Considerations All major information technology hardware is covered under standard property and casualty insurance for the College. All computers that are covered under a maintenance contract are also covered by an insurance policy. 9.0 Recovery Team In case of a disaster, the team will use the emergency call list. General duties of the disaster recovery coordinator are overviewed in Section 9.1. Recovery team leaders have been assigned in each major area and provided with a list of responsibilities as designated in Sections 9.3 and 9.4. The team leader in each designated area will make specific task assignments during the recovery stage. 9.1 Disaster Recovery Coordinator The Vice President for Information Systems and Technology or his designated alternate will serve as Disaster Recovery Coordinator. The major responsibilities include: Establishing the Recovery Team and providing the appropriate training. Determining the extent and seriousness of the disaster, notifying the President and Executive Vice President immediately and keeping them informed about the activities and recovery progress. Invoking the Disaster Recovery Plan after approval of the President or Executive Vice President. If neither of them is available, an appropriate member of the President s Cabinet will contacted. Coordinating the re-routing of all critical data and network services with alternate recovery locations/institutions (if warranted). 6
Coordinating the appropriate personnel to assist with all hardware and data recovery efforts (if deemed appropriate). Supervising the recovery activities. Coordinating with the President and Cabinet about priorities for clients while going from partial to full recovery. Naming replacements, when needed, to fill in for any disabled or absent Disaster Recovery Team members. Any members who are out of town and are needed will be notified to return. The Vice President for Information Systems and Technology will keep clients apprised of the recovery activities. 9.2 Disaster Recovery Team Headquarters If McLeod Hall is accessible, the recovery team will meet in McLeod 108 (Vice President for Information Systems and Technology) and McLeod 115 (Training Room) will be set up as the operational control center. If McLeod Hall is inaccessible due to hazardous conditions (physical, chemical or structural), the team will meet in the President's Conference Room located in the Willis J. King Administration Building. If the Administration Building is inaccessible due to hazardous conditions (physical, chemical or structural), the Disaster Recovery Coordinator will be responsible for locating another meeting place on campus. If all campus facilities are inaccessible due to hazardous conditions (physical, chemical or structural), it is presumed that the disaster is of such proportions that recovery of computer support will take a lesser priority. The Disaster Recovery Coordinator will make appropriate arrangements. 9.3 Academic Systems Recovery Team Leader Responsibilities The Director of Academic Information Technology will serve as Academic Systems Recovery Team Leader. The responsibilities in this area include recovery in case of complete or partial disruption of services from the central academic systems. Further, with the many academic labs, this group will be responsible for providing services for any disabled academic lab using information technology and educational technology resources. Responsibilities include: Coordinating hardware and software replacement with the academic hardware and software vendors. 7
Coordinating the activities of moving backup media and materials from the offsite security files and using these for recovery when needed. Keeping the Academic Affairs office informed of the extent of damage and recovery procedures being implemented. Coordinating recovery with client departments, those using the academic computers and those using labs. Keeping the Disaster Recovery Coordinator informed of the extent of damage and recovery procedures being implemented. 9.4 Administrative Systems Recovery Team Leader Responsibilities The Director of Administrative Information Technology will serve as Administrative Systems Recovery Team Leader. Responsibilities include: Coordinating hardware and software replacement with the administrative hardware and software vendors. Supervising retrieval of backup media and materials from the off-site storage location and using these for recovery when needed. Coordinating recovery with client departments. Coordinating recovery of administrative software with all administrative units. Providing support for the re-routing of all critical data and network services with alternate recovery locations/institutions (if deemed appropriate). Coordinating schedules for administrative programming, production services, and computer job processing. Keeping the Disaster Recovery Coordinator informed about the extent of damage and recovery procedures being implemented. 10.0 Preparing for a Disaster This section contains the minimum steps necessary to prepare for a possible disaster and as preparation for implementing the recovery procedures. An important part of these procedures is ensuring that the off-site storage facility contains adequate and timely computer backup tapes and documentation for applications systems, operating systems, support packages, and operating procedures. 10.1 General Procedures Responsibilities have been given for ensuring that each of following actions have been taken and that any updating needed is continued: 8
Maintaining and updating the disaster recovery plan. Ensuring that all information technology personnel are aware of their responsibilities in case of a disaster. Ensuring that periodic scheduled rotation of backup media is being followed for the off-site storage facilities. Maintaining and periodically updating documentation and systems information stored in the off-site areas. Maintaining a current status of equipment in the main equipment rooms in McLeod Hall. Informing all technology personnel of the appropriate emergency and evacuation procedures from McLeod Hall. Ensuring that all security warning systems and emergency lighting systems are functioning properly and are periodically checked by operations personnel. Ensuring that fire protection systems are functioning properly and that they are checked periodically. Ensuring that Uninterruptible Power Systems (UPS) are functioning properly and that they are being checked periodically. Ensuring that the client community is aware of appropriate disaster recovery procedures and any potential problems and consequences that could affect their operations. Ensuring that the operations procedure manual is kept current. Ensuring that proper temperatures are maintained in equipment areas. 10.2 Software Safeguards Network Backups In accordance with ISTD standard operating procedures, daily backups are conducted at the close of business Monday through Thursday at 5:00 pm and weekly backups are scheduled every Friday afternoon. The daily and weekly backups are stored directly on the Storage Area Network System (SANS). The full monthly backup is conducted on the last working day of the month and stored on a magnetic tape, which is transported at the close of that business day to the safety deposit box at Bank One, Marshall, Texas. The first backup of each month is retained for one year. Daily and weekly backups are retained in ISTD until the next full backup. 9
Database Backups (Jenzabar) The Jenzabar EX software uses transaction capturing during the daily online processing. Thus, recovery is possible by using the last night's differential backup and the transaction audit to any particular time during the day. Telecommunications Backups Full backups that run at 3:00 AM Tuesday through Saturday secure long distance software and data. The Saturday full backups are transported each Monday morning to ISTD. The first full backup of each month is retained for one year. Backups are stored on 4mm DAT tapes and other compact media. A special backup is done immediately before each monthly billing cycle. These backups are overwritten before the next monthly billing cycle. Call records from six months previous are archived to a 4mm DAT tapes at the end of every billing cycle. Disposal dates for the saved sets are not currently implemented. Call records are routed through a solidstate recorder. This captures call records while the long distance computer is unavailable. A full backup secures telephone switch software and data each night to a diskette. The diskette left in the telephone switch is overwritten each night. Each Monday morning, the diskette is removed and transported to the ISTD. A copy of the full backup is also stored in a safety deposit box at Bank One, Marshall, Texas. There are three diskettes in rotation for the full backups. A full backup to diskette secures VoiceMail software and data. Each Monday morning, this diskette is transported to the ISTD. A copy of the full backups is also stored in a safety deposit box at Bank One, Marshall, Texas. 11.0 Recovery Procedures 11.1 Central Facilities Recovery Plan An incident classified under the Disaster Recovery Plan affecting standard ISTD managed business continuity systems centralized in McLeod Hall and/or the Thomas Winston Cole Library may serve as the impetus to place this plan into action. An incident may be of the magnitude that the facilities are not usable and alternate site plans are required. In this case, the alternate site portions of this plan must be implemented. It is obvious that all major support sections in information technology areas will need to function together in a disaster, although a specific plan of action is written for each section. 11.2 Systems & Operations This portion of the disaster recovery plan will be set into motion for computing services when an incident has occurred that requires use of the alternate site, or the damage is such 10
that operations can be restored, but only in a degraded mode at the central site in a reasonable time. The President s decision to activate the Disaster Recovery Plan will be made based upon evidence and advice from the Vice President for Information Systems and Technology. In case of a move to an alternate site, or the implementation of a plan to continue operations at the main site, the following general steps must be taken: Determine the extent of the damage and whether additional equipment and supplies are needed. Obtain approval for expenditure of funds to bring in any needed equipment and supplies. Notify local vendor marketing and/or service representatives if there is a need for immediate delivery of components to bring the computer systems to an operational level even in a degraded mode. If it is judged advisable, check with third-party vendors to determine whether a faster delivery schedule can be obtained. Notify vendor hardware support personnel that a priority should be placed on assistance to add and/or replace any additional components. Notify vendor systems support personnel that help is needed immediately to begin procedures to restore systems software. Order any additional electrical cables needed from suppliers. Rush order any supplies, forms, or media that may be needed. Once the alternative site has been identified, the following actions should occur: Notify officials where the alternate site is located. Coordinate moving of equipment and support personnel into the alternate site with appropriate personnel. Bring the recovery materials from the off-site storage to the alternate site. As soon as the hardware is up to specifications to run the operating system, load software and run necessary tests. Determine the priorities of the client software that need to be available and load these packages in order. Factors that generally determine these priorities are the time of the month and semester in which the disaster occurs. 11
Prepare backup materials and return these to the off-site storage area. Set up operations in the alternate site. Coordinate client activities to ensure that the most critical jobs are being supported as needed. As production begins, ensure that periodic backup procedures are being followed and materials are being placed in off-site storage periodically. Work out plans to ensure all critical support will be phased in. Keep administration and clients informed about the status, progress, and any problems. Coordinate plans with Wiley College administration, alternate site officials, and staff to outline measures to restore all data and network operations. 11.3 Degraded Operations at Central Communications Site In the event of degraded operations at the central communications site, it is assumed that an incident has occurred, but that degraded operations can be set up at McLeod Hall. In addition to the general steps that are followed in either case, special steps need to be taken. Evaluate the extent of the damage, and if only degraded service can be obtained, determine how long it will be before full service can be restored. Replace hardware as needed to restore service to at least a degraded service. Perform system installation as needed to restore service. If backup files are needed and are not available from the on-site backup files, they will be transferred from the off-site storage. Work with the various vendors, as needed, to ensure support in restoring full service. Keep the administration and clients informed of the status, progress and problems. 11.4 Use of Alternate Sites If the central site is destroyed, support of critical academic computing activities will be given from the alternate sites. Additional computer systems will be brought in as needed. Some steps necessary in this process are listed below: 12
Determine the priorities of client needs and upgrade computers at the academic labs. Set up for operations support. Coordinate the installation of additional equipment and moving support personnel. Coordinate restoration of any network communications with Computer & Network Services. Coordinate client-computing support with clients. As production begins, ensure that backup procedures are followed and periodic backups are stored off site. Work with the Vice President of Academic Affairs, the President, and clients in coordinating long-range plans for restoring full support by the academic computing resources. 11.5 Networks Redundancy continues to be built into the computer and telecommunications systems. Currently, the College does not have complete redundancy, but critical systems are equipped with backup policies and associated equipment. Since most of the telephone and computer communications lines are buried and in conduits across campus, connecting lines to alternate sites and to critical areas cannot be done rapidly. For example, it is estimated that if ISTD had to move, it would take 72 hours to restore critical data and voice communications lines. Some general steps that must be taken in case of a network communications disaster at the central site and/or other parts of the communications network are given: Assessment of the damage and an evaluation of steps needed to restore services. Assignment of personnel to disaster crews and assignment of tasks. If present supplies and equipment on hand are not adequate to restore service as needed, obtain approval for funds needed and contact vendors for priority shipment. Coordinate repairs of data communications disaster affecting specific areas of technology support with the recovery team leader of that area. 13
Keep the Disaster Recovery Coordinator and team leaders of support areas informed about the extent of the communications damage and recovery procedures being implemented. A diagram of all communications networks is located in the off-site storage area and periodically updated. 11.6 Personal Computer Recovery Plan Faculty and staff members are encouraged to periodically backup all files to diskette, CD or an assigned network drive. The backup procedure is the responsibility of the user and guidance will be provided by the Help Desk. Backed up media should be stored in a secure location. For network storage, users should arrange for training, access, storage space, and procedures with the Help Desk. 11.7 Computer Lab Recovery Plan In case of an event affecting only a lab, this section of the disaster plan will be executed. For recovery purposes, labs, by definition, will mean a computer area supporting a number of clients as contrasted to an area containing only a few personal computers. An event can occur in an area not defined as a lab; however, it is assumed recovery of services in this situation can be carried out in a routine manner. An area may be considered a lab even if it is in an administrative service area and there are a large number of personal computers involved. A disaster will be declared in a lab when a large portion of the units in the lab is affected to the extent that recovery in that area in a reasonable time with normal procedures is not possible. General steps that will be followed in recovery of a lab are listed. The team leader of the computer area with support duties over the lab affected will assume prime responsibility in the recovery process. Determine the extent of the damage in the lab and whether alternate lab services will be needed while recovery is taking place. Obtain College approval for any funds needed to replace equipment and supplies. Determine whether adequate equipment is available on campus, either from the Campus Store or other areas, to restore even partial services in the lab affected. Coordinate recovery of the center with Computer & Network Services if communications lines are involved in the lab. 14
If alternate services are to be provided for clients of the lab, coordinate activities between groups affected. Keep the Disaster Recovery Coordinator informed of the status of the lab and the recovery process. 11.8 Emergency Procedures In case an incident has happened or is imminent that will drastically disrupt operations, the following steps should be taken to reduce the probability of personal injuries and/or limit the extent of the damage, if there is not a risk to employees. Similar steps should be followed, where appropriate, in incidents occurring in a satellite center. An announcement should be made to evacuate the building, if appropriate, or move to a safe location in the building. As a preparation for a potential disaster, all technology area personnel should be aware of the exits available. If there are injured personnel, ensure their evacuation and call emergency assistance as needed. If the computers and air conditioning have not automatically powered down, initiate procedures to shut down systems orderly when possible. When possible and if time is available, set up damage-limiting measures. Designate available personnel to initiate normal lockup procedures. 11.9 Alternate Computing Services Facility In the event of a catastrophe that would require the development of an alternative computing facility, the ISTD will attempt to provide equipment that will meet the needs of the impacted area. ISTD will work to ensure that the type of equipment needed can be relocated and configured to meet the critical needs of the area impacted. The intent would be to survey the needs at the time of disaster, and place critical equipment where staff would have the best access to network services. This may include the use of telephone lines to establish dial-up access. Access would be limited to the applications and processes most critical to the mission of the institution. 11.10 Off-Site Storage All central file backups are made on magnetic tapes or other compact media using an appropriate backup strategy and stored in the server room in McLeod Hall on campus at WC. ISTD employees have access to keys both to the exterior doors and to the server room where tapes are stored. A copy of the full backups is also stored in a safety deposit box at Bank One, Marshall, Texas. 15