Disaster Recovery Plan for Office of Information Technologies Draft 3

Transcription

1 Disaster Recovery Plan for Office of Information Technologies 2010 Draft 3

2 TABLE of CONTENTS 1.1 Mission Statement Disaster Recovery Planning Recovery Objectives Assumptions of the Plan Overview of the Disaster Recovery Plan DISASTER RISKS AND PREVENTION Earthquake Fire Smoke Flood or Water Damage Power Outage Terrorist Activity or Sabotage Sudden Loss of key personnel DISASTER PREPARATION Disaster Recovery Planning Warm Sites for the Fourth Avenue Building Data Center Replacement Equipment General Backup Information Backup Procedures Offsite Storage Agreement Documentation of Current Systems Storage of DRP DISASTER DETECTION AND INITIATION Red Cross Information Disaster Recovery Teams Disaster Detection and Determination Disaster Notification When to Activate the Plan What to Do When a Crisis Erupts...18 OIT Disaster Recovery Plan, 2010 Page 2 of 59 6/14/2010

3 5.0 ACTIVATING THE PLAN DISASTER RECOVERY STRATEGY Recovery Procedures Team Plans Defined Team Plans by Service Central Services Recovery Plan PLAN MAINTENANCE AND TESTING Appendix A: EMERGENCY CONTACT LIST Appendix B: VENDOR LIST Appendix C: NETWORK DIAGRAMS Appendix D: WARM SITE AGREEMENT Appendix E: AGREEMENT WITH IRON MOUNTAIN OFFSITE STORAGE Appendix F: WARM SITE EQUIPMENT AND CONFIG...46 Hardware Network & Cabling Storage Appendix G: DATA CENTER STARTUP/SHUTDOWN PROCEDURES Appendix F: OIT CONTINUITY OF OPERATIONS PLAN OIT Disaster Recovery Plan, 2010 Page 3 of 59 6/14/2010

4 1.0 INTRODUCTION Portland State University depends heavily upon information and the ability to process and analyze this information. The university increasingly depends on computer-supported information processing and services. Technology and automated systems are often used to process and analyze information and their disruption for even a few hours could cause severely affect the overall performance of the institution. This dependency on IT services will only continue to grow. 1.1 Mission Statement The Office of Information Technologies (OIT) is a support organization dedicated to providing state-of-the-art information technology and communications to students, faculty, researchers, and staff for instruction, general research, administration, and public service in support of the University's plan for excellence. Towards that end, OIT provides a myriad of computing and communication services to the University and some other educational entities. The continuing goals of OIT are to meet the changing and expanding computing and communications needs of the University and to provide outstanding service to all University constituents. 1.2 Disaster Recovery Planning A disaster is an adverse incident that in some way causes the loss of the ability to perform a specific or group of business functions or activities. This incident could be the result of a natural event, a human mistake or willful damage. A disaster recovery plan for Technology Infrastructure Services (TIS) must respond to each of these events. The Technology Infrastructure Services Disaster Recovery Plan is a comprehensive statement of actions to be taken before, during and after a disaster. This plan is designed to reduce the risk to an acceptable level by ensuring the restoration of critical functions and services within a short time frame, and all essential production within a longer, but permissible, time frame. This plan identifies the critical functions and services for the university and the resources required to support them. Guidelines and recommendations are provided for ensuring that needed personnel and resources are available for disaster preparation, assessment and response to permit the timely restoration of services. OIT Disaster Recovery Plan, 2010 Page 4 of 59 6/14/2010

5 1.3 Recovery Objectives The reason for having a disaster recovery plan is to minimize business interruption. The risk and size of the impact on university services is also minimized and subsequently, the cost implications of an interruption in these services to students, faculty and staff is minimized as well. This Disaster Recovery Plan protects the university in the event that all or part of its information technology operations is rendered unusable. The objectives of this document are: present a course of action for restoring critical systems to Portland State University within a minimum number of days of initiation of the plan minimize the disruption of IT operations and services describe an organizational structure for carrying out the plan identify the equipment, procedures and other items necessary for recovery ensure an orderly recovery after a disaster occurs minimize risk of lost production or services provide a standard for testing the plan minimize decision-making during a disaster This recovery plan and the associated documents provide a measure of security for the services, information and other non-computer assets of Technology Infrastructure Services. The activities associated with the preparation of this plan include: Identification of the risks to the Fourth Avenue Building which may affect the critical functions Identification of the likely impacts should a disaster occur, and the likelihood of their occurrence After identification, determination of a reasonable level of expenditure on the business recovery planning process, and prevention and recovery Determination of suitable prevention and protection processes Demonstration of the validity of the plan by testing OIT Disaster Recovery Plan, 2010 Page 5 of 59 6/14/2010

6 1.4 Assumptions of the Plan This document plans for the major/worst case disaster. However, if an outage of services occurs to a lesser degree, this plan will cover the incident. NOTE: This plan does not guarantee zero data loss! Recovery efforts in this plan are targeted at getting the critical systems functional using the last available off-site backup tapes or other sources. Considerable effort will be required after critical systems are restored to restore data integrity to the point of the disaster and to synchronize that data with any new data produced from the point of the disaster forward. In addition, this recovery plan is predicated on the following three assumptions: The situation that caused the disaster is localized to the Fourth Avenue Building. It is not a general disaster affecting a major portion of the greater Portland metropolitan area. It should be noted, that this Plan will still be functional and effective even in an area-wide disaster. Even though the basic priorities for restoration of essential services to the community will normally take precedence over the recovery of Portland State University s IT services, this plan will still outline ways in which the services can be brought back online quickly. The Plan is based on the availability of a warm site. The accessibility of this site is a significant requirement. This plan is also based on the concept that the technical teams tasked with Data Center management and Networking will provide the baseline infrastructure needs in the event of an emergency. These teams will ensure a warm site is activated or another location on campus will be made available to house sensitive equipment. The Data Center/Network Teams will ensure that Fourth Avenue Building or any other site will have proper electrical, networking and security for housing systems that will process sensitive University information. The teams and their plans are detailed later in this document. 1.5 Overview of the Disaster Recovery Plan This plan will address the following areas in the event of a disaster that destroys or severely cripples the main computing center for Portland State University: data center recovery planning warm site recovery planning OIT Disaster Recovery Plan, 2010 Page 6 of 59 6/14/2010

7 network recovery planning central services recovery planning recovery of Banner processes Personnel Immediately following a disaster, a planned sequence of events begins. Key TIS personnel are notified and recovery teams are grouped to implement the plan. Personnel currently employed are listed in the plan. However, the plan has been designed to be effective if some or all of the personnel are unavailable. Portland State University must take special pains to ensure that the recovery workers are provided with resources to meet their physical and emotional needs. If the disaster is one that affects the greater metropolitan area, many local support agencies such as the Police and Fire Departments and the Red Cross will be involved. PSU will make efforts to work with any or all of these outside agencies to protect life and property and to ensure security. Salvage Operations at Disaster Site Early efforts are targeted at protecting and preserving computer equipment. In particular, any storage media (hard drives and backup tapes) are identified and either protected from the elements or removed to a clean, dry environment away from the disaster site. Designation and Activation of a Warm Site A survey of the disaster scene is done by the appropriate personnel to determine which warm site will be activated. If the disaster is contained to strictly Fourth Avenue Building, TIS will move critical systems/functions to the alternate locations on campus. If the disaster is campus wide, the Disaster Recovery Manager will determine if it is needed to activate the warm site agreement with Western Washington University. If it is determined to do so, the Associate CIO for Technology Infrastructure Services will contact WWU to alert them to PSU's status. Key personnel will begin to restore and activate our critical systems that are physically located at WWU. This may require that one or more employees leave and restore on site at Western Washington University. During this emergency restoration of critical systems, a survey of the disaster scene is done by appropriate personnel to estimate the amount of time required to relocate all OIT operations of FAB to a location some distance away from the scene of the disaster where computing and networking capabilities can be temporarily restored until the primary site is ready. Work begins almost immediately at repairing or rebuilding the primary site. This may take months, the details of which are beyond the scope of this document. OIT Disaster Recovery Plan, 2010 Page 7 of 59 6/14/2010

8 Purchase New Equipment The recovery process relies heavily upon vendors to quickly provide replacements for the resources that cannot be salvaged. The University will need to provide expedited procurement procedures (approved by the University's purchasing office and the Office of State Purchasing) to quickly place orders for equipment, supplies, software, and any other needs. Begin Reassembly at Recovery Site Salvaged and new components are reassembled at the recovery site according to the instructions contained in this plan. If vendors cannot provide a certain piece of equipment on a timely basis, it may be necessary for the recovery personnel to make lastminute substitutions. After the equipment reassembly phase is complete, the work turns to concentrate on the data recovery procedures. Restore Data from Backups Data recovery relies entirely upon the use of backups stored in locations off-site from FAB. Early data recovery efforts focus on restoring application and university data (financial, student information, etc.) from the backup tapes or other sources. Some applications and/or data may be available only to a limited few key personnel. Specific departmental information will be restored as appropriate and may require involvement with outside administrators to ensure that data is restored properly. Move Back to Restored Permanent Facility If the recovery process has taken place at the warm site, physical restoration of the Fourth Avenue Building Data Center (or an alternate facility) will have begun. When that facility is ready for occupancy, the systems assembled at the warm site are to be moved back to their permanent home. This plan does not attempt to address the logistics of this move, which should be vastly less complicated than the work done to do the recovery at a warm site. 2.0 DISASTER RISKS AND PREVENTION As important as having a disaster recovery plan is, taking measures to prevent a disaster or to mitigate its effects beforehand is even more important. This portion of the plan reviews the various threats that can lead to a disaster, where our vulnerabilities are, and steps we should take to minimize our risk. There are many forms of catastrophic loss that can occur. This section lists some of the events and situations that are considered when determining what to include in the plan. 2.1 Earthquake Earthquakes could result in partial or total loss of data for an extended period. Recovery could be slow or impossible. The probability of an earthquake in the greater Portland area is low but the severity of loss and damage in the event of an earthquake is high. OIT Disaster Recovery Plan, 2010 Page 8 of 59 6/14/2010

9 Preventive Measures Building construction makes all the difference in whether the facility will survive or not. Even if the building survives, earthquakes can interrupt power and other utilities for an extended period of time. The Fourth Avenue Building, where the Data Center is located, is served by redundant power feeds. PGE serves the building from the North and South from separate substations. In the event of a failure of both feeds, the building is serviced by a turbine that can run the complex for several days before requiring refueling. The cooling systems are run by well water drawn from sources internal to the building, a benefit in the event the external water supply to the buildings is compromised. 2.2 Fire Fire can also result in partial or total loss of data for an extended period. The probability of fire within the Fourth Avenue Building Data Center is high based on the high power consumption requirements of the equipment and heat generation in the room. Preventive Measures The Fourth Avenue Building is equipped with a sophisticated fire alarm system, with ceiling-mounted smoke detectors scattered widely throughout the building. Hand-held fire extinguishers are available. The Data Center is equipped with a air-sampling VESDA system to detect the early onset of smoke particles. Temperature sensors are also available to trigger on heat thresholds. These alert systems trigger the building alarm panels and notify campus facilities. IT systems also monitor the alarms and notify OIT personnel. If the fire is not dealt with in time, overhead sprinkler systems are deployed. Building management personnel perform periodic maintenance checks of the fire alarm systems. 2.3 Smoke Smoke particles on magnetic media can render it useless. The damage from smoke occurs much faster than damage from the actual fire or water. A relatively small amount of smoke can cause a huge degree of loss in terms of data. It is imperative that smoke be contained to the smallest possible area. Preventative Measures The preventative measures for smoke detection are the same as for fire detection. The Fourth Avenue Building is equipped with a sophisticated fire alarm system, with ceilingmounted smoke detectors scattered widely throughout the building. Hand-held fire extinguishers are available. The Data Center is equipped with an air-sampling VESDA system to detect the early onset of smoke particles. Temperature sensors are also available to trigger on heat thresholds. These alert systems trigger the building alarm panels and notify campus facilities. IT systems also monitor the alarms and notify OIT OIT Disaster Recovery Plan, 2010 Page 9 of 59 6/14/2010

10 personnel. If the fire is not dealt with in time, overhead sprinkler systems are deployed. Building management personnel perform periodic maintenance checks of the fire alarm systems 2.4 Flood or Water Damage The possibility of floods from natural causes is small for the Fourth Avenue Building. However, there is the risk of broken water and sewer lines causing major water or flood damage. Being able to detect flooding and the presence of water in the data center could help curtail serious damage to some costly pieces of equipment. Preventative Measures: Humidity levels in the Data Center are monitored by the CRAC units and reported to facilities. Water sensors are embedded under the raised floor and monitored by a NetBot which alerts IT personnel. Cramer Hall has water sensor contacts installed in the central campus router room in order to have quick detection of flooding. 2.5 Power Outage The likelihood of a power outage in the downtown Portland area is high. A three unit UPS system provides clean power to the rooms and protects against any power surges. Preventative Measures: The Fourth Avenue Building, where the Data Center is located, is served by redundant power feeds. PGE serves the building from the North and South from separate substations. In the event of a failure of both feeds, the building is serviced by a turbine that can run the complex for several days before requiring refueling. The UPS system dedicated to the Data Center can keep things running for up to an hour while the turbine spins up. In the event of a turbine system failure during an emergency, the one hour of UPS time allows IT personnel an opportunity to shut systems down in an orderly fashion. 2.6 Terrorist Activity or Sabotage It is a reality that irrational things can be done that would adversely affect Technology Infrastructure Services ability to provide IT services to the campus community. Physical damage to the Fourth Avenue Building Data Center data or facilities by disgruntled employee(s) (or students, or hackers) can pose a serious threat to data integrity. To minimize these risks, an effective guideline for the handling of human relations issues and labor disputes, in conjunction with good data protection procedures, will minimize the exposure to these risks. OIT Disaster Recovery Plan, 2010 Page 10 of 59 6/14/2010

11 Preventative Measures Portland State University uses an HID proximity card-based access system for secure areas. Every employee of OIT requiring access to the Data Center is issued a proximity card with a number that is unique to that person. The Fourth Avenue Building Data Center is protected by having these card readers on all doors into the facility. Every entry to any door that is used with these cards is logged to a database. Card access reports are looked over once a month for any odd activity and to ensure that only key personnel have access to the data center. There are also two cameras within the data center and perimeter that monitor the center at all times. The images are kept for a period of 30 days before they are deleted. The Banner system that houses all student information and financial information for the university has been placed behind a tightly controlled firewall. Most systems in the Data Center are also firewalled to some extent. All systems and services are consistently monitored through Nagios. Any unusual activity or high bandwidth traffic pages system administrators in order for someone to investigate the activity. 2.7 Sudden Loss of key personnel The loss of key personnel through death, unexpected departure or job termination is a valid and serious exposure in Technology Infrastructure Services. This type of loss can be minimized by cross training in each team and through the documentation of the services and processes TIS provides to the university. Preventative Measures If needed, consulting can be acquired from the major vendors for the university, i.e. SungardHE Banner, Microsoft, etc. Recommendations Documentation of critical systems and services should be kept in an offsite location. All critical system passwords should be documented and taken to the offsite storage location. These passwords should be updated in the document on a regular basis. Cross training in each team in TIS should be a high priority for the staff. 3.0 DISASTER PREPARATION In order to facilitate recovery from a disaster that destroys all or part of the data center in Fourth Avenue Building, certain preparations have been made in advance. This document describes what has been done to lay the way for a quick and orderly restoration of the facilities that TIS operates. OIT Disaster Recovery Plan, 2010 Page 11 of 59 6/14/2010

12 3.1 Disaster Recovery Planning The first thing to do is to have a plan. This document is part of an overall plan that Portland State University will use in response to a disaster. The extent to which a business continuity plan can be effective, however, depends on disaster recovery plans by other departments and units within the University. Every other business unit within the university should develop a plan on how they will conduct business, both in the event of a disaster in their own building or a disaster in OIT that removes their access to data for a period of time. Those business units need means to function while the computers and networks are down, plus they need a plan to synchronize the data that is restored on the central computers with the current state of affairs. 3.2 Warm Sites for the Fourth Avenue Building Data Center If Fourth Avenue Bulding is either totally or partially destroyed in a disaster, repair or rebuilding of the building and data center may take an extended period of time. In the interim it will be necessary to restore computer and network services at an alternate site. The university has a number of options for alternate sites. Each option has a cost associated with it. Western Washington University There is a warm site agreement between Portland State University and Western Washington University. This agreement stipulates that Banner related equipment is held in the other's facility. In the event of a major catastrophe that affects the entire Portland State campus, the financial data for Portland State will be recovered at the Western Washington site. The formal agreement between Portland State and Western Washington has been renewed. The details of the agreement can be found in Appendix D. A new set of equipment has been placed in this remote location (see Appendix F), a snapshot of the Oracle and Banner software trees are periodically transferred to the remote systems and an incremental data backup is performed nightly. Cramer Hall Development of this on-site warmsite has resumed. This local warmsite will be built with a focus on dealing with a localized disaster in the Fourth Avenue Building that leaves the rest of the campus intact. This planned local warmsite is located in the Urban Center Building.. The expansion to this area will provide two racks worth of capacity, with related power and linkage to the data center network infrastructure. Some infrastructure components for this site have been purchased. The Urban Center warm site is designed to be a temporary measure until the main data center is brought back on line. There isn t enough space in this location to re-build the entire data center capacity. OIT Disaster Recovery Plan, 2010 Page 12 of 59 6/14/2010

13 Disaster Partnerships One of the most critical issues involved in the recovery process is the availability of qualified personnel to oversee and carry out the recovery. This is often where disaster partnerships can have their greatest benefit. Through cooperative agreement, if one partner loses key personnel in the disaster, the other partner can provide skilled workers to carry out recovery and restoration tasks until the disabled partner can hire replacements for its staff. There is an informal agreement with the University of Oregon (UO) to host DNS services for Portland State University in the event of an emergency. UO currently hosts one of PSU s secondary DNS servers. When possible, formal agreements should be made between the departments in OIT to outside partners. 3.3 Replacement Equipment This plan contains a complete inventory of the components of each of the servers and network systems that must be restored after a disaster. Where possible, agreements have been made with vendors to supply replacements on an emergency basis. To avoid problems and delays in the recovery, every attempt should be made to replicate the current system configuration. However, there will likely be cases where components are not available or the delivery timeframe is unacceptably long. Although some changes may be required to the procedures documented in the plan, using different models of equipment or equipment from a different vendor may be suitable to expediting the recovery process. 3.4 General Backup Information New hardware can be purchased. New buildings can be built. New employees can be hired. However, the data that was stored on the old equipment cannot be bought at any price. It must be restored from a copy that was not affected by the disaster. There are a number of options available to help ensure that such a copy of OIT s critical data, currently residing the Data Center, survives a disaster at the primary facility. Design of the Current Backup Systems OIT backs up all servers hosted in the data center to two Sun StorageTek towers. Each tape backup library tower can hold 84 LTO-3 tapes. Each tape can approximately hold gigabytes of data depending on the types of files to be saved. Currently, backup compression occurs on the backup host. Windows Server Backups Windows servers are backed up using CommVault backup software. Most servers have an incremental backup each night and a full backup every two weeks. The current retention is at least 45 days for all servers. To speed up recovery in a disaster situation, CommVault runs a disaster recovery backup each night at 5 PM. This information is sent OIT Disaster Recovery Plan, 2010 Page 13 of 59 6/14/2010

14 to the WWU site and contains a copy of the CommVault database and associated settings for faster restore in a disaster situation. UNIX/Linux Server Backups Unix/Linux server backups, which include all Portland State University s financial and other data from Banner, are backed up using Legato. There are nightly incremental backups and a full backup every two weeks. A copy of the Legato configuration is also periodically included in the backup tapes that are sent off site. The current retention is 2 to 3 months. 3.5 Backup Procedures Well documented backup procedures help to ensure that recovery time is kept to a minimum. The different teams in TIS have documented procedures for backing up the critical data sets for OIT and Portland State University. 3.6 Offsite Storage Agreement OIT has contracted with Iron Mountain to store the backup tapes offsite in a secure location in a different area of Portland. The current frequency of the offsite rotation is to have full backup tape sets for all servers sent offsite every week. This will allow for the oldest data set to be 7 days or less in the event that restores need to occur. 3.7 Documentation of Current Systems Maintaining current documentation of the file and directory systems will also ease the process of recovery. In the event that key personnel are lost or injured, alternates or replacements will be able to understand the configuration of the systems. Each team in TIS maintains documentation on the structure of the services provided to the campus. The IS Team maintains their own documentation about the structure and recovery procedures for Oracle and SungardHE Banner. 3.8 Storage of DRP An up-to-date paper copy of the Disaster Recovery Plan will be stored at the offsite Iron Mountain location. A copy of this entire document, burned onto CD s, will be given to each member of the Disaster Recovery Management team as well as to the Western Washington University IT Director. OIT Disaster Recovery Plan, 2010 Page 14 of 59 6/14/2010

15 4.0 DISASTER DETECTION AND INITIATION In almost any disaster, hazards and dangers can abound. While survival of the disaster itself can be a harrowing experience, further injury or death following the disaster stemming from carelessness or negligence is senseless. Safety of personnel will be the top priority of TIS. This section will discuss the possible hazards to be aware of during an emergency. 4.1 Red Cross Information As disaster workers seek to meet the needs of victims and communities following any type of disaster, they are surrounded by and exposed to disorganization, confusion, scenes of destruction, and the tears and the pain of victims. Disaster workers have the potential to become "secondary victims," as they work long, hard hours under poor conditions. In some cases, physical dangers exist for responders. Worker accommodations may be poor when they are near or within the affected area, or may require an hour or more of travel when located outside the affected area. Personal support systems are left at home, and new supports must be formed while on the operation and while time is scarce. Supervisory styles are different from person to person; administrative organization and regulation often must change with little warning, adding additional stressors as workers try to satisfy the needs of the clients and of the organization. Most disaster workers are dedicated individuals who also tend to be perfectionists. Because of this, they are at risk of pushing themselves too hard and of not being satisfied with what they have accomplished. With so much yet to do, they often fail to take credit for the amount of work completed and the effort contributed to the operation. Frustration is common, and our usual sense of humor is often stretched beyond limits. Workers become exhausted, and anger comes easily to the surface. The anger of others -- workers, victims, and media -- becomes difficult to deal with, and may be seen as a personal attack on the worker rather than as a normal response to exhaustion. Survivor guilt may emerge as workers see the losses of others when they have suffered none themselves. COPING: Remember that you are giving those victimized by the disaster a gift of yourself -- your time and your caring -- a gift you could not give if you were also a victim. This may be your first experience with scenes of great destruction or high levels of injury and death. These are realities we don't often face, and methods of coping with these are not developed overnight. In each of us, there is an unconscious fear that a victim could be you or a loved one. You need to understand and appreciate the intensity of your emotions, and talk about your feelings to others. OIT Disaster Recovery Plan, 2010 Page 15 of 59 6/14/2010

16 Although we may function in superhuman ways during a disaster operation, the stress associated with our jobs takes its toll. We get tired... and confused... and hurt... and scared. It is critical both for ourselves and those we try to help that we understand the effects of stress and make every effort to deal with it. Stress-relieving activities are not as difficult or time consuming as we may think. A 15- minute walk during a lunch or coffee break; talking to a co-worker, supervisor, or mental health worker; going out to dinner or a movie; or just learning and using deep breathing exercises can significantly reduce stress. During the operation, it's important to eat nutritional foods, avoid drinking large amounts of caffeine and alcohol, get some exercise whenever possible, and get as much sleep as you can. That way you'll be better able to continue meeting the challenges of your job. Your supervisors will be attempting to juggle schedules so that you can have some time off to yourself to sleep, read, or just sit in the sunshine. If you feel that you need this time off before you're scheduled for it, just ask. If you need a change of assignment or setting, just ask. And, hard as it may be to turn over your duties to someone else, when it is time for your shift to be over, leave and take time to recharge. 4.2 Disaster Recovery Teams To function in an efficient manner and to allow independent tasks to proceed simultaneously, the recovery process will be handled by teams. This plan calls for teams that work together, but for which specific portions of the recovery are assigned. Disaster Recovery Management Team The Disaster Recovery Management Team oversees the whole recovery process. The members of this team should be comprised of personnel who are extremely familiar with the structure, systems, and services that Technology Infrastructure Services provides to Portland State University. The DR Manager leads the DR Management Team. In the case of TIS, the DR Manager will be the current Associate CIO for Technology Infrastructure Services. The DR Manager has the final authority on technical decisions that must be made during the recovery but works closely with the Incident Commander dedicated to OIT, typically the CIO, to ensure organizational goals are being met while dealing with the disaster. The DR Manager is responsible for appointing the other members of the Recovery Management Team. If appropriate, the DR Manager will ask additional university staff from Facilities or other areas to participate on the team. If the University Disaster Response Team has been mobilized, the DR Manager will take direction from the Incident Command center. Damage Assessment Team OIT Disaster Recovery Plan, 2010 Page 16 of 59 6/14/2010

17 The Damage Assessment Team will be comprised of personnel who are knowledgeable about the hardware and equipment located in the Fourth Avenue Building Data Center. Likely choices for this team would be a member(s) from Physical Plant, NTS, CIS teams and the IS team. The primary thrust for this team is to do two things: Provide information for the Recovery Management Team to be able to make the choice of the recovery site and provide an assessment of the recoverability of major hardware components. This team will also be the main group involved with salvaging any equipment in the data center. Based on this assessment the DR Management Team can begin the process of acquiring replacement equipment for the recovery. Facility Recovery Team The Facility Recovery Team should be led by a member in Facilities but will also need to include members from TIS. This team will be responsible for the details of preparing the recovery site to accommodate the hardware, supplies, and personnel necessary for recovery. They will be responsible for the oversight of the activities for the repair and/or rebuilding of Fourth Avenue Building or a secondary site. It is anticipated that the major responsibility for this will lie within Facilities and contractors. However, this team must oversee these operations to ensure that any facility is repaired to properly support data center operations. All infrastructure recovery (networking, power, AC, UPS, generator, security, etc.) will be the responsibility of Facilities and relevant TIS Teams. The recovery of specific services and data will be the responsibility of the individual teams in OIT (NTS, CIS and IS). Each team has separate recovery plans for restoring services quickly. Overall Data Center restart strategy can be found in Appendix G. 4.3 Disaster Detection and Determination The detection of an event which could result in a disaster affecting production or information processing systems at Portland State University is the responsibility of the Associate CIO for Technology Infrastructure Services, CIS, NTS or whoever first discovers or receives information about an emergency situation developing in one of the functional areas of Technology Infrastructure Services. 4.4 Disaster Notification Whoever detects the disaster should notify the Associate CIO for Technology Infrastructure Services or the Associate Director for Computing Infrastructure Services (CIS) who is responsible for the Data Center. In addition to providing some fault tolerance in initial response, this role sharing enables effective use of shifts during the OIT Disaster Recovery Plan, 2010 Page 17 of 59 6/14/2010

18 disaster recover process. The Associate CIO for TIS or Associate Director for CIS will monitor the evolving situation and, if appropriate, will then notify the Disaster Recovery Teams in TIS. The complete emergency contact list for the university is included in Appendix A. 4.5 When to Activate the Plan The plan should be activated if any of the following circumstances occur: Any damage to the Fourth Avenue building for any reason, which includes but is not limited to fire, water, acts of terrorism and/or sabotage. Furthermore, this Disaster Recovery Plan could be activated whenever an action occurs that hampers Technology Infrastructure Services ability to provide service to the campus community for a period greater than 48 hours. 4.6 What to Do When a Crisis Erupts As soon as a potential crisis situation develops, the first person to be alerted should be the Associate CIO for Technology Infrastructure Services who will also be the DR Manager. If it is determined that a crisis situation has occurred, that person will alert TIS personnel in order to help activate the Disaster Recovery Plan. The first phase begins with the initial response to a disaster and activation of the plan. During this phase, the existing emergency plans and procedures of Portland State s Campus Public Safety Office direct efforts to protect life and property, the primary goal of initial response. Security over the area is established as local support services such as the Police and Fire Departments are enlisted through existing mechanisms. Once access to the facility is permitted, an assessment of the damage is made to determine the estimated length of the outage. If access to the facility is precluded, then the estimate includes the time until the effect of the disaster on the facility can be evaluated. If the estimated outage is less than 48 hours, recovery will be initiated under normal operational recovery procedures. Use of facilities in the Urban Center or Cramer Hall should be investigated to see if systems can be brought online in that location. If the service outage is longer than 48 hours, the DR Manager will decide upon the appropriate warm site for recovery (on-site at another location, Western Washington or elsewhere). OIT Disaster Recovery Plan, 2010 Page 18 of 59 6/14/2010

19 5.0 ACTIVATING THE PLAN The DR Recovery Manager sets the plan into motion. Early steps to take are as follows: The Recovery Manager should retrieve the Disaster Recovery Plan. Copies of the plan should be made and handed out at the first meeting of the DR Recovery Management Team if possible. Determine Personnel Status One of the Recovery Manager's important early duties is to determine the status of personnel working at the time of the disaster. Safety personnel on site after the disaster will affect any rescues or first aid necessary to people caught in the disaster. However, the Recovery Manager should produce a list of the able-bodied people who will be available to aid in the recovery process. Taking care of people is a very important task and should receive the highest priority immediately following the disaster. While we will have a huge technical task of restoring computer and network operations ahead of us, we can't lose sight of the human interests at stake. Equipment Protection and Salvage A primary goal of the recovery process is to restore all computer operations without the loss of any data. The Damage Assessment Team can immediately set about the task of assessing the damage to the data center, and protecting and salvaging any equipment or hardware, especially those on which data may be stored. This document contains information on procedures to be used immediately following an incident to preserve and protect resources in the area damaged. It is important that any equipment or hardware in the Fourth Avenue Building be protected from the elements to avoid any further damage. Some hardware may be salvageable or repairable and save time in restoring operations. The Damage Assessment team should cover all computer equipment to avoid water damage. Ask the police to post security guards at the primary site to prevent further damage. All salvageable equipment will need to be moved to a secure location or the warm site. As soon as practical a complete inventory of all recovered equipment must be taken, along with estimates about when the equipment will be ready for use (in the case that repairs or refurbishment is required). This inventory list should be given to the DR Manager who will use it to determine which items from the disaster recovery hardware and supplies lists must be purchased to begin building the recovery systems. Establish the Recovery Control Center The Recovery Control Center is the location from which the disaster recovery process is coordinated. The Recovery Manager should designate where the Recovery Control Center is to be established. Depending on the extent of damage, the center may be located off of the campus. Initial Steps of the Disaster Recovery Management Team OIT Disaster Recovery Plan, 2010 Page 19 of 59 6/14/2010

20 The DR Manager is to call a meeting of the Recovery Management Team at the Recovery Control Center or a designated alternate site. Each member of the team is to review the status of their respective areas of responsibility. The DR Manager briefly reviews the plan with the team. Any adjustments to the Disaster Recovery Plan to accommodate special circumstances are to be discussed and decided upon. Each member of the team is charged with fulfilling his/her respective role in the recovery and to begin work as scheduled in the DR Plan. Each member of the team is to review the makeup of their respective recovery teams. If key personnel on any recovery team are unavailable, the DR Manager is to assist in locating others who have the skills and experience necessary, including locating outside help from other OUS institutions or vendors. The next meeting of the Recovery Management Team is scheduled. The DR Management team should meet at least once each day for the first week of the recovery process. An assessment can be made at the end of the first week to decide the frequency of additional meetings. The DR Management Team members are to immediately start the process of calling teams together to begin the recovery process. Cell phones and two-way radios will be important during the early phases of the recovery process. Some departments in OIT have two-way radio units that may be available if damage is not severe to the Fourth Avenue Building. 6.0 DISASTER RECOVERY STRATEGY The disaster recovery strategy pertains specifically to a disaster disabling the main computing facility in the Fourth Avenue Building. This functional area provides the infrastructure and major server support to Portland State s administrative applications. 6.1 Recovery Procedures The time required for recovery of the functional area and the eventual restoration of normal processing depends on the damage caused by the disaster. The time frame for recovery can vary from several days to several months. In either case, the recovery process begins immediately after the disaster and takes place in parallel with back-up operations at the designated warm site. The primary goal is to restore critical operations as soon as possible. 6.2 Team Plans Defined OIT Disaster Recovery Plan, 2010 Page 20 of 59 6/14/2010