Disaster Recovery Policy - External
Disaster Recovery Policy Modified by: Rob Wray, IT and Development Manager November 2008 Approved by: Alan Matthews, Managing Director Georgia Cogdon, Executive Assistant to the MD December 2008 Objectives of the MIS Contingency Plan The role of this plan in these objectives is to document the pre-agreed decisions and to design and implement a sufficient set of procedures for responding to a disaster that involves the production and IT systems and their services. A disaster is defined as the occurrence of any event that causes a significant disruption in production capabilities. The central theme of the plan is to minimise the effect a disaster will have upon on-going operations. This plan responds to the most severe disaster, the kind that requires moving off site to a backup facility. Occurrences of a less severe nature are controlled at the appropriate management level as a part of the total plan. Assumptions of the Contingency Plan No matter how many precautions are implemented and to what extent they are enforced, most people in the data processing field agree that no computers are completely secure. The operations of a data centre could be suddenly disrupted by events we have little or no control over, involving people, mechanics, electronics, or natural disasters. It is important that you realise the exposure to your organisation at the loss of your data centre would be, and that you take steps to minimise the costs resulting from loss or damage to its resources and capabilities, and the costs to the departments and customers it serves by their losses or a reduction in computer services. Disaster Recovery Policy External V5 2 nd December 2008 1/10
The computer room is the heart of a data centre. Any threat in or near the computer room can affect the critical flow of information from this nerve centre. The location of the disaster could be more important than the amount of damage it causes. A small problem at a critical location could cripple a data centre and require it to re-establish operations at a backup facility. This plan assumes that a catastrophic event has severely crippled the main production site, forcing it to re-establish full operations at a backup facility. As soon as hardware can be installed in a cold site, processing will be moved from the backup facility to the less costly cold site. All applications will eventually be processed at the backup location, even those not classified as critical. Concurrent with the backup facility processing is the reconstruction of the original or alternate permanent facility, and the planning for the final move back to this site. Although this plan follows the assumption of a catastrophic disaster, the plan can be quickly altered to handle a less severe emergency as determined by management. Production / IT Environment The MAIN MPH production facility at the Media Centre contains both a secure area containing the group s main servers and processing systems. Aligned with this are the main Print and Braille areas which produce mono and colour standard and alternative media documents. Contingency Plan for Major Disasters The cycle from the occurrence of a disaster to the full restoration of normal processing has four phases; initial response, preparation for temporary; backup site operations, backup site fully operational, restoration and return to permanent facility. Disaster Recovery Policy External V5 2 nd December 2008 2/10
Detection and Reaction As soon as an emergency situation happens, the on-site personnel should contact the appropriate emergency authorities and then take the necessary steps to minimise property damage and injury to people in the vicinity. Following these procedures, they will then contact the Contingency Management Team so that the team can personally make an on-site evaluation of the disaster. Identifying the Problem; Notifying the Authorities Emergency Services Telephone the following numbers to reach local authorities for emergency situations such as fire, explosion, etc. Non-Emergency Numbers: Fire Department (emergency 999) 0191 454 7555 Police Department (emergency 999) 0191 232 1224 Ambulance (emergency 999) 0191 482 0000 Environment If a problem is detected concerning the computer room environment, such as electrical, water damage, excessive heat, cold, or humidity, contact the following authorities: Don Matthews, Maintenance Manager (See APPENDIX 5) Physical security If you are aware that an unauthorised person is in a secured area of the computer complex, notify the following: Rob Wray, IT & Development Manager (See APPENDIX 5) Disaster Recovery Policy External V5 2 nd December 2008 3/10
Reducing exposure Following the procedures below are used to reduce the organisation s exposure to additional losses because of actions not taken by on-site personnel. These actions are targeted at emergencies concerning airconditioning, fire, or electrical or water damage. Air-conditioner Failure A graphic temperature-monitor is located in the computer room and operates 24 hours a day. The temperature is checked each morning and periodically if heat increase is noticed. The computer room has two air-conditioning units; the fans in both units will normally operate at all times to maintain the proper airflow. If one unit fails, the second unit will carry the load. The failing unit will be repaired as soon as possible to take the strain off the only operating unit. The normal temperature for a computer room is between GS and 76 degrees. If the temperature rises above 76 degrees, take the following precautions: 1. Advise the Maintenance Manager that the temperature is above the normal operating range. He will notify the Maintenance Department for corrective action and then notify the Operations Manager. 2. If the temperature rises above 84 degrees, the hardware vendor must be notified. The IT Manager will decide which non-critical applications may continue to be processed. External contractors normally perform maintenance for air-conditioning units each month. They will clean the filters, check the gas, check the belts and hoses, and do normal visual inspections. If any problems occur between scheduled maintenance operations, the Maintenance Department should be notified. Disaster Recovery Policy External V5 2 nd December 2008 4/10
Fire Alarm Procedures Should fire or smoke be detected in the computer room, do the following: 1. Wait for automatic fire suppression do not enter computer room. 2 Power down servers from external terminal located in different room. 3. Confirm that automatic alarm system has called fire services. 4. If time permits: - Remove current tapes from computer room to a safe place. - Cover all hardware with large plastic sheets. - Remove as many tapes as possible. Flood and Water Damage Although the building is high above water level, is not prone to flooding and does not use water based sprinkler systems for fire suppression, the following rules are in place. 1. If flooding/water release occurs: a. Power down the computer. b. Push the Emergency Power Off switch next to the door of the computer room. c. Cover all hardware with plastic covers stored in the computer room. d. Call the site manager and tell him that the sprinkler is discharging and no fire is apparent. e. Have him turn off the sprinkler system and notify the necessary maintenance people. f. Make sure the vault is secured. g. Contact the Production Manager. 2. If the water damage exposure is not caused by the sprinkler system, but has affected the computer hardware: Disaster Recovery Policy External V5 2 nd December 2008 5/10
a. Power down the computer. b. Push the Emergency Power Off button to stop electric power to the computer room. c. Place plastic covers over all equipment if water is coming from above. d. Close and lock the vault door. e. Notify the IT Manager who will contact the hardware vendor to have the equipment checked for damage before the equipment is powered up. 3. Advising the Emergency Management Team of the situation: As soon as possible after a disaster, notify the Emergency Management Team. It is the responsibility of the IT Manager to make sure the team is advised of the situation. If the on-site person was unable to contact the operations management, that person will now be responsible for contacting the Emergency Management Team. The team members will be phoned in the following sequence until someone is reached. The person reached will continue to call the remaining team members. EMERGENCY TEAM MEMBERS: Alan Matthews MD (See APPENDIX 5) Mike Hewett Production Manager (See APPENDIX 5) Rob Wray IT Manager (See APPENDIX 5) The team will personally visit the site and make an initial determination of the extent of the damage. Based on their assessment, all or part of the Operations Contingency Plan will be initiated. The team will decide: a. If production can be continued at the site and repairs can be started as soon as possible. b. If production can be continued or restarted with the assistance of only certain recovery teams. c. If a limited production operation can be continued at the site and plans started to repair or replace unusable equipment. Disaster Recovery Policy External V5 2 nd December 2008 6/10
d. If the production centre is destroyed to the extent that the backup recovery facility must be used and the full Production Contingency Plan initiated. The Management Team will decide on its plan of action and then notify senior management. If the action plan requires the assistance of other recovery teams, those teams will be notified. Emergency Management Team: Initial Response Co-ordinate initial response using office procedures to protect life and minimise property damage. Assess the damage. Determine extent to which production contingency will be utilised. Minor damage processing can be restarted in a short time with no special recall of personnel. Anticipated downtime is less than one day. Damage could be to hardware, software, mechanical equipment, electrical equipment, or the facility. Major damage selected teams will be called to direct restoration of normal operations at current site. Estimated downtime is two to six days. Major damage to hardware or facility: Catastrophe - damage is extensive. Restoration will take upwards from one week. Computer room or facility could be completely destroyed. All Team Leaders will be called to begin a total implementation of the MIS Contingency Plan: - Notify senior management. - Notify users. - Prepare regular status reports for senior management. - Notify users of projected time for becoming operational. Disaster Recovery Policy External V5 2 nd December 2008 7/10
Initiation of Backup Site Procedures (Emergency Management Team notify other teams) Following an emergency at the computer centre, the operational personnel on site will take the appropriate initial action and then contact a member of the Emergency Management Team. They will determine the action to take and will notify senior management. If a determination is made to notify all other teams, the Emergency Management Team will phone the other teams using a predefined pyramid contact system. The team will at this point also make the decision to run contingency at: Site 1. Secondary building Site 2. External site supported by contractor As detailed above, the Emergency Team will make this decision based upon the scale of outage/disaster encountered. Site 1 Emergency Checklist Group A will be followed (see Appendix 3) Site 2 Emergency Checklist Group B will be followed (see Appendix 4) Contingency Plan for Major Disasters Establishment of Full Recovery at alternative backup site: a. All planned software, hardware, and resources in place at backup site, and the applications tested. b. Communications network and other equipment fully operational. Make arrangements with the telephone company and other communication vendors for delivery and installation of temporary equipment. Vendors that specialise in used equipment can deliver their equipment in a very short time. Conduct a complete series of tests to ensure full recovery of the communication network capabilities. Provide for full restoration of service at the original or new alternate facility. Disaster Recovery Policy External V5 2 nd December 2008 8/10
Alternative Facilities: Site 1 Secondary Building (use MPH alternative building) Site Address: MPH Training & Conference Centre Unit 2B Stonehills Shields Road Gateshead NE10 0HW Contact Details: Dee Matthews (Centre Manager) (See APPENDIX 5) Equipment on site: 1 A4 production printer (mono) 1 A4 printer (colour) 1 Brailler 9 Dell Vostro laptops equipped with software 1 FTP/SFTP server 1 backup production server containing all programming code for production of alternative media billing 1 backup comms line with 15 fixed IP addresses for data failover Site 2 External Agreement with Print Contractor Site Address: Total Business Group Albany House, Albany Road East Gateshead Industrial Estate Gateshead NE8 3DG Disaster Recovery Policy External V5 2 nd December 2008 9/10
Contact Details: Mr Robert S. Winter (Managing Director) Tel: 0191 490 0822 MPH has an agreement with our print suppliers to provide additional print capacity in the event that precludes the use of the second MPH site for recovery purposes. In this event we will be provided facilities to produce work either at the office area at Total Business Group or alternatively, to have a unit supplied to a specified address within 24 hours. Additional Equipment: Backup laptop, stored at the IT manager s home address, containing production code allowing transcription of alternative media billing to meet service level agreements. Disaster Recovery Policy External V5 2 nd December 2008 10/10
The Media Centre Stonehills Shields Road Gateshead NE10 0HW Tel: 0191 438 6063 Fax: 0191 469 1955 Minicom: 0191 438 1122 www.mph-uk.com