DISASTER RECOVERY PLAN FOR SOLENT MORTGAGE SERVICES. DRAFT VERSION 2 August 16th 2013

Transcription

1 DISASTER RECOVERY PLAN FOR SOLENT MORTGAGE SERVICES DRAFT VERSION 2 August 16th 2013

2 Section One: Introduction.1Objective of Plan The objective of this Recovery Plan is to enable the operation of Solent Mortgage Services to be restored as quickly as possible in the event of any disaster occurring that leads to a loss of premises or of critical support services. The initial plan will be for resources to be allocated to various locations as so as a resumption services can be implemented with 24 hours of a disaster occurring. This will involve use of all possible locations including both branch offices, and working from home. A communication network will also be devised as so as all members of staff can be accessed at all times. It is envisaged that another Basepoint office be used initially as a communication/recovery centre. The immediate resumption of I.T. facilities is essential to the plan. The overall aim of the plan will be to provide a resumption of services as soon as possible using whatever locations and facilities that are available at the time and then a gradual transfer to more permanent temporary accommodation as soon as the facilities have been made available..2definition of Disaster A disaster can take many forms; its effects upon the business varying from minimal disruption to potential long-term closure. The following are non-exhaustive lists of the types of disaster and their effects that may impact upon Solent Mortgage Services Types of Disaster Natural Disaster Terrorist Action Fire Criminal Action Other accidents.2 Possible effects of Disaster Loss of use of premises Loss of facilities: Telephone / Facsimile Information Services Computer Network / Server Hard Copy - Client Paperwork/Files Hard Copy - Company Paperwork/Files Invoicing and Accounting Paperwork Contracts Loss of key service providers

3 1.3 Structure of Disaster Recovery Plan This plan includes details of; Notification of disaster and activation of Recovery Plan Use of alternative premises Recovery of facilities Recovery of Stored Data Recovery of Documentation Provision for New Business Key service providers Client/supplier contact lists Re-Direction of Postal Services Initial Directives to Staff in the event of a Disaster Action plan 1.4 Scope Of The Action Plan This plan details the overall actions required by key members of staff in the event of any disaster. Many decisions that need to be made may only be able to be considered at the time of the disaster. This plan details the safeguards put in place with regards recovering data and information and details the general actions required in order to restore data and services to a managable level whilst action is taken to restore the original facilties. It is to be expected that some of the arrangements initally put in place may not be sustainable in the longer term and the directors or managers overseeing the continuation of the business may need to take this into account when assessing the overall needs of the company in anything but a short term basis. This document in the main deals with a total loss of all facilities at head office with sub sections detailed at the end of this document which will deal with individual losses of facilities which, in the main, will be variations of the total loss plan contained herein.

4 Section Two: Notification of Disaster to main contacts and staff.1notification to Head Office Directors Personnel. In the event of any disaster the following personnel need to be contacted in the first instance advising them of the current situation; Ian Balfour The above are responsible to assess the current situation with regards the event and decide if to implement the disaster recovery plan. 2.2 Notification by staff to main contacts In the event of any member of staff becoming aware of a disaster they must notify the Directors contact names immediately Memo to all staff regarding Notification of Disaster. See Appendix Notification of all staff by Directors Upon notification to the Directors of a Disaster, they must immediately notify all other staff of the circumstances and known facts and advise them that the Disaster Recovery Plan is to be activated. The Directors available to be contacted shall in the first instance notify each other of the incident and then call the remainder of the Staff. The full list of staff in contained within Appendix 3. The company insurers will also be notified of any event by the acting PA. SMS I.T. and the Directors are to initially meet on the site (or as close as) and assess the situation and implications. They are then to decide a course of action and then report to the designated meeting area to initiate the plans contained herein. Staff contact details are kept by all Directors and IT Staff Closure of Main Office due to Disaster In the event that the Basepoint Office is unavailable for any reason or inaccessible due to any disaster, all other staff will be asked to return home and await instructions..3 Notification to Customers

5 Customers are defined as broker introducers. will be restored as a priority and messages sent to everybody informing them of the situation and that further information will follow. Important customers and customers identified as having urgent cases will be identified using the broker contact list. Further details will be forwarded to these contacts as soon as contact centres are established. Copies of these lists should be kept by all Directors at there home address at all times. Section Three: Premises In the event of the loss of the normal operating premises of Basepoint Office, Fareham, temporary accommodation must be sought whilst steps to resolve the situation regarding the above premises are taken..1provision of temporary accommodation.1.1 The following premises have been identified as potential temporary accommodation.1.2 for Mortgage Services. These premises have been assessed as having the correct.1.3 facilities available at short notice in order for Solent Mortgage Services to trade Basepoint Havant Basepoint Waterlooville Basepoint Southampton Provision for Insurance Appendix 8 contains details of the SMS insurance policy. This currently allows for 50,000 in relocation expenses in the event of a disaster Provision for use of existing Premises The initial overall plan will be to re-allocate all processing staff to one of the above.1.10 offices, as quickly as possible. This arrangement will be considered a short term.1.11 measure to ensure continuity until a more permanent arrangement can be made.

6 .1.12 Provision for staff working from home It may be possible that some staff may be able to work from their own home. The requirements for this would be that they have use of . This is possible over any Internet Browser by accessing Mobile phone and also a minimum of broadband capabilities for access to the web. Appendix 9 details staff currently having these facilities and expressing a desire to work from home should the need arise. This list will be periodically updated. Staff working from home shall be given expenses as so as to provide themselves with stationary and postage fees. 3.7 Continuation of business. It shall be considered the priority of the company to set up a network of staff at various locations to ensure that new and existing enquiries and applications are handled in an efficient manner as circumstances can allow. The initial priorities will be to establish; A location for as many staff as possible to continue working. An effective means of communication to and from the staff Concentrate on the priorities new and existing business Re-assemble existing files and IT equipment Informing customers generally and personally of the situation. Details of all of the above are contained within this document. 3.8 Review of Long-term premises solution Decisions on the suitability of the temporary accommodation shall be reviewed by the Directors periodically and decisions made as to alternative arrangements be considered. Overall consideration must be made to the short, medium and long term requirements of the company assessed at any given time. Section Four: Facilities.1 Recovery of facilities Computer facilities Hardware and Software Refer to the Backup and Disaster Recovery Procedure for Servers (IT_WI_021) for details. Copies of the backup images on tape, may be held by certain member sof staff, and these will need to be made available for Data Restoration.

7 Section 5: Key Service Providers 5.1 Details of the Key service providers are contained within the I.T. disaster recovery plan. Section 6: Recovery of Documentation and Continuation of Existing Business COLLECTION AND RE-ASSEMBLING CASE FILES. It will be a priority to replace as many case files as soon as possible. This will involve identifying and contacting all known suppliers of the information previously held. This will have the effect of tying up fax machines for considerable periods while large faxes are sent. It will be the responsibility of the processing staff to manage the reproduction of case files as quickly and efficiently as possible making use of all resources available to them. The processing staff will be re-located to the other branches as soon as possible as so as to use there facilities for processing and collection of data. Overall, a strategy must be decided at the time of the disaster, bearing in mind the facilities and resources available at the time, on how to deal with the short, medium and long term requirements of the company. It must also be noted that re-assembling files from business derived from Satellite or Direct Submission sources should be easier due to the fact the brokers would of collected all the information themselves and thus collecting any 3 rd party references should not be an issue. the files. The process should also be quicker than other brokers we process for. Definements For the purpose of disaster recovery documentation shall be divided into recoverable and non recoverable documentation. Recoverable documentation are identified as documents that can be reproduced either internally or externally by Solent Mortgage Services or any supplier of information to Solent Mortgage Services. It is to be considered that any mortgage application and/or supporting documentation, any 3 rd party references (such as pay references), any information or communication from solicitors or lenders or any other source has the ability to be reproduced from wherever it originated. Non recoverable documentation is to be considered any documentation or data stored at the site of the head office which either cannot be reproduced anywhere else or is legal in its nature. Accountancy information is considered non recoverable as would be any legal documentation such as passports, visa s, etc.

8 Legal and non recoverable documentation is either not to be stored on Site at head office or be placed in fire proof safes. All departments within head office have confirmed they do not hold any non replaceable documentation. (See appendix 10) ACTIONS In the event that documentation ascertaining to mortgage applications are recoverable arrangements should be made as soon as practically possible to recover all possible files and data. On the basis that some or all of the documentation and files be destroyed, the following course of events should occur after workstation facilities have been restored; Using restored data, all files currently been worked on and considered live should be identified. A plan of action be drawn up at the time of the disaster as so as to concentrate business from certain individual customers to a new focal point as so as provide a single point of contact for the customer. The plan must take into account the facilities available both locally and regionally and also the profile of the customer. The plan must identify which department or individuals are dealing with which type of business and from whom. A sketch plan as to the possible re-allocation of resources regarding staff is enclosed on Appendix 9 The re-allocation plan is a model is based on staff volunteers to be re-allocated to other branches / locations (see new business) and needs to take into account the resources available at the time of disaster. Confirmation of staff willingness to relocate in the event of a disaster is contained in appendix 14. After the plan has been instigated, all brokers, solicitors, lenders and 3 rd parties should be identified and requested to forward duplicate copies of all the information supplied to Solent Mortgage Services prior to the disaster at the address allocated to the individual dealing with the case. PROVISION FOR NEW BUSINESS Should any disaster occur at head office the following course of action should occur in order to maintain a continual flow of business to Solent Mortgage Services. All new applications received should be considered to be transferred to the regional offices for them to commence the processing of the applications as normal. On identification of the new business the brokers concerned shall be notified by telephone that the case will be transferred to the selected office.

9 Regional offces shall notify the broker of receipt of any application and deal with the case as they would do normally. For accountancy purposes any cases transferred to regional offices shall be identified by the short code or broker unique number. Compensation for inconvenience shall be determined by Solent Mortgage Services at the time of disaster but will, as a minimum, entitle the member of staff to have accomodation, expenses and travelling paid for. Regional offices shall at all times have at their disposal at least one spare work station that may be used in the event of a disaster at head office. Regional offices shall make additonal workstations available for use for head office staff if required to do so. Regional managers will need to possibly re-organise internal staff at their offices in order to accommodate this. This will be under the direction of the Directors of Solent Mortgage Services. Location of the disaster recovery plans. Directors and heads of departments should keep full updated copies of this disaster plan at there home addresses for retreival should the circumstance occur. These should contain; Memo to staff in the event of a disaster Department contact list they belong to Memo to staff reference re-location and facilities available at Home. SECTION 7 RE-DIRECTION AND CO-ORDINATION OF POSTAL SERVICES Solent Mortgage Services will appoint a postal co-odinator to work from the relocated Basepoint Centre. It will be the position of the co-ordinator to assign post of any kind either to any temporary or regional office identified as dealing with any application as per the plans instigated above.

10 Backup and DR Procedure Iss 4 IT_WI_021 PRODUCED BY: Mike Tinney IT Manager REVIEWED BY: APPROVED BY:

11 Dept: Solent Mortgage Services IT WORK INSTRUCTION Procedure No. IT_WI_021 SUBJECT Backup and Disaster Recovery Procedure Date: 16/08/13 Issue: 4 No of Pages: 11 of 17

12 Introduction The Backup and Disaster Recovery policy installed here is based around Backup Exec System Recovery Ver 7.0/8.0. This product comes in two parts; the main software and a separate Recovery Boot CD. The software is based on bit level disk imaging. An initial or base image is first created which is an entire copy of the Server or hard drive. For the remainder of the week a much smaller image (slice) is taken which just includes the changes since the last backup. At the end of the week you will have 1 base image and 6 further slices. Slices are then integrated into the base image and kept under control by choosing the amount of weekly data to be kept on record. This varies depending on how dynamic or crucial the data is, ie. the File Server (Datastore) 3 weeks worth of data is kept (in case anyone wants to recover a file they deleted by mistake), the Gravity Server on the other hand, we would only need the most up-to-date version, we would not want to go back further than the day previous. We currently have 6 licences, all of which allow us to restore a Server image to a completely different hardware platform and we have at least one standby DR Server available at head office strictly for this purpose. The following servers are covered by this software: Gravity Datastore SORCE Orac Store2 Restoring lost files and Disaster Recovery In the case of a simple file recovery or recovery, the Symantec Software comes with a file viewer utility. Follow the procedure detailed in IT_WI_015 to restore a file or folder. For full blown Disaster Recover of a failed server, the procedure outlined below is to be used. To start the process you will need the Backup Exec System Recovery CD which is labeled and kept in the Software Cupboard in the IT Office (there is also a copy at Cromer) For a schedule of the various Backup s, please refer to Appendix A Dept: Solent Mortgage Services IT WORK INSTRUCTION Procedure No. IT_WI_021 SUBJECT Backup and Disaster Recovery Procedure Date: 16/08/13 Issue: 4 No of Pages: 12 of 17

13 Server Disaster Recovery Procedure Method Restoring to alternative Hardware Pre Checks 1 Ensure the destination server has enough hard disk capacity to accommodate the source server. This may not necessarily mean the same number of hard disks, although in tests, this was by far the easiest solution. It may be necessary to format the disks clean prior to the server recovery operation. If beginning a new Disk or Array, press F8 as the server boots to invoke the ROM array configuration.. 2 Link an Ethernet cable from the Backup Server (smsbackup - Gosport, smsbackup2 - Cromer) direct to the destination server, this will prevent the restoration from affecting the rest of the LAN, and prove to be far more successful. There is a 10m cable for this job. Method 1 Boot the destination Server to the original Backup Exec System Recovery CD, agree to the licence when prompted, and leave the default networking option to occur. 2 We need to load the NIC driver. From the Home tab choose Load a driver. Click on the System icon on the left and select the CD Drive from the list. Go to \DDB\COMPAQ NC67XX-77XX and select the Q57WIN32 driver. 3 Now choose the Network Tab. Click on Configure Network Settings and enter an IP address and mask similar to that of the backup server ( ), I suggest with network mask , and the def gateway and DNS server should both be Click on Map a Network Drive, and enter the following path, \\ \DATA \\ \Backup_Area2(3) - Gosport Office - Cromer Office When prompted for security password, use the following format: Username: sms-hq\administrator Solent Mortgage Services Dept: IT WORK INSTRUCTION Procedure No. IT_WI_021 SUBJECT Backup and Disaster Recovery Procedure Date: 16/08/13 Issue: 4 No of Pages: 13 of 17

14 5 5.1 Go to the Recover Tab. 5.2 Choose Recover My Computer. 5.3 Choose multiple drives (if more than one partition to copy), or choose Single Drive if doing drives one by one. 5.4 Browse to the newly mapped drive and select the latest.v2i file (use details view to verify this). Add each v2i file as required and choose an appropriate Destination drive (it needs to be large enough to support the Sorce partition). 5.5 If it is a System (C:) drive, then ensure that the Primary Partition and Set Drive for Booting, and Restore Original disk Signature options are all checked. If it is not a system drive, just the Restore Original disk Signature should be checked. There is no need to select verify the image as this just wastes time. 6 If doing Partitions separately as single Drives, do each in turn without rebooting, and only reboot when both, or all, drives have been restored. 7 Remove the Recovery CD, and reboot the new Server. If the Primary drive and its associated IDE cabling is correct, then Windows should boot and reconfigure itself. This can take some time and may involve a few re-boots, but it should sort itself out. 8 Check in System Manager that all drivers have been loaded successfully. Troubleshooting: If problems occur during the recovery check the following. Check that the destination server and the backup server (smsbackup(2)) are interconnected directly with an Ethernet cable. Ensure that destination Drives are large enough, partitioned correctly and formatted. Check the BIOS of the Destination machine and check that all the Drives are recognised, and are showing the correct size. Change BIOS Settings, IDE cable arrangements, or hard drive jumpers to correct these issues. Using the Network Tab of the recovery CD, ensure that you can ping (smsbackup), (smsbackup2). Dept: Solent Mortgage Services IT WORK INSTRUCTION Procedure No. IT_WI_021 SUBJECT Backup and Disaster Recovery Procedure Date: 16/08/13 Issue: 4 No of Pages: 14 of 17

15 Known Errors Failure: Error EA39070A: The internal structure of the image file is invalid or unsupported. This error has occurred on a DELL destination server, but the error was ambiguous and actually referred to a connectivity problem. Should this error reoccur, instead of mapping a drive within the Network Tab. Go straight to Recover my Computer, and use a UNC path direct to the image file. If this fails, copy the image file onto a USB hard drive, and use this to transfer the image, it s slower but works. Failure: Error EOBC0002: Object ~ not found Caused by not connecting the destination and source servers together with a single Ethernet connection. Failure: Process The pipe is being closed. Have not yet loaded the NIC driver, see section 2 above. Failure: Unable to verify image Remove the verify image before restoring check box. This is unnecessary as the images are verify at time of backup. PRODUCED BY: Mike Tinney IT Manager Dept: Solent Mortgage Services REVIEWED BY: WORK INSTRUCTION Procedure No. IT_WI_021 SUBJECT APPROVED BY: Date: 16/08/13 Issue: 4 No of Pages: IT Backup and Disaster Recovery Procedure 15 of 17

16 APPENDIX A Backup Files kept on Disk (smsbackup/smsbackup2) Schedule for SYS drive (c:) Data Retention Period Schedule for Data drive 1 Data Retention Period Drive Name Sorce 23:00 1 Week 23:30 1 Week D: Drive Datastore 22:00 1 Week 22:30 1 Week E: Drive 02:00 1 Week 02:45 1 Week E: Drive Gravity 20:00 1 Week 20:40 1 Week S: Drive Orac 19:00 1 Week 19:45 1 Week D: Drive Store2 04:00 1 Week 00:00 1 Week F: Drive * A separate Exchange backup is run using a Backup Exec Exchange Agent. This runs daily at 19:00 and completes a secondary backup of the Exchange Information Store. In doing this, the Exchange Log files are depleted in accordance with Microsoft. The Symantec product is not able to control the log files. Backup files kept on Tape Offsite DR Backup Copy Location Held Sorce Daily Cromer Backup Location Datastore Daily Cromer Backup Location Exchange Daily Cromer Backup Location Gravity Daily Staff home Orac Daily Cromer Backup Location Daily Cromer Backup Location Store2 Daily Staff home PRODUCED BY: Mike Tinney IT Manager Solent Mortgage Services Dept: REVIEWED BY: WORK INSTRUCTION Procedure No. IT_WI_021 SUBJECT APPROVED BY: Date: 16/08/13 Issue: 4 No of Pages: IT Backup and Disaster Recovery Procedure 16 of 17

17 APPENDIX B Disaster Recovery Testing Schedule Description of test Date Comments Completed a Restore of the Active Directory Server (SERVER-AD) onto different Hardware, but the Server could not find the Active Directory Part. Successfully completed a Restore of the SORCE Server onto different Hardware Successfully completed a Restore of SORCE on different hardware (DELL server). This was in order to create a clone Server for Development purposes. 16/08/ /08/ /01/2007 Need to attempt this exercise again. Problems occurred in that we could not copy the image file to the new disk over the Network (even directly connected), but we could transfer from a USB drive. This spreadsheet is kept in the S:\IT\Critical Information folder. Refer to this folder for up to date information. PRODUCED BY: Mike Tinney IT Manager REVIEWED BY: APPROVED BY: