IT SERVICES DISASTER RECOVERY POLICY AND PROCEDURE

Transcription

1 IT SERVICES DISASTER RECOVERY POLICY AND PROCEDURE Originator: Chris Parkin Date: 12 March 2014 Approved by: SMT Type: Policy Due for revision: 2016/17

2 CONTENTS 1. Introduction 3 2. Locations 3 3. Risk Management 3 4. Roles and Responsibilities 4 5. Backups and Restores 5 6. Disaster Recovery Testing 5 7. Disaster Recovery Process 6 APPENDIX A - Communications 7 APPENDIX B Systems & Servers 8 2

3 1. Introduction Leeds College of Art (the College ) utilises an increasing number of systems and devices across its buildings and networks. This document outlines the recovery processes to be undertaken by IT Services staff in the event of a major disruption to computer or network services and the polices in relation to disaster recovery testing. Although the risks to the College infrastructure vary in nature and not all can be anticipated the processes involved in recovering from any detrimental event are largely the same. This policy acts as a guide, rather than a prescribed set of instructions that staff must follow. Not all instructions within this document are appropriate to every DR scenario. It is important to understand that should a DR event occur, resumption of normal service could possibly take several months and even a reduced service may take up to a week to organise. 2. Locations The College currently has four buildings: Walk Court Vernon Street Broderick Building This houses the main infrastructure for the College. All external IT services are delivered through this building. Court is linked to Walk via a fibre link. In the event of damage to Walk, Court would lose connectivity. The lease expires on this building in September Vernon Street has a small server room but would be able to contain some additional IT infrastructure should Walk no longer be available. This building is linked to Vernon Street via a laser link. In the event of damage to Vernon Street, this building would lose connectivity. The lease on this building expires in July Risk Management 3.1 Primary Risks The main risks that could lead to a DR scenario occurring are: Loss of power / power surge Fire Water damage Physical damage to equipment (either malicious damage or accidental damage from building work) Denial of service (virus attacks, hacking) Hardware failure Air Conditioning failure 3

4 3.2 Mitigation As far as possible the above risks are reduced by the following factors: The server rooms at both Walk and Vernon Street are locked and are only accessible to the IT Services team and also the Estates team. Local network cabinets are locked (or are within lockable rooms) and accessible to only delegated members of staff. Both server rooms are located on upper floors of their respective buildings, reducing the likelihood of water damage due to flooding. However, any potential damage to the roof in Walk could lead to a source of water damage. The Firewall is actively managed and there are strict protocols as to what data is allowed onto the College networks from external sources. The College utilises up to date virus software to reduce the risk of accidental or malicious infection. Power to the servers is regulated by a UPS system, which will perform a controlled shutdown of the servers in the event of a long power disruption. The UPS also shields the servers from any power spikes that may otherwise cause damage. Both server rooms contain CO 2 fire extinguishers. IT Services are in the process of procuring a fire-suppression system for the Walk server room. In the event of either server room being partially damaged, some operations can be moved to the other site. The College only purchases reputable brands of hardware that comes with a warranty (minimum 3 years). The air conditioning units in the server rooms are routinely serviced and the temperature in both rooms is monitored. Alerts notify staff of any significant increase in temperature. Full Backup tapes are located off-site (currently Court, to be moved to Vernon Street in 2014) in a fire-retardant safe. Incremental backup tapes are stored at Walk in another fire-retardant safe. The firewall servers are under a fast-response contract which would have them swapped out in the event of failure. A backup ISDN exists for the phone system at Vernon Street. 4. Roles and Responsibilities All Disaster Recovery activities will be performed by the IT Services team. Specific responsibilities will be delegated as below: 4

5 Head of IT Senior Infrastructure Support Engineer Infrastructure Support Technician Web Services & Support Technician Application Support Analyst Oversee restoration effort, co-ordinate with other teams and suppliers involved in recovery process. Manage the communication strategy. Purchasing of new equipment as required. Identify requirements for returning to business as usual. Build servers and VM Servers for software installs. Responsible for organising restore from tapes, building Backup Tape server if required. Assist with server builds. Assist with recovery process as required. Recovery and configuration of Agresso, Resource and HR system. 5. Backups and Restores In the event of either a hardware failure or other physical damage to College servers, it may be necessary to restore data from the nightly backup. A backup is taken of Staff data on both the Leeds-Art and networks. Students are responsible for backing up their own data to either cloud servers or personal storage media. 5.1 Schedule A full backup of College data occurs each Friday. On Monday, Tuesday, Wednesday and Thursday an incremental backup takes place, capturing only changes in data since the last full backup. No backups are performed on Saturday or Sunday. All backups are verified once completed. 5.2 Process The backup process runs in two stages. Initially the backup server (Oscar) performs a backup of data onto its hard drives (either a full or incremental backup). This is then backed up concurrently to tape. Currently an incremental backup is about 300GB in size. This fits on a single tape. The complete backup, as mentioned, is about 5TB in size and requires 3 tapes. 5.3 Retention Backup tapes are kept for a period of time depending on what they are storing and when the backup occurred: Monday Thursday Incremental: Stored for 4 weeks (16 tapes) Friday Full Backup: Stored for 4 weeks (4 tapes) Final Friday of Month: Stored for 12 Months (12 tapes) Each week is numbered 1-4 and tapes are used in that order. Data which was captured overnight in an incremental backup or weekly full is therefore available for restore for up to one month. Data captured in the full monthly backup is available for restore for up to one year. The tapes are stored in a fire-proof safe either in the server room or at Court. Tapes are moved to Court at the end of each week. Upon the lease on this building expiring, the safe will be located at Vernon Street. 5

6 6. Disaster Recovery Testing DR Testing is routinely done to a Virtual Server environment (unless a test server exists for the system involved). Once the system has been installed (where possible), the system owner or an equally appropriate member of staff is required to confirm that the system has been restored correctly. Restore processes are documented in the IT Wiki, of which a hard copy is stored in both IT safes. IT Services will perform a test of the three core systems (Agresso, Resource and HR) at least once in a three year period. All other servers will be tested once in a 5 year cycle. Part of this testing will be incorporated into migrating systems with lower specification server requirements onto virtual servers. A full disaster recovery test is not currently considered necessary due to the low risk of such an event occurring and current backup procedures. However, this will be periodically reviewed as the risks and complexity of systems increases. Dates of planned DR Testing and the outcomes will be posted onto the IT section of Portal. A full report of the testing will be considered by SMT. 7. Disaster Recovery Process There are a number of scenarios which would require the invocation of the Disaster Recovery Process. These are: 1. An event which has caused localised damage to either (or both) server rooms and possibly the physical servers and infrastructure in them. 2. An event which has damaged the building housing the server room. 3. An event which has damaged one or more servers, but not the server rooms, leading to loss of data or failure of systems. In addition, there are two scenarios which would require some DR measures by the College as a result of external factors: 4. An event causing damage to the University of Leeds which causes a failure to the JANET link. 5. An event causing damage to Basinghall Street which impacts upon the phone lines. The ability to recover from these events and the timescale that is achieved in is directly related to the nature of the event and having safe access to the systems involved. IT Services will only begin the recovery process upon being told by the College Health & Safety Officer or Head of Estates that it is safe to do so. Should the backup server be damaged, there would be further delay as this would have to be recovered first. 7.1 Outline 1. Plan the requirements for the new network infrastructure. 2. Co-ordinate with Janet, BT and Freedom Communications any new external network connections that may be required. Get new firewall from ECSC. 6

7 3. Purchase new servers as required or repurpose existing servers where possible. 4. Restore servers in the following order: a. Backup Exec Server. b. Network servers (e.g. AD Server). c. Communication servers (firewall, phones). d. File Server. e. Server. f. Finance System. g. Agresso System. h. HR System. i. Other Core systems (Library, Moodle). j. Non-core systems (e.g. In situations where it is more crucial to get one system recovered and operational over another (e.g. annual student return / HESA etc.) the above order can be changed with SMT approval. Actions A & B cannot be changed as they are required in order for other systems to be recovered and made operational. 7

8 Appendix A Communications Internet The internet link is provided by Janet UK. In the event of WAN loss, Janet will need to install a new router and fibre link to the College. In the even the College is utilising a new location, if a WAN link exists then it could be utilised. Contact: Telephones The phone system is provided / supported by Freedom Communications (UK). In the event of a disaster, the main College line can be diverted to a temporary BT line (or equivalent). Contact: / Network The base / backbone of the network is controlled by a centralised Firewall / Router. This is provided by ECSC. There are 2 Dell PowerEdge Servers being used for this. ECSC holds backups of the configuration along with a copy of Leeds-art.ac.uk DNS and.lcad DNS zones. ECSC will need to arrange on-site consultancy to install and configure new firewalls. Contact:

9 Appendix B Systems & Servers The College has 43 logical servers (32 physical excluding tape drives) along with Alcatel communications equipment which will need accommodation should the server room be damaged. This should be an air-conditioned (or well-ventilated and spacious) environment to maintain temperatures. The current list of servers is: Physical Servers Name Network Make Model Purchase Date DAVINCI HART MENDEZ RAPHAEL VINCENT BEAKER BOBO BUNSEN CHEF Functions HP DL360 G4p 02/05/2006 DC DNS DHCP HP DL360 G5 31/05/2007 Sophos for academic FlexLM licensing: Maya AutoCAD Alphacam licensing (dongle) Deep Freeze Norton ghost Gerber (depracated?) HP DL585 01/06/2008 BW student file server HP DL360 G6p 11/03/2010 print server (Papercut master) HP DL360 G5 11/04/2007 DC HP DL360 G6p 11/03/2010 Resource 3200 (finance system) + Deed (purchase order website) HP DL380 G5 21/12/2006 Crystal reports HP DL360 G5 01/07/2008 Liberty (LRC/Loans system) HP ProLiant DL360p Gen8 06/12/2012 Exchange 2010 ( server) CRM-DB CRM-TESTDB HP DL360 G7 15/11/2011 Agresso CRM DB Agresso Students DB Agresso Financials DB HP DL380 G7 15/11/2011 Test Agresso CRM DB Test Agresso Students 9

10 DROOP ERNIE FOZZIE GONZO OSCAR RIZZO ROWLF STRANGEPORK GAUDI KAHLO LEONARDO SWEETUMS Vernon Vernon Vernon Vernon DB Test Agresso Financials DB Test Agresso CRM Application hp DL360 G5 21/02/2007 DC DNS HP DL360 17/02/2007 Celcat (timetabling) Chris 21 (HR system) HP DL360 G5 01/05/2008 DC (Infrastructure Master) DHCP DNS WINS HP StoreEasy 26/09/2013 staff file server 1830 HP DL380 G5 29/07/2008 Backup Exec Backup Exec System Restore Veeam HP DL360 G4p 02/05/2006 DC DNS HP DL360 G5 13/02/2008 Alcatel management software HP DL380 G5 20/06/2009 Sophos for leeds-art WUS Cisco works (not used) Monitoring software (SNMP/Pinger) HP X1800 G2 02/06/2011 RS/VS student storage HP DL360 G4p 23/06/2006 Print server (papercut slave) HP DL360 G4p 23/06/2006 DC DNS DHCP HP DL360 G5 11/04/2007 DC DHCP WINS DNS BERT DMZ HP DL360 G5 31/05/2007 Open day booking system MISSPIGGY DMZ HP DL360 G6p 11/03/2010 Portal Moodles DRVMWARE Trunked HP DL370 G6p 20/02/2010 Disaster recovery 10

11 VMWARE_BEE Trunked HP DL380 G6p 11/03/2010 [ see virtual servers ] MER VMWARE_VSA1 Trunked HP DL380G7 16/01/2012 [ see virtual servers ] VMWARE_VSA2 VMWARE_VSA3 FIREWALL1 Router Dell PowerEdge FIREWALL2 R710 22/03/2011 Virtual Servers Host Name Network Functions Beamer HARING quota website MICHELANGELO Sassafrass K2 (software license monitoring) academic NPS (WiFi auth) WEBPRINT Papercut web print component CLIFFORD SAM Kronos Gatekeeper (swipe system) Pro Achieve Pro General leeds-art NPS (WiFi auth) VSA VSAMANAGER WEBSENSE CRM- STUDPORTAL Manages VSA cluster Websense management server Online Enrolments KYOCERA Kyocera contract copier management ANIMAL Helpdesks Asset tracker Wiki 11

12 Print configs Short course database CRM-MSCRM CRM- STAFFPORTAL CRM-TEST-WEB Agresso CRM application Agresso Web Services Test Agresso students applications Test Agresso financials applications CRM-WEB Agresso students applications Agresso financials applications Photos website 12