Disaster Recovery Planning and Implementa6on. Chris Russel Director, IT Infrastructure and ISO Compu6ng and Network Services York University

Transcription

1 Disaster Recovery Planning and Implementa6on Chris Russel Director, IT Infrastructure and ISO Compu6ng and Network Services York University

2 Agenda Background for York s I.T. Disaster Recovery Planning and Ini6al Hurdles Major Implementa6on Steps Expanding outside the main data centre Physically diverse cable plant Resilient/redundant networks and systems Lessons Learned Next Steps

3 Star6ng Out with Disaster Recovery Planning Educause survey results in 2007: 17% of responding ins6tu6ons had completed DR plans for central I.T. If it was easy, someone would have already done it.

4 Project start: January 2003 Genesis of York s I.T. DRP Ini6a6ve Sponsored by CIO and VP Finance and Administra6on Intended to be a mul6 phase, mul6 year effort Scope: Systems: key informa6on systems Scenarios: localized disaster or failure A lot of defini6on s6ll to be desired

5 Where DO you start? Easy to feel overwhelmed Best Prac6ces variety of sources ITIL, COBIT, ISO17799, etc. Good informa6on but these will not take into account resource considera6ons, compe6ng priori6es, ins6tu6onal poli6cs, etc. Scope (Systems, Scenarios) must s6ll be defined

6 Disaster Recovery Threat Assessment Proximity to heavy industry Oil depot across street Near intersec6on of major highways (400 & 407) Freight train corridor (chemical spill 1980) Main Campus is on airport flight path Main data centre in basement of old building High traffic area (library, with washrooms) located directly above Worst-case scenario chosen: Loss of building containing main data centre

7 Typical DRP Phases 1: Risk Assessment 2: Architecture/Recovery Strategies 3: Implementa6on 4: Tes6ng 5: Sustain Program (loop back to 1) First What systems are you planning for?

8 First Phase: Business Impact Assessment/Cri6cality Analysis Business Con6nuity vs. Disaster Recovery not the same thing: BC: Considers the academic, research and business func6oning of the ins6tu6on as a whole. Includes risk assessment, and plans for func6onal units and business processes. Poten6ally wider variety of scenarios to consider. DR: Ac6vi6es to enable recovery to an acceptable condi6on aner a disaster. BC includes DR. DR requires guidance from BC to direct priori6es and set scope.

9 Cri6cality Assessment (CA) Engaged func6onal unit leaders and I.T. support areas Asked to gauge effect on loss of I.T. systems on business processes, loss of business processes on the mission of the University Asked to iden6fy maximum tolerable outages and data loss Results Greater than 50% of business processes ranked cri6cal very high cost to implement DR measures Reality Check: everyone s work is cri6cal at some point the findings did not agree with observed impacts from lesser scale outages VP and AVP consulta6ons final step to ra6onalizing CA Costs of DR implementa6on < Risk of delayed business processes

10 Risk Management

11 Service Classifica6ons Category Objec.ves Service Types 1 RTO,RPO<4hr Safety, Primary Communica6ons 2 RTO<72hrs Cri6cal Customer and Partner Interfaces 3 RTO<7 days Primary Customer and Partner Interfaces 4 RTO<14 days Secondary Customer and Partner Interfaces 5 RTO<30 days Departmental 6 RTO>30 days All Others RTO = Recovery Time Objective, max time between incident and system recovery RPO = Recovery Point Objective, max acceptable data loss between incident and data restore point Major systems are grouped into these categories

12 DRP Phase 2: Architecture Task: Design systems and infrastructure to support recovery objec6ves Infrastructure not categorized; recovery objec6ves variable depending on reliant services Clear vulnerabili6es: Main Data Centre Fibre/Cable Plant Network (LAN, WAN)

13 Architecture: Recovery Strategies Category Recovery Strategy 1 On or Off campus Hot site, ac6ve mirror or synchronized data copy 2 On or Off campus Hot site, async data copy or restore from disk backup 3 On or Off campus Hot site, promote QA, restore from backup 4 On or Off campus cold site, repurpose hardware, restore from backup 5 On or Off campus cold site, replacement or consolidated hardware 6 On or Off campus cold site, replacement or consolidated hardware

14 DRP Phase 3: Implementa6on Infrastructure work completed for: Cable Plant Off campus Data Centre On campus alternate Data Centre WAN (in progress) Some design work s6ll progressing (Network) None of this is men6oned in any DR category, it just has to be there

15 Steacie Data Centre

16 Cable Plan Modifica6ons Historical: low cost star configura6on, no DR capability or rerou6ng op6ons Solu6on: retrofit ring like topology Start with selected physical fibre aggrega6on points Added new fibre to provide alternate connec6vity to these points, leading to new data centre in another building (500m distant, not in basement, former dance studio)

17 Cable Plant

18 Cable Plant DR Modifica6ons

19 Power Redundant power feeds, but no generator University co genera6on plant expected to provide backup power to building in the event of a loss This proved false, co gen does not have capability to direct output to selected buildings, and does not have capacity to serve en6re campus. Only 6me in 40 years both feeds have been down was Aug New alternate data centre has generator, more robust than main facility

20 Cooling Data centre usage always growing Density increasing along with server count helps reduce increased need for physical space High density equipment need more power/area More power/area means more cooling/area Airflow also a concern cubic metres/min Failure of cooling as bad as power failure S6ll working on refinements to resiliency Make sure to size backup power to include A/C

21 Co loca6on Site RFP in 2007, 9 racks of equipment Sufficient for priority systems: SIS, ERP, , Web, LMS, virtual tape library and part of backup system Telus IDC Selected, 15km distant Deployment as hot/produc6on site in progress WAN upgrade with L2 na6ve fibre connec6vity (currently 1Gbit) plus redundant connec6ons to the Keele campus

22 Network Survivability Designing alternate network core to be located at alternate on campus data centre WAN Upgrade Greatly enhanced performance L2 na6ve fibre connec6vity to Telus IDC Physically diverse links to off campus, and to fibre aggrega6on points on campus where possible

23 Off Campus Storage and Backup System Redesigning Backup and storage systems to u6lize WAN + on & off campus data centres Disk staging/virtual tape for backup system Stop duplica6ng tapes, they are already offsite Mirroring and/or replica6on for short RPO systems

24 Telecom System Survivability TDM Technology Not going to rip & replace Not going to expand for redundancy U6lize IP telephony and leverage DR capable network TDM/IP gateways to bridge non IP phones Arrangements with telecom provider to temp reroute calls Finally, deploy DR capable telecom system to use both on campus data centres similar to network

25 DRP Documenta6on Refresh Revising classifica6on of systems to match new technical capabili6es Adapt to changing business processes and needs Tes6ng Phase: Walkthrough exercises at minimum (done) Bare metal rebuilds/restores as SOP DR fire drill poten6al

26 Planets must be aligned Lessons Learned Execu6ve, Management (technical and business), Staff Resources all commiwed lack of any probably means it won t happen Risk Assessment par6cipa6on by business units and execu6ve review Technical: Virtualiza6on helps solve a lot of problems, but S6ll runs on real hardware, s6ll need all the usual elements: alternate sites, power, cooling and the network to get there

27 Lessons Learned Recovery is good, resiliency is bewer Some6mes not much extra effort to go from fast recovery to simply not going down at all When can you claim victory? Be realis6c No plan survives contact with the enemy Failure to plan is planning to fail Can t plan for every scenario, but the act of DR planning can be as useful as the completed plans affects thinking, start asking what if much more

28 Next Steps Finish what we started with infrastructure build out WAN/network connec6vity upgrade Resilient/Redundant Network Redesigned Backups and Storage Systems for Geo diversity Deployment to hot site Refresh of DR Plans to occur annually Incorporate applica6on level into planning Improve linkages with University Emergency Planning Provide more basis for Business Con6nuity Plans

29 Ques6ons