New Clerk Academy August 13, 2015
Disaster Recovery OVERVIEW
Presentation Agenda Introduction and Definitions DR Motivators and Drivers Recovery Challenges Scope of Disasters Components of Recovery Plans Business Continuity Planning Process FACCSG Data Vaulting Solution 3
What is a disaster? Definitions: The interruption of business due to the loss or denial of the information assets required for normal operations. A sudden, unexpected emergency requiring immediate action. An event that renders the system inoperable. A loss or interruption to business functions. A failure within the infrastructure which interrupts business resulting in unplanned downtime. 4
DR Motivators and Drivers Audit requirements Government mandates Insurance requirements Business impact Legal liabilities Protection of business assets Protection of business relationships Impact to customers Recent disasters Increased regulation from external sources 5
Recovery Challenges Why do planning efforts fail? Degree of dependence not understood Potential impact not recognized Lack of management commitment The won t happen here syndrome DR is never a priority 6
Recovery Challenges (cont d) Shrinking recovery windows Seamless or transparent recoveries Growth of E-commerce Internet dependencies Too big to recover syndrome Regional and global recoveries 7
Why Move Forward? Potential Impact Tangible costs: Lost revenue Lost productivity Legal implications and fines Lost wages Intangible costs: Political image Reputation Credibility Employee morale 8
Planning Questions What types of interruptions could we have? Which of our business functions are critical? How are we defining critical? How would a disaster affect the quality of our service? If you had 15 minutes to vacate your office, do you know what you would take with you? 9
Planning Questions (cont d) How long could your organization survive in the event of a disaster? Are recovery tasks adequately organized and assigned? What happens if there s a disaster at 2:00 pm on a holiday weekend? 10
Types of Disasters blizzard bomb threat chemical spill civil disturbance contractual failure dam collapse earthquake epidemic hardware failure explosion fire flooding freezing heat high winds hostage taking hurricane network failure nuclear war power failure riot sabotage strike telecommunications failure terrorism theft tidal wave tornado volcano water damage 11
Scope Layers of Protection Region/Area Building/Site Data-Center Network System Application database Data Emergency Operations Center (EOC) Crisis management Internal hot-site Commercial hot-site Remote fail-over Work area recovery Redundant networks Reroutable networks Network backups Database backups Database logging Component redundancies Local fail-over systems Data back-ups Off-site media storage Disc mirroring (remote) Electronic vaulting(remote) 12
Components of an IT Recovery Plan System Recovery Database Recovery Application Recovery Data Recovery Network Recovery Telecommunications Recovery LAN Recovery User Work Area Recovery 13
Components of a Business Continuity or Emergency Response Plan Employee Safety and Awareness Emergency Response Crisis Management Site or Facilities Recovery Relocation of business units Vendor-supplier recovery Public Relations/Media Response IT Recovery is a component of the BCP 14
Disaster Recovery Plan Definitions A predefined, tested set of procedures to implement in the event of a disaster. Documented procedures which outline the who, why, what, and how to plan and recover from a disaster. Purpose is to increase the chances of survival and to decrease the amount of loss. 15
A Disaster Recovery Plan Does provide: Blueprint for survival of the business after a disaster Does not: Duplicate a normal business environment Provide business-as-usual, in survival mode 16
Business Continuity Planning Process Management Sponsorship Scope and Objectives Risk Management and Disaster Avoidance Recovery Requirements Design and Development of the Plan Rehearsal, Maintenance, and Reviews 17
Management Sponsorship Strategies for selling senior management Identify risks and vulnerabilities Describe the benefits of having a plan in place Identify impact in not having a plan Substantiate downtime in dollars Clarify insurance coverage boundaries Increase awareness 18
Scope and Objectives Defining scope What are you attempting to protect? Business continuity versus IT recovery Define scope of plan Defining objectives Define how much you are willing to lose Define disaster scenarios Classify disasters: minor, major, catastrophic Define acceptable, tolerable downtime Define project management plan Define project team members Define methodology to be used 19
Disaster Recovery Project Team Structure - Sample Outside Authorities Business Continuity Manager Senior Management Business Continuity Recovery Coordinator System Recovery Team Communication Facilities Data Control Administration User-Liaison Operating System Database Data Communications Voice Communications LAN Site/ Building Security Clean-up / Restoration Tape Librarian Data Entry Personnel, Insurance, Public Relations, Legal Petty Cash, Legal Purchasing, Audit Users Application Support Application Development 20
DR Methodology - Sample BACKUP CRITICAL APPLICATIONS ALTERNATE PROCESSING RECOVERY PROJECT MANAGEMENT MANAGEMENT PROCEDURES PROCEDURES DISASTER RECOVERY PLAN TESTING TRAINING MAINTENANCE EVENT DETECTION REVISIONS AND UPDATES INVENTORY 21
Risk Management and Disaster Avoidance Identify risks, threats, and exposures Assess current environment Conduct disaster avoidance review Assess cost-benefit ratio for accepting risk versus implementing controls Implement controls to mitigate risk and potential disasters 22
Recovery Requirements Define recovery needs and requirements Conduct Business Impact Analysis (BIA) define key functions develop and validate questionnaire conduct interviews identifies tolerable, maximum amount of downtime identifies potential workarounds Categorize and prioritize business functions vital, critical, important, deferrable disaster classifications: minor, major, catastrophic Conclusions from the analysis 23
Recovery Requirements (cont d) Define Recovery Time Objectives (RTOs) Define recovery strategies to support potential disasters For example: For minor disasters for important business functions, the recovery strategy is to do nothing for a period of 4 hours or less, not to exceed 24 hours. For minor disasters for critical business functions, the strategy will be to alert stand-by processing and wait for 1 hour or less. For major disasters affecting vital business functions, the strategy is to assemble the disaster management team and to invoke the recovery plan within the hour. 24
Recovery Timeline DISASTER Backup Facility Activated Full Recovery Alternate Processing Interim Processing Normal ELAPSED TIME 25
Problem Escalation Process DISASTER ALERT 2 DECLARE DISASTER INITIATE RECOVERY PLAN ALERT 1 NOTIFY: - VP, Information Systems - Mgr, Change Management - User Liaison Team Leader PROBLEM OPERATOR: - Attempt to diagnose and resolve problem REFER, AS NEEDED, TO: - Shift Supervisor - Operations Manager NOTIFY: - Dir, Computer Ctr Opns - Mgr, Computer Ctr Opns - Mgr, Info Sys Network CALL IN: - Outside resources CONTINUE: - Diagnosis and problem solving REPORT ON-SITE ASAP: - Dir, Computer Ctr Opns - Mgr, Op Supp & Control - Disaster Recovery Manager - Mgr, Computer Ctr Opns - Mgr, Info Sys Network CONTINUE: - Resolution efforts EVALUATE CRITICALITY AND DECIDE ON INITIATING RECOVERY PLAN? 15 MIN * If resolved, write incident report. * If NOT resolved within allowable time, escalate to next level. 4 HRS MAXIMUM TOLERABLE DOWNTIME (24 HRS) 26
Define Backup Alternatives or Options Define backup alternatives, options, workarounds for potential scenarios. Depending upon the type of disaster, will dictate which backup option is appropriate. Some of include: do nothing recover at time of disaster recover using manual processing, service bureau recover using stand-alone, PC-based system recover at another company owned facility recover at a commercial hot-site recover at an internal hot-site NONE OF THESE ARE MUTUALLY EXCLUSIVE combination of the above 27
Design and Develop Recovery Plan Plan to include: Scope and objectives Escalation and notification procedures Recovery scenarios Project team roles and responsibilities Recovery priorities Recovery procedures Alternate processing Notify key business units owners Notify key vendor and supplier contacts Return to Normal procedures 28
Disaster Recovery Process Flow RECOVERY TEAMS DISASTER MANAGEMENT TEAM RECOVERY PROCEDURES DISASTER MANAGEMENT PROCEDURES NOTIFICATION & CONTROL CENTER DAMAGE/IMPACT ASSESSMENT RECOVERY DIRECTIVE INTERIM PROCESSING FULL RECOVERY ALTERNATE PROCESSING EMERGENCY PROCEDURES APPLICATION TEAMS 29
Rehearsals, Maintenance and Reviews Rehearsals Practice-practice-practice Design test plans define evaluation criteria for success define areas to be rehearsed (people, procedures) Planned rehearsals versus unannounced Use variety of techniques: partial to full-scale evacuation phone lines/email table-top exercises 1-2 applications or business units full scale rehearsals 30
Rehearsals, Maintenance and Reviews (cont d) Maintenance Schedule regular updates Incorporate feedback from the rehearsals Incorporate into change control process Reviews Validate against Service Level Objectives (SLAs) Conduct pre-audit of existing recovery plans Conduct objective review 31
Civitek Data Vaulting Solution Civitek has partnered with Hayes Computer Systems and EMC to provide a data vaulting solution to Clerks Each participating Clerk will purchase or lease a local storage appliance manufactured by EMC The local device will be networked with the central vault in Alpharetta, Ga This device can be easily integrated into the existing IT environment
How does the service work? Data is replicated from Clerks local EMC appliance to a secure CAT 3 rated facility in Alpharetta, Ga Utilizes Clerk s current WAN connected to all 67 counties throughout Florida Data replication requiring more network bandwidth can utilize a secure vpn connection
34
Benefits Improve Disaster Recovery- data off site, replace tape based DR with replication Reduce backup costs Reduce backup and recovery risks Speed up your backups Simplify backup and recovery operations
Benefits Improve Disaster Recovery- data off site, replace tape based DR with replication Reduce backup costs Reduce backup and recovery risks Speed up your backups Simplify backup and recovery operations
Disaster Recovery Replicate data offsite Replace Tape based DR
Reduce Backup Costs Replace costly media Tape drives Backup application licensing costs
Reduce Backup and Recovery Risks Replace unreliable media Tapes get lost Tapes get damaged High percentage fail RTO RPO
Speed up backups Efficient technology DD Boost software EMC Networker EMC Avamar NetBackup Backup Exec Dell NetVault Dell vranger HP Data Protector
Simplify backup and recovery Scalability Less to manage Less infrastructure
Options Buy server sized for your environment and projected growth. Lease- server sized for your environment, lease to buy, replace as growth requires Cloud protect- server sized for your environment, replaced with larger server as needed, charged for amount of data stored.
Contacts David Porter -Marketing porter@flclerks.com Earl Donaldson Network Engineer edonaldson@flclerks.com Tab Bradford- TAP Administrator bradford@flclerks.com