Disaster Recovery Best Practices & Lessons Learned

Transcription

1 Disaster Recovery Best Practices & Lessons Learned Paul Sullivan, VP & General Manager Agility Recovery For Audio: (1) Listen through PC speakers, OR (2) Dial and use access code

2 President & CEO Agility Recovery MBA from McColl School of Business B.S. from Davidson College Formerly Controller for SunCom Group, VP at NationsBank (Bank of America) Formerly Vice President & General Manager at Muzak prior to joining Agility Recovery Has transformed Agility Recovery and led the growth into a nationally recognized leader in Disaster Recovery Solutions Bob Boyd Agility Recovery President & CEO

3 Disasters Happen with Little to NO Warning

4 Disasters Happen with Little to NO Warning

5 1. Communication 2. The Realities of a Disaster 3. Site Vulnerability Agenda

6 Communication Create a Crisis Communication Plan Online Message Boards Emergency Contact Lists Your Website Quick Reference Guides Inform those Outside your Organization Media Relations Communication is the Key

7 Reassure Stakeholders Ensure efficient, swift recovery Reduce or eliminate mistakes and inefficiencies Protect organizational reputation Save time and money Enhance trust & faith in the organization Communication is the Key

8 Develop a process to make sure all stakeholders (internal and external) are aware of decisions and expectations. Ensure redundancies independent of cell or terrestrial networks as much as possible 24 hour phone tree Password protected web page (centralized emergency status) E mail alert Call in recording system Text/Data Alert system Manage member and key vendor/partner communications. Prepare a media communications plan. Create a Crisis Communications Plan

9 MEMBER BEST PRACTICES Password Protected Provides Information Allows employees to provide status and location updates Virtual Meeting Place Online Message Boards

10 MEMBER LESSONS LEARNED Ensure redundancies independent of Cell or Terrestrial Networks as much as possible. Alabama Tornadoes April 2011 Entire Terrestrial Phone network disabled, including 911 Centers Forced to rely on Text Messaging, no voice communications Entire recovery efforts coordinated via text messaging For critical communications, consider satellite phones Crisis Communications Plan Redundancies

11 Google Person Finder Japan Quake/Tsunami

12 What information to include: Home Phone Alternate Mobile Personal E mail Family Contact Info Assigned Rendezvous Point (if multiple locations) Store remotely for easy access Set Up an Alert Notification Program Explain the Purpose Test Regularly Update with any changes to organization Emergency Contact List

13 Do you have remote access to your Web site? Post critical information on Home and Contact pages Provide employees, clients, customers and business partners with timely information about your company during a crisis Ensure your site has contingencies for any potential spike in traffic during emergency events Your Organization s Website

14 MEMBER BEST PRACTICES T WCCD Channel 18 Employee Wallet/Quick Reference Cards

15 Let the public know about your recovery strategy Establish Yourself / Your Company as a Thought Leader. Help Clients / Customers and Partners with Their Recovery Program Development. Inform local media about your strategy Inform those Outside Your Organization

16 Establish a team of spokespersons with succession order Establish a policy for employee interaction with media Once operations are reestablished, contact media again Establish yourself and your business as a thought leader in recovery planning Issue CONCISE media alerts and press releases Follow up with phone calls Inform Media During Disasters

17 Arrival of Government/Municipal Assistance Availability of Key Personnel Follow Your Plan Regardless Take Simple Precautions The Risk of Self Performance The Impact of Supply Chain Disaster Reality

18 Kentucky Ice Storm Example Assumption is that supplies, assistance & services happen immediately Reality is it may be 72 hours OR MORE before any assistance arrives Often, Priority is Search/Rescue before basic provisions TAKE AWAY Assume self support for up to 72 Hours before assistance arrives Arrival of Government/Municipal Assistance

19 Disaster affects work & home Devastating events could render some personnel inaccessible or unavailable Regional events could make contact/communication with key personnel difficult Cross Train / Test your plan Availability of Key Personnel

20 People aren t rational at time of crisis Decision making ability is impaired Base decisions on the pre defined timeframes and contingencies in your recovery plan. If it says to declare a disaster after 4 hours of downtime, do so even if the power company continues to tell you just 15 more minutes and we ll have you back up. Maintain your plan Follow regular review of plans, especially following major changes to your organization Maintain assets for readiness Follow Your Plan

21 The question isn t CAN you, but SHOULD you? Bank 34: The Peripheral Effect Alamogordo, New Mexico MEMBER EXAMPLE Arsonist entered the building. Force entry into 2 nd floor tenant s office. (An attorney.) Set two fires. Crime happened in ~ 6 minutes. Sprinklers were activated. Six inches of water accumulated in branch. No way to resume operations without help. The Drawbacks of Self Performance

22 MEMBER BEST PRACTICES Talk to your key vendors & suppliers about their recovery plans Kentucky Ice Storm Massive ice storm leaves entire cities without power throughout Kentucky Those with Generators were left without assistance from local fuel providers who relied on electricity to pump fuel into trucks Fuel had to be brought in by backup fuel providers located 50+ miles away due to power outage The Impact of Supply Chain

23 Historical Perspective Facility Location Site Vulnerability

24 Hazardous material spills Transportation accidents Fires Severe weather Earthquakes Hurricanes Tornadoes Terrorism Utility Outages RMS Catastrophic Risk Map Historical Perspective

25 Consider your proximity to: Flood plains Seismic faults Dams Power grids Nuclear plants Major transportation Companies that produce, store, use or transport hazardous materials Facility Location

26 MEMBER BEST PRACTICES 400 Person Call Center: 1 Ounce of Prevention Nashville, Tennessee Building located in floodplain Entire facility flooded with 5 feet of water during 2010 floods Extensive planning to alleviate risk of flooding including elevated IT equipment storage, redundant facilities, fail safe phone system, etc. HOWEVER, still needed to relocate employees, and chose to recover locally in mobile offices rather than lay off or transfer employees Know Your Location s Risk

27 Questions? Bob Boyd, President & CEO, Agility Recovery Today s session has been recorded. Links to the archived recording will be ed to all registrants automatically tomorrow afternoon. For copies of the slides presented during today s session, please scott.teel@agilityrecovery.com