WALDO CANYON FIRE Is Your Business Ready for a Disaster? Dan Martinez VP, Information Technology
Intro / Context Data Center is in PRCA HQ Building Critical Systems Include: AS/400 Legacy Rodeo Applications & Data Member Portal -.NET Based Self-Service Application JDE Financials Package Avaya IP Office Phone System w/call Center Setup Managed Services Provider (engagement 1 year old) DR Efforts never survive the budget process Failing Cooling System 2
Saturday, June 23 - Noon 3
Conditions Weather Report High of 99 F Humidity of 6% South Winds @ 15 MPH PRCA Activity Company Picnic Office Open / Call Center Only Evacuation Orders 4
Business Continuity Actions Discussion Only Initial conversations between myself, COO and Commissioner Agreed to watch and wait 5
Garden of the Gods 6
Sunday, June 24 th 7
Business Continuity Actions Discussion Only Continued conversations between myself, COO and Commissioner Fire moving north, not threatening the city Evacuation Orders Agreed to watch and wait Board Meeting Started at PRCA Offices 8
Monday, June 25th Fire continued to spread north. No new alerts / concerns to city. 9
First Footprint Reports 10
Business Continuity Actions Board Meeting Continued at PRCA Offices IT Actions Prepare a pull and run list Items that we wanted to grab in case we got evacuated Put vendors & key partners on notice Continue to watch and wait Evacuation Orders Cooling System Tripped Alerts from Data Center of Rising Temps Water Turned On 11
Tuesday, June 26th Forecast calls for 4 th straight day of near 100 degree temps Increased chance of Thunderstorms Brisk and variable winds. Evacuated areas opened for limited access. 12
Business Continuity Actions IT Actions Finalize our pull and run list Items that we wanted to grab in case we got evacuated Review with vendors & key partners Continue to watch and wait Idea of having to evacuate had faded. Fire expanded significantly, mostly to the North then 13
Fire Exploded to the East Thunderstorm 40 miles north comes through midafternoon, winds shift now blowing to the east at 60+ mph Coincidentally, the flames exploded as officials in charge of fighting the fire held an afternoon news conference to discuss their tactics and their concerns over the fire s rapid spread north around the Queen s Canyon area. The briefing was interrupted with a stark, horrifying announcement: Everyone in Mountain Shadows and Peregrine needed to evacuate immediately. 14
15
Tuesday, June 26 th 4:30 pm News of evacuations spreads PRCA still not in evacuation zone All staff instructed to bring laptops home IT begins working pull and run list Critical backups kicked off Key Vendors notified Monitor Police & Fire bands Can t save that structure, move to the next one. 16
5:10 PM -- ½ mile north of PRCA offices Looking to the West 17
Tuesday, June 26 th 6:00 pm News of multiple, countless structures on fire. No indication of evacuation order expanding further east Backups running We had to kick people off the system and out of the building to complete our data collection processes. 18
6:10 PM -- Parking Lot of PRCA offices Looking to the West 19
Tuesday, June 26 th 6:30 pm Smoke is becoming very thick near the offices Health advisories repeated across local media outlets IT Leadership Meeting We grab what we can, once the evacuation order comes, we leave with whatever we have. No questions. Call from Commissioner & COO Don t be heroes. 20
Tuesday, June 26 th 7:00 pm Evacuation Area Expands Now Includes PRCA Offices Backups not complete No server hardware pulled yet We have software, older backup tapes, manuals and our prayers. 21
Tuesday, June 26 th Evening Managed Services Partner Start pulling data from selected servers via dedicated VPN tunnel Task force assembled to make changes as needed PRCA Leadership Team Setup Command Post at Commissioner s house on Wednesday AM. 22
23
24
25
26
Unprecedented Behavior This is a firestorm of epic proportions. Colorado Springs Fire Chief Rich Brown It exploded far beyond what could have been predicted. Colorado Springs Mayor Steve Bach The fire breached primary and secondary containment lines. Incident Commander Rich Harvey 27
28
Wednesday, June 27 th Morning Local News reporting fire s reach No Word on Lifting Evacuation Zones Command Post Established at Commissioner s House Team Focus: How to manage a few days away from the office? Communication: Staff / Members / Fans All deadlines for 7/27 pushed by 24 hours IT Activities: Keep systems running, keep pulling backups Monitor Data Center Temperature Design Setup of Remote Operating Location System Access Phone Re-routing 29
Wednesday, June 27 th Afternoon July 4 th Rodeos / Cowboy Christmas Biggest week in Rodeo outside of NFR WPRA (Women s Pro Rodeo Association) Offers small Conference Room Laptops Identified & IT Support Lined Up VPN Access No Call Center on 7/28 Deadlines pushed another 24 hours Impact to our Cowboys & Rodeo Committees 30
Thursday, July 28 th Setup Remote Work Area at WPRA Offices Word of a potential call center at Gazette building Coordinating with Newspaper on the fly? Groundwork for Setting up Remote Call Center Local Hotel Conf Rooms Cell Phones + Land Lines Laptops & VPN Access Overnight Setup for Friday Operation 31
Temporary Access & Reprieve Can access building (with Police escort) on Thursday afternoon Pull Backup Device Left Over Laptops Cell Phone Inventory News comes after 4 PM Daily Briefing Evacuation Lifted! 32
Impact 346 Homes Lost 2 People Died No Businesses Lost! Lost Tourism Down est. 15% from Summer 2011 18,247 Acres Burned Cost to Fight: over $17 Million Cause: Still Under Investigation 33
We Dodged a Bullet Data Center Cooling never tripped Water was inadvertently left on by Facilities Power was never lost We have no generator on site SMB Outfit with Limited IT Budgets Board not supportive of additional expenses (historically) Business Partners 34
So Now What? Business Continuity Project Initiated Defining Cost of Being Out of Business Establishing a Recovery Point Objective (RPO) and Recovery Time Objective (RTO) Outline of Plan to be Prepared before 2013 Budget Submission Includes upgrade to Cooling System At a minimum, a written plan will be in place. Testing of Phone Redirection Planned 35
Other Considerations Limited IT Budgets Champagne taste on a beer budget Lack of Business Buy In Lack of Resources Info-Tech Research 28% of Organizations don t have a DR Plan in Place. 36
Are You Ready? Is your DR Plan Current? Written Down? Easily Found? RPO & RTO Established? Last reviewed? Key Staff (not just IT) identified? Communication Plan? Discussions with Key Vendors? Test your DR Plan Walkthrough Simulation / Parallel Full Testing 37
Thank You Q& A 38
Photo Credits Slides 4, 7 Ethan Beute - http://www.flickr.com/photos/ethanbeute/ Slide 6: Tom Mullenaux - http://www.flickr.com/photos/tmullenaux/ Slides 9, 12, 19, 23 & 25 Dan Martinez - http://www.flickr.com/photos/dan_martinez/ Slide 15 Russ Jackson - http://www.flickr.com/photos/bluehiker/ Slide 24: Clotaire Domy - http://clotairedamy.com Slide 26: Jake Stewart - http://www.jakestewartphoto.com 39