Developing A Disaster Recovery Plan The Basics

Transcription

1 One of the most challenging management projects is the development of an enterprise-wide disaster recovery plan. Such a plan involves issues as wide ranging as systems and information technology replication or back-up, to relocation logistics, and even to the availability in such circumstances of critical personnel. There is a significant challenge in many cases in simply getting managers to respond to the need to contribute time and energy to such a project. Unless your business is located in a particularly risk-prone area, the Gulf Coast or California, for instance, many managers will simply put such a project on the lower end of the priority list. This can be a fatal mistake as statistics show that a very significant percentage of businesses fail to reopen after a major disaster. In this article we will describe the basic of disaster plan development, the Wortham disaster recovery plan development, and our successful experience in dealing with Hurricane Ike which closed our headquarters offices for several weeks. Also, very importantly, we will outline insurance coverages that can significantly mitigate the cost of such a disaster. At Wortham we know the potential for disaster. Not only have we seen clients have to deal with major disasters, fires and the like, our headquarters is in Houston, with Gulf of Mexico hurricane exposure. Also, we have seen our offices in Dallas and Fort Worth have to deal with short term closures due to snow, ice and power outages, and our Fort Worth office was decimated by the Fort Worth tornado in We know it can happen to anyone!

2 Developing A Disaster Recovery Plan The Basics The development of a disaster recovery plan can be a very complex undertaking but the basics of such a plan can be described as follows: Determine how to most effectively duplicate/restore any indispensable activities of your company diversify locations, have back-up facilities available, etc. Back up IT systems and develop a plan to restore communications with third parties. Develop a plan that is employee oriented determine the availability of employees under all business interruption scenarios you can imagine, and how/where/when these human resources will be best utilized. Develop a communications plan for employees. Determine how you will handle financial issues access cash, fund payroll and other expenses, maintain cash flow. Brainstorm all scenarios you can practically imagine and make sure your internal communication plan and the restoration of indispensable activities can respond in all cases. Test the plan as completely and realistically as possible to learn about any problems/shortcomings, and to be well practiced in the implementation. The business recovery plan usually has two parts: 1. A technical plan for the restoration of business activities, IT systems, and communication with third parties, and 2. The people plan, which is essentially a common sense plan for dealing with the personal issues employees may have to deal with as well as the work issues. An outline at the end of this article also lists these issues and is a convenient, simple checklist. Development of the Wortham Disaster Recovery Plan and the Hurricane Ike Experience Living and working along the Texas Gulf Coast makes everyone aware of the potential dangers of hurricanes and tropical storms along with the need to take safety precautions during hurricane season. At Wortham, those preparations take on a significant added dimension. We always knew that if a major catastrophe hit Houston, we would immediately receive calls from clients. In many cases, these calls would be among the first our clients, who comprise a significant percentage of businesses and individuals in Houston, would make after a disaster.

3 The Wortham business recovery plan as a whole has been a tremendous success, and clients in Houston and beyond have been the beneficiaries, said Bob Hixon, Chairman. Thanks to all who participated in the recovery planning for such a disaster. Even with many employees dealing with issues concerning their own homes and families, they were able to respond and take care of business here at the firm. I have said it before and I ll say it again, everyone at Wortham is the Greatest. Thanks to all of you. At Wortham, we take business recovery planning very seriously with special emphasis on making our personnel, particularly claims personnel and client account executives, accessible after a catastrophic event. The events of 9/11 also made it clear to Wortham that digitized information on insurance policies and other important data, combined with ready access, is crucial. Wortham developed a business continuity plan that includes a hardware configuration in our Austin office which completely backs up our Houston headquarters computer systems and on a real-time basis. Under our plan, in case our headquarters were damaged or inaccessible, we could restore operations using a fully-equipped business continuity mobile unit at our Houston facility or another safe location. In addition, we made plans for employees to evacuate to Austin, San Antonio, or the Dallas-Fort Worth area to use our offices and resume business activities. Preparing for Ike Like many Houston firms, Wortham began business continuity implementation when Hurricane Ike appeared to be headed towards the upper Texas coast. By Thursday, September 11, 2008, we secured our headquarters in the Wortham Tower, just west of downtown Houston, placing paperwork in boxes in interior hallways and moving priceless artifacts including portraits of founders Gus S. Wortham and his father John L. Wortham, a painting of downtown Houston in the early 20th century, and an original desk from the old Lloyd's building in London, in safe locations.

4 Hurricane Ike struck the Houston- Galveston area as a Category 2 storm on Friday night, September 12. The next morning, the Wortham business recovery team members traveled the debris-strewn roads of Houston to reach the Wortham Tower. Upon arriving, they discovered our top six floors had been buffeted by tremendous winds and hard rain, breaking several windows. The building s upper two floors were the most severely damaged as interior walls caved in on desks, ceilings had collapsed, and ceiling tiles and wiring were scattered. Water damaged computer and electrical systems, filing cabinets and carpets. The top two floors were a total loss. The Mobile Headquarters On Sunday, the team launched a major component of our business continuity plan, setting up a high-tech mobile unit that would serve as the firm's headquarters. The unit from Rentsys Recovery Services, an IT equipment rental and business continuity firm in College Station, Texas, was prepared over the weekend. By Sunday evening, the unit arrived in the Wortham Tower parking lot. The mobile unit features 46 workstations with PCs and telephones, and ancillary office equipment such as printers and a fax machine. Fully powered with diesel generators and a 10-ton air conditioner, the unit was a very welcome relief to employees most of whom did not have power at their homes. The unit s satellite dish and ample bandwidth enabled quick connections through a satellite uplink with the backed-up Wortham IT systems in our Austin office. Through a pre-arranged disaster recovery system, our Houston office phone system was forwarded to the mobile unit.

5 Wortham s periodic disaster plan rehearsals - the mobile unit and satellite link were fully tested by the Wortham IT staff before hurricane season - and a phone tree communication system for employees, paid off. The end result: By mid-morning on Monday, September 15, the first business day after the hurricane, Wortham s headquarters team was back up and running. Working from the mobile unit, phones calls were answered and claims were submitted. Business activities not related to Ike, such as handling a marine claim from India and ensuring payment of a multi-million dollar London premium, also were conducted from the mobile unit. Conference Room With Sunscreen "We were basically running our full operation," said Charles Flournoy, Vice Chairman. To accommodate the large number of Wortham employees working at the mobile unit, we set up additional chairs, tables and a canopy outside for the partners. It was the first conference room I've been in where you needed sunscreen." Jim Gleaves, Managing Director, pointed out that most of the firm s employees faced the same hardships no power at home and concerns about gasoline, ice and food as experienced across Houston. Despite these problems, our employees made themselves available for the company and our clients, Gleaves said. Our internal communications worked well. Wortham departments had implemented phone tree communications processes, and employees and clients could also be updated on developments through the Wortham website home page. Remediation Efforts Get Underway Over the next several days, Wortham Vice Chairman of Sales Richard Blades and Managing Directors Chad Dodd and Lee Doggett directed efforts to salvage contents of the two most heavily damaged floors, and began remediation efforts on other floors, which suffered water damage. While floors 23 and 24 were a total loss, the remaining floors had carpet and lower sections of sheetrock removed and were then habitable. By Friday, September 26, most occupants of the four less damaged floors were back in their offices.

6 On the ground floor of Wortham Tower, Wortham leased vacant space to temporarily house employees from floors 23 and 24. Thanks to herculean efforts by the Wortham IT staff, Mr. Dodd and Mr. Doggett, this space was furnished and equipped with PCs and telephone system by early October, allowing for release of the mobile unit. Insurance Considerations in Disaster Recovery Planning There are very helpful insurance structures available to assist in indemnifying expenses related to disaster recovery costs. Through virtually all commercial property insurance policies the insured can also acquire coverage for the indemnification for income related to the business interruption and for the extra expenses related to issues after a disaster. Business interruption insurance can provide coverage for lost income for a period of time after a disaster. Since this coverage is part of the property insurance policy there typically must be property damage for the business interruption coverage to incept. Also there is a form of coverage known as contingent business interruption coverage that provides indemnification for the loss of income when another entity, generally a supplier or other business that the insured is dependent upon, has a property loss that interrupts their ability to provide the necessary goods or services. Extra expense coverage is also typically triggered by property damage of the insured, and generally covers certain predefined or estimated costs incurred by the insured after the disaster and property loss. Extra expense coverage can be very helpful in covering costs associated with the disaster recovery, for instance, the potential costs of relocation to a disaster recovery center, the cost of a mobile recovery unit, and other costs not incurred in the normal course of business but related to the recovery of the business.

7 Outline - Business Continuity Plan Issues I. Purpose of the Plan Determine and prioritize the primary objectives of the plan. How does the plan fit your business model? II. Communication Internal Communication Use of phone trees, employee hotline, 800 number, website, twitter.com, public media, emergency number cards External Communication Hotline, 800 number, website, twitter.com, blast fax or Communication to recovery site III. Data Management and Back-Up Information management systems Back-up processes IV. Recovery Sites Fixed locations Mobile recovery units V. Specific business Issues Personnel involved Staffing the team Cash flow and banking Lockbox, cash sweep account, line of credit, check stock Payroll Third party service, direct deposit Insurance Business interruption, extra expense VI. Testing the Plan