Pre-Disaster Recovery Plans

Transcription

1 Pre-Disaster Recovery Plans Presentation to: NACRC Legislative Conference March 3, 2007 Carol Foglesong, Orange County Comptroller s Office, Orlando, FL and Paul Ketz, Broward County Records Division, Ft. Lauderdale, FL

2 Agenda Introductions Purpose Unpredictability of Disasters Business Continuity Planning Definitions Reasons for a Pre-Need Contract What do you need to consider? Orange County s Process to Contract Other Resources Questions

3 Purpose We are not here because we are experts in the field of Business Continuity/Disaster Planning. We are here because all of us are struggling with this topic. We would like to share some of our experiences with you, and learn from the experiences of others here today.

4 Cycle of Disaster Planning RISK REDUCTION PREVENTION MITIGATION PREPAREDNESS RECOVERY RESPONSE

5 So, what s a disaster? Disasters come in all sizes and shapes Big events: Hurricanes, Tornadoes, Earthquakes, Blizzards, Floods, Vandalism, Terrorism Smaller events: Fires, Water Pipes, Roof Leaks Disasters impact A large geographic area A localized area (county or city) A particular office operation/location

6 Hurricane Wilma Exposed office suites are seen in the damaged Regents Bank Plaza on Tuesday, October 25, 2005 in Fort Lauderdale. (Sun- Sentinel/Joe Cavaretta)

7 Hurricane Wilma The Broward County School Board building as seen from the 30th floor of the AutoNation Building in downtown Fort Lauderdale. (Brian Hekman, Coconut Creek)

8 Hurricane Wilma Virtually every window on the west side of the Broward School Board building in downtown Fort Lauderdale was blown out by Hurricane Wilma. (Sun-Sentinel /Carl Seibert)

9 Hurricane Wilma Repairs are underway at the Broward County Courthouse offices and parking garage in downtown Fort Lauderdale. (Sun- Sentinel.com)

10 What s a common theme? Disasters are typically unexpected Disasters happen when you have or are making other plans Disaster size or impact can t be readily predicted Disasters aren t all BIG events; it can be the little ones that cause the most concern and frustration

11 Disaster Readiness Business Continuity Plan Staffing plans How to get up and running again Disaster Recovery Plan Setting priorities on what has to be tackled first Papers versus computers Forecast Disaster Plan Before the event planning Office preparations when a disaster is predicted

12 But who do you call for help? You ve made a list Businesses with addresses, names, phone numbers, faxes, and addresses Services offered and needed You ve identified responsibilities of staff Who calls whom when Who s in charge Do you and your staff keep a copy of your plan/phone lists with you? It won t help if your list is at the office!

13 How to guarantee availability? If a large area is impacted, there will be many organizations needing help; you may need to get in line If a small area is impacted, will you get the specific help you need quickly? If it s a particular building or location impacted, what s the best solution?

14 One answer A PRE-NEED CONTRACT A commitment to your organization by a service provider You have a spot on the top of their list You ve pre-negotiated prices and services You ve already checked out who the provider is and their references

15 Getting the Pre-Need Contract You will have to convince your purchasing/contract group that a preneed contract is not only possible, but logical You knowingly commit to going through the purchasing process of your organization even with no known disaster on the horizon

16 Cost Be clear: no services = no dollars But money has to be available when services are, in fact, necessary Does your organization require a set aside? Does your organization have a reserve fund? Contract has to be renewed Annually? Every 3 years? What are your renewal options?

17 Who s in charge of the procurement process? You may be the designated Records Management Officer for your organization, but you will need: The purchasing/contract group Likely users must buy in Agreement on who sets the priorities for service Recognition that the disaster may be small or large scale Approval by your governing body Who will negotiate the contract? Do they have a solid understanding of your business needs to secure the best vendor?

18 Other Considerations Can you opt for higher priority by negotiating a retainer? Will the retainer be applied against services rendered? If something happened to your materials, how long would it take to access them? Do you have 24/7 access? How do you get someone to the facility? Is HIPAA a consideration in the custody/restoration of your records?

19 Response Time Do you have a Continuity of Operations Plan (COOP)? What kind of time will it take to respond to an event? For Emergency Services Fire, Plumber, Repairmen For you and your staff For the Paper-Recovery Vendor

20 What events are important based on your geography? Where are you located and what kinds of events do you expect every year? Florida Hurricanes California Earthquakes Northeast Snow Low lying Flooding What is it for you???

21 Vendor Capabilities How many linear feet of paper can be recovered per hour per piece of equipment? Is the Vendor s equipment wholly owned by company, or is some subcontracted? Does the Vendor have a schedule of who will respond after they are first notified? Process for chain of custody of records Where will the vendor set up base camp? What kind of turn-around time

22 Vendor Experience How much experience will you require? What kinds of references do you want to see? Past Performance Program Name Owner / Architect Contract Amount / Percent Complete Completion Date

23 Be clear about the scope What media do you want used to recreate your records? Film Electronic Back to paper?!? Who can activate the contract Do you have other agencies/business units who need access to this contract? If multiple agencies/business units can access this contract, have you told the vendor who has priority? Whose budget is used to pay for services?

24 Writing the RFP Plagiarize! Don t reinvent the wheel unnecessarily Each organization has their own standard language and format Stick with the assistance particulars that you might actually need Have a way to compare apples and oranges between vendors/proposers

25 Computers vs Papers? Typically, there s already a recovery plan for computers, servers, databases, etc. Communication is key talk to your IT people and be sure they have a plan too! Computer recovery involves not only the data, but also the equipment (machines, hardware) There are specialized companies who handle this specialized service Make sure your organization has an active and realistic plan in place

26 Paper or Paper Plus? Paper in a warehouse, awaiting its retention time or its microfilming Paper in all those office filing cabinets and individual desk drawers Microfilm? Microfiche? CDs? Maps or big drawings? Other media besides computers?

27 Which documents first? Are your documents prioritized? Do you store records for multiple agencies? What gets saved first? Are your boxes segregated? All things being equal, whose records are most important? Does it make sense to secure your truly irreplaceable records elsewhere?

28 What type of paper recovery process will be used? There are multiple types of document recovery/restoration techniques Vacuum Freeze Drying Molecular Sieve treatment Desiccant dehumidification HVAC decontamination Gamma Radiation (Remove Mold/Bacteria) Are there techniques that cannot/should not be used on your records?

29 Paper Contamination Factors Debris Sewage Silt Haz-Mat materials Asbestos PCB Other petroleum products

30 Extras Re-filing, Re-indexing, Re-shelving Daily equipment rentals Small equipment purchases (shovels) Will they provide their own generators, or does the vendor require electricity (water)? Freight / Transportation Costs Travel / Per Diem or subsistence costs General Debris removal at your site to gain building access Warranty on services performed/document reproduction quality

31 Who does what? Inventory Packing Sorting/Identifying Prioritizing Labeling Loading/Unloading Re-shelving Security

32 Price per cubic foot Pack out Gamma Radiation Cleaning Drying Ozone Deodorization

33 Price per each item Audio tapes Blueprints Maps Compact disks Diskettes Manuscripts Laser disks Photographs Slides Microfiche Microfilm Negatives X-rays Pack out boxes Pack in boxes

34 Price per hour Will any of the vendor s services be provided per hour in consultation fashion? Does the vendor have per hour charges for anything not specifically covered in your contract?

35 Where will the actual restoration and recovery take place? Where will the vendor set up base camp? Onsite Remote Location Vendor s Site How will the materials be transported? What kind of turn-around time does the vendor give for recovery onsite? Offsite?

36 Before you go back in Who declares whether the building is safe to re-enter? Fire Marshal Building Inspector Facilities Management Some other authority? Where are your emergency triage supplies? What will you provide? What will the vendor provide? Where will the vendor set up base camp?

37 Before you go back in (cont d) Will you need assistance to secure the structure itself? Water removal Temporary Power Temporary building repairs Securing Walls Roof Repairs What is your plan if your staff balk at going back in?

38 One solution Set up scenarios and make pricing those responses a part of the evaluation process Pick 3 events of varying significance Mix up the media to be recovered

39 Orange County FL Sample Full RFP and evaluation grading sheets being lifted to NACRC document library Understand that we decided our biggest problem was going to be water damage We don t get snow or many tornadoes Substitute your own boiler plate front end

40 Scope of Work Location for services (Orange County FL; mostly mid-orange County) Volume of work undetermined (who knows what s going to happen?) Not for computer systems What we asked for There shall be no retainer paid in order to keep the contract in effect. The resulting contract will be on an as-needed basis and used only in case of a disaster or emergency as so deemed by the County. The Contractor shall provide 24 hours per day, 7 days per week emergency response service. Response by phone after the first notification shall be within 2 hours. The Contractor shall have a representative on site after first notification within six (6) hours with the necessary equipment to be transferred within twelve (12) hours.

41 Selection Criteria and Points CRITERIA Qualifications of Staff Qualifications of Firm Technical Approach M/WBE Utilization Location Fee Proposal TOTAL Welfare Recipient Hires WEIGHT (minimum allowed) 10 (minimum allowed) bonus points

42 Qualifications of Staff Include a listing of all staff to be assigned to provide the required services and resumes for each describing experience, training and education in the required consulting services. Reasoning and/or recommended changes Tell them to make the bios BRIEF You don t need or want 5 pages per person! Clarify the kind of training or certification staff has WHEN did they get that training or certification? Identify staff experience working with governmental entities and list those projects. Reasoning and/or recommended changes Hold the government entities and projects for the firm s qualifications You just get the same information in two places

43 Qualifications of Firm #1 Provide a description and history of the firm focusing on previous governmental experience. Provide the number of years in business showing proof of a minimum of 5 years in document recovery, the current number of full-time employees and the excess labor that can be provided in case of an emergency. Reasoning and/or recommended changes Does governmental experience actually matter? Five years in business seemed a logical threshold so that we weren t getting a firm that sprung up in response to our 2004 hurricanes Full-time employees seemed important to evaluate whether the firm could respond to more than one disaster Excess labor? Probably would just hire locally and teach/train while on the job

44 Qualifications of Firm #2 Identify your equipment inventory available to Orange County within the Southeastern United States. Identify the capacity to process in linear feet per hour for each piece of equipment at each location. Identify equipment wholly owned by your firm, or which is available through subcontracting. Reasoning and/or recommended changes Making up the equipment list was VERY interesting We decided on Southeastern US as a logical staging area for us We learned you can lease/rent anything!

45 Qualifications of Firm #3 List at least five references, with a minimum of two from governmental entity experience, for which the firm has performed similar work including the contact name, address, telephone number, , and date of the contract. Reasoning and/or recommended changes 5 references seems about the standard 2 government entities seemed to make purchasing people happier Make sure to ask for so you can readily contact the references Send the reference organizations questions and then call to interview them

46 Technical Approach #1 Provide a brief description of the firm s approach to the project. Identify the equipment and recovery methodologies you employ to recover water-damaged paper documents, microfilm, CDs, maps and plans, etc. Provide a schedule (not later than) of arrival of your equipment in hours after first notification. Identify how you ensure records are properly inventoried and identified as well as the chain of custodianship.

47 Technical Approach #2 Identify the methods and instruments used to assess moisture content of records. Provide an operational plan for rapid deployment of required supervisors to provide direction to County employee first responders to assess damage and take interim steps to minimize loss. Include number of people, and number of hours for arrival. Should no power be available, contractor shall be responsible for providing all power required to operate equipment.

48 Technical Approach #3 Provide a plan for how your clients are prioritized in the event of a widespread emergency in which there is multiple-client demand. Identify your permanent dehumidifying equipment locations in the Southeastern United States and the capacity in pounds of water extracted per hour for each piece of equipment at each location. Present options available to provide dehumidifying of buildings and mold and mildew treatment.

49 Technical Approach #4 Submit your warranty or guarantee of your services. Confirm the firm s agreement to meet the minimum requirements of this Request for Proposal. Proposers may offer alternative solutions/options to achieve successful completion of the scope of work herein.

50 Technical Approach Notes This area provided us with the greatest flexibility in evaluating proposals and the largest points to award/withhold. We don t know if we got this area right as we haven t had to use the contract. But we did get it in writing in advance from the firm with which we contracted. We drew upon notes from prior NACRC and disaster recovery sessions. We tried to imagine many/most scenarios.

51 Fee Schedule #1 Submit a rate schedule for equipment, supplies, and staff services. Travel, lodging, and meals shall be at rates allowed to public employees under state law per state statute Identify the cost per cubic foot, separately, for each of the elements as listed in item below. Provide a cost for all recovery services. freeze conventionally blast freeze freeze dry rapid freeze treat for mold and mildew packing and unpacking treatment for soot

52 Fee Schedule #2 Identify mobilization and demobilization rates for equipment and staff. Identify your markup for any equipment rented, and supplies purchased. Identify freight cost per mile by type of equipment. Provide the cost to implement the recovery process. Provide the cost of the Recovery Service. Describe, if offered, your pre-registration service and cost associated for such service.

53 Fee Schedule #3 In order to adequately evaluate and compare services provided by proposers, three (3) hypothetical scenarios have been created for which proposers must provide a written response using prices submitted on Attachment A, the scope of work provided, and the technical approach of your firm. These scenarios are located in Attachment B.

54 Scenario #1: Roof Leak It has just been discovered that the roof of the County s Records Center has been leaking for some time now. It is estimated from the buckling ceiling tiles that this has occurred over the past six (6) months. When the actual damage was assessed, 200 boxes (dimensions: 10x12x15 = 1.2 cubic feet) containing paper documents and microfilm had been severely damaged by water, resulting in mold and mildew. Contents of the boxes are as follows: 7/8 copier and bond paper and file folders, and 1/8 microfilm. Based on the limited information in this scenario, the scope of work, and the services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the 240 cubic feet of documents. It is understood that the type or method of restoration will vary depending upon the type of records involved.

55 Scenario #2: Water Pipe Burst A water pipe has just burst in a County building, which houses permits and application forms of the County. Standing water, approximately 8 deep has seeped into the records (i.e., CDs, microfilm, paper, maps, and plans, etc.). The damaged records consist of 1/2 copier and bond paper and file folders, 1/8 CDs, 1/8 microfilm, and 1/4 building maps and plans. The room size for this County office is approximately 20,000 sq. ft. Housed in this office space are records contained in 10 four-drawer vertical letter-size filing cabinets (@ 6.0 cubic feet per unit for approximately 60.0 cubic feet total); 10 five-drawer lateral, lettersize filing cabinets (@ 13.0 cubic feet per unit) for approximately cubic feet of records.

56 Scenario #2 (continued) This space also contains approximately 50 workstation cubicles at 100 sq. ft. each, which is equivalent to an estimated 4 to 5 cubic feet of records housed at each workstation. The 50 workstations contain approximately 250 cubic feet of water-damaged records. Based on the limited information in this scenario, scope of work, and services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the records. It is understood that the type or method of restoration will vary depending upon the type of records involved.

57 Scenario #3: Small Fire A coffee pot with a potential electrical shortage problem was left plugged into an outlet too long and sparked a small fire. Smoke from the fire triggered the sprinkler system, which caused water damage to some of the records. When the actual damage was assessed, 300 boxes (dimensions: 10x12x15 = 1.2 cubic feet) containing paper documents, microfilm, CDs, and maps had been moderately damaged by both water from the sprinklers and soot from the fire. The damaged records consist of ¾ copier and bond paper and file folders, 1/8 CDs, and 1/8 microfilm. Based on the limited information in this scenario, scope of work, and services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the records. It is understood that the type or method of restoration will vary depending upon the type of records involved.

58 Fee Schedule Notes Having the scenarios was very important We had trouble figuring out how to evaluate all the standard information the proposers submitted The scenarios gave us a window into comparing apples to apples side-by-side

59 Checking References You really have to do this part We designed a series of 10 questions (lifted to the NACRC document library) which we sent to the reference people in advance of our call to them We had at least 2 people and usually 3 people on each call and then we compared notes

60 Doing the Evaluation We created spreadsheets and allocated points among the criteria for each category (lifted to the NACRC Document Library) Each of the evaluators completed their own spreadsheet separately We then met together and compared scores; we were very close to each other What one person couldn t find or figure out, another person did

61 The Contract We selected our finalist just as Hurricanes Katrina and Rita plowed in Getting our contract fully executed dragged out for 3 months and we only needed their properly executed insurance form! Since the contract was finally and fully signed in January 2006, we have not had to use the contract But we re sure sleeping better!

62 Additional Thoughts Start now; don t wait for your season Anticipate delays along the way From first concept to actual completion was 2+ years for us Writing the scenarios and considering all the possibilities requires multiple brains and various styles

63 Lessons Learned from Katrina Having a pre-need contract crosses the T s and dots the I s in advance. Remember to look at the whole process New Orleans Parish experience with their vendor, The work that they did was excellent but the collection department was awful. They relied on a technical line in the work authorization to keep our books in Chicago until they were paid eliminating the opportunity for an audit of their bill. I suggest that any pre arrangements include reviewing the billing and collection methods.

64 Use your NACRC Contacts Use NACRC as a resource to find vendors Talk to other jurisdictions about their experiences Conferences Listserv Conference Calls

65 Questions & Answers What experiences have you had? Good or bad? When will you have a pre-need contract? How will you develop your potential proposer list to which to send the RFP? Can you join with other organizations in your area? Can you tag along to an existing contract someone else has?

66 Additional Resources NACRC Florida DOS State Library & Archives ry/#hvac National Parks Service Wet Collections Recovery wet-recovery.htm#

67 Thanks and for any follow-up Carol Foglesong, Assistant Comptroller, Orange County Comptroller s Office, PO Box 38, Orlando, FL 32802; ; carol.foglesong@occompt.com Paul Ketz, Broward County Records Division, 115 S. Andrews Ave., Room 120, Ft. Lauderdale, FL 33301; ; pketz@broward.org