Can Your Electrical Infrastructure Weather a Natural Disaster?

Transcription

1 Can Your Electrical Infrastructure Weather a Natural Disaster? May 2013/1910DB1206 By Chad Kennedy, Industry Standards Manager, Power Equipment

2 Summary 1.0 Introduction Natural Disaster Definition & Statistics Three Steps to Electrical Disaster Recovery Planning Step 1: Knowledge of the Electrical System... 5 Step 2: Develop (or Update) an Electrical Safe Work Practices Policy... 6 Step 3: Electrical Emergency Action Plan Developing an Electrical Emergency Action Plan Conclusion... 10

3 1.0 Introduction Businesses are under increasing pressure to maximize profits and minimize downtime. Therefore, it is extremely important to have a contingency plan for continued operations in the event of a natural disaster or emergency. Actions taken during the first 24 to 48 hours of a disaster are critical in determining whether or not a business fully recovers. As many as 50% of businesses close down following a disaster according to the latest research 1. While natural disasters cannot be prevented, having a detailed emergency recovery plan can limit the financial and personal havoc they can cause. A good starting point is to address the following key areas: Ensure electrical equipment is properly maintained Identify the electrical equipment that is critical to operations Be aware of the most current natural disaster recovery codes and standards Know the effects of water damage to electrical equipment Develop a safety plan that incorporates emergency procedures Develop an electrical emergency action plan The National Fire Protection Agency (NFPA) and the Occupational Safety and Health Association (OSHA) provide guidelines to develop disaster recovery, emergency response, and safety plans. This paper will incorporate those guidelines to help in the creation of both short-term and long-term restoration plans. The number one priority for both plans is to safely restore power. Contingency planning for continued business operations is a multi-faceted risk management function. While natural disasters cannot be avoided, their impact may be somewhat lessened if businesses are better prepared. This paper identifies pre-planning exercises companies can complete to help restore electrical distribution and control equipment efficiently and safely. White paper on Emergency Response & Disaster Recovery 3

4 2.0 Natural Disaster Definition & Statistics Natural disasters can be broken down into four categories, as shown below. Each type of disaster has a unique severity level. Category Geophysical Meteorological Hydrologic Climatologic Type of Disaster Earthquakes, tsunami, volcanic eruption Storm, lightning Flood, mass movement Extreme temperature drought, forest fire Since the early 1990s, total economic losses from natural disasters in the United States have averaged tens of billions of dollars per year 2. These disasters cause death and injury, property damage, business interruptions or downtime and loss of revenue. As an example, in May, 2010, Nashville, Tennessee experienced an unprecedented 500-year flood killing more than 30 people and devastating property with over $2 billion of damage. Two years later, some businesses have just re-opened for business. There is increasing evidence that climate change is leading to more frequent and severe weather events, which points to an increased natural catastrophe risk. Figure 1 illustrates U.S. natural disasters for the past 30 years. In some cases, the increasing numbers can be attributed to better technology to identify natural disasters. However, climate changes are considered to have a significant impact on natural disaster occurrences. For additional information on your location, the Federal Emergency Management Agency (FEMA) publishes disaster and emergency declaration by year and state. Figure 1 Natural Disasters In The United States, (Number of events, Annual totals vs. first six months 2011) Natural disasters can damage electrical distribution equipment in multiple ways (fire, vibration, water, etc). The most frequently-occurring events in the U.S. involve water in the form of hurricanes and storms, with subsequent flooding. Since water and electricity do not mix, restoring power to water-damaged equipment can be a dangerous undertaking. Relevent Codes & Standards NFPA 1600 is the overarching standard and primary document on disaster recovery, emergency management, and business continuity. For workplace safety and planning, OSHA references (and cites to) NFPA 70E, Standard for Electrical Safety in the Workplace. NFPA 70B, Recommended Practice for Electrical Equipment Maintenance, is planning to add a specific chapter on disaster recovery and emergency response. In addition, the National Electrical Manufacturer s Association (NEMA) has published Evaluating Water- Damaged Electrical Equipment and is currently working on a guide for Evaluating Fire- and Heat- Damaged Electrical Equipment. Source: 2011 Munich Re. NatCatSERVICE. 4 White paper on Emergency Response & Disaster Recovery

5 3.0 Three Steps to Electrical Disaster Recovery Planning Step 1: Knowledge of the Electrical System As part of an electrical disaster recovery plan, businesses should: Have a current single-line drawing of their electrical distribution system Identify which electrical equipment is critical to the electrical infrastructure Understand which equipment must be replaced and that which can be reconditioned, as shown in Figure 2 Electrical equipment exposed to water can be extremely dangerous if re-energized without proper reconditioning or replacement. Assessing the damage goes beyond visual water indications. Moisture and weather exposure can affect the equipment s integrity. Water can be contaminated with sewage, chemicals, salt, debris, and other substances. Foreign debris could possibly remain inside the electrical enclosure. These contaminants may cause loss of dielectric spacing and can be a hazard upon re-energization. Whether the equipment is to be replaced or reconditioned, all services should be performed by qualified personnel familiar with the equipment s operation and construction. It is important to note that the ability to recondition equipment will vary; it may include the repair or replacement of internal components. Reconditioned equipment should be tested per ANSI / IEEE standards prior to re-energization. Figure 2 It is important that restoration activities are performed by qualified personnel as defined by OSHA and NFPA. Hiring outside contractors to assist or perform electrical work does not relieve the facility owner (host employer) from being liable for safe work practices. NFPA 70E states that known hazards must be communicated to the contracted worker(s). White paper on Emergency Response & Disaster Recovery 5

6 Step 2: Develop (or Update) an Electrical Safe Work Practices Policy An Electrical Safe Work Practices (ESWP) policy is a written document created by the employer that covers all areas of the company s electrical safety practices. While having a safety policy has been a requirement of NFPA for several revision cycles, the 2012 Edition further refines the requirements for developing and auditing the policy. The ESWP policy is not a one-size-fits-all policy to cover a company with multiple locations. For example, voltage, energy level, circuit conditions and hazard levels are different in each facility. The policy should identify detailed information specific to that location. This may include the location of the safety policy for employee (or contractor) access, who is the onsite authority having jurisdiction (AHJ) for decisions, etc. Developing and auditing an ESWP policy is critical to business continuity and disaster recovery. Since creating a comprehensive program may seem overwhelming, it may be helpful to use the following guidelines to get started. 1) Facility - Includes company policies and systems regarding: Equipment maintenance Tools Testing Repairs Clearance requirements Safe working conditions 2) Personnel - Focuses on actual work practices and addresses: Qualified and unqualified personnel Proper care and use of PPE Job preparedness Training and continuing education 3) Procedures - References on the job procedures and includes (but not limited to): Performing energized work De-energizing and re-energizing Lock-out / Tag-out Job planning Arc flash hazard analysis Equipment labeling Reporting safety concerns Recordkeeping The employer must verify on a regular basis that each worker is complying with the safety-related work practices required by NFPA 70E. Managers and supervisors must embed ESWP as part of their DNA, attend electrical safety training classes, and ensure all employees practice safe work practices everyday. Finally, should a natural disaster strike, employers and employees should be very familiar with emergency procedures and work practices. Step 3: Electrical Emergency Action Plan The purpose of the electrical emergency action plan (EEAP) is to understand the electrical assets, critical operational infrastructure, risks, and short- and long-term power restoration execution plans. Some of the benefits include: Reduces time to restore short-term and long-term power most quickly and safely Reduces uncertainties when a disaster occurs Increases understanding of electrical assets, available emergencies services, and replacement market availability Being able to know the immediate financial implications should a disaster occur 6 White paper on Emergency Response & Disaster Recovery

7 NFPA 1600 provides a framework for a site wide disaster recovery program regardless of the risk type. Although it does not include specific guidelines for any particular system or hazard, the provisions within help users manage potential disasters by having a common set of guidelines for managing all disaster recovery plans for a facility. From an electrical system perspective, NFPA 70B should be a first step in the process for the electrical system recovery plan. NFPA 70B provides guidance on creating a maintenance program for the electrical system in a facility which will contain information and data essential to the electrical system recovery plan. In recognition of this relationship between maintenance and disaster recovery, NFPA 70B will introduce a disaster recovery chapter in the 2013 edition which will help users determine potential catastrophic events and categorize potential impacts to the system. For now, electrical disaster recovery planning is based on experiences of the individual or team who has electrical responsibilities. One purpose of this whitepaper is to provide guidelines on developing an electrical emergency action plan and examples how it saved or could have saved facilities from excessive downtime. Two real industry examples follow. Downtime, loss of revenue and headaches could have been reduced from a robust EEAP program. Note: Facility names will remain anonymous. Example 1: Transformer Disaster Recovery Industrial Facility Background: Transformers are considered by corporate risk managers and insurers as the most critical equipment in facilities because of the large quantity of oil in contact with high voltage elements. Service entrance transformers are essential to operations. Many facilities would experience downtime or lack of productivity if the service entrance transformer is being serviced or a disaster hindered its performance. Furthermore, transformers can be custom-designed for utilities and facilities which makes finding a replacement or spare parts complicated, especially on an emergency basis. What Happened? A manufacturing facility that produced over $1 million per day in product was struck by lightning. The strike caused extensive electrical equipment damage, which included catching a 25 MVA transformer on fire.the fire was not contained in time and spread to an adjacent transformer. No standby generators were available. (Even so, standby generation at most industrial facilities would only be able to power vital processes, data rooms, and key emergency services.) Without standby generators and spare transformers, the facility experienced a complete blackout. To get back online as soon as possible, the team assessed the damages, identified an available emergency transformer, coordinated logistics and financial approvals, acquired the additional electrical equipment needed, and hired an experienced electrical disaster recovery team. Lessons Learned: 1. An EEAP should provide details to understand immediate actions, the sequence of engagement with recovery teams, available immediate spares at the facility, primary, secondary market and the financial impact to communicate to senior management. 2. To restore power as quickly as possible, identify critical electrical assets and the nearest available spares, rates, and terms for both short- and long-term requirements. For example, temporary 25MVA transformers are hard to find and once located, lease for approximately $35,000 per month (excluding delivery charges). In addition, lead times on new replacement equipment can take months. 3. Outside resources may be required to help manage power restoration, should another disaster occur. Partner with organizations that have the qualifications, resources and experience to handle the job. Items such as pre-negotiated commercials terms and conditions, defined scope of work, equipment pricing, lead times, availability, and service agreements streamline the power restoration process. White paper on Emergency Response & Disaster Recovery 7

8 Example 2: Generator Disaster Recovery - Commercial Building with Businesses and Data Center Background: On May 1-2, 2010, Nashville, Tennessee received a record 13 inches of rain, which caused a historical 500-year flood. Thirty people were killed and damages exceeded $2 billion. Also known as Music City USA, Nashville is a key tourist area for music lovers around the world. Due to its proximity to the Cumberland River, downtown Nashville was hit hard and floodwaters damaged many commercial buildings. This example focuses on a class A commercial high-rise in downtown Nashville that houses multiple businesses and a data center. What Happened? The following timeline of events depicts how the flooding affected the commercial high-rise building. Times shown serve as general guidelines. Sunday, May 2 9:00 pm Municipal power was no longer able to serve power to downtown Nashville. The commercial building automaticly switched to emergency power and only served vital data center operations. Teams were deployed to the site to closely monitor data center and weather conditions. Monday, May 3 12:00 am Metropolitan police ordered a mandatory evacuation in downtown, which left data center operations and security vulnerable. 4:30 am Selected personnel were allowed back into downtown to perform key operations, which included refueling generators. Although rain had stopped, the Cumberland River was steadily rising. 10:00 am A decision was made to start soft shutdown actions, but it was too late. Generators flooded, which began a 31-hour data center outage. Through a key relationship, a temporary generator was located 20 miles south the city. Tuesday, May 4 4:30 pm The data center was brought back on-line. Wednesday, May 5 - Still with no lights, air conditioning, or restroom facilities, a 24-hour staff rotation was set up to monitor the data center and the condition of the temporary generator. The water level was still too high to assess damage to the building s generator. Thursday, May 6 - The building s generator was drained and the damage was assessed. Spare parts were not immediately available. Five days later the generator was repaired and power switched back to the building generator. However, the temporary generator was retained as a back-up for a few additional weeks. Thursday, May 13 - Nashville Electric Services (NES) restored power to the building and switched from generator to municipal power. 8 White paper on Emergency Response & Disaster Recovery

9 Lessons Learned: Figure 3 summarizes three lessons learned from the aftermath of this historical flood. 1. Defined Priorities: The EEAP plan must have a clear definition of what constitutes an emergency and when to execute the EEAP. In this case, only three hours were available before the mandatory evacuation order was issued. Prioritizing critical functions is essential to efficiently and safely restoring power. 2. Emergency Service Contracts: Searching for temporary equipment after a disaster occurs slows down the power restoration process and can be very expensive. The EEAP should include emergency service contracts to guarantee disaster recovery team response time, critical equipment pricing, lead times, and a deployment strategy with details on setting up and operating a command center to meet an organization s needs.critical Spare Parts: Having a current single-line electrical diagram of the power distribution system is crucial to efficiently restoring power. Use the drawing to pinpoint the electrical equipment critical to business operations. Identify the equipment s critical spares parts availability, pricing and lead times for custom-made parts. Figure 3 Total outage: 11 days Lost power, data center running on temp gen. Mandatory evacuation from Metro Police Water rose and caused a complete outage Found and restored temp power to data center Assessed generator damage; awaited parts Parts received and generator repaired Switched to municipal power May 2 nd 9:00pm May 2 nd 12:00am May 3 nd 10:00am May 4 nd 4:30pm May 6th May 11th May 13th 3 hours before mandatory evacuation EEAP Priorities 31 hours data center down time Emergency Service Contracts 5 days to bring back building generator Critical Spare Parts Lessons Learned In some cases, EEAP programs should be signed off by customers and facility management companies. Why? Consider this example: A customer leases multiple floors (one floor being a data center) from a large building management company. The customer s IT manager may have no control over the equipment feeding power to the office space and data center. In this type of situation, clear roles and responsibilities, work scopes, and agreed upon emergency actions are vital to a long-term building owner and customer relationship. White paper on Emergency Response & Disaster Recovery 9

10 4.0 Developing an Electrical Emergency Action Plan The development of an EEAP may seem overwhelming. Below is a step-by-step guide created by Schneider Electric Services to help customers be prepared in the event of an emergency. 1. Define the Criteria of an Emergency Lighting strikes on service entrance transformers and flooding of backup generators clearly indicate operational states of emergency. Are nuisance tripping on circuit breakers or power quality issues considered emergencies? When productivity is impacted, it can be confusing as to when a state of emergency should be declared. The EEAP should clearly define what constitutes an electrical emergency. 2. Identify Electrical Equipment that is Critical to Business Operations On the single-line diagram, trace the power from the incoming utility source to every piece of electrical equipment feeding critical business operations. Perform an analysis on each of these critical assets to include availability in the market, lead times, and a plan of action if and when the equipment is no longer functional. For most critical assets, it is necessary to understand cost of temporary rentals, logistics, contacts, and testing requirements for both temporary and permanent power restoration. For example, a special transformer s spare parts may not be available in the primary market, i.e., the local electric municipality. Therefore, it is a good practice to determine available suppliers, acquire budgetary pricing, and lead times on the secondary market. 3. Selection of Outside Vendors and Pre-Negotiated Commercial Terms & Conditions In the aftermath of a disaster, it may be too late to negotiate pricing or lead times for the required resources to restore electrical power. Without pre-negotiated emergency service contracts, companies may suffer from overpricing and insufficient support. In addition to pre-negotiated normal and emergency rates, due diligence should include estimated response time and procedures for large-scale project coordination (see Step 4). The selected vendor(s) should have the depth and experience to handle major disasters. 4. Define Internal and External Responsibilities The EEAP should clearly define who has responsibility for what in restoring power to the facility. They also have details on the customer s responsibility such as providing electrical singleline diagrams, energization procedures, and coordination and communication activities between multiple vendors through a central contact. Third-party vendors should also provide a clear procedure on how they will approach an emergency at a facility that includes assessing damage, mobilizing resources, appointing a project manager, and establishing a command center. All parties involved should fully understand the safety plan that is put in place. 5. Define the Equipment and Service Scope This section will further define the equipment and associated work scope. For example, an electrical distribution service company can define the equipment scope to be from the utility service entrance (13.8 kv) to low voltage switchgear (480 V). The associated work scopes include equipment installation and commissioning in both temporary and permanent scenarios. 6. Emergency Contract Terms Emergency contracts should be written for a specific time period, with an expiration date. It is also recommended to add expiration dates to the EEAP plan for self-auditing purposes. A suggested best practice is to audit the EEAP when the ESWP is being audited which is every three years according to NFPA 70E Contact Information This section of the EEAP should include the latest contact information for anyone (internal or external) who has a defined responsibility in restoring power due to an emergency. 10 White paper on Emergency Response & Disaster Recovery

11 5.0 Conclusion When a natural disaster strikes, its impact on individuals, communities and businesses can be devastating. Restoring electrical power is a crucial part of the recovery process. Regardless of the industry or facility type, having a detailed Electrical Safe Work Practices (ESWP) policy and an Electrical Emergency Action Plan (EEAP) can help recovery efforts. Multiple standards exist from OSHA, NFPA and NEMA to serve as guidelines for businesses to help them understand and develop a contingency plan in the event of an emergency or natural disaster. References 1. ServPro Managing the escalating risks of natural catastrophes in the United States Lloyd s Insurance Report, October 2011 White paper on Emergency Response & Disaster Recovery 11