Hydro One Major Incident Report. Operational response and incident review of flooding at Manby and Richview Transformer Stations on July 8, 2013

Transcription

1 Hydro One Major Incident Report Operational response and incident review of flooding at Manby and Richview Transformer Stations on July 8,

2 Toronto Flood, July 8, 2013 On July 8, 2013, the Greater Toronto Area was hit by record rain fall with Pearson Airport reporting 90 mm of rain between 4:23 1 p.m. and 6:00 p.m. and 123 mm over a 24 hour period. The rainfall broke the previous daily record of 121 millimeters of rain set in 1954 by Hurricane Hazel. The rain came down with no advanced warning of the severity from Environment Canada weather services and caused major damage to the infrastructure serving the West GTA. The flash flooding that came with the rain overwhelmed rail, subway, sewers and drainage across the affected area as well as major electricity infrastructure serving the western portion of the Greater Toronto Area. Hydro One Networks Inc. experienced severe flooding at its Richview and Manby Transformer Stations (TS) causing interruptions of twenty-six 230,000 volt and eight 115,000 volt circuits supported by these stations and damaging communications equipment within these stations. Water damage continued to wreak havoc on operations as a building housing internal computer systems and telecommunications infrastructure filled with water from cable conduit penetrations, insufficient weeper sump pump capacity and loss of AC power to the weeper sump pumps. The flooding was significant, and reports from the response team indicate that the water removed from in and around Manby TS would have filled 15 Olympic-sized swimming pools. Hydro One declared a Level 2 Transmission Emergency after 3,400 MW of load was interrupted, causing widespread power outages to more than half a million households and businesses in the GTA. Six local distribution companies who receive their supply from Hydro One were directly affected by this event: Toronto Hydro, Hydro One Brampton, Enersource, Oakville Hydro, PowerStream, and Veridian. Four transmission connected customers were also directly affected: Ontario Power Generation Kipling Complex, Ford Oakville, IBM and Kinetrics HV Lab. During this time, Hydro One s Ontario Grid Control Centre (OGCC) was inundated with more than 126,500 alarms and 9600 phone calls. The unprecedented rush of water flooded into buildings at Richview TS (specifically into the OGCC Transmission and Distribution back-up control facilities) damaged and/or rendered critical equipment unavailable. Compounding this situation, the ITMC control room had to evacuated to their back up control room. Additionally, corporate IT systems were impacted, including mobile access on non-blackberry devices and Call Centre numbers. 1 All times are in Eastern Daylight Time (EDT) 2

3 Restoration began immediately and over the following five days the GTA power system was reconfigured and stabilized while station repairs were made and flood damage was assessed. Chronology of Restoration Hydro One crews were immediately dispatched, however with transportation systems across the city impacted by severe flooding, the speed of the response was hampered. Throughout the evening and into the night, crews made their way to these stations, at times receiving escorts from Toronto Police Services. The restoration focus was to transfer the affected 115 kv Toronto West load to Leaside TS and restore 230 kv circuits from remote stations outside of Richview TS and Manby TS. Hydro One worked closely with Local Distribution Companies (LDCs) and the Independent Electricity System Operator (IESO) to coordinate the system reconfiguration. By 4:00 a.m. on July 9, about 10 hours after the intense rain front hit the GTA, the electricity system was 91% restored with only 300 MW 2 directly supplied load out of Richview TS and Manby TS remaining unrestored, roughly representing10,000 homes and businesses served mostly by Toronto Hydro. On site work continued around the clock as crews removed water from flooded buildings, dried and inspected damaged equipment. Additionally, the OGCC dealt with the predetermined 115 kv corridor thermal limit issues that would knowingly ensue as the normal load profile demand would increase the following day after the storm. To alleviate the loading issues on these corridors voltage reductions and load shedding was implemented at the associated stations throughout the GTA. By 3:24 p.m. on July 10, all affected load had been restored, however the GTA power system required significant work to bring it back to normal configuration. The Transmission Emergency Level 2 remained in effect until 5:34 p.m. on July 12 th as Hydro One worked to reinforce the system with restored transmission connections between Richview TS and its remote terminal stations: Trafalgar TS, Cherrywood TS, Parkway TS and Claireville TS. This provided redundant supplies and vastly improved network security. Further remediation continued at both Richview and Manby over the following few days and at 2:44 p.m. on July 15 th the system was officially back to normal status. 2 Some of the 300 MW was restored through internal load transfers within the LDC systems. 3

4 Communication with the public during the event was vital and Hydro One depended on the news media to help tell the story. Hydro One s media relations team conducted more than 200 media interviews, issued 6 media releases and tweeted 36 times. Hydro One s President and CEO Carmine Marcello participated in an on-site press conference with the Mayor of Toronto and the President and CEO of Toronto Hydro. Status updates and conference calls were held to provide updates to government officials and the City of Toronto s Emergency Operations Centre. Summary of Root Causes The OGCC and Asset Management conducted investigations into the cause and response to this event. The OGCC assessed the response and actions taken from an operational perspective including IT, Corporate Communications and Security, identifying valuable lessons learned. Asset Management identified the causes for the flooding and recommended corrective actions to prevent a recurrence. This summary report provides high level information taken from their detailed internal analyses. Weather There was no prior warning of serious rainfall or potential for flooding issued by any of the weather forecasting providers used by the OGCC. Water from the flash flooding primarily entered impacted buildings via cable penetrations in building basements. AC powered sump pumps at Manby TS and Richview TS Building O were quickly rendered inoperable due to loss of AC station service power at both stations. Employees were significantly delayed in reaching Richview TS and Manby TS to aid in the restoration efforts during the onset of this event due to extreme weather and road closures. Local news updates on the road conditions were indicating road closures, sink holes, nonfunctioning traffic signals and high traffic volume. On site Construction crews at Manby TS proactively called for vacuum trucks to help with the flood relief. Even though some on-call employees had been able to respond within minutes from dispatch, due to amount of rainfall received, they could not prevent the resultant water damage. 4

5 Hot weather during the days following the initial flooding led to high energy usage across the GTA. As a result transformers and transmission lines were heavily loaded as restoration efforts progressed. The reconfigured circuits had additional loading, which led to heavy thermal loading on several corridors. Communication challenges Incident Command Centre-Transmission was not readily initiated and could have been initiated earlier; however the functional requirements of ICCT were already being implemented and managed by the OGCC frontline support and the first responders without an official corporate wide declaration. The initiating conditions were satisfied with the Level 2 SENS but declaration of establishment corporately was not immediately conveyed. OGCC and IESO complied with proper communication and policy protocols throughout the event; however review of these protocols will take place to explore opportunities of increased operating automomy during localized events which would further decrease restoration times. Response to station alarms was affected due to the high number in a short period. As a result, communications to customers did not occur immediately as would under normal operating procedures. Customers that contacted the OGCC for status updates were kept apprised. Constant requirement for information between involved agencies led to delays and discrepancies (for example, forecasted restoration times). The Hydro One media line was inundated with calls and triage was conducted with the largest outlets with the greatest reach receiving priority. Equipment Condition and Design Both site drainage systems at Manby TS and Richview TS were designed for a 100 year one-hour storm event. Immediately, prior to the July 8, 2013, storm, there was no evidence to suggest that either site drainage system was not operating as designed. At Manby TS, the site drainage system was recently inspected, flushed and upgraded ($4.7M spent, completed fall 2011). Similar investments were not required at Richview TS, as this site had no history of significant site or building flooding. At Manby TS, although the drainage capacity of the site drainage system and the City of Toronto stormwater sewer system was likely exceeded during the storm peak, this was not the primary cause of building flooding. The primary cause was 100+ year rainfall plus aging/plugged cable trench underdrainage systems. As a result, stormwater from flooded cable trenches in the switchyards entered buildings via unsealed/poorly sealed cable penetrations in basement walls. 5

6 Water rendered inoperable critical power system equipment located in these basements (terminal racks, battery rectifiers, etc). At Richview TS, as a result of a 100+ year rainfall, stormwater once again flooded the yard cable trenches and entered both the East and West Relay Buildings via cable penetrations in the basement walls. Additional stormwater entered the West Relay Building via a poorly sealed window well. Flooding of terminal racks in these Relay Basement buildings caused circuits to be removed from service and the loss of AC power at Richview TS and Manby TS. Stormwater also entered the Richview TS Telecom Building via cable penetrations, sump pit overflow/backflow and the loss of AC power to the sump pumps. Follow-up Actions Following the event a rigorous review was undertaken to document the full scope of damage and the resulting system repairs completed in the days following the initial damage. To conduct this review information was collected from Operating and Field staff, various lines of business, affected customers, the Network Management System information and storage retrieval system, and the review of 9,600 phone calls relating to the event from the OGCC control room audio log system. The detailed report provides a breakdown of all affected stations, line reconfigurations, system impacts related to Corporate IT infrastructure and relationship management with external agencies and customers. Follow up actions related to the below Key Learning s are documented and will be tracked on an ongoing basis. Follow-up actions with regards to the events surrounding the GTA flood will support Hydro One s compliance commitments and service level agreements and goal of operating a safe and reliable system for all customers. Key Learnings and Recommendations Findings, relevant comments and concerns, lessons learned and recommendations arising from this review of the GTA blackout that occurred on July 8 th have been documented and continue to be implemented into regular operating procedures and future emergency response planning. Operational recommendations Review and evaluate back up options for key computer and telecommunication services, and back-up systems integral to OGCC s 6

7 operability of the power system. Business Information Technology will assess opportunities to build further redundancies in the system thus avoiding future vulnerability to the Corporate IT services. Response time for Hydro One staff can be reduced by engaging local police services to explore options for faster transfer of people and equipment to required sites. Operations staff may be directed to drive personal vehicles to a non-impacted common location to rendezvous with a police escort in order to get to the site of emergency quicker. A review of the benefit of creating a standard system emergency message that could be issued at the onset of extreme events to notify customers and internal staff that Hydro One is not in a normal operating state. This could assist in reducing low priority calls and would ensure that the customers have been notified in a prompt manor. Clarification is required on declaration of ICCT dependent on the classification of the level of emergency. As an example, when an ICCT is initiated non-emergency transmission calls could be re-routed to other sectors not affected by the event. Other required logistical processes are put in place under ICCT that assist with dissemination of communications and coordination efforts. Coordinate joint messaging with Toronto Hydro and emergency response organizations to ensure consistent messaging is delivered during other large-scale events to avoid public confusion. Explore opportunities for IESO pre-approved load restoration plans during localized emergencies. Asset Management recommendations To prevent the entry of stormwater during severe rainfall events into building basements containing critical power system/it equipment, Hydro One is undertaking the following actions: Seal watertight all cable penetrations and window wells at Richview TS and Manby TS. Re-design the AC supply system to the Richview TS Telecom Building sump pumps to ensure automatic back-up supply via the diesel generators. 7

8 Review consultant s recommendations and compare against as built site drainage system/sump pump upgrades at Manby TS. Identify all transmission station buildings with critical power system equipment located in basements and, where appropriate, develop a mitigation plan for incorporation into the 5 year investment plan. This plan may include the following: 1) sealing and maintaining cable penetrations; (2) relocating new/replacement equipment and/or cable entrances to above grade locations; 3) inspecting/replacing/maintaining basement check valves (or backflow prevention valves); 4) upgrading cable trench under drainage systems and/or 5) upgrading building sump pits/pumps. Determine if changes to the Hydro One design criteria for transmission station site drainage systems are required (ie, design for a 200 year one-hour storm event). As part of planned infrastructure investment projects, replace the Richview TS 230 kv air blast circuit breakers and relocate protection and control equipment to new relay buildings. 8