NERC s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining

Transcription

1 NERC s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system

2

3 Bulk Power System Operations Lesson Learned SOL and IROL Monitoring Tool Leads to Unnecessary Manual Load Shedding Problem Statement A breaker fault on a shunt reactor of a high voltage transmission line caused the line to trip. This event led to the exceedance of two interconnection reliability operating limits (IROLs). Immediate actions were taken to clear those IROL exceedances within the required 30 minutes.

4 Corrective Actions The entity conducted system operator training that featured additional stressful circumstances to more accurately reflect real life conditions that can occur on the grid during an event. The entity reviewed the system operator s understanding of several topics as well, including: The differences between the types of limits (SOLs versus IROLs) on a particular interface and how to monitor them The emergency operations control actions list The system operators training guide regarding the monitoring of SOL and IROL limits to determine if additional clarification is required

5 Corrective Actions cont The entity also conducted a re evaluation of its monitoring tools. The entity verified that the EMS monitoring display s contents are adequate to clearly differentiate between the SOL and IROL limits. The entity created a new tool for displaying the SOL and IROL limit timers on the control room timer display for SOL/IROL exceedances. The entity developed a new display for the transmission and balancing system that would help to quickly select the required control actions according to the type of SOL/IROL exceedance. The entity also re evaluated its operating procedures. It clarified the definition of the SOL and IROL limits in its operating procedures and instructions. It clarified the application of the emergency condition in its operating procedures and instructions.

6 Lessons Learned Clearly differentiate the SOL and IROL limits in the EMS displays and the control room timer display for SOL/IROL exceedances to ensure they will not lead to system operator confusion. Review system operator understanding of the SOL and IROL limits and the emergency operations control actions permitted in each situation on a regular basis. More simulation training for system operators is needed with added stressful circumstances in order to enhance system operator situational awareness and response during emergency operations.

7

8 Control Network Communication Path Problem Statement Following a failover exercise being conducted pursuant to EOP 008 1, the TOP s router bandwidth for network traffic between control centers saturated and led to a flat line condition, preventing communication between the active control center (ACC) and the active data center (ADC). The ADC serves the ACC s monitoring and controlling capability by housing the active supervisory control and data acquisition (SCADA) servers and associated equipment. As a result of the bandwidth saturation, the ACC lost the ability to monitor and control its portion of the Bulk Electric System for approximately 39 minutes.

9 Corrective Actions SCADA administrators failed back over to the PDC to resolve the issue. Going forward, the entity is instituting technical controls to prioritize SCADA communication network traffic over other network traffic on the out of band router to reduce the likelihood that bandwidth saturation will interrupt SCADA communication.

10 Lessons Learned Entities running workstations that are remote from EMS/SCADA servers should prioritize network traffic such that situational awareness traffic is prioritized over other network traffic, such as cybersecurity logging traffic. Technical controls can be established with quality of service configuration on network devices which allows prioritization of situational awareness traffic and traffic levels. Additionally, entities should consider if there should be a prohibition on nonemergency changes when operating remotely from EMS/SCADA servers.

11

12 Generation Facility Event Lesson Learned Human Error Leads to Evacuation of Primary Control Room Problem Statement A utility had to evacuate its primary control center due to smoke from a nearby fire that was caused by maintenance worker failing to follow proper hot work procedures.

13 Corrective Actions All personnel have been retrained on the hot work permit system. The utility also plans to implement the measures necessary to prevent smoke from traveling into the control center, like appropriate fire stops and ventilation changes.

14 Lessons Learned Workers should properly assess all work conditions before beginning any maintenance activities and follow established hot work permit processes. Periodic training on hot work procedures should be given to all maintenance employees. Control center ventilation equipment and fire stops should be assessed on a regular basis to verify that proper precautions have been taken to ensure that smoke from internal or external fires cannot reach the control center. In instances when control centers are not completely separate, but are located adjacent to other active facilities, consideration should be given to the impact of these facilities on control centers.