Safety in the Data Center

Transcription

1 Safety in the Data Center HEPIX University of Michigan Tony Wong Brookhaven National Lab

2 Background BNL is a multi-disciplinary lab with projects at various stages RHIC operational since 1999 NSLS-II currently building (ETA is 2015) Several new buildings supporting research efforts since ~2008 Diverse workforce White collar (engineers, scientists, clerical workers) Blue collar (technicians, tradesmen, unskilled labor) Aging infrastructure Some BNL buildings date back to ~1950 s. Many built in 1970 s and 1980 s out of compliance with current building codes, but grandfathered in New research facilities are significantly dependent on old systems (chilled water plant, power substations, outdated safety features, etc)

3 BNL A few, isolated high profile accidents recently Arc-flash incident at RHIC (Apr. 2006) Building explosion (Oct. 2008) propane leak Worker fall 16 ft. from scissor lift (Nov. 2011) Safety culture at BNL has been changing Minimize downtime and productivity losses due to accidents Developing processes to record, analyze past accidents (and near-misses) and decrease occurrence of future ones Emphasis on work-planning and accountability

4 Arc Flash Accident Damaged Switch Undamaged switch

5 Scissor Lift

6 Rack Tip-Over Accident Rack fell on its front face during installation procedure on Sept. 30, 2013 No one injured Safety officers notified within 2 hours of accident (BNL internal requirement) Since servers are loaded from the front, we could not even remove servers one by one and then attempt to lift up the empty rack full rack weighs over 1000 lbs (454 kg) Severed power cables underneath raised floor and damaged nearby CRAC unit No water leak in damaged CRAC unit Power cables not energized no electrical hazards

7 Racks Before Accident

8 Fallen Rack

9 Fallen Rack

10 Damaged Power Cable

11 Damaged CRAC Unit

12 Contributing Factors Typical of how accidents normally occur a sequence of conditions that culminated with the incident Tile in front of rack removed to access power and network cables Rack wheels not immobilized Tile in front not replaced after cables are fetched or adjusted Two staff maneuver rack forward and falls into open hole

13 Aftermath Rack raised up with specialized equipment Hydraulic jacks Crane Detailed physical inspection ensued Rack front door damaged beyond repair discarded One hot-plug disk was re-seated Extensive tests so far find NO DAMAGE to any of the servers Fully loaded rack worth ~US$100k Warranty still valid Mitigating factors Rack slid down slowly due to open hole and CRAC unit took some of the momentum away Raised floor only 12 in (30 cm) high, not deep enough for impact to cause significant damage

14 Hydraulic Jacks

15 Undamaged Servers

16 Effect on Data Center Operations Three power cables replaced on Oct. 14 delayed energizing of PDU All rack installation interrupted ~3 weeks until investigating was completed and remediation procedures were approved on Oct. 21 Remediation must be auditable increase in paperwork and more time-consuming (see next slide) Internal discussion on hiring external, qualified helpers Procedures still in the process of being formalized and implemented A review of safety procedures will occur in 3 years (part of remediation)

17 (In)adequate Training? Currently in our data center training course: Close open tiles as soon as possible, no later than the end of each day. Make sure that the floor is flat and there are no raised edges to trip on when the tiles are replace. Use safety cones, barricades, caution tape, or other safety equipment or devices to direct people away from hazardous areas, especially when a floor tile is removed and the under floor area is open with the potential for some to fall. Consider using Masonite or plywood to protect the surface and physical strength of the raised floor when moving heavy equipment like loaded computer racks. Reinforcement must be used when moving very heavy objects like PDUs and air conditioners.

18 Remediation Procedures Administrative and engineering measures A 3 rd person assisting Modified training for data center operations Work plan (permit) in advance Immobilize rack wheels when nearby tiles are removed Understand best practices in the field Contacted rack manufacturers Discuss with peer institutions Further measures possible to mitigate contributing factors or conditions that cause accidents

19 Wheel Chocks Successfully used to immobilize wheels after operations were resumed Have a limited number of sets need to purchase or manufacture more

20 Follow-Up What do other data centers do to prevent these kinds of accidents? Risk of equipment damage Risk of serious injury Are there more effective administrative or engineering procedures? The damaged rack was filled with 14 2-U servers and weighed ~1000 lbs. Racks filled with 1-U or blades would weigh ~1500 lbs. The risk of damage/injury is even higher

21 Summary First serious accident at the RACF in ~15 years of operation Bad timing against backdrop of enhanced safety accountability Minor equipment damage but increased scrutiny and oversight by upper management Delays to facility operations increased operational complexity as end result

22 Questions? Ofer Rind (c. 2004) University of Michigan alumnus