AIS Power Reliability

Transcription

1 Case Study AIS Power Reliability How a Wayward Raccoon Sparked a Major Power Distribution System Upgrade Company executives welcome open, frank discussion and want to address all questions and concerns about reliability, failover processes, backup systems, and service continuity. AIS (American Internet Services) is a leading provider of enterprise-class data center, cloud, and connectivity services known for its robust network, security, and solutions engineering capabilities. For over 20 years AIS has been helping companies get the most out of their information technology initiatives. Headquartered in San Diego, AIS operates seven secure SSAE 16-compliant, SOC 1,2,3-audited facilities in San Diego, Los Angeles, and Phoenix. The company s mission is to provide innovative and trusted enterprise-class IT infrastructure solutions that enable clients to focus on their core business. The following describes what took place before, during, and after a brief clientaffecting incident on March 1, The outage gave AIS a temporary black eye in terms of perceived reliability but since that time the company has undergone intense introspection, remediation, and self-imposed scrutiny in order to fully understand, and move beyond, the root causes of the event. The purpose of this piece is to provide a summary of what occurred and how AIS has addressed the situation. Company executives welcome open, frank discussion and want to address all questions and concerns about reliability, failover processes, backup systems, and service continuity. Indeed, AIS stands by its enterprise-class service reliability records and is happy to compare itself to any other data center in the industry. It is this culture of continuous improvement that defines who we are and why we are different.

2 AIS Case Study Power Reliability 2 Power Distribution Systems Expansion Ken Carter, vice president of data center operations and the San Diego Business Journal 2012 IT Executive of the Year, had been working on AIS s power distribution systems expansion design for over two years. A professionally certified data center design electro-mechanical engineer, Mr. Carter is responsible for the continuous reliability, availability, and productivity for each of the AIS data centers. His efforts include planning and implementing a replacement of the main electrical distribution and standby power systems serving the company s data centers. Ken Carter VP Data Center Operations American Internet Services (AIS) During the debilitating regional outage in September 2011 knocking out power to a large portion of Southern California AIS successfully transferred to enginegenerator and operated seamlessly for 13 hours without issue. AIS subsequently earned accolades from clients and the local business community for superior performance during the extended period. To ensure unsurpassed reliability and security for all clients, AIS data center infrastructure systems have an N+2 level of redundancy in generator capacity, expanding to a 2N level of capacity. In practical terms, this means that AIS has eight permanently-installed 644 kw engine-generators with 16,000 gallons of on-site diesel fuel at the company s flagship 57,000 square-foot Lightwave Data Center facility. The company s utility service feed at Lightwave is 12.5 MW capable and 9.1 MW usable a massive amount of power which places AIS near the top of the San Diego Gas & Electric (SDG&E) largest local customers list. One area of emphasis for Mr. Carter had been making sure that Lightwave could transfer power automatically and seamlessly from the SDG&E electrical grid to the on-site AIS power supply. This capability was accomplished by implementing PowerSecure s Interactive Distributed Generation (IDG) power system which was designed for synchronized and parallel operation with SDG&E for peak load management and emergency preparedness purposes. A bigger challenge for Mr. Carter had to do with improving the reliability of an aging emergency stand-by system in place at Lightwave the engine-generators. One of the AIS criteria for the new system was that it, must be capable of transferring to generator remotely upon command without disruption to the system load which required a closed transition transfer capability with remote monitoring and transfer control. The upgrade project, which began in early 2009 and which Mr. Carter assumed in late 2010, was completed in April During the commissioning of the system, the loss of SDG&E source (utility) power was simulated numerous times and the system functioned as designed. As a matter of fact, during the debilitating regional outage in September 2011 knocking out power to a large portion of Southern California AIS successfully transferred to engine-generator and operated seamlessly for 13 hours without issue. AIS subsequently earned accolades from clients and the local business community for superior performance during the extended period.

3 AIS Case Study Power Reliability 3 Along Comes a Raccoon On Thursday, March 1, 2012, a wayward raccoon found its way into an SDG&E substation near Lightwave and generated a line-to-ground fault, thereby causing a fault across a feeder to the AIS facility. The fault occurred on the SDG&E kv circuit feeding AIS at the Kearny Mesa sub-station vault located at the intersection of Overland and Lightwave avenues, approximately 200 yards from the AIS facility. To put it delicately, the raccoon did not survive but our memory of that fateful critter lives on. Raccoon (Procyon lotor) All told, AIS invested nearly $1 million in remediation, sparing no expense in implementing infrastructure upgrades to bolster reliability and regain client trust. The company knows that a solid reputation takes a long time to build but only a brief moment to lose. AIS is committed to regaining that trust. The sub-station feed caused the fault that impacted across the AIS facility load-break connects that directly support the facility. The line-to-ground short circuit fault tripped all systems off-line and resulted in a partial loss of client loads, predominately for those pieces of equipment whose power was single-sourced. Translation: an electrical outage occurred for certain client equipment that had only one power feed. The immediate impact of the fault was that all medium voltage utility service breakers and engine generator bus breakers were tripped and over-current protection relays were locked out on high over-current, requiring a subsequent manual reset by AIS personnel. The reset process began 25 minutes after the outage occurred and was completed 10 minutes later but not until two of four Uninterruptible Power Supply (UPS) systems serving AIS customers lost battery back-up power and tripped offline. Power was subsequently restored after manually resetting all relays and breakers. The total client outage lasted from 1-20 minutes depending on the specific client, and all power to all points within the facility was restored 55 minutes after the start of the SDGE outage. By then, however, the damage had been done in terms of interruption to client service the worst fear of any data center operator. Investigating, Evaluating & Remediating Immediately after the outage, AIS engaged CCG Facilities Integration Incorporated, an independent third-party firm specializing in power systems reliability, to perform a complete analysis of the company s power distribution system and prepare a recommended solution to preclude this type event from ever occurring again. Consulting engineer Mike Mosman confirmed a design flaw in the system PowerSecure had implemented for AIS which failed to account for the impact of a ground fault caused by the installation of generator step-up transformers using a grounded secondary. Because the transformer design and installation were flawed, it allowed for three times the current that was modeled for in the design, leading to improper transfer control breaker settings and thus causing the system to not function as intended with the resultant loss of the facility load. In other words, a design flaw in the original system enabled the outage.

4 AIS Case Study Power Reliability 4 CCG Facilities recommended that AIS retrofit and reengineer the existing step-up transformer configuration to remove the path for ground current during a short circuit by the utility. The solution provided transfer control that would facilitate engaging the transformer when generator was required and would also provide proper grounding and lightning arrestors to the transformers. This solution, which AIS implemented immediately, will indeed prevent this issue from ever occurring again. AIS also undertook the following remediation steps: By aggressively addressing the situation, AIS now has an emergency bus system that is consistent with a Tier-3 data center design capable of responding to all voltage anomalies and faults. n Re-engineered the system design to ensure that the step-up transformers remain de-energized unless a call for generator is initiated. n Made sure the source power for battery chargers and lighting is now derived from an internal downstream source, not the (formerly) energized transformers. n Relocated three 200-amp breakers feeding certain step-down transformers and installed feeds to transformer breakers. n Re-configured transfer control logic such that all utility service anomaly sensing (UV, OV, UC, OC, etc.) will cause a utility service breaker to open, start generators, and transfer to engine-generator. All told, AIS invested nearly $1 million in remediation, sparing no expense in implementing infrastructure upgrades to bolster reliability and regain client trust. The company knows that a solid reputation takes a long time to build but only a brief moment to lose. AIS is committed to regaining that trust. Aftermath The Commissioning Field Report submitted by CCG Facilities in June 2012 stated, The [AIS] Standby Engine-Generator System is fully functional and able to protect the data center from utility outages. The exposure to utility service ground faults (as which caused the last outage) has been removed. Furthermore, having learned to trust but verify, AIS proceeded to subject the vastlyimproved power distribution system to the most demanding sequence of testing possible. Numerous simulated outages and transfers were initiated and all were completed without issue. AIS and CCG Facilities even conducted a rigorous, marathon series of tests over three nights where no less than 40 outage and fault scenarios were simulated and the system responded as designed in each case. Consulting engineer Mike Mosman summed it all up: By aggressively addressing the situation, AIS now has an emergency bus system that is consistent with a Tier 3 data center design capable of responding to all voltage anomalies and faults. Hallelujah.

5 AIS Case Study Power Reliability 5 Conclusion It has been said that every cloud has a silver lining. The March 1, 2012 outage helped AIS create a power distribution system that is more robust than ever and second to none in the Southwest. Now completely rock solid, the remediated system is fully capable of handling the challenges of today and beyond. Tim Caulfield CEO American Internet Services (AIS) Tim Caulfield, chief executive officer of AIS, summed things up: Present and prospective AIS customers can take comfort in knowing that the company is now wiser, more experienced, and more vigilant than ever. Indeed, there are those data center providers who have experienced a major client-affecting incident and those who will; fortunately, AIS personnel and processes are fully steeped in the knowledge, training, and experience to prevent, mitigate, identify, and respond to potentially clientimpacting events faster than ever before. The lesson was painful but now seared into the company s collective consciousness. We are more vigilant than ever. What s Next Continuous improvement is the name of the game at AIS. Indeed, Mr. Carter has already commenced work on the next phase of power distribution expansion at Lightwave that includes four additional engine-generators paralleled with the existing eight engine-generators. The goal is to provide 5.4 MW of on-site generation at N+2 to accommodate future growth. He is also working on matching the power distribution system end-state design with a modular mechanical plant environmental systems expansion to provide a 2N configured and paralleled chilled water system. The upshot of all this investment is even greater electrical and cooling capacity for AIS clients in the Lightwave facility. Note: The AIS acceptance testing results report can be made available upon request consistent with meeting authorization criterion. Please contact your AIS representative for more information.