Consulting Associates, Inc. Page 1 EXCERPT - THE DATA CENTER BATTERY DILEMMA Developed by Marco W. Migliaro, PE, Life Fellow IEEE Visit: stationarybatterytraining.com esaconsulting.com Copyright 2014 by Marco W. Migliaro All rights reserved 1 Foreword 2 2 The information and photographs included in this presentation are targeted for those involved with data center high performance lead-acid stationary battery installations (typically UPS batteries). Make certain that a copy of the battery manufacturer's data sheet and installation, operation and maintenance instructions for the specific cell or battery to be installed, inspected, maintained or tested are available. Always read and understand the battery manufacturer's recommendations, including safety requirements, provided in its installation, operation and maintenance instructions or other document and in the battery manufacturer s warranty. Industry standards that provide recommendations for safety, installation, inspection, maintenance and testing are also available and should be readily available for reference and use. Safe work practices should always be followed including wearing proper PPE and using insulated tools. Stationary batteries are very reliable; however, problems can and do occur. The problems experienced may be a result of improper application, selection, installation or maintenance, or not understanding the actual design life of the battery. They may also be due to a product design or manufacturing issue. The photographs in this presentation may show a number of batteries in service, sometimes including the markings of the battery manufacturer. Test equipment may also appear in some of the photographs. These photos are provided for informational or educational purposes only and are not meant to endorse any one battery or equipment manufacturer, nor are they meant to imply that any battery manufacturer produces a superior or inferior product. The developer makes no warranties expressed or implied about the materials contained in this or other presentations. The contents of this and other presentations are owned by the developer and are protected by United States and international copyright laws and international trade provisions.
Consulting Associates, Inc. Page 2 DATA CENTER BATTERY SPECIFICATIONS Engineers and owners prepare specifications that: Require vented lead-acid batteries to have a 20 year design (expected, or service) life Deliver the largest kw/lb in the smallest physical footprint with the lowest cost Battery manufacturers respond by offering high performance vented lead batteries that: Have a 20 year warranted life (typically 1 yr full, 19 yrs pro-rata or 3 yrs full, 17 yrs pro-rata) The reality is that these batteries have expected lives of, at best, 10 to 12 years 3 3 DATA CENTER BATTERY SPECIFICATIONS (cont d.) 4 4
Consulting Associates, Inc. Page 3 20 YEAR STATIONARY BATTERIES? Battery manufacturers do make vented lead-acid products that have a 20 year design life In these cells, the positive plate thickness is generally 0.30 inch vs. 0.17 to 0.20 inch in high performance cells and plate thickness is the key to determining design life (thick plates = longer life) Some of the products may use a different type of positive plate, e.g., a Planté Some industries require battery manufacturers to demonstrate 20 year design life by test, using naturally or artificially aged cells. The test generally requires the 20 year aged cells to be capacity tested and deliver at least 80% of their nameplate capacity, after having gone through a series of seismic tests on a shake table 5 5 CAN TRUE 20 YEAR LIFE BATTERIES BE APPLIED IN DATA CENTERS? The answer to this is yes, however: Almost all of these cells types are in single cell jars, although there are a few types in two-cell jars The jars are physically larger and will require much larger battery rooms Their cost (on a per cell basis) is much greater than the high performance cells used in data centers primarily because there is more lead in each cell The typical duty cycle of 10 to 15 minutes in a data center will utilize very little of the lead in the cell during a discharge (i.e., discharging the surface but not the depth of the plate) The owner therefore pays for a lot of lead that is never used 6 6
Consulting Associates, Inc. Page 4 SO WHAT IS THE REAL ISSUE? Users and engineers: Do not understand that any battery manufacturer s high performance lead-acid battery does not have a 20 year life and battery salespersons continue to tell them the batteries have a 20 year life Continue to write specifications requiring a 20 year life Battery manufacturers: Understand that in most bidding situations, they will likely lose the order if they take exception to the 20 year life requirement Know that if they offer a true 20 year life battery, it s not the right type for application at a data center and they will lose the order based on price 7 7 WHAT IS THE IMPACT ON THE DATA CENTER? The high performance vented lead-acid batteries installed will function normally for the first 8 to 10 years This assumes the batteries have been properly sized, installed, maintained and tested, and the cells do not have any manufacturing defects Once the batteries pass the 8 to 10 year mark, failures begin to occur, often developing shorts The users believing the battery life is 20 years, think the battery manufacturer provided a defective product Maintenance costs will increase, sometimes dramatically, as more and more cells need to be replaced 8 8
Consulting Associates, Inc. Page 5 WHAT IS THE IMPACT ON THE DATA CENTER? (cont d.) The real reason why the number of cell failures is increasing, is that the installed batteries are not defective, but are approaching end-of-life The user should be planning entire battery replacements at this time instead of cell replacements for cells that are nearing end-of-life Battery maintenance organizations may not understand: the cells are at end-of-life and may recommend these replacements; may not recognize cells are developing plate shorts; and may recommend additional maintenance that will ultimately do nothing to improve the battery, all of which add to the data center s operating costs 9 9 WHAT IS THE IMPACT ON THE DATA CENTER? (cont d.) Data Center Managers and Supervisors: Will often ask maintenance personnel what they did wrong to cause these 20 year life batteries to start failing at 10 or 12 years Assuming an effective maintenance program, the conclusion is typically that they must have a defective product The reality is that no matter how good the maintenance program is, the high performance vented lead-acid battery will not have a 20 year design life 10 10
Consulting Associates, Inc. Page 6 LACK OF ATTENTION TO DETAIL MAINTENANCE ORGANIZATIONS The following examples illustrate issues that should be identified and brought to the owner s attention by their maintenance organization but were not These are taken from data for four, 192 cell high performance lead-acid UPS batteries with nominal 1.215 SG electrolyte Each battery was the same model number Two batteries were 11 years old and the remaining two were 8 years old One of the two 11 year old batteries had gone into thermal runaway and the second had started to go into thermal runaway 2 months after the quarterly inspection for the data shown was performed 11 11 LACK OF ATTENTION TO DETAIL MAINTENANCE ORGANIZATIONS (cont d.) A review of the battery maintenance reports for the previous 18 months (the start of the current maintenance organization s contract) showed the issues that lead to the thermal runaway and near runaway began to show up in the maintenance data over a year before the event The maintenance organization did not recognize the issues and continued to provide the owner with recommendations that would accelerate the degradation of the batteries Some maintenance recommendations were based on averages rather than baselines At no time was there a recommendation to contact the battery manufacturer or to have the manufacturer s service department inspect the batteries 12 12
Consulting Associates, Inc. Page 7 LACK OF ATTENTION TO DETAIL MAINTENANCE ORGANIZATIONS (cont d.) Although from the owner s perspective, the batteries were being regularly maintained, based on the events, it is likely that at least one of the four batteries and possibly two could have failed to perform their design function had they been required to do so The maintenance program did not include discharge testing of the batteries It is incumbent on the owner to review and question if necessary, the maintenance reports, including the recommendations provided and not simply file them away 13 13 EXAMPLE 1 MAINTENANCE REPORT 240 CELL UPS BATTERY These data are from the quarterly report for one 11 year old battery that was submitted to the owner and approximately 7 weeks later, the battery experienced a thermal runaway and had to be immediately removed from service The maintenance organization had to bring in a temporary battery and an unplanned (unbudgeted) replacement of this battery and the second 11 year old battery which was found to be starting into thermal runaway had to be performed A recommendation made in the quarterly maintenance report is likely to have accelerated the failure of the battery 14 14
Consulting Associates, Inc. Page 8 EXAMPLE 1 RECOMMENDATIONS 15 15 EXAMPLE 1 DATA EXCERPT 16 16
Consulting Associates, Inc. Page 9 EXAMPLE 1 OBSERVATIONS These observations should not be considered a detailed evaluation of the data Although these are the maintenance data, the cells were visually in distress, with signs of plate growth, slivering, plate cracking and this should have been obvious to those performing the quarterly inspection There was no recognition that the battery was at end-oflife and there was no recommendation made to replace it (or even selected cells), although there was a recommendation to capacity test it; however, if there was a need to capacity test, the test should not have been performed until the cells needing replacement were replaced 17 17 EXAMPLE 1 OBSERVATIONS (cont d.) There were 11 cells in the battery with cell voltages 2.09 Vdc (one of these cells was <2.0V) As important as low voltage cells, there were 30 cells with voltages 2.34 Vdc (with 12 of these cells 2.40 Vdc), but these are often overlooked and not thought of as being an issue High voltage cells are as much of a problem as low voltage cells, since the higher voltage will cause excessive gassing (which will increase the need for watering in these cells), degrading the cells The reason for these extremely high voltage (the highest 2.45 Vdc) cells is that with the battery floating at constant voltage, as the voltage of the shorted cells drop, the remaining cell voltages will increase since the overall charge voltage remains constant 18 18
Consulting Associates, Inc. Page 10 EXAMPLE 1 OBSERVATIONS (cont d.) There is no recommendation to replace any cells although some replacements were needed, to ensure the battery could perform until replaced E.g., cell no. 81 which is clearly in trouble Since this is a quarterly report, only 10% of the cell SGs and internal resistances are measured There is no recommendation to measure the SGs of the cells that are shorted to assess their state-of-health The recommendation to equalize the battery will accelerate the degradation of both shorted cells and high voltage cells Although it should be obvious to a knowledgeable person that some of these cells will never recover, the recommendation was made none the less 19 19 EXAMPLE 1 OBSERVATIONS (cont d.) Four cells are identified as having higher SG than other cells with a recommendation to reduce the SG of the cells Reviewing past maintenance data, this issue had been on every maintenance report since this maintenance organization had assumed responsibility for maintenance 18 months ago The reason the SG of these cells was high was that they were nominal 1.250 SG cells in a string with 1.215 SG cells (and the cell nameplate had a different model no.) It is presumed that the previous maintenance organization either purchased the wrong model cells or the battery manufacturer delivered the wrong model cells and they were installed in the battery The current maintenance organization never clearly identified the issue to the owner and never took action 20 20
Consulting Associates, Inc. Page 11 EXAMPLE 2 - MAINTENANCE REPORTS IDENTIFY HIGH RESISTANCE CELLS These data are from the quarterly report for the four UPS batteries previously discussed The maintenance organization uses deviation from average values to determine cell issues, instead of a baseline value with specific limits As a consequence, the same model cells in two different batteries with the identical internal resistance will be judged acceptable in one case and identified as a problem in the other. There can also be cells with even higher resistance judged to be acceptable in the battery with the higher average resistance Evaluation and analysis of data in this manner is incorrect! 21 21 EXAMPLE 2 DATA EXCERPT 22 22
Consulting Associates, Inc. Page 12 EXAMPLE 2 DATA EXCERPT (cont d.) 23 23 WHAT NEEDS TO BE DONE? Users and engineers: Need to recognize that any battery manufacturer s high performance vented lead-acid battery is the right choice for data center UPS applications; however, they will never have a 20 year design life Need to prepare specifications that call for a 10 year design life Battery manufacturers: Can effectively respond to proposals, understanding the 10 year expected life requirement is part of the UPS battery specification and this levels the playing field for all manufacturers that will respond to the bid 24 24
Consulting Associates, Inc. Page 13 WHAT NEEDS TO BE DONE WITH MAINTENANCE REPORTS AND DATA? Users: Should select a maintenance organization that is knowledgeable in stationary battery maintenance, including visual inspection and analysis of the data taken, to specific baseline values for the particular cell type installed Should require the battery maintenance organization to not only analyze these data but trend these data and virtually no battery maintenance contracts require this today Alternately, the owner could perform this trending analysis itself Consider having an independent 3 rd party review of the battery maintenance program periodically, possibly triennially, or as frequently as annually 25 25 HOW WILL A UPS BATTERY SPECIFICATION REQUIRING 10 YEAR LIFE CHANGE DATA CENTER OPERATION? Users: Should experience no major issues with the UPS batteries, unless there is a design or manufacturing defect Will understand they need to plan for replacement batteries in 10 to 12 years Should see a reduction in operating and maintenance costs and unnecessary maintenance should be eliminated Battery manufacturers: Will not have to deal with unrealistic user and engineer battery design requirements for a cell type that will never have a 20 year design life 26 26
Consulting Associates, Inc. Page 14 STATIONARY BATTERY One type within the classification of industrial batteries Manufacturers in North America generally classify vented lead-acid cells as* Long duration: telecommunications ( 0.30 to 0.35 inch thick positive plate) General purpose: switchgear tripping, generating station, substation, industrial control, industrial UPS ( 0.25 to 0.30 inch thick positive plate) High performance: commercial UPS ( 0.17 to 0.20 inch thick positive plate) *NiCd cells are classified as long, medium, high and ultra high 39 39 THE HIGH PERFORMANCE LEAD-ACID STATIONARY BATTERY Battery manufacturers have done everything to optimize this battery type for high current, short duration discharges, such as: Using thin plates to allow more plates to be installed in the jar to maximize the plate surface area available for discharge Used flat pasted plates with an optimal design to produce the greatest amount of discharge current/power Used plastic vs rubber separators to minimize separator thickness and some eliminated retainers Minimized plate spacing and strap/post height Use pure copper or copper inserted posts Increased electrolyte specific gravity 40 40
Consulting Associates, Inc. Page 15 END OF USEFUL LIFE Assuming that a stationary battery has been properly sized, installed, maintained and tested, and that the battery itself has no design or manufacturing defects: The end of useful life for a lead-acid stationary battery is corrosion of its positive plates. Signs of end of life include positive plate growth, plate cracks and sulfate The end of useful life of a nickel-cadmium battery is carbonization of its electrolyte 41 41 EXAMPLE OF A BATTERY MAINTENANCE COMPANY MISSING THE OBVIOUS The second slide after this one shows a portion of a commissioning report for a 240 cell UPS battery that has been installed and open circuit cell voltages have been measured before the freshening (i.e., initial) charge is applied It clearly shows some cells are not charged and were likely not recharged after their factory discharge test They were shipped to the site in an uncharged condition (there were 40 cells affected) The organization making the measurements did not recognize the obvious, did not bring the condition to the attention of any party and performed the initial charge 59 59
Consulting Associates, Inc. Page 16 EXAMPLE OF A BATTERY MAINTENANCE COMPANY MISSING THE OBVIOUS (cont d.) By not recognizing the issue, the initial charge will not be effective and not charge the cells properly, but more importantly, the uncharged cells may fail prematurely or may have already been irreparably damaged, depending how long they have been in their present condition (including time in storage) The correct action here would have been to notify the appropriate parties, and contact the battery manufacturer s service department and have the cells examined If the manufacturer determined the uncharged cells could be recovered, they would be placed on a charge by themselves before the entire string was placed on initial charge 60 60
Consulting Associates, Inc. Page 17 PLATE SHORTS Plate shorts can occur in a cell if a positive plate and negative plate contact each other as a result of: Positive plate growth, including growth of grid elements like forming bars A defect in a separator or a mispositioned separator Paste lumps or other material buildup (e.g., blisters) on the plates Lead runs or flash not removed from a cell Sediment buildup Plate shorts are difficult to detect visually but if they can be seen, the positive plate will turn grey in the area of the short The easiest way to detect a plate short is by measurement of cell voltage If a short develops, the cell voltage will drop to near open circuit 62 62 PLATE SHORTS (cont d.) Although plate shorts are not a desired condition, the presence of a plate short does not mean that the cell (or battery) does not have enough capacity to meet its design function Any shorted cell must be replaced, but if detected early it may be possible to allow the cell to remain in the battery until the replacement cell is received If the cell is allowed to remain in the battery, then periodic measurement of its electrolyte specific gravity must be made to determine if it may remain in the battery or needs to be removed from the battery 63 63
Consulting Associates, Inc. Page 18
Consulting Associates, Inc. Page 19