Data Centre Reliability and Redundancy. What do they mean and are you are getting what you need?

Transcription

1 Data Centre Reliability and Redundancy What do they mean and are you are getting what you need?

2 What's the message? Why consider Reliability and Redundancy? How do you know what Reliability and/or Redundancy is required for your Data Centre? How do you communicate to the design engineer? How does the design engineer communicate to you? What are the typical issues and what can typically go wrong? How to consider Reliability and Redundancy? Summary.

3 Why? Why do we need to consider Reliability and Redundancy? 1. Equipment or system Reliability can not be guaranteed. 2. All equipment and systems require some form of maintenance, repair and/or replacement at some time. 3. Data centre users require certainty of operation.

4 Reliability vs Redundancy 100% None Reliability Redundancy None 100% or more

5 How? How do you know what Reliability and/or Redundancy is required for your Data Centre? 1. Understand the intended Data Centre use. 2. Understand stakeholder expectations. 3. Understand the reliability of the equipment proposed. 4. Understand the Reliability of the available energy and water supplies. 5. Understand the available budget limitations.

6 How? How do you communicate to the design engineer? How does the design engineer communicate to you? Agree on a common language!

7 What are the typical issues? Very complicated operational and servicing inter-relationships. Poor briefing / wrong language used. Inadequate budget to meet brief. Downtime for repairs or maintenance. Cheaper equipment may be less reliable and therefore require greater redundancy. Key stakeholders on different pages. Testing and Commissioning. Who will operate and maintain the systems? Understanding the various classifications, 999, TIA Tier 3, Uptime.

8 What can typically go wrong? Design does not meet the project requirements.

9 Single Point of Failure not properly analysed.

10 Partial or Multiple equipment failures not considered properly.

11 Installation does not meet the design.

12 Equipment is not tested or commissioned properly.

13 What else can go wrong? Complex systems difficult for technicians to understand and maintain. Design makes maintenance difficult

14 Routine and/or monitored equipment maintenance is not undertaken properly.

15 End user compromises design through poor change management.

16 End user doesn t understand design principles.

17 Human Error!

18 How to consider Reliability and Redundancy? Mean Time To Repair (MTTR) Mean Time To Failure (MTTF) Mean Time Between Failure (MTBF) Availability (MTBF / MTBF + MTTR) MIL-STD-756B Single Point Of Failure (SPOF) Failure Mode Analysis (FMA) Steady State Operation (SSO) TIA 942 Tier system

19 What is the relationship between MTBF and Reliability? Reliability theory is a statistical approach to predicting equipment failures. It can become very difficult to interpret in relation to Data Centre infrastructure. Basic Theory is; MTBF = Mean Time Between Failure This is actually defined as the inverse of the failure rate over a given time period. That is failure rate (say failures per hour) λ = 1/MTBF This then allows Reliability over a particular time period to be estimated using an exponential equation of R(t) = e -t/mtbf

20 MTBF Misconceptions Often thought to mean a certain guaranteed time between failures. THIS IS NOT CORRECT! It is a calculated statistical average assuming a constant failure rate. MTBF does NOT account for Infant Mortality (Burn in) that occurs during factory batch testing and specific project commissioning activities.

21 More MTBF Misconceptions MTBF is NOT service life. Service life is determined by specific component usage requirements and is addressed by monitored and/or routine maintenance or replacement regimes. An example of this is UPS battery replacement and generator bearing replacements. It is important to note that in some cases MTBF can exceed the service life of a component.

22 LED light fitting example MTBF = 200,000 hours Duration of service (t) = 8,760 hours (1yr) Number of units = 1000 Reliability = 95.62% Failure rate = 4.38% Expected failures = 43

23 MTBF UPS System example = 100,000 hours Duration of service (t) = 43,800 hours (5y) Number of units = 2 Reliability = 65% Failure rate = 35% Expected failures = 1

24 Result It is therefore important to understand that it is almost certain that one UPS will fail over a 5 year period. Based on this understanding there must be an expectation that any particular item of critical plant will fail unexpectedly at some point in time.

25 What does Reliability mean? Different things to different people! ICT vs Power vs Cooling. Short term power failure can result in long term ICT failure. Short term cooling failure may not be critical. Can be defined as determining the risk of failure.

26 Redundancy requirements Need to respond to equipment and / or system failures. Depends on the specific Data Centre requirements. Depends on the equipment installed. Depends on the tolerable ICT downtime permissible. Depends on the specific system design characteristics.

27 Communicating Redundancy We need a common language that communicates redundancy There are very few recognized standards that can be used One that has been useful is ANCI/TIA/942 This is an American Standard and therefore needs to be used carefully in Australia.

28 Overview of Data Centre Critical Systems ANSI/TIA 942 has an appendix that addresses various aspects of redundancy It expands on the Uptime Institute Tier System. The appendices cover components of a Data Centre that affect reliability and redundancy including Architectural, Structural and fire systems in addition to Mechanical and Electrical.

29 TIA 942 Architectural (1) Uptime Tiering Reference Architectural Site Selection Proximity to flood hazard as mapped on the federal flood hazard boundry or flood insurance map No Requirement Not in 50 year Flood hazard area Not within 100 year flood hazard area and greaterthan 91m (300ft) from 50 year flood hazard area Greater than 91m (300ft) from 100 year flood hazard Proximity to costal or inland No Requirement No Requirement greater than 91m (300ft) Greater than 0.8km (1/2 mile) waterways Proximity to major highway No Requirement No Requirement greater than 91m (300ft) Greater than 0.8km (1/2 mile) trafic arteries Proximity to major airports No Requirement No Requirement Greater than 1.6km (1 mile) and less than 48km (30 miles) Greater than 8km/5 miles and less than 48km/30 miles Proximity to major metropolitan area Parking Separate visitor and employee parking areas No Requirement No Requirement less than 48km (30 miles) less than 16km (10 miles) No Requirement No Requirement yes (physically separated with separate enteries) Separate from loading docks No Requirement No Requirement yes (physically separated with separate enteries) Proximity of visitor parking to data centre perimiter building walls Tier 1 Tier 2 Tier 3 No Requirement No Requirement 9.1m (30 ft) minimum separation with physical barrier to prevent vehicle from driving closer Multi-tenant occupancy No Restriction Allowed if occupancies non- hazardous Allowed if tenants are Data Cenntres or Telecommunications companies Tier 4 yes (physically separated by fence or wall with separate enteries) yes (physically separated by fence or wall with separate enteries) 18.3m (60 ft) minimum separation with physical barrier to prevent vehicle from driving closer Allowed if tenants are Data Cenntres or Telecommunications

30 TIA 942 Architectural (2) Building Construction Type of construction No Restriction No Restriction Type IIA,IIIA or VA Type 1A or 1B Fire Resistive Requirements Exterior bearing walls Code requirement Code requirement 1 Hour Minimum 4 Hours Miinimum Interior bearing walls Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Exterior nonbearing walls Code requirement Code requirement 1 Hour Minimum 4 Hours Miinimum Structural frame Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Interior non-computer room Code requirement Code requirement 1 Hour Minimum 1 Hour Minimum walls Interior computer room walls Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Shaft enclosures Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Floors and Floor-ceilings Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Roofs and roof-ceilings Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Compliance with NFPA 75 No Requirement Yes Yes Yes Miscellanous Building Components Vapour barriers for computer No Requirement Yes for walls no Yes Yes room requirements for ceilings building entrances with security checkpoints No Requirement No Requirement Yes (primary building entrance manned) Yes (primary building entrance manned) Access Floor Panel construction (when provided) No Requirement No Requirement Computer grade all steel Computer grade all steel or or computer grade steel with concrete filled Understructure (when access floor is provided) No Requirement No Requirement Bolted stringer Bolted stringer with 1.2m x 1.2m (4 ft x 4 ft) basket weave patten

31 TIA 942 Architectural (3) Ceilings with in Computer areas (when provided) Ceiling construction No Requirement No Requirement If provided - suspended with clean room class 10M-100M perforated tiles Suspended with clean room class 100 non perforated vinyl coated gypsum board tiles Ceiling Height (above raised access floor if provided) 2.6m (8.5 ft) minimum 2.7m (9.0 ft) minimum 3.0m (10 ft) minimum (not less than 460mm (18 in) above tallest piece of equipment) 3.0m (10 ft) minimum (not less than 600mm (24 in) above tallest piece of equipment) Roofing Class No Restrictions Class A Class A Class A Type No Restrictions No Restrictions Non-redundant with Noncombustible deck (no mechanically attached systems) Double redundant with concrete deck (no mechanically attached systems) Wind Uplift resistance Minimum code requirements FM I-90 FM I-90 minimum FM I-120 minimum Roof slope Doors and Windows Fire Rating Door Size Windows on perimiter of computer room Minimum code requirements Minimum code requirements Minimum code requirements Allowed with minimum Code required fire rating Minimum code requirements 1:48 (1/4 in per foot) minimum 1:24 (1/2 in per foot) minimum Minimum code Min Code Req, not less than 0.75 Min Code req, not less than requirements hour at Computer room 1.5 hours at Computer Room Minimum code Min Code Req, not less than 1m Min Code Req, not less than requirements wide and 2.13m High 1.2m wide and 2.13m High Allowed with minimum interior windows allowed with interior windows allowed Code required fire rating minimum 1 hour fire rating no with minimum 1 hour fire exterior windows allowed rating no exterior windows allowed

32 TIA Architectural (4) Entry Lobby Physically Separate from No Requirement Yes Yes Yes other areas of Data Centre Fire separation from other Minimum code Minimum code Minimum code requirements (not Minimum code requirements areas of Data Centre requirements requirements less than 1 Hour) (not less than 2 Hour) Security Counter No Requirement No Requirement Yes Yes (physically separated) Single person interlock to No Requirement No Requirement Yes Yes prevent pass back Adminstrative Offices Physically Separate from No Requirement Yes Yes Yes other areas of Data Centre Fire separation from other Minimum code Minimum code Minimum code requirements (not Minimum code requirements areas of Data Centre requirements requirements less than 1 Hour) (not less than 2 Hour) Security Office No Requirement No Requirement Yes Yes Physically Separate from No Requirement No Requirement Yes Yes other areas of Data Centre Fire separation from other Min Code Req Min Code Req Not less than 1 Hour Not Less than 2 Hours areas of Data Centre 180 deg peepholes on No Requirement Yes Yes Yes Security equipment and rooms Dedicated and hardened security equipment and No Requirement Yes Yes, with 16mm (5/8) plywood walls and solid core door Yes, with 16mm (5/8) plywood walls and solid core door monitoring rooms Operations Centre Operations Centre physically No Requirement No Requirement Yes Yes Separate from other areas of Data Centre Fire separation from other No Requirement No Requirement 1 Hour 2 Hours areas of Data Centre Proximity to computer room No Requirement No Requirement Indirectly accessable (maximum of 1 adjoining room) Directly accessable

33 TIA Architectural (5) Restrooms and Break room areas Proximity to computer room and support areas No Requirement No Requirement If immediately adjacent, provided with leak prevention barrier Fire separation from computer room and support areas Minimum code requirements Minimum code requirements Minimum code requirements (not less than 1 Hour) Not immediately adjacent and provided with leak prevention barrier Minimum code requirements (not less than 2 Hour) UPS and Battery Rooms Aisle widths for maintenance, repair, or equipment removal No Requirement No Requirement Minimum code requirements (not less than 1m (3 ft) clear) Minimum code requirements (not less than 1.2m (4 ft) clear) Proximity to computer room No Requirement No Requirement Immediately adjacent Immediately adjacent Fire separation from other Minimum code Minimum code Minimum code requirements (not Minimum code requirements areas of Data Centre requirements requirements less than 1 Hour) (not less than 2 Hour) Required Exit Corridors Fire separation from computer Minimum code Minimum code Minimum code requirements (not Minimum code requirements room and support areas requirements requirements less than 1 Hour) (not less than 2 Hour) Width Min Code Req Min Code Req Minimum code requirements and not less than 1.2m (4 ft) clear Shipping and Receiving area Shipping and receiving area physically Separate from Fire separation from other areas of Data Centre Physical protection of walls exposed to lifting equipment traffic Minimum code requirements and not less than 1.5m (5 ft) clear No shipping and receiving area provided no Yes Yes Min Code Requirement Minimum code Not less than 1 Hour Not Less than 2 Hours if shipping and requirements receiving area present No Requirement No Requirement Yes (minimum 19mm (3/4 in) Yes (steel bollards or similar plywood wainscot) protection) Number of Loading docks No Requirement 1 per 2500sqm of computer room Generator and Fuel Storage areas Proximity to computer room and support areas Proximity to publicly accessable areas 1 per 2500sqm of computer room (2 Minimum) No Requirement No Requirement If within Data Centre Building - min 2 Hour separation from all other areas 1 per 2500sqm of computer room (2 Minimum) Separate building or exterior weatherproof enclosures with Code required building separation No Requirement No Requirement 9m (30 ft) or greater separation 19m (60 ft) or greater separation

34 TIA 942 Security (1) Uptime Tiering Reference Security Tier 1 Tier 2 Tier 3 Tier 4 System CPU UPS capacity No requirement Building Building Building + 8 hour Battery minimum Field Panels UPS capacity No requirement Building + 4 hourr Battery minimum Building + 8 hour Battery minimum Building + 24 hour Battery minimum Field Device UPS capacity No requirement Building + 4 hourr Battery minimum Building + 8 hour Battery minimum Building + 24 hour Battery minimum Dedicated Security staff per shift No security staffing provided 1 per 3000 sqm (1 Minimum) 1 per 2000 sqm (2 Minimum) 1 per 2000 sqm (3 Minimum) No of 8 hour security shifts No requirement 1 (minimum 5 days per week) 2 (7 days per week) 3 (7 days per week) Security Access Control/Monitoring at: Generators Industrial grade lock Intrusion Detection Card Access Card Access UPS, Telephone & MEP Industrial grade lock Intrusion Detection Card Access Card Access Rooms Fiber Vaults Industrial grade lock Intrusion Detection Intrusion Detection Card Access Emergency Exit Doors Industrial grade lock Monitor Delay egress per code Delay egress per code Accessible exterior windows/openings No Monitoring Intrusion Detection Intrusion Detection (with offsite monitoring during shifts when no security staff is present) Intrusion Detection (with offsite monitoring during shifts when no security staff is present) Security Operations Centre No requirement No requirement Card Access Card Access Network Operations Centre No requirement No requirement Card Access Card Access Security Equipment Rooms No requirement Intrusion Detection Card Access Card Access

35 TIA 942 Security (2) Doors into computer rooms Industrial grade lock Intrusion Detection Card or biometric access for ingress Perimiter Building doors No requirement Intrusion Detection Card access if main entrance; intrusion detection all others Main door onto computer Industrial grade lock Card Access Single person Interlock to room floor prevent passback Card or biometric access for ingress and egress Card access if all entrance Single person interlock to prevent passback, with biometric Bullet resistant walls, windows & doors Security counter in lobby No requirement No requirement Level 3 minimum Level 3 minimum Security counter in shipping No requirement No requirement No requirement Level 3 minimum and receiving CCTV Monitoring Building Perimiter and No requirement No requirement Yes Yes parking Generators No requirement No requirement Yes Yes Access Controlled doors No requirement Yes Yes Yes Computer room floors No requirement No requirement Yes Yes UPS, Telephone & MEP No requirement No requirement Yes Yes Rooms CCTV CCTV recording of all No requirement No requirement Yes - Digital Yes - Digital activity on all cameras Recording rate (Frames/sec) No requirement No requirement 20 frames/sec min 20 frames/sec min

36 TIA Structural Uptime Tiering Reference Structural Tier 1 Tier 2 Tier 3 Tier 4 Any seismic zone acceptable, but may require No Restriction No Restriction No Restriction No restriction special support mechanisms Facility designed to seismic zone requirements No Restriction No Restriction No Restriction In seismic zone 0, 1, 2 to Zone 3 requirements, in siesmic zone 3 & 4 to Zone 4 requirements Site specific response spectra - degree of local seismic accelerators No No With operation status after 10% in 50 year event With operation status after 5% in 100 year event Importance factor I=1 I=1.5 I=1.5 I=1.5 Telcommunications equipment racks/cabinets No Base only Fully braced Fully braced anchored at base or supported top and bottom Deflection limitations on telcommunications No No Yes Yes equipment within acceptable limits Bracing of electrical conduits and cable trays Per code Per code with importance factor Per code with importance factor Per code with importance factor Bracing of mechanical system major duct runs Per code Per code with importance factor Per code with importance factor Per code with importance factor Floor loading capacity superimposed live load 7.2kPa (150lbf/sqsf) 8.4kPa (175lbf/sqft) 12kPa (250lbf/sqsf) 12kPa (250lbf/sqsf) Floor hanging capacity for ancillary loads 1.2kPa (25lbf/sqsf) 1.2kPa (25lbf/sqsf) 2.4kPa (50lbf/sqsf) 2.4kPa (50lbf/sqsf) suspended from below Concrete slab thickness at ground 127mm 127mm 127mm 127mm Concrete topping over flutes for elevated floors affects size of anchor which can be installed 102mm 102mm 102mm 102 Building LFRS (Shearwall/braced frame/moment frame) indicates displacement of structure Building Energy Dissipation - Passive Dampers/base isolation (energy absorption) Battery/UPS floor vs building composition. Concrete floors more difficult to upgrade for intense loads. Steel framing with metal deck and fill much more easily upgraded Steel deck and Fill/ Pre tension concrete/ CIP Mild - PT slabs much moore difficult to install anchors Steel/Conc MF Conc.Shearwall/Steel BF Conc.Shearwall/Steel BF Conc.Shearwall/Steel BF None None Passive dampers Passive dampers/base isolation PT Concrete CIP Mild Concrete Steel Deck & Fill Steel Deck & fill PT Concrete CIP Mild Concrete Steel Deck & Fill Steel Deck & fill

37 TIA 942 Electrical systems (1) Uptime Tiering Reference Electrical Tier 1 Tier 2 Tier 3 Tier 4 General Ststem allowance concurrent maintenance No Down to but not including UPS Output Panelboards Down to but not including Power Distribution Units Throughout Distribution Systems Single points of failure Power Syatem Analysis Computer and Telecommunications Equipment Power Cords One or more single points of failure fo rthe distribution systems serving electrical equipment or mechanical systems Short Circuit Study Coordination Study Arc Falsh Analysis Single cord feed with 100% capacity One or more single points of failure fo rthe distribution systems serving electrical equipment or mechanical systems Short Circuit Study Coordination Study Arc Falsh Analysis Single cord feed with 100% capacity No more than single points of failure for the distribution systems serving electrical equipment or mechanical systems Short Circuit Study Coordination Study Arc Falsh Analysis Load Flow Study Redundant cord feed with 100% capacity on remaining cord or cords No single points of failure fo rthe distribution systems serving electrical equipment or mechanical systems Short Circuit Study Coordination Study Arc Falsh Analysis Load Flow Study Redundant cord feed with 100% capacity on remaining cord or cords Utility Utility entrance Single feed Single feed N+1 Redundant Feed 2N Redundant Feed from different utility substations Main Utility Switchboard Service Shared Dedicated Dedicated Dedicated Construction Panel Board with bolt on circuit breakers Switchboard with stationary circuit breakers Switchboard with drawout circuit breakers Switchboard with drawout circuit breakers Surge Suppression Optional Optional Yes Yes Generators correctly sized for UPS Yes Yes Yes Yes Uninterruptible Power Supply System

38 TIA 942 Electrical systems (2) Redundancy N N N+1 2N Topology Single or Parallel Single or Parallel Modules Modules with Static Bypass Automatic Bypass None Non dedicated maintenance bypass feeder to UPS output switchboard Maintenance Bypass arrangement Output Power Distribution Battery String Battery Type Parallel Redundant modules, Distributed Redundant Modules or Block Redundant Systems Dedicated maintenance bypass feeder to UPS output switchboard Parallel Redundant modules, Distributed Redundant Modules or Block Redundant Systems Dedicated maintenance bypass feeder to UPS output switchboard Bypass from same source Bypass from same source Bypass from same source Bypass from reserve UPS system Panelboard incorporating Panelboard incorporating standard thermal standard thermal magnetic trip magnetic trip breakers breakers Common string for multiple modules 5 year valve regulated lead acid Dedicated string for each module 10 year valve regulated lead acid or flood type Panelboard incorporating circuit breakers that are coordinated under overload and ground fault current condition Dedicated string for each module 15 year valve regulated lead acid or flood type Panelboard incorporating circuit breakers that are coordinated under all overcurrent conditions Dedicated string for each module 20 year lead acid flood type Battery Minimum Backup 5 minutes 7 minutes 10 minutes 15 minutes Time Battery Monitoring System Optional Optional String level by UPS system Centralised automated system to check each cell for, voltage and impedance or resistance Flywheel Allowed Allowed Allowed Allowed Power Distribution Unit Transformer Optional K-rated or harmonic cancelling K-rated or harmonic cancelling, Front accessable for maintenance K-rated or harmonic cancelling low inrush, Front accessable for maintenance Automatic Static Transfer Switch Overcurrent Device NA Fuse Circuit breaker Circuit breaker Maintenance Bypass NA Manual non Guided Manual Guided Manual Guided Output NA Single circuit breaker Dual circuit breaker Dual circuit breaker

39 TIA 942 Electrical systems (3) Grounding Lighting Protection System Based on Risk analysis Based on Risk analysis Yes Yes Service entrance grounds Yes Yes Yes Yes and generator grounds fully comply to NEC or local codes Data Centre grounding Optional Optional Yes Yes infrastructure in Computer room Computer Room Emergency Power Off (EPO) System Installation Do not install if not required by Authorities having juristriction Do not install if not required by Authorities having juristriction Do not install if not required by Authorities having juristriction Do not install if not required by Authorities having juristriction Test Mode Optional Optional Yes Yes Alarm Optional Optional Yes Yes Abort Switch Optional Optional Yes Yes Central Power Monitoring Monitored Points None Utility, UPS, Generator Utility, Main Transformer, UPS, Generator, Automatic Static Switch, PDU, Automatic Transfer Switches Utility, Main Transformer, UPS, Generator, Automatic Static Switch, PDU, Automatic Transfer Switch,es, Surge Protection Devce, Critical Load Branch Circuits Notification Method NA Control Room console Control Room console, Pager, , Text message Control Room console, Pager, , Text message

40 TIA 942 Electrical systems (4) Battery Room Separate room No Yes Yes Yes Individual battery banks No Yes Yes Yes isolated from each other Shatterproof viewing glass No No No Yes in Battery room door Standby Generating Systems Generator Sizing Sized for UPS System only Sized for UPS & mechanical System Sized for total building load with N+1 redundancy Sized for total building load with 2N redundancy Generators on single bus Yes Yes Yes No Load Bank Installation None Provision for portable Provision for portable Permenant Equipment Tested NA Generator Generator, UPS Generator, UPS Auto Shutdown NA None Automatic upon failure of utility Automatic upon failure of utility Testing Factory Acceptance Testing Optional Optional Yes Yes Site Circuit Breaker Testing Optional Optional Contact resistance test for all critical circuit breakers in critical and essential paths, 225 Amps and higher Commissioning Optional Component level, system level Component level, system level, Integrated system Primary injection and Contact resistance test for all critical circuit breakers in critical and essential paths, 225 Amps and higher Component level, system level, Integrated system

41 TIA Maintenance Uptime Tiering Reference Equipment Maintenance Tier 1 Tier 2 Tier 3 Tier 4 Maintenance Staff On site day shift Only. On call at other times On site day shift Only. On call at other times On site 24 hrs M-F, on-call weekends Onsite 24 x 7 Preventative Maintenance None None Limited preventative maintenance program Facility Training Programs None None Comprehensive training program Comprehensive preventative maintenance program Comprehensive program including manual operation procedures if it is necessary to bypass control system

42 TIA 942 Mechanical systems (1) Uptime Tiering Reference Mechanical General Routing of water or drain piping not associated with the Data Centre Equipment in data centre spaces Positive pressure in comuter room and associated spaces relative to outdoors and non data centre spaces Floor drains in computer room for condensate drain water, humidifers flush water and sprinkler Mechanical discharge systems water on Standby generator Water Cooled Systems Indoor terminal Air Conditioning units Humidity control for Computer rooms Electrical Service to Mechanical Equipment Tier 1 Tier 2 Tier 3 Tier 4 Permitted but not recommended Permitted but not recommended Not Permitted Not permitted No requirement Yes Yes Yes Yes Yes Yes Yes No requirement Yes Yes Yes No Redundant units One redundant AC unit per critical area Qty of A/C units sufficent to mainain critical area during loss of one source of electrical power Qty of A/C units sufficent to mainain critical area during loss of one source of electrical power No requirement Humidification provided Humidification provided Humidification provided Single path of Electrical power to A/C equipment Single path of Electrical power to A/C equipment Multiple paths of electrical power to AC equipment. Connected in checkerboard fashion for cooling redundancy Multiple pathsof electrical power to AC equipment. Connected in checkerboard fashion for cooling redundancy

43 TIA 942 Mechanical systems (2) Heat Rejection Dry Coolers (where applicable) Closed- circuit fluid coolers (where applicable) Circulating pumps Piping system Chilled Water System Indoor Terminal Air Conditioning units Humidity Control for Computer rooms Electrical Service to Mechanical Equipment Heat Rejection Chilled Water Piping System No redundant dry coolers No redundant fluid coolers No redundant condenser water pumps Single path condenser water system No Redundant AC units One redundant unit per system One redundant unit per system One redundant unit per system Single path condenser water system One redundant AC unit per critical area Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Headed parallel piped condenser water system Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Qty of units sufficent to mainain critical area during loss of one source of electrical power Qty of units sufficent to mainain critical area during loss of one source of electrical power Qty of units sufficent to mainain critical area during loss of one source of electrical power Dual path condenser water system Qty of A/C units sufficent to mainain critical area during loss of one source of electrical power No requirement Humidification provided Humidification provided Humidification provided Single path of Electrical power to A/C equipment Single path chilled water piping system Single path of Electrical power to A/C equipment Single path chilled water piping system Multiple paths of electrical power to AC equipment Dual path ladder drop chilled water system with isolation valves Multiple paths of electrical power to AC equipment Dual path chilled water system Chilled Water Pumps No Redundant units One redundant unit per system Air-cooled Chillers No Redundant units One redundant unit per system Water Cooled Chillers No Redundant units One redundant unit per system Cooling Towers No Redundant units One redundant unit per system Condenser Water Pumps No Redundant units One redundant unit per system Condenser Water Piping System Single path condenser water system Single path condenser water system Qty of units sufficent to mainain critical area during loss of one source of electrical power Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Qty of units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Headed parallel piped condenser water system Qty of units sufficent to mainain critical area during loss of one source of electrical power Qty of units sufficent to maintain critical area during loss of one source of electrical power Qty of units sufficent to maintain critical area during loss of one source of electrical power Qty of units sufficent to maintain critical area during loss of one source of electrical power Qty of units sufficent to maintain critical area during loss of one source of electrical power Dual path condenser water system

44 TIA 942 Mechanical systems (3) Air-Cooled Systems Indoor Terminal Air Conditioning units/outdoor Condensors No Redundant AC units One redundant AC unit per critical area Qty of AC units sufficent to provide minimal temporary capacity to critical area during loss of one source of electrical power Qty of A/C units sufficent to mainain critical area during loss of one source of electrical power Electrical Service to Mechanical Equipment Humidity control for Computer rooms HVAC Control System HVAC Control System Power source to HVAC control system Plumbing (for Water-cooled heat rejection) Make-up Water Points of connection to condenser water system Single path of Electrical Single path of Electrical Multiple paths of electrical power Multiple paths of electrical power power to A/C equipment power to A/C equipment to AC equipment to AC equipment No requirement Humidification provided Humidification provided Humidification provided Control system failure will interrupt cooling to critical area Single path of Electrical power to HVAC control system Control system failure will not interrupt cooling to critical area Redundant, UPS electrical power to BMS control Control system failure will not interrupt cooling to critical area Redundant, UPS electrical power to BMS control Control system failure will not interrupt cooling to critical area Redundant, UPS electrical power to BMS control Single water supply with no Dual sources or one source Dual sources or one source + onsite Dual sources or one source + on- on-site storage + on-site storage storage site storage Single point of connection Single point of connection Two points of connection Two points of connection

45 TIA 942 Fuel Oil Uptime Tiering Reference Fuel Oil Tier 1 Tier 2 Tier 3 Tier 4 Bulk Storage tanks Single storage tank Single storage tanks Multiple storage tanks Multiple storage tanks Storage tanks, pumps and piping Generator fuel capacity at full load Single pump and/or supply pipe Multiple pumps and multiple supply pipes Multiple pumps and multiple supply pipes Multiple pumps and multiple supply pipes 8 hrs 24 hrs 72 hrs 96 hrs

46 TIA 942 Fire Suppression Uptime Tiering Reference Fire Suppression Tier 1 Tier 2 Tier 3 Tier 4 Fire Detection System No Yes Yes Yes Fire Suppression System When required Pre-action (when required) Pre-Action (when required) Clean agents listed in NFPA Gaseous suppression system No No Early warning smoke detection system No Yes Yes Yes Water leak detection system No Yes Yes Yes Pre-Action (when required) Clean agents listed in NFPA

47 TIA Items beyond the designers control THIS IS A LIST OF ITEMS BEYOND THE CONTROL OF THE DESIGNER ITEM Tier 1 Tier 2 Tier 3 Tier 4 SECURITY Dedicated Security staff No security staffing 1 per 3000 sqm (1 1 per 2000 sqm (2 1 per 2000 sqm (3 Minimum) per shift provided Minimum) Minimum) No of 8 hour security No requirement 1 (minimum 5 days 2 (7 days per week) 3 (7 days per week) shifts per week) MAINTENANCE Maintenance Staff Onsite 24 x 7 On site day shift Only. On call at other times On site day shift Only. On call at other times On site 24 hrs M-F, on-call weekends Preventative Maintenance Facility Training Programs None None Limited preventative maintenance program None None Comprehensive training program Comprehensive preventative maintenance program Comprehensive program including manual operation procedures if it is necessary to bypass control system

48 TIA Additional items beyond designers control These items are also beyond the control of the designer as they are determined by site selection rather than data centre design ITEM Tier 1 Tier 2 Tier 3 Tier 4 Site Selection Proximity to flood hazard as mapped on the federal flood hazard boundry or flood insurance map No Requirement Not in 50 year Flood hazard area Not within 100 year flood hazard area and greaterthan 91m (300ft) from 50 year flood hazard area Greater than 91m (300ft) from 100 year flood hazard Proximity to costal or inland No Requirement No waterways Requirement Proximity to major highway trafic No Requirement No arteries Requirement Proximity to major airports No Requirement No Requirement Proximity to major metropolitan area STRUCTURAL Telcommunications equipment racks/cabinets anchored at base or supported top and bottom No Requirement No Requirement greater than 91m (300ft) Greater than 0.8km (1/2 mile) greater than 91m (300ft) Greater than 0.8km (1/2 mile) Greater than 1.6km (1 mile) Greater than 8km/5 miles and less than 48km (30 miles) and less than 48km/30 miles less than 48km (30 miles) less than 16km (10 miles) No Base only Fully braced Fully braced Floor loading capacity 7.2kPa 8.4kPa 12kPa (250lbf/sqsf) 12kPa (250lbf/sqsf) superimposed live load (150lbf/sqsf) (175lbf/sqft) Concrete slab thickness at ground 127mm 127mm 127mm 127mm Concrete topping over flutes for elevated floors affects size of anchor which can be installed 102mm 102mm 102mm 102 Building LFRS (Shearwall/braced frame/moment frame) indicates displacement of structure Steel/Conc MF Conc.Shearwall /Steel BF Conc.Shearwall/Steel BF Conc.Shearwall/Steel BF Building Energy Dissipation - Passive Dampers/base isolation (energy absorption) None None Passive dampers Passive dampers/base isolation ELECTRICAL Utility entrance Single feed Single feed N+1 Redundant Feed 2N Redundant Feed from different utility substations Factory Acceptance Testing Optional Optional Yes Yes Site Circuit Breaker Testing Optional Optional Contact resistance test for all critical circuit breakers in critical and essential paths, 225 Amps and higher Primary injection and Contact resistance test for all critical circuit breakers in critical and essential paths, 225 Amps and higher

49 TIA Design parameters that are affected by capital cost These are examples where TIA makes decisions that may be over ruled by cost or practicality ITEM Tier 1 Tier 2 Tier 3 Tier 4 ELECTRICAL Battery Type 5 year valve regulated lead acid 10 year valve regulated lead acid or flood type 15 year valve regulated lead acid or flood type 20 year lead acid flood type Factory Acceptance Testing Optional Optional Yes Yes Site Circuit Breaker Testing FUEL SYSTEM Optional Optional Contact resistance test for all critical circuit breakers in critical and essential paths, 225 Amps and higher Primary injection and Contact resistance test for all critical circuit breakers in critical and essential paths, 225 Amps and higher Bulk Storage tanks Single storage tank Single storage tanks Multiple storage tanks Multiple storage tanks Storage tanks, pumps Single pump and/or supply Multiple pumps and Multiple pumps and multiple Multiple pumps and multiple and piping pipe multiple supply pipes supply pipes supply pipes Generator fuel capacity 8 hrs 24 hrs 72 hrs at full load 96 hrs FIRE Fire Suppression System When required Pre-action (when required) Pre-Action (when required) Pre-Action (when required) Gaseous suppression system No No Clean agents listed in NFPA Clean agents listed in NFPA

50 TIA Non Australian references These are examples where TIA942 calls up standards that are not familiar to Australian designers. In some cases it is not possible to easily interpret these requirements ITEM Tier 1 Tier 2 Tier 3 Tier 4 Building Construction Type of construction No Restriction No Restriction Type IIA,IIIA or VA Type 1A or 1B roofing Class No Restrictions Class A Class A Class A Fire Resistive Requirements Exterior bearing walls Code requirement Code requirement 1 Hour Minimum 4 Hours Miinimum Interior bearing walls Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Exterior nonbearing walls Code requirement Code requirement 1 Hour Minimum 4 Hours Miinimum Structural frame Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Interior non-computer room Code requirement Code requirement 1 Hour Minimum 1 Hour Minimum walls Interior computer room walls Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Shaft enclosures Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Floors and Floor-ceilings Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Roofs and roof-ceilings Code requirement Code requirement 1 Hour Minimum 2 Hours Minimum Compliance with NFPA 75 No Requirement Yes Yes Yes Roofing STRUCTURE Building LFRS (Shearwall/braced frame/moment frame) indicates displacement of structure Building Energy Dissipation - Passive Dampers/base isolation (energy absorption) Conc.Shearwall/St eel BF Conc.Shearwall/Steel BF Conc.Shearwall/Steel BF Steel/Conc MF None None Passive dampers Passive dampers/base isolation ELECTRICAL Power Distribution Unit - Transformer Optional Panelboard incorporating standard thermal magnetic trip breakers K-rated or harmonic cancelling Panelboard incorporating standard thermal magnetic trip breakers K-rated or harmonic cancelling, Front accessable for maintenance Panelboard incorporating circuit breakers that are coordinated under overload and ground fault current condition K-rated or harmonic cancelling low inrush, Front accessable for maintenance Panelboard incorporating circuit breakers that are coordinated under all overcurrent conditions UPS Output Power Distribution

51 TIA Common design deficiencies (1) THE FOLLOWING LIST OF ITEMS FROM TIA REPRESENT AREAS WHERE DESIGNS ARE OFTEN DEFICIENT ITEM Tier 1 Tier 2 Tier 3 Tier 4 Ststem allowance concurrent maintenance No Down to but not including UPS Output Panelboards Down to but not including Power Distribution Units Throughout Distribution Systems Single points of failure Power Syatem Analysis Construction One or more single points of failure fo rthe distribution systems serving electrical equipment or mechanical Short Circuit Study Coordination Study Arc Falsh Analysis Panel Board with bolt on circuit breakers One or more single points of failure fo rthe distribution systems serving electrical equipment or Short Circuit Study Coordination Study Arc Falsh Analysis Switchboard with stationary circuit breakers No more than single points of failure for the distribution systems serving electrical equipment or Short Circuit Study Coordination Study Arc Falsh Analysis Load Flow Study Switchboard with drawout circuit breakers Grounding Lighting Protection System Based on Risk analysis Based on Risk analysis Yes Yes Service entrance grounds Yes Yes Yes Yes and generator grounds fully comply to NEC or local codes Data Centre grounding Optional Optional Yes Yes infrastructure in Computer room Computer Room Emergency Power Off (EPO) System Do not install if not required by Authorities having juristriction Do not install if not required by Authorities having juristriction Do not install if not required by Authorities having juristriction No single points of failure fo rthe distribution systems serving electrical equipment or mechanical systems Short Circuit Study Coordination Study Arc Falsh Analysis Load Flow Study Switchboard with drawout circuit breakers Do not install if not required by Authorities having juristriction Installation Test Mode Optional Optional Yes Yes Alarm Optional Optional Yes Yes Abort Switch Optional Optional Yes Yes Electrical Service to Mechanical Equipment Single path of Electrical power to A/C equipment Single path of Electrical power to A/C equipment Multiple paths of electrical power to AC equipment. Connected in checkerboard fashion for cooling redundancy Multiple pathsof electrical power to AC equipment. Connected in checkerboard fashion for cooling redundancy

52 TIA Common design deficiencies (2) HVAC Control System HVAC Control System Control system failure will interrupt cooling to critical area Control system failure will not interrupt cooling to critical area Control system failure will not interrupt cooling to critical area Control system failure will not interrupt cooling to critical area Power source to HVAC control system Single path of Electrical power to HVAC control system Plumbing (for Water-cooled heat rejection) Make-up Water Single water supply with no on-site storage Points of connection to condenser water system MAINTENANCE Maintenance Staff Single point of connection Redundant, UPS electrical power to BMS control Dual sources or one source + on-site storage Single point of connection Redundant, UPS electrical power to BMS control Dual sources or one source + on-site storage Two points of connection On site day shift Only. On call at other times On site day shift Only. On call at other times On site 24 hrs M-F, oncall weekends Preventative Maintenance None None Limited preventative maintenance program Facility Training Programs None None Comprehensive training program Redundant, UPS electrical power to BMS control Dual sources or one source + on-site storage Two points of connection Onsite 24 x 7 Comprehensive preventative maintenance program Comprehensive program including manual operation procedures if it is necessary to bypass control system

53 Summary (design) Establish clear and concise redundancy requirements. Use a language understood by all parties. Get a clear understanding of all stakeholder expectations including the budget and all risks. Design to meet the brief within the budget. Ensure designers understand the intended maintenance regime.

54 Summary (operation) Install, commission and operate in accordance with the design. Establish a regular failure mode testing regime. Undertake both monitored and proper routine maintenance. Ensure the end users understand the specific design principles. Ensure the Data Centre has an effective change management process.

55 Questions