Data Center Designs and Hospital Operations

Data Center Designs and Hospital Operations Designing for Data Center Availability and the Effect on Daily Operations Joe Harris, PE, RCDD Senior Engineer, AECOM Data Centers and Hospitals June 10, 2015

AECOM Healthcare Practice Industry Leader. 2014 ENR RANKING 400 Hospitals 300 Ambulatory Care Centers 200 Medical Group Practice Centers 40 Academic Medical Centers 10,000 Patient Rooms #1 DESIGN FIRM #1 PURE DESIGN #1 HEALTH CARE #1 GENERAL BUILDING Top 10 Worldwide, Modern Healthcare Survey 2014 Page 2

About Me 20+ Years Industry Experience Data Center Design across multiple industries: Healthcare Military Commercial Industrial Professional Engineer (Ohio) Registered Communications Distribution Designer Registered Telecommunications Project Manager Page 3

Learning Objectives: Understand the reliance on the data center Learn why we need to design for reliability and availability Understand how design decisions affect availability based on: Space planning Infrastructure Security concerns Services and Maintenance June 10, 2015 Data Centers and Hospitals Page 4 Client logo

Agenda Data Center Definition Hospital and Data Center Trends Availability and Reliability Design Considerations Conclusion Page 5 Client logo

Industry Acronyms TIA Telecommunications Industry Association ANSI American National Standards Institute BICSI Building Industry Consulting Service International PUE Power Usage Effectiveness ASHRAE American Society of Heating, Refrigerating, and Air- Conditioning Engineers NEC National Electric Code NFPA National Fire Protection Association NEBS Network Equipment-Building System CRAC Computer Room Air Conditioner Page 6 Client logo

Data Center Defined

The Data Center Is: Large Heat Source Mechanical Engineer Large Electrical Load Electrical Engineer The Heart of Our Network Hospital IT Staff Our Most Critical Space CIO Expensive CFO RISK! Security Director It Depends On your Viewpoint Page 8

Official Definition Data Center A building or portion of a building whose primary function is to house a computer room and its support areas 1. 1. ANSI/BICSI 002-2014 Page 9

Data Center Components Computer Room Network Core Switching Servers Storage Cabling Power Distribution Cooling Distribution (Most Critical IT Space) Support Areas Entrance Room(s) Electrical Room(s) Mechanical Rooms(s) Operations Center Crisis Response Room Maintenance Areas Support Staff Offices Loading Docks Receiving/Storing Areas Battery Rooms Telecommunications Rooms (Enables everything that happens) Page 10

Expanded (Functional) Definition Data Center A building or portion of a building whose primary function is to house a computer room and its support areas 1, designed as a set of interdependent systems operating together to create the processing and networking capabilities required to meet the current business demands. 1. ANSI/BICSI 002-2014 Page 11

Hospital and Data Center Trends

Hospital Trends Electronic Health Records More Processing More Bandwidth More Storage More Risk Big Data Electronic Health Information Exchange Radiology reports Lab results Clinical Care Summaries Medication lists F Data sharing inside and outside of the creating organization Page 13

Hospital Trends Bottom Line It s a connected world Healthcare providers rely on connected systems to provide and receive patient information Services are being delivered electronically Patient diagnostics and historical data are online Critical systems are integrated Everything Relies on the Availability of Electronic Data Page 14

Data Center Trends THE OLD DAYS (15 Years Ago) Simple Backups 24 Hour Replacement Contracts Disaster Recovery Fire, flood, tornado Checklist Management Anticipate every disaster Hot sites TODAY Tier IV = 2(N+1),.4 hour,.99.999% Worms, Hackers, Backdoors Terrorism Political Concerns Power grid failure Liability and Regulatory Compliance Cascading Events Handle unexpected disaster Fwithout downtime Page 15

Data Center General Trends Greater focus on availability and reliability Extreme power and cooling requirements Increased Focus on preventing Data Loss Consolidation of servers Virtualization Struggles with predicting growth rates Converging low-voltage systems More Focus on Efficiency and Cost of Operations PUE Dark Data Centers Page 16

Coming Together Electronic Data Bandwidth Connectivity Complexity Availability Reliability Services Efficiency Everything Relies on the Availability of Electronic Data Page 17

Availability and Reliability

What Happens When our Data Center Goes Down? We lose: Link to the outside world Server based applications Storage availability Ability to work remotely Connectivity Systems availability Reputation Everything Relies on the Availability of Electronic Data Page 19

Availability vs Reliability Reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time Availability is the probability that a component or system is in a condition to perform its intended function affected by more events than a failure requiring repair or replacement of a component or system Availability = Uptime / Total Time (within the observation interval) Page 20

Reliability Risk Metrics Failure Risks Mean Time to Failure (MTTF) Mean Time to Repair (MTTR) Susceptibility to natural disasters Fault tolerance Single Points of Failure in a system Maintainability Maintenance Programs Page 21

Availability Classifications 99.999% Availability = 5 Minutes of Downtime per Year ANSI/BICSI-002-2014 and TIA 942 5 Classes, 5 Sub groups New Industry Standard 0 = No redundancy 4 = Component AND System Redundancy Uptime Institute Tier Standards and Certifications Proprietary to Uptime Institute Tier 1 = Basic Site Tier 4 = Fault Tolerant 2N+1 Redundancy = 2 Complete Systems + 1 extra component Page 22

Tiering For Dummies TIER I/Class 0 and 1 Car without a spare tire. Stop the car and wait for it to be fixed. TIER II/Class 2 Four tires and a spare. Stop, fix the flat, drive on. TIER III/Class 3 Run Flat tires. Drive slower until you can replace the tire. TIER IV/Class 4 Change the tires, transmission, and engine while driving at 60MPH. Page 23

9 s and N s 9 s of availability Measure of uptime expressed as a percentage (i.e. 99.9%) Five 9 s = 99.999% Availability o Equates to five minutes of downtime per year o Each additional 9 adds 50% to the data center cost. N s of redundancy Redundancy applied to components, systems, services, etc. Can be UPS systems, Switches, CRACs, etc. 2N+1 = 2 times the required number + 1 extra Page 24

How Much Redundancy is EnoughE ETo Guarantee Availability? Page 25

Design Considerations

Design Considerations Space Planning Infrastructure Security Services and Maintenance Page 27

Design Considerations Space Planning Infrastructure Security Services and Maintenance Page 28

Space Planning Site Selection Where should you put the DC? Adjacencies What should be near the DC? What should not? Layout How should the DC space be arranged? Page 29

Space Planning Site Selection Stand Alone DC Location Criteria Availability of and access to reliable Telecommunications Availability of and access to reliable Power Other Utilities (Water, Sanitation, Natural Gas, Etc.) Transportation Access Property adjacencies Environmental Influences (Seismic, Flooding, Air Quality, Wind) Separation from your primary or backup data center Page 30

Space Planning Site Selection DC in a Hospital Location Criteria No patient access Not on ground floor or below Protected from outside walls Access to freight elevator Not near EMI/RFI sources Not near sensitive equipment No foreign piping systems in/over the space Away from large equipment (air handlers, MV Transformers, etc.) No Drains running through the computer room! (Avoid all water sources) Page 31

Space Planning Site Selection Inside a Hospital Public Space Secure Hospital Staff Secure IT Staff Computer Room Semi-Private Semi-Public Page 32

Space Planning Adjacencies Computer Room Control Room Data Storage Symb Adjacency Computer Room Control Room Data Storage Lobby X X Z X Y Z X Y N Lobby Z Z N X Y Z N Connected Near Away From No Relation Page 33

Space Planning Data Center Layout Work Room Secure Storage Unpacking Staging Computer Room HVAC / ELEC Computer Room Secure Corridor Control Room WAR Data Storage Telecom Entrance Room Secure Receiving Security Center IT Personnel Spaces Page 34

Design Considerations Space Planning Infrastructure Security Services and Maintenance Page 35

Infrastructure Power UPS Systems Power Distribution Cooling Computer Room Air Conditioners (CRACs) Hot (or cold) Air Containment Structured Cabling Fiber, Copper, Routing Page 36

Power Power Company Generators Uninterruptable Power Supply Automatic Transfer Switches Power Distribution Units Power Whips and Strips Power Cords Redundancy Redundancy Page 37

Cooling ASHRAE TC 9.9, ANSI/TIA-942, NEBS GR-3160 All slightly different Operational Temp Max Rate of Change Operational RH Max Dew point ANSI/TIA- 942 68ºF 77ºF 9ºF/hour 40% - 55% 69.8ºF NEBS GR- 3160 65ºF 80ºF 54ºF/Hour 55% Max 82ºF ASHRAE TC 9.9 64.4ºF 80.6ºF 9ºF/Hour 60% Max 42ºF - 59ºF Page 38

Cooling / Thermal Management The primary goal is to deliver properly cooled air, in the correct volume, directly in front of all active equipment. Page 39

Without Isolation of Return Air Path Page 40

Return Air Path Isolation Hot air is physically segregated from cold air in return air duct Rear door is sealed to prevent exhaust air from leaking into room Air Dams and filler panels in front of cabinet prevent air recirculation inside cabinet Room A/C is easy to manage, with little regard to concentrated heat loads Cabinets can be placed and oriented any way desired Cold air delivery can be shared throughout room with fewer zone issues Page 41

Cooling / Thermal Management HVAC Design should provide for Control Filtration Direct cooled equipment Avoidance of recirculation and bypass air Redundant cooling systems to meet the class design. Page 42

Cooling and Structured Cabling Heat is the #1 cause of electronic component failures Data Centers and Hospitals June 10, 2015 Page 43

Under Floor Separation of Media Page 44

Defined Cable Management Data Centers and Hospitals June 10, 2015 Page 45

Structured Cabling The fiber and copper cabling that connects all of the networked components and distribution areas of the computer room. Consists of: Fiber Single Mode and/or 50 Micron Laser Optimized Multi-Mode (OM4 Recommended) Copper Category 6A or better Proprietary Cabling (some switch manufacturers) 10 Gig backbone minimum Migration path to 40/100 gig (25 gig?) Pre-terminated cabling ORGANIZATION Page 46

Structured Cabling Areas of the Computer Room MDA Main Distribution Area Network Core and Cross Connect HDA Horizontal Distribution Area Backbone connections to serve multiple cabinets (typically end of row or middle of row) ZDA Zone Distribution Area Cross connect point similar to a consolidation point EDA Equipment Distribution Area Connections to the end equipment (servers, SAN, etc.) Page 47

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 FAN STATUS -48 V TO -60 V -40 A 1800 VA Cisco Systems, Inc. 1 2 3 4 5 6 7 8 9 INPUT OK FAN OK OUTPUT FAIL Switch must be in off "O" position to Install/Remove power supply. Fastener must be fully enaged prior to operating power supply. Power Supply 1 Power Supply 2 Catalyst 6500SERIES -48 V TO -60 V -40 A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1800 VA Cisco Systems, Inc. INPUT OK FAN OK OUTPUT FAIL Switch must be in off "O" position to Install/Remove power supply. Fastener must be fully enaged prior to operating power supply. MDA U47 U46 U45 U44 U43 U42 U41 U40 U39 U38 U37 U36 U35 U34 U33 U32 U31 U30 U29 U28 U27 U26 U25 U24 U23 U22 U21 U20 U19 U18 U17 U16 U15 U14 U13 U12 U11 U10 U09 U08 U07 U06 U05 U04 U03 U02 U01 Front View Catalyst 6500 SERIES U47 U46 U45 U44 U43 U42 U41 U40 U39 U38 U37 U36 U35 U34 U33 U32 U31 U30 U29 U28 U27 U26 U25 U24 U23 U22 U21 U20 U19 U18 U17 U16 U15 U14 U13 U12 U11 U10 U09 U08 U07 U06 U05 U04 U03 U02 U01 INSTALL RUN Rear View Fiber Termination INSTALL RUN Includes the core switching and main cross connect for the data center Hub of cabling infrastructure for the Data Center Has backbone cabling to all HAD s and TR s Page 48

HDA HDA A RACK #1 RACK #2 RACK #3 RACK #4 RACK #5 RACK #6 HDA B FIBER PATCH PANEL A 48 UTP PATCH PANEL A SIDE (28 ACTIVE) 48 UTP PATCH PANEL A SIDE (28 ACTIVE) 48 UTP PATCH PANEL A SIDE (28 ACTIVE) 48 UTP PATCH PANEL A SIDE (28 ACTIVE) 48 UTP PATCH PANEL A SIDE (28 ACTIVE) 48 UTP PATCH PANEL A SIDE (28 ACTIVE) FIBER PATCH PANEL ROUTE B UTP PATCH PANEL #1 48 UTP PATCH PANEL B SIDE (28 ACTIVE) 48 UTP PATCH PANEL B SIDE (28 ACTIVE) 48 UTP PATCH PANEL B SIDE (28 ACTIVE) 48 UTP PATCH PANEL B SIDE (28 ACTIVE) 48 UTP PATCH PANEL B SIDE (28 ACTIVE) 48 UTP PATCH PANEL B SIDE (28 ACTIVE) UTP PATCH PANEL #1 UTP PATCH PANEL #2 24 UTP PATCH PANEL KVM & MGMT 24 UTP PATCH PANEL KVM & MGMT 24 UTP PATCH PANEL KVM & MGMT 24 UTP PATCH PANEL KVM & MGMT 24 UTP PATCH PANEL KVM & MGMT 24 UTP PATCH PANEL KVM & MGMT UTP PATCH PANEL #2 UTP PATCH PANEL #3 UTP PATCH PANEL #3 UTP PATCH PANEL #4 UTP PATCH PANEL #4 UTP PATCH PANEL #5 UTP PATCH PANEL #5 UTP PATCH PANEL #6 UTP PATCH PANEL #6 UTP PATCH PANEL #7 UTP PATCH PANEL #7 UTP PATCH PANEL KVM & MGMT RACKS 1 & 2 SERVER SPACE SERVER SPACE SERVER SPACE SERVER SPACE SERVER SPACE SERVER SPACE UTP PATCH PANEL KVM & MGMT RACKS 7&8 UTP PATCH PANEL KVM & MGMT RACKS 3 & 4 UTP PATCH PANEL KVM & MGMT RACKS 9 &10 UTP PATCH PANEL KVM & MGMT RACKS 5 & 6 UTP PATCH PANEL KVM & MGMT RACKS 11 & 12 SWITCH SPACE SWITCH SPACE Contains the horizontal cross connect Distribution point for the equipment distribution areas Has horizontal cabling to all EDA s Typically contains switching equipment to support the EDA Page 49

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 ZDA 12 12 11 11 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 Connection point between HAD and EDA Contains no active electronics Limited to 288 connections Can be installed in cabinet, under floor, over head, etc. Comparable to Consolidation point located in the Work Area in 568 2 2 1 1 Page 50

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 EDA U47 U46 U45 U44 U43 U42 U41 U40 U39 Front View U47 U46 U45 U44 U43 U42 U41 U40 U39 Rear View Fiber Termination Area allocated for the end equipment Includes horizontal cable and connections to the HAD U38 U37 U36 U35 U34 U38 U37 U36 U35 U34 Point-to-point cabling permitted U33 U32 U31 U30 U29 U28 U27 U26 Space for Optional Keyboard & Monitor U33 U32 U31 U30 U29 U28 U27 U26 Comparable to the location of end equipment in 568 U25 U25 U24 U24 U23 U23 U22 U22 U21 U21 U20 U20 U19 U19 U18 U18 U17 U17 U16 U16 U15 U15 U14 U14 U13 U13 U12 U12 U11 U11 U10 U10 U09 U09 U08 U08 U07 U07 U06 U06 U05 U05 U04 U04 U03 U03 U02 U02 U01 U01 Page 51

Structured Cabling Areas of the Computer Room Page 52

Structured Cabling Redundancy Page 53

Design Considerations Space Planning Infrastructure Security Services and Maintenance Page 54

Security Three Main Components Architectural Electronic Operational Page 55

Data Center Security Accreditations? Courtesy Altus Technologies Page 56

Security Architectural Landscaping Lighting Doors/Locks/Glazing Building features that contribute to security Electronic Access Control Intrusion Detection CCTV Duress Operational Security Staffing Post Orders Guard Tours Policies and procedures Contingency Planning Page 57

Information Security (InfoSec) Confidentiality Availability Integrity The best configured firewall in the world can t stand up to a well placed brick. Page 58

Design Considerations Space Planning Infrastructure Security Services and Maintenance Page 59

Services and Maintenance Testing Periodically throughout the life of the system Include backup systems Scheduled for off peak times System Maintenance includes Cabling Electrical HVAC IT Equipment Security Page 60

Services and Maintenance Cleaning is critical and often overlooked All spaces (above and below) Part of maintenance contract Data Centers and Hospitals June 10, 2015 Page 61

Services and Maintenance An insufficient maintenance plan will result in extended downtime. Regularly scheduled maintenance must be carefully planned and performed by persons familiar with the entire system and all inter-dependencies Page 62

Conclusion

Design for Availability Hospitals require a high level of data availability to operate efficiently Data center designs need to focus on systems availability By designing for availability in the data center, we can help ensure hospital operations are not interrupted due to lack of electronic services and data Everything Relies on the Availability of Electronic Data Page 64

Thank You Joe.harris@aecom.com Data Centers and Hospitals June 10, 2015