Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-942

Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-942 Chris DiMinico cdiminico@ieee.org MC Communications

Data Center Standards ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers Co-chairs: Chris DiMinico & Jonathan Jew Published 2005 available through TIA at www.tiaonline.org ANSI/NECA/BICSI-002 Data Center Design and Implementation Best Practices co-chairs: chairs: Jonathan Jew & John Kacperski best best practices complements TIA-942 2007 target

Who Developed TIA-942? Developed by the TIA TR-42.1.1 Network Distribution Nodes subcommittee as Project No. 3-00923 Participants included: Architecture & Engineering Firms Consultants End End Users Manufacturers

Purpose of TIA-942 Encourage early participation of telecom designers in data center design process Fill a void by providing standards for planning of data centers, computer rooms, server rooms, and similar spaces. The standard encompasses much more than just telecommunications infrastructure. Close to half of the technical content deals with facility specifications.

Purpose of TIA-942 Define a standard telecommunications infrastructure for data centers Structured cabling system for data centers using standardized architecture and media Accommodates a wide range of applications (LAN, WAN, SAN, channels, consoles) Accommodates current and future protocols (e.g., 10+ GbE ) Replaces unstructured point-to to-point cabling that uses different cabling for different applications

Purpose of TIA-942 Specifications for data center telecommunications pathways and spaces Recommendations on media and distance for applications over structured cabling Establish a standard for data center tiers to replace several proprietary standards. The TIA data center tier standard is: A tool to evaluate existing data centers A tool to communicate design requirements

I8 I7 I6 I5 I4 I3 I2 I1 FILE CABI NET FI L E FILE FILE AT &T DEFINIT Y UNIT Unstructured Cabling A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS Heater - A/C Heater - A/C Heater - A/C 1 2 3 4 5 6 7 8 9 10 11 Heater - A/C POMONA A/C UNIT #8 TABLE DESK DESK CAB INE T CA BIN ET TAPE RACK DASD DASD CABINE T STORAGE NE TWORK TAPE RACK NE TWORK NE TWORK S-390 R34 TABLE 247-7589 H8 H7 H6 H5 H4 H3 H2 H1 TABLE G7 G6 G5 G4 G3 G2 G1 POMONA A/C UNIT #7 F6 F5 F4 F3 F2 LAN F1 E7 E6 E5 E4 E3 E2 E1 CAB 3746-900 3745-17A SAA D2 SAA D1 3745-A11 3745-41D 3745-L13 POMONA A/C UNIT #6 C ABINET C6 C ABINET C5 C ABINET C4 C ABINET C3 C ABINET C2 C ABINET C1 CD ROM SCANNER B7 B6 B5 B4 B3 B2 B1 HP PR IN TER C ABINET A6 C ABINET A5 A4 C ABINET A3 C ABINET A2 PAC BELL A1 CI SC O ROU TE R CONS OLE HALON PAN EL F/X 247-7543 HALON ABORT HALON R EL EASE 12 13 14 15 STOR AGE 247-7538 TAPE RACK AS -400 AS-400 INSTALL A CABLE WHEN YOU NEED TABLE IT (SINGLE-USE, UNORGANIZED CABLING) STORAGE LIEBERT PDU-C STORAGE 3174 3174 3174 3174 3174 3174 LOCKER 16

I8 I7 I6 I5 I4 I3 I2 I1 FILE CABI NET FILE FIL E FILE AT&T DEFINI TY UNIT Structured Cabling A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS Heater - A/C Heater - A/C Heater - A/C 1 2 3 4 5 6 7 8 9 10 11 Heater - A/C POMONA A/C UNIT #8 TABLE DESK STORAGE DESK NE TWORK CAB INE T NE TWORK CA BIN ET NE TWORK CAB IN ET TAPE RACK TAPE RACK CA BINET CA BINET S-390 DASD DASD R34 NE TWORK IDF IDF Mainframe IDF TABLE 247-7589 H8 H7 H6 H5 H4 H3 H2 IDF H1 TABLE G7 CABIN ET G6 G5 CABIN ET G4 CABIN ET G3 CABIN ET G2 CABIN ET IDF G1 POMONA A/C UNIT #7 IDF FIBER MDF & CORE F6 F5 CABIN ET F4 CABIN ET F3 CABIN ET F2 LAN F1 COPPER MDF E7 E6 E5 E4 E3 E2 E1 3745s CAB 3745-A11 3746-900 3745-41D 3745-17A 3745-L13 SAA D2 SAA D1 POMONA A/C UNIT #6 C6 C5 C4 C3 C2 IDF C1 IDF CD ROM HP SCANN ER PRINTER B7 B6 B5 B4 B3 B2 IDF B1 IDF A6 A5 A4 A3 A2 PAC BELL A1 CISCO ROUTER CONSOLE HALON PANEL F/X 247-7543 HALON ABORT HALON RELEASE 12 13 14 15 STORAGE 247-7538 TAPE RACK AS-400 AS-400 STRUCTURED CABLING SYSTEM (ORGANIZED, TABLE REUSABLE, FLEXIBLE CABLING) LIEBERT STORAGE PDU-C STORAGE 3174 3174 3174 3174 3174 3174 LOCKER 16

Design Elements Cabling Design Facility Design Network Design Informative annex s: Provide best practices Annex A - Cabling Design Considerations Annex B-B Telecommunications infrastructure administration Annex C-Access C provider information Annex D-D Coordination of equipment plans with other engineers Annex E-E Data center space considerations Annex F-F Site selection

Design Elements Cabling Design: Copper and fiber cabling performance Connectors, cables, distribution hardware Cabling distances Space management Facility Design: Data center sizing Power distribution methodologies Pathways and spaces HVAC, security, operations, and administration. Flexibility, scalability, reliability and space management

Design Elements Network Design: Support of legacy systems Enable rapid deployment of new and emerging technologies such as 10 GbE and 10+ GbE copper and fiber applications.

Relationship of Spaces BUILDING SITE BUILDING SHELL GENERAL OFFICE SPACE TELECOM ROOMS & EQUIPMENT ROOMS for spaces outside data center OFFICE BUILDING SUPPORT SPACE DATA CENTER SUPPORT STAFF OFFICES ENTRANCE ROOM(S) DATA CENTER ELECTRICAL & MECHANICAL ROOMS OPERATIONS CENTER TELECOM ROOM(S) for data center support spaces STORAGE ROOMS & LOADING DOCKS COMPUTER ROOM

Data Center Topology Entrance Room Entrance Facility Carriers Active Equip interconnect Carriers Offices, Operations Center, Support Rooms Secondary Entrance Room Active Equip interconnect MC Active Equip interconnect Carriers Carriers Back bone Cabling IC HC IC IC IC HC HC HC interconnect Active Equip Storage Area Devices interconnect Active Equip Horizontal Cabling interconnect Active Equip Zone Distribution interconnect Active Equip Data Center

TIA-942 Spaces Entrance Room (ER) - location of interface with campus and carrier entrance facilities Main Distribution Area (MDA) location of main cross-connect connect (MC) Horizontal Distribution Area (HDA) location of horizontal cross-connect connect (HC) Zone Distribution Area (ZDA) location of zone outlet (ZO) or consolidation point (CP) Equipment Distribution Area (EDA) location of equipment cabinets and racks

Data Center Cabling MC Main Distribution Area Horizontal Cabling HC IC Backbone Cabling IC HC Horizontal Distribution Area interconnect Zone Distribution Area interconnect Active Equip interconnect Active Equip Active Equip Equipment Distribution Area Horizontal cabling is the cabling from the horizontal cross-connect (in the main distribution area or horizontal distribution area) to the outlet in the equipment distribution area or zone distribution area.

Horizontal and Backbone Cabling Recognized Cables: a) 100-ohm twisted-pair cable (ANSI/TIA/EIA-568- B.2), category 6 recommended (ANSI/TIA/EIA- 568-B.2-1) b) multimode optical fiber cable, either 62.5/125 micron or 50/125 micron (ANSI/TIA/EIA-568-B.3), 50/125 micron 850 nm laser optimized multimode fiber is recommended (ANSI/TIA-568-3-1) c) singlemode optical fiber cable (ANSI/TIA/EIA- 568-B.3) d) 75-ohm (734 and 735 type) coaxial cable (Telcordia Technologies GR-139-CORE)

Horizontal cabling distances The maximum horizontal distance is 90 m independent of media type. The maximum channel distance including equipment cords is 100 m. The maximum cabling distance in a data center not containing a horizontal distribution area is: 300 m for an optical fiber channel including equipment cords. 100 m for copper cabling including equipment cords.

Backbone Cabling Includes cabling from MDA to ER, HDA Optional cabling between HDAs allowed Maximum backbone cable lengths depend on applications to be supported Centralized optical fiber cabling supported with interconnect, splice, or pull-through at the HDA Star topology with no intermediate cross- connects Various topologies permit redundancy and flexibility to support various data center sizes

Computer Room & Entrance Room Requirements Min clear height of 2.6m/8.5 ft Min door size 1m/3ft wide 2.13/7ft high Min dist floor loading 7.2 kpa/150lbf/ft2, recommended min 12 kpa/250 lbf/ft2 20 degrees C to 25 degrees C 40% to 55% relative humidity (reduces ESD) Any sprinkler systems must be pre-action system Common bonding network (CBN) equipotential ground reference

Main Distribution Area Location of Main Cross-Connect Connect (MC), the central point of distribution for data center structured cabling system Centrally located to avoid exceeding maximum distance restrictions (typically for E-E 1s, E-3s, E T-1s T and T-3s) T Install separate racks for Fiber, UTP, and coaxial cable distribution

Horizontal Distribution Area Location of Horizontal Cross-Connect Connect (HC), the distribution point for cabling to equipment distribution area Distribution LAN, SAN, KVM switches and console servers located in HDA MDA may also include an HC for nearby equipment distribution area Number of HDAs depends on the density of cabling and the size of the data center

Horizontal Distribution Area The capacity of the cable tray system and the size of the cross-connect connect creates practical limits on the size of the HC Guideline is maximum of 2,000 4-pair 4 UTP or coax cable terminations per HDA Arrange patch bays to minimize patch cable lengths and to simplify cable management Separate Separate racks for fiber, UTP, and coax Locate Locate switches and patch panels to minimize patch cord lengths

Zone Distribution Area Rack, cabinet, or under floor enclosure that houses a zone outlet (ZO) or consolidation point (CP) ZO - structured cabling termination for floor-standing equipment that cannot accept patch panels (e.g. mainframes and large servers). CP - intermediate termination point (e.g. cabling to areas where floor plan is uncertain or dynamic) No cross-connects connects within the ZDA No active equipment shall be located in the ZDA Maximum of 144 connections in a ZDA Maximum of one ZDA within a horizontal cable run

Equipment Cabinets Front rails of cabinets must be recessed to provide adequate room for patch cables and wire managers Recommend 1-to-1 ratio of patching to cable management Arrange switches and patch panels to minimize patching between cabinets & racks Perforated tiles at front of cabinets One edge of cabinets placed at edge of tile

Raised Floor Better appearance than overhead cabling. Allows higher power densities, better control of cooling, and more flexibility in location of cooling equipment Most stand-alone alone computer systems are designed for cabling from below Coordinate under floor cabling with mechanical & electrical engineers Recommend wire basket cable trays in hot aisles for telecom cabling

Example of Wire Basket Cable Trays For Cabling Under Raised Floor 24" RAISED FLOOR TILE 1.375" 9" 4" DIVIDER 18" 5" COPPER FIBER CABLING SUPPORT STRUT 1" 36" 12" 4" 5" FIBER 18" COPPER CABLING SUPPORT STRUT 1" Open Area 13.5" 24" On Center

Overhead Cable Trays Less expensive than raised floor systems Cable trays can be attached to the top of racks and cabinets (if they are uniform in height) Cable trays suspended from the ceiling provides more flexibility for supporting cabinets/racks of various heights and for adding and removing cabinets/racks. Cable trays can be installed with several layers Coordinate location with lighting, ducts, overhead conduits, overhead power distribution

Overhead Cable Tray Example 3 Layer cable tray system: Bottom layer signal Middle layer power Top layer fiber Signal Reference Grid in brackets attached to lower layer of trays Fiber patch cables may be in fiber duct attached to threaded rods

Infrastructure Administration Informative annex with TIA-606 606-A A standards compliant labeling scheme for all components. Labeling scheme extended for use in data centers Cabinets and racks labeled by location using tile grid or row/position identifiers All cabinets, racks, patch panels, cables, and patch cords should be labeled

Labeling Example "X" COORDINATE AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS AT AU AV AW AX AY AZ BA BB BC BD 01 02 03 Cat 6 x 24 04 05 06 07 08 RIGHT FRONT CORNER AJ05 AQ03 09 10 11 12 13 "Y" COORDINATE Patch Panel for 24 Cat 6 from 1 st Panel in Cab AJ05 to 2 nd Panel in Cab AQ03 AJ05-A to AQ03-B Ports 01-24 A 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 A

Site Selection Informative annex with guidelines for selection of a site for a data center Architectural ElectricalElectrical Mechanical Telecommunications SecuritySecurity OtherOther

Facilities Specifications & Tiers Informative annex with general architectural, structural, electrical, mechanical, and telecommunications recommendations requirements Annex includes detailed architectural, security, electrical, mechanical, and telecommunications recommendations for each Tier Recommended specifications by tier are a uniform way to rate aspects of a data center design and are a starting point for initiating design requirements with qualified architects and engineers

Data Center Tiers Tier 1 basic data center no no redundancy Tier 2 redundant components Single Single distribution path with redundant components Tier 3 concurrently maintainable Multiple Multiple distribution paths with only one active Tier 4 fault tolerant Multiple Multiple active distribution paths

Redundant Topologies Carriers Carriers Carriers Carriers Primary Customer Maintenance Hole (Tier 1 and higher) Secondary Customer Maintenance Hole (Tier 2 and higher) Data Center Tier 1 Tier 3 Tier 2 Tier 3 Primary Entrance Room (Tier 1 and higher) Tier 3 Secondary Entrance Room (Tier 3 and higher) Offices, Operations Center, Support Rooms Tier 1 Tier 4 Tier 3 Tier 4 Telecom Room Main Dist Area (Tier 1 and higher) Tier 4 Secondary Dist Area (Optional for Tier 4) Computer Room Horiz Dist Area Optional Cabling Zone Dist Area Horiz Dist Area Horiz Dist Area Equip Dist Area Equip Dist Area Equip Dist Area

Data Center Tiers Higher tiers correspond to higher availability, but also have higher construction costs Data Center can have different tier ratings for different portions of its infrastructure (architectural, security, mechanical, electrical, telecommunications) The overall rating for the data center is equal to the lowest tier rating Capacity of systems may need to be upgraded to maintain tier rating as data center load increases Human error and operating procedures have a major impact on availability

Conclusion TIA-942 is the first standard to specifically addresses data center infrastructure. Primarily a telecom infrastructure standard, but about half of the content deals with facility requirements. Provides a flexible and manageable structured cabling system using standard media. Builds on existing standards, when applicable Guidelines on a wide range of subjects useful to someone designing or managing a data center. An official tiering standard for determining the quality of a center. A way to objectively compare one center with another.