Planning, Designing and Installing a Pre-Terminated Copper and Optical Cabling System for the Data Center
Presenters Charlie Bogolawski, RCDD, CDT Director of Technical Sales, Leviton Danbury, CT Bo Conrad, RCDD Director of Project Management, E2Optics Ashburn, VA
House Cleaning items Please quiet all phones and loud conversations Please hold all questions until Q+A section Please raise your hand when you have a question
Discussion on Planning Planning Definition of a Data Center Roles and Responsibilities Project Delivery Methods The Architecture The Transport Technologies Migration Strategy
The Design Discussion on Design Performance Standards and Codes Pathways and Spaces Overhead Under Floor Media Copper Fiber Patching Environment Interfaces Density
Discussion on Installation The Installation Timelines and Milestones Field and/or Factory Terminated False sense of security Performance Warranties and Guaranties Contract Requirements
Define a Data Center
Is this a Data Center?
A Data Center is A place that stores electronic information that is mission critical to my business. A place that provides telecommunications access to my electronic information without any delay or interruption. A place that protects my electronic information. A place that helps reduce my business cost and gets me closer to my customers.
What is being asked of a Data Center? 1. Improve application reliability 2. Align closer with business needs 3. Reduce complexity 4. Reduce cost Get me to my volumes of information faster, better and reduce my cost. Anywhere and anytime I need it.
Planning
The Owner Principle stake holder, End User and Operator of Data Center Provide program for Data Center utilization Issues Purchase Orders for The Work and authorizes Project Change Orders
The Owner Owner s Project Team should consist of not only IT personnel but also liaisons from every single Business Unit (BU) within the Organization. Every BU needs to be a stake holder along with IT for success of project. IT should be asking - What are the short term and what are the long term business requirements for each business unit. Know and communicate the requirements often.
The Owner s analysis Size of facility Build or Co-Lo Application performance needs Service Level Agreements (SLAs) Servers Storage Cabling Infrastructure Power Cooling
The General Contractor Is issued and holds the Primary Contract to build / construct the Data Center and its systems for power, cooling, cabling, and Service Provider connections for the Owner. Has a Team of in-house or sub-contracted experts to help fulfill all details of the contractual obligations of the Contract.
The Low Voltage Contractor Constructs and/or assembles a Data Center cabling system that will support various transport technologies as specified by electrical and/or optical performance characteristics and specifications as outlined in the Construction Documents, also known as The Work. Contractors need to have their Work deemed acceptable by Owner before payment of rendered services/products.
The Manufacturer Manufactures, as well as, assembles cabling systems and components that meet performance requirements that are specified in construction documents. Ensures tolerances and quality of finished goods are produced as expected and in a timely manner.
The Distributor Facilitates the logistics of materials to ensure specified products are delivered to the LV Contractor. Provides credit for project builds, materials purchase / return and maintains inventory.
Project Delivery Methods Advantages / Dis-advantages
Project Delivery Method Design-Bid-Build Architect / MEP / Consultant Owner General Contractor Sub- Contractor
Project Delivery Method Design-Build Owner Architect / MEP / Consultant General Contractor Sub- Contractor
Linear track toward construction Design-Bid-Build More time up front for design development and bid analysis Changes required to do The Work is slow Design-Build Fast track build Changes can be done quickly Need very strong management
The Architecture
TIA 942 A / Traditional "This content, from the draft ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
TIA 942 A / Fat-tree "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
TIA 942 A / Fat tree w/port extenders "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
TIA 942 A / Fat tree Pods "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
TIA 942 A / Full-mesh "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
TIA 942 A / Interconnected mesh "This content, from the ANSI/TIA-942-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
TIA 942 A / Centralized "This content, from the draft ANSI/TIA-942-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
TIA 942 A / Virtual switch "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
Transport Technologies
Current and Emerging Technologies over Single mode Application Ethernet 1000BASE-LX Nomimal Wavelength Channel Attenuation Channel Distance 1310 nm 4.5 db 5000 meters (16405 feet) Fiber Type TIA 492CAAA OS1 Supports Fiber Type TIA 492CAAB OS2 Supports Ethernet 10GBASE-LX4 1310 nm 6.3 db 10000 meters (32810 feet) Supports Supports Ethernet 10GBASE-L 1310 nm 6.2 db 10000 meters (32810 feet) Supports Supports Fibre Channel 100-SM-LC-L (1062 Mbaud) 1310 nm 7.8 db 10000 meters (32810 feet) Supports Supports Fibre Channel 200-SM-LC-L (2125 Mbaud) 1310 nm 7.8 db 10000 meters (32810 feet) Supports Supports Fibre Channel 400-SM-LC-L (4250 Mbaud) 1310 nm 7.8 db 10000 meters (32810 feet) Supports Supports Fibre Channel 1200-SM-LL-L (10512 Mbaud) 1310 nm 6.0 db 10000 meters (32810 feet) Supports Supports
Current and Emerging Technologies over Single Mode continued Application FDDI SMF-PMD ANSI X3.184 Nomimal Wavelength 40GBase-LR4 1264.5 to 1277.5 1284.5 to 1297.5 1304.5 to 1317.5 1324.5 to 1337.5 100GBase-LR4 1294.53 to 1296.59 1299.02 to 1301.09 1303.54 to 1305.63 1308.09 to 1310.19 100GBase-ER4 1294.53 to 1296.59 1299.02 to 1301.09 1303.54 to 1305.63 1308.09 to 1310.19 Channel Attenuation Channel Distance 1310 nm 10 db 10000 meters (32810 feet) 6.7 db 10000 meters (32810 feet) 6.3 db 10000 meters (32810 feet) 18 db 40000 meters (131234 feet) Fiber Type TIA 492CAAA OS1 Supports Supports Supports Supports Fiber Type TIA 492CAAB OS2 Supports Supports Supports Supports 10x10MSA 1521 to 1525 1529 to 1533 1537 to 1541 1545 to 1549 1553 to 1557 1561 to 1565 1569 to 1573 1577 to 1581 1585 to 1589 1593 to 1597 2.6 db 5 db 2000 meters (6562 feet) 10000 meters (32810 feet) Supports Supports Supports Supports 40GBase-FR 1550 serial? 2000 meters (6562 feet) Supports Supports
Current and Emerging Technologies over Multimode Application Nomimal Wavelength Channel Attenuation Channel Distance Fiber Type TIA 492AA AA OM1 Fiber Type TIA 492AA AB OM2 Fiber Type TIA 492AA AC OM3 Ethernet 10 / 100 BASE-SX Ethernet 100BASE-FX Ethernet 1000BASE-SX Ethernet 1000BASE-LX Ethernet 10GBASE-S Ethernet 10GBASE-LRM 850 nm 4.0 db 300 meters (984 feet) 1300 nm 11.0 db OM1 6.0 db OM2 6.0 db OM3 850 nm 2.6 db OM1 3.6 db OM2 4.5 db OM3 2000 meters (6560 feet) varies 1300 nm 2.3 db 550 meters (1804 feet) 850 nm 2.4 db OM1 2.3 db OM2 2.6 db OM3 varies 1300 nm 1.9 db 220 meters (720 feet) Supports Supports Supports Supports Supports Supports Supports up to 275m (900 feet) Supports up to 500m (1804 feet) Supports up to 800m (2625 feet) Supports Supports Supports Supports up to 33m (108 feet) Supports up to 82m (269 feet) Supports up to 300m (984 feet) Supports Supports Supports Ethernet 10GBASE-LX4 1300 nm 2.5 db OM1 2.0 db OM2 2.0 db OM3 300 meters (984 feet) Supports Supports Supports
Current and Emerging Technologies over Multimode continued Application Fibre Channel 100-MX-SN-I (1062 Mbaud) Fibre Channel 200-MX-SN-I (2125 Mbaud) Fibre Channel 400-MX-SN-I (4250 Mbaud) Fibre Channel 1200-MX-SN-I (10512 Mbaud) 16G Fibre Channel 1600-MX-SN (10512 Mbaud) Nomimal Wavelength Channel Attenuation 850 nm 3.0 db OM1 3.9 db OM2 4.6 db OM3 850 nm 2.1 db OM1 2.6 db OM2 3.3 db OM3 850 nm 1.8 db OM1 2.1 db OM2 2.5 db OM3 850 nm 2.4 db OM1 2.2 db OM2 2.6 db OM3 850 nm 1.6 db OM2 1.9 db OM3 Channel Distance varies varies varies varies Fiber Type TIA 492AA AA OM1 Supports up to 300m (984 feet) Supports up to 150m (492 feet) Supports up to 70m (230 feet) Supports up to 33m (108 feet) Fiber Type TIA 492AA AB OM2 Supports up to 500m (1640 feet) Supports up to 300m (984 feet) Supports up to 150m (492 feet) Supports up to 82m (269 feet) varies Not supported Supports up to 35m (115 feet) Fiber Type TIA 492AA AC OM3 Supports up to 860m (2822 feet) Supports up to 500m (1640 feet) Supports up to 270m (886 feet) Supports up to 300m (984 feet) Supports up to 100m (328 feet) FDDI PMD ANSI X3.166 1300 nm 11.0 db OM1 6.0 db OM2 6.0 db OM3 2000 meters (6560 feet) Supports Supports Supports
Current and Emerging Technologies over Multimode continued Application Nomimal Wavelength Channel Attenuation Channel Distance Fiber Type TIA 492AA AA OM1 Fiber Type TIA 492AA AB OM2 Fiber Type TIA 492AA AC OM3 Fiber Type TIA 492AA AD OM4 40GBASE-SR4 850 nm 1.9 db OM3 1.5 db OM4 100GBASE-SR10 850 nm 1.9 db OM3 1.5 db OM4 Varies No support No support Supports up to 100m (328 feet) Varies No support No support Supports up to 100m (328 feet) Supports up to 150m (492 feet) Supports up to 150m (492 feet) 4 x 25 Gigabit TBD TBD Varies No support No support TBD TBD
Current and Emerging Technologies over Copper Application Media Distance IEEE 802.3i 10BASE-T IEEE 802.3u 100BASE-T IEEE 802.3ab 1000BASE-T IEEE 802.3ak 10GBASE-CX4 IEEE 802.3an 10GBASE-T SFF-8431 SFP+ Fiber Channel / Ethernet IEEE 802.3ba 40GBASE-CR4 100GBASE-CR10 Category 3 twisted pair or better Category 5 twisted pair or better Category 5e twisted pair or better Twinaxial Category 6A twisted pair or better Twinaxial Shielded balanced twisted pair 100m (328 feet) 100m (328 feet) 100m (328 feet) 15m (49 feet) 100m (328 feet) 7m (23 feet) 7m (23 feet)
Migration Strategy
Market Drivers - Multi-Gigabit Adoption Rates Source: IDC
Market Drivers Network Port Forecast By 2015, 100G will account for 25% of network ports Source: Infonetics Research
40/100 Gigabit Architecture Source: Brocade
100G Line-Side Equipment Available Brocade MLX Series Juniper T1600 Series Cisco CRS-3 Series
40G Client-Side Equipment Available Extreme Networks Summit TOR & Black Diamond Core Switch Cisco Nexus 3064 TOR Switch Blade Networks/IBM G8264 TOR Switch
40G Client-Side Equipment Available Force 10 S4810 TOR & E-Series Core Switch Arista 7050 TOR Switch Hitachi Apresia Switch
Extreme Networks 40G Architecture
GBICs / SFP
Equipment Interfaces 40G CFP QSFP 100G CFP CXP CXP Transceiver CFP Transceiver
Migration strategies need to account for: Performance of existing 10G networks Channel Insertion Loss Budget Channel Distance Connector Insertion Loss System Polarity Reduced fiber strand loss Design flexibility for equipment architecture Four 40 Gig Ports per Edge switch
The Design
The Design Discussion on Design Performance Standards and Codes Topology Pathways and Spaces Overhead Under Floor Determining cable length Media Copper Fiber Patching Environment
Codes, Standards, References
Codes International Fire Code NFPA 70: National Electrical Code, 2011 NFPA 70E: Standard for Electrical Safety in the Workplace, 2012 NFPA 72: National Fire Alarm and Signaling Code, 2010 NFPA 101: Life Safety Code, 2012 NFPA 232: Standard for the Protection of Records, 2012 NFPA 75: Standard for the Protection of Information Technology Equipment, 2009 NFPA 76: Standard for the Fire Protection of Telecommunications Facilities, 2012
Standards TIA Common, Premise, & Component Standards TIA 942 A BICSI 002 Data Center Design and Best Practices NECA / BICSI 607 Grounding and Bonding Standard BICSI 009 Building Information Modeling BICSI ITSIMM 6 th Edition BICSI TDMM 12 th Edition IEEE P802.3ba EN 50600-2-2
ASHRAE References ASHRAE 62.1 Ventilation for Acceptable Indoor Air Quality 2007 ASHRAE Best Practices for Datacom Facility Energy Efficiency 2009 ASHRAE Design Considerations for Data and Communication Equipment Centers 2009 ASHRAE Gaseous and Particulate Contamination Guidelines for Data Centers 2009 ASHRAE Structural and Vibration Guidelines for Datacom Equipment Centers 2008 ASHRAE Thermal Guidelines for Data Processing Environments 2009
Understand the components of the performance standard you are trying to attain Common Standards Premise Standards Component Standards ANSI/TIA-568-C.0 Generic Telecommunications Cabling for Customer Premises ANSI-TIA-568-C.1 Commercial Building Telecommunications Cabling Standard ANSI-TIA-568-C.2 Balanced Twisted-Pair Telecommunications Cabling and Components Standard TIA-569-B Commercial Building Standard for Telecommunications Pathways and Spaces ANSI-TIA-570-B Residential Telecommunications Infrastructure Standard ANSI-TIA-568-C.3 Optical Fiber Cabling Components Standard ANSI-TIA-606-A Administration Standard for Commercial Telecommunications Infrastructure ANSI-TIA-758-A Customer-Owned Outside Plant Telecommunications Infrastructure Standard ANSI-TIA-568-C.4 Broadband Coaxial ANSI-TIA-607-B Telecommunications Grounding (Earthing) and Bonding for Customer Premises ANSI-TIA-942 Telecommunications Infrastructure Standard for Data Centers ANSI-TIA-862 Building Automation Systems Cabling Standard for Commercial Buildings ANSI-TIA-1005 Telecommunications Infrastructure Standard for Industrial Premises ANSI-TIA-1179 Infrastructure Standard for Healthcare Facilities
Codes, Standards References continued IEEE determines and establishes transmission protocols/methods and component performance for network transport systems to support communication reliably. TIA determines component, link and channel performance. They also specify testing methods to validate performance. BICSI develops standards with a mix of best practices
Topology
Topology Key items Flexibility Scalability Modularity Manageability Network Architecture Independent
TIA 942 Topology "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
Direct Connect no structure
Structured Discrete connections
Structured with MTP
Bring the server connections back to MDA / HDA
Using fiber out to server rows 1U 48 Channel panel Port 1-24 Port 1-24 48 strand 48 strand P65 3/26/2014
Own the MDA
Conveyance
Conveyance Systems Allow maximum flexibility Allow for Growth / Contraction Minimal disruption Mechanical Systems Power Distribution Systems Maintenance, Adds and Changes Consider ALL Code, Standard, Ratings, Technology Performance Accessibility Long Term Management
Conveyance Design Considerations TIA-942 mandates minimum ceiling height of 8.5ft AFF to allow for 7 foot racks and cabinets Also mandates minimum clearance of 18 of clearance above pathways (for building systems) Potential limitation If frequent changes expected consider using trunk cables to overhead ( zero-u ) patch panels Design around life safety, mechanical systems What will the impacts be to air flow
If you go overhead, tier it!!!
Keep it simple!
Underfloor plan it right
Additional Considerations Are there any Seismic Restrictions / Requirements Can overhead be supported structurally Does a floor mounted raised pathway support system work Multi-Tier Conveyance Systems Media segregation Tennant asset separation Is there space available above / below floor
Overhead or Underfloor Overhead Pro Lower Construction Cost Easier MAC Easier to Trace cabling Separates from high voltage pathways (if under floor) Con Must accommodate cabinet height variations Not all ceilings can support Design around other building systems Requires ladders etc. for MAC work Raised Floor Presents clean appearance More secure pathways (limits access and tampering) Greater path flexibility Potential airflow disruption Limited room for cabling Higher Construction Cost (when paired with flooring)
Pathway systems Ladder Rack Cable Tray Optical Trough Pro Greater structural support Higher material cost Ease of tiering Lower installation cost Lower material cost Vestal elevation changes Resists environmental variables Better protects fiber installations Con Limited feature set Higher installation cost Difficult to tier High installation cost Limited growth capacity
Media
Considerations Equipment Costs Cable Fill Power Consumption Scalability / Flexibility Life Cycle
Copper What is the Application What is the life cycle Environmental conditions What if it could be scaled with the DC IS COPPER DEAD?
The Categories TIA classification of cabling & components: Category 1 is specified up to 1 MHz. Category 2 is specified up to 4 MHz. Category 3 is specified up to 16 MHz. Category 4 is specified up to 20 MHz. Category 5/5e is specified up to 100 MHz. Category 6 is specified up to 250 MHz. Category 6A is specified up to 500 MHz. Category 7 not specified Category 8 to be developed up to 2 GHz possibly. IEC/ISO classifications of cabling & components: Class A is specified up to 100 khz. Class B / Category 1 is specified up to 1 MHz. Class C / Category 3 is specified up to 16 MHz. Class D / Category 5 is specified up to 100 MHz. Class E / Category 6 is specified up to 250 MHz. Class EA / Category 6A is specified up to 500 MHz. Class F / Category 7 is specified up to 600 MHz. Class FA / Category 7A is specified up to 1000 MHz. (under development) Note: Class = Cabling / Category = Components
Fiber What is the application What is the life cycle Density Cost Analysis SM vs. MM Polarities
Connector Types
MTP End-Face Geometry Verification
Loss Budgets Fatal mistakes Total Loss = 4.8dB
Loss Budgets Multiple hops Total Loss = 2.0dB
Loss Guarantees on pre-term
ANSI/TIA-568C.0-2009 page 25 "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
ANSI/TIA-568C.0-2009 page 26 "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
ANSI/TIA-568C.0-2009 page 27 "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
ANSI/TIA-568C.0-2009 page 28 "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
ANSI/TIA-568C.0-2009 page 29 "This content, from the ANSI/TIA-942 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association."
Q & A Session