Data Centre Link Survey Alan Flatman Principal Consultant LAN Technologies, UK a_flatman@compuserve.com Contribution to IEEE 82.3ba Higher Speed Ethernet Task Force 23-25 Jan 8, Portland, OR
Survey Objectives 1. establish length distribution requirements for Higher Speed Ethernet links 2. establish link speed & connectivity requirements 3. focus on recent data centre designs/installations 4. provide greater granularity for the first m of horizontal/client links than existing studies 5. assess level of standards compliance for cabling 6. examine differences to existing cabling studies
Flatman Cabling Surveys for IEEE 82.3 July 1999 In-premises Optical Fibre Deployment to Jan 3 Installed Copper Cabling Forecast to end 5 July 3 Installed Horizontal Cabling Length Distribution (office & medium-sized data centre) Mar 4 In-premises Optical Fibre Installed Base to 7
Data Centre Network Infrastructure Storage Storage Servers Clients Access LAN/NAS/SAN switches Distribution Routers/L3 switches Core Gateways/Firewalls
Data Centre Cabling Standards EIA/TIA-942: Data Centre Cabling» published in 5 CENELEC EN 5173-5: Data Centre Cabling» approved early 7 ISO/IEC 24764: Data Centre Cabling» based on CENELEC EN 5173-5» approval anticipated in 8
ISO/IEC 24764: Data Centre Cabling ENI MD ZD LDP (optional) EO Equipment ENI network access cabling subsystem main distribution cabling subsystem zone distribution cabling subsystem equipment cabling ENI = Equipment Network Interface MD = Main Distributor ZD = Zone Distributor LDP = Local Distribution Point EO = Equipment Outlet
ISO/IEC 24764: Data Centre Cabling Example of MD to ZD to EO implementation showing equipment interfaces
Standard Data Centre Cabling Models C Patch cord C Patch Panel Crossconnect Permanent Link Equipment Equipment Equipment cord Equipment cord Channel C C Patch Panel Crossconnect Patch cord C Patch cord C Patch Panel Crossconnect Permanent Link Equipment Equipment Equipment cord Equipment cord Channel C C Permanent Link Equipment Equipment Equipment cord Equipment cord Channel C
Data Centre Size Classification IDC (6) identifies 4 different types, distinguished by size: Small Data Centre ~15, ft 2 raised floor 35-5 volume servers 1-3 high end servers Medium Data Centre ~, ft 2 raised floor 1,5-1,7 volume servers 4-5 high end servers Large Data Centre ~35, ft 2 raised floor 2,-2,5 volume servers 6-7 high end servers Very Large Data Centre >1, ft 2 raised floor <25, volume servers >8 high end servers
Oct 7 Data Centre Cabling Survey Summary 9 enterprise data centres from US, UK, Germany total data centre floor space was 715, sq. feet 56k servers, 25k switch ports, 9k patch ports small, medium, large, v. large sizes (IDC classes) v. large had 74% links large had % links medium had 5% links small had 1% links data centre cabling mostly standards compliant some servers had direct connections to top of rack switches (i.e. no patch cords/patch panels involved)
Data Centre Link Speed the next 5 years 7 (by site) 1M 1G 1G G 1G Client-to-Access Links Access-to-Distribution Links Distribution-to-Core Links 78% 1% 11% % % % 8% 6% % % 1% 7% 8% % % 1 (by site) 1M 1G 1G G 1G Client-to-Access Links Access-to-Distribution Links Distribution-to-Core Links 11% 1% 78% % % % 7% 9% % 1% % 6% % 1% % 12 (by site) 1M 1G 1G G 1G Client-to-Access Links Access-to-Distribution Links Distribution-to-Core Links 11% 78% 8% % 1% % % 8% % % % % 8% % 6%
Enterprise Data Centre Cabling Lengths Client-to-Access Permanent Links 1 All V.Large Large Medium Small 9 Total of 25, links 8 7 % Links 6 5 1 91-1m 81-9m 71-8m 61-7m 51-6m 41-5m 31-m 21-m 11-m -1m
Enterprise Data Centre Cabling Lengths Client-to-Access Permanent Links 1 All V.Large Large Medium Small 9 Total of 25, links 8 7 % Links 6 5 1 91-1m 81-9m 71-8m 61-7m 51-6m 41-5m 31-m 21-m -m
Enterprise Data Centre Cabling Lengths Client-to-Access Permanent Links 1 All V.Large Large Medium Small 9 Total of 25, links % Links 8 7 6 5 1 good correlation for first m 91-1m 81-9m 71-8m 61-7m 51-6m 41-5m 31-m -m
Prior Data Centre Cabling Survey (IEEE 82.3an DiMinico_1_113) Note: This is Permanent Link data 7 Data Centres Total = 325, sq. feet
Prior Data Centre Cabling Survey (IEEE 82.3an DiMinico_1_113) Note: This is Permanent Link data 11 Data Centres Total = 448, sq. feet
Enterprise Data Centre Cabling Lengths Client-to-Access Permanent Links 1 Flatman All DiMinico J+M average DiMinico M+A average 9 8 7 % Links 6 5 1 91-1m 81-9m 71-8m 61-7m 51-6m 41-5m 31-m -m
Enterprise Data Centre Cabling Lengths Client-to-Access Permanent Links 1 Flatman All DiMinico J+M Av. DiMinico M+A Av. Cumulative % Links 9 8 7 6 5 1 Shorter length distribution for Flatman survey may be due to shift from pedestal to rack servers over 5 years (period between 2 surveys) 1 5 6 7 8 9 1 Length (m)
Enterprise Data Centre Cabling Lengths Client-to-Access Permanent Links 1 Flatman All Flatman V.Large Flatman Large Flatman Medium Flatman Small DiMinico J+M Av. DiMinico M+A Av. 9 8 Cumulative % Links 7 6 5 1 1 5 6 7 8 9 1 Length (m)
Enterprise Data Centre Cabling Lengths Client-to-Access Permanent Links & Channels Flatman All Permanent Links Flatman All Channels 6 % Links 5 used to calculate channel lengths Average Perm Link = 15.8m Average Cordage = 4.4m Minimum Cordage = 3.m Maximum Cordage = 9.m Average Channel =.2m Min # Cords/Channel = 1. Max # Cords/Channel = 4. Avg # Cords/Channel = 1.6 1 91-1m 81-9m 71-8m 61-7m 51-6m 41-5m 31-m 21-m 11-m -1m
Enterprise Data Centre Cabling Lengths Client-to-Access Channels 1 9 8 Flatman All 82.8% Total of 25, links Cumulative % Links 7 6 5 1 33.3% 1 5 6 7 8 9 1 Length (m)
276-m 251-275m 226-25m 1-225m 176-m 151-175m 126-15m 11-125m 76-1m 51-75m 26-5m -25m 45 35 25 15 1 5 Enterprise Data Centre Cabling Lengths Access-to-Distribution Permanent Links All V.Large Large Medium Small Total of 16, links % Links
Prior Data Centre Cabling Study (IEEE 82.3 HSSG Kolesar_1_96)
Prior Data Centre Cabling Study (IEEE 82.3 HSSG Kolesar_1_96)
Prior Data Centre Cabling Study (IEEE 82.3 HSSG Kolesar_1_96) 45 35 Optical Fibre Permanent Links; assumed to be switch-to-switch. Derived from cumulative length distribution & converted into m. % Links 25 15 1 5 smoothed tail 276-m 251-275m 226-25m 1-225m 176-m 151-175m 126-15m 11-125m 76-1m 51-75m 26-5m -25m
Prior Data Centre Cabling Study (IEEE 82.3 HSSG Swanson_1_116) Note: Data based on several 1 s Corning trunk cable shipments, simulation designed to represent actual data centre deployments (using concatenated trunks) 5% links have 2 trunks % links have 3 trunks % links are point-point
Prior Data Centre Cabling Study (IEEE 82.3 HSSG Swanson_1_116) 276-m 251-275m 226-25m 1-225m 176-m 151-175m 126-15m 11-125m 76-1m 51-75m 26-5m -25m 45 35 25 15 1 5 Optical Fibre Permanent Links; assumed to be switch-to-switch. Derived from cumulative length distribution. % Links
276-m Enterprise Data Centre Cabling Lengths Access-to-Distribution Permanent Links Flatman All Kolesar Swanson 251-275m 226-25m 1-225m 176-m 151-175m 126-15m 11-125m 76-1m 51-75m 26-5m -25m 45 35 25 15 1 5 % Links
Enterprise Data Centre Cabling Lengths Access-to-Distribution Permanent Links Flatman All Kolesar Swanson 1 9 8 Cumulative % Links 7 6 5 1 25 5 75 1 125 15 175 225 25 275 Length (m)
Enterprise Data Centre Cabling Lengths Access-to-Distribution Permanent Links 1 9 Flatman All Flatman V.Large Flatman Large Flatman Medium Flatman Small Kolesar Swanson 99% links Cumulative % Links 8 7 6 5 1 1% links 25 5 75 1 125 15 175 225 25 275 Length (m)
Enterprise Data Centre Cabling Lengths Access-to-Distribution Permanent Links & Channels Flatman All Permanent Links Flatman All Channels 45 % Links 35 25 used to calculate channel lengths Average Perm Link = 46.1m Average Cordage = 1.4m Minimum Cordage = 3.m Maximum Cordage = 24.m Average Channel = 56.5m Min # Cords/Channel = 2. Max # Cords/Channel = 4. Avg # Cords/Channel = 2.7 15 1 5 276-m 251-275m 226-25m 1-225m 176-m 151-175m 126-15m 11-125m 76-1m 51-75m 26-5m -25m
Enterprise Data Centre Cabling Lengths Access-to-Distribution Channels Flatman All 1 9 8 89.% Total of 16, links Cumulative % Links 7 6 5 1 25 5 75 1 125 15 175 225 25 275 Length (m)
Enterprise Data Centre Cabling Lengths Distribution-to-Core Permanent Links All V.Large Large Medium Small 276-m 251-275m 226-25m 1-225m 176-m 151-175m 126-15m 11-125m 76-1m 51-75m 26-5m -25m 1 9 8 7 6 5 1 Total of 3, links % Links
Enterprise Data Centre Cabling Lengths Distribution-to-Core Permanent Links & Channels Flatman All Permanent Links Flatman All Channels % Links 6 5 used to calculate channel lengths Average Perm Link = 37.6m Average Cordage = 11.1m Minimum Cordage = 4.m Maximum Cordage = 12.m Average Channel = 48.7m Min # Cords/Channel = 2. Max # Cords/Channel = 4. Avg # Cords/Channel = 2.1 1 276-m 251-275m 226-25m 1-225m 176-m 151-175m 126-15m 11-125m 76-1m 51-75m 26-5m -25m
Enterprise Data Centre Cabling Lengths Distribution-to-Core Channels Flatman All Cumulative % Links 1 9 8 7 6 5 1 84.8% Total of 3, links 25 5 75 1 125 15 175 225 25 275 Length (m)
Data Centre Cabling Market Research detailed data centre cabling analysis by BSRIA research conducted in the USA during fall 7 assesses deployment trends 7, 8 & 9 report planned to be published 31 January 8 see Alan Flatman for prospectus & other details