10G and Beyond in the Data Center with OM3 Fiber Doug Coleman, Manager Technology and Standards doug.coleman@corning.com
Overview Optical Trends in Data Center Networks The time is NOW for 10G optical connectivity 10GBASE-SR compared to 10GBASE-T A 10G Link Connectivity Comparison What s next? Ethernet Fibre Channel Infiniband MTP Connectivity Solutions For Today and the Future Summary
Data Center Environment - Today Higher Speeds Higher Density Higher Reliability Lower CAPEX Lower OPEX
Optical Trends in Data Center Networks Standards have evolved to meet increasing bandwidth requirements with cost-effective solutions Standards Roadmap 10Gb/s required new OM3 fiber specifications and measurement methods Data rate (mb/s) 100000 10000 1000 100 10 1 Below 10 Gb/s, application standards used then-current multimode fibers to design network solutions 10M Ethernet Token Ring FDDI 1985 100M Fast Ethernet ATM LEDs 1G GbE Fiber Channel Lasers 2G FC 10G 1995 1999 2002 10 GbE Sonet/SDH InfiniBand 4G FC 20/40G InfiniBand 8G FC 40/100G IEEE We expect next step in data rates to primarily use current fiber specifications 2005 2010
Optical Trends in Data Center Networks Multimode Fiber Nomenclature TIA-568 Rev C adoption
Optical Trends in Data Center Networks (=< 300m) Expect lower-cost solutions in data center networks will remain at 850 nm 850 nm VCSELs have won the 1G premises market 850 nm 10G VCSELs just entering high-volume manufacturing cycle Will continue to be low-cost solution for 10G 1300 nm solutions will capture some market share in premises with legacy systems (10GBASE-LX4 and10gbase-lrm) Relative Cost 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0 10G Transceivers 850 nm optics 1300 nm optics 2004 2005 2006 2007 2008 2009
Optical Trends in Data Center Networks 80% 70% 60% 50% 40% 30% 20% OM3 OM2 OM1 SMF 10% 0% 2004 2005 2006 2007 Source: CCS
Optical Trends in Data Center Networks Laser-Optimized 50 Micron Fiber (OM3) Core Size: Attenuation: Bandwidth: 50 Micron 3.0/1.5 db/km @ 850/1300 nm OFL 1500/500 MHz km @ 850/1300 nm EMBc 2000 MHz km @ 850 nm Distance: Gigabit Ethernet 1000/600 meters @ 850/1300 nm Serial 10 Gigabit Ethernet: 300 meters @ 850 nm CWDM 10 Gigabit Ethernet: 300 meters @ 1300 nm
The Time Is NOW for 10G Optical Connectivity 10 Gigabit Ethernet 10GBASE-S Multimode fiber, serial transmission at 850 nm Lowest cost for new installs (=<300/550 m) Data centers and building/campus backbones 10GBASE-LX4 Multimode or single-mode fiber, WWDM transmission in the 1300 nm region (300 m) Multimode fiber solution intended for legacy systems 10GBASE-L Single-mode fiber, serial transmission at 1300 nm (10 km) Campus backbones 10GBASE-E Single-mode fiber, serial transmission at 1550 nm (30-40 km) Metro area networks
The Time Is NOW for 10G Optical Connectivity 10 Gigabit Ethernet 10GBASE-CX4 Twin axial coax cable (15m) 10GBASE-LRM Multimode fiber, EDC, FDDI fiber, 1300 nm, 220 m Multimode fiber solution intended for legacy systems
The Time Is NOW for 10G Optical Connectivity!!!!! 10GBASE-S optical connectivity is the choice solution when compared to 10GBASE-T copper connectivity Premier transmission performance Data rate scalability Pathway and space utilization Electronics port density, power and cooling efficiencies Ease of installation and testing
The Time Is NOW for 10G Optical Connectivity!!!!! Copper Cable Raw Material Costs Increasing Raw Materials Copper $ 3.86/lb (3/31/08) Hydrocarbon base plastic raw materials $100+ per barrel of oil Copper $$$/pound Copper $$$/pound Drives higher copper connectivity cable $$$$ Optical fiber cable prices trending down 4 day average 9 day average 18 day average
The Time Is NOW for 10G Optical Connectivity!!!!! Lower the total cost of network ownership Fiber optic cabling is less expensive to operate Less power consumption Decreased cooling requirements Smaller transceiver size Higher electronic port densities Fiber ~1-4 W 10 Gb/s Operating Cost Fiber v. Copper Power Consumption Cooling Requirements Transceiver Size Copper ~8-15W Some estimates blame up to 60% of all data center downtime on heat-related issues -- IDC $ Data Center Area $$$$
The Time Is NOW for 10G Optical Connectivity!!!!! Bandwidth headroom for what s now and what s next Design & Installation Complexity versus Bandwidth Cost and Complexity Bandwidth Demand and Time Copper Fiber Optics Copper cabling Increasingly complex engineering and installation to keep up with bandwidth demands Fiber optic cabling is trending the other way Becoming easier and easier to install while effortlessly keeping up with bandwidth demand
The Time Is NOW for 10G Optical Connectivity 10GBASE-T Transmission Issues Operates across 500 MHz frequency spectrum Increased insertion loss Extensive data encoding and signal processing required to achieve acceptable BER 10-12 Complex electronic digital signal processing (DSP) for internal noise impairments Contributes significantly to inherent time delay known as latency External noise such as alien cross-talk and EMI cannot be corrected with electronics due to random nature
The Time Is NOW for 10G Optical Connectivity 10GBASE-T Transmission Issues 10G optical has 1000 times better latency performance than copper Typical 10G optical physical (PHY) latency in the nanosecond range (10-9 sec.) Typical10G copper PHY latency in microseconds (10-6 sec.) A one-millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm, Information Week, April 2007 Copper s Road LATENCY!!! Fiber s Road
The Time Is NOW for 10G Optical Connectivity 10GBASE-T Electronics Issues Significant switch power requirements 10G copper 10-15 watts per port Major silicon chip development required to reduce power Expect 3-4 watts to be lowest achievable power level per port independent of distance 10G optical switches 1-4 watts per port Significant server adapter card power requirements 10G copper 25 watts per server adapter card 30m service distance 10GBASE-SR optical <9 watts per server adapter card 300m service distance
The Time Is NOW for 10G Optical Connectivity 10GBASE-T Power Requirements High power requirements result in higher generated heat, contributing to higher cooling needs EPA states that for every KW of electronics power, an equal unit of power is required for cooling Higher power requirements result in higher CO2 emissions 1.6 lbs CO2 per KW-Hr The result is significantly higher energy costs operation and cooling -- with 10GBASE-T electronics Yearly Energy Cost Total Electronics and Cooling Energy Cost and Savings Comparison for 10GBASE-SR and 10GBASE-T $80,000 $70,000 $60,000 $50,000 $40,000 $30,000 $20,000 $10,000 $0 76% 82% 84% 84% 85% 86% 48 96 144 192 240 288 Port Count 87% 85% 83% 81% 79% 77% 75% 73% 71% 69% Copper Switch Fiber Switch % Energy Reduction OPTICAL CONNECTIVITY Enables The Green Data Center!!! Energy Savings
The Time Is NOW for 10G Optical Connectivity 10GBASE-T Reduced Switch Port Density Projected 4-8 ports per 10G copper card Maximum 100m distance Low density drives need for more line cards and chassis, driving up power and space utilization 10G optical switch density X2: 16 ports, XFP: 36 ports, SFP+ 48 ports per line card 10GBASE-SR 300m distance One Fiber Line Card = 48 Ports Six Copper Line Cards = 48 Ports
The Time Is NOW for 10G Optical Connectivity Emerging Optical Electronics 10G Transceiver Trends Move signal processing from module to the line card Reduce module size Reduce power Reduce price Source: Intel Source: CMP Media XENPAK & X2 XFP SFP+
The Time Is NOW for 10G Optical Connectivity 10GBASE-T Pathway and Space Issues Larger cable OD and heavier 1 CAT6A UTP -- 0.354 inch diameter and 46 lbs/1000ft weight A 216-fiber ribbon cable, 0.76 inch OD, 200 lbs/1000ft 108 circuits at 200ft 108 CAT6A cables, 1000lbs, typical effective diameter 5.0 inch 216-fiber optical cable, 40 lbs, effective diameter 0.76 inch
Let s Build a Link! Scenario: CAT6A UTP Copper and 10G Electronics Scenario: 108 10G Copper Ports This will require 6U of rack space per rack dedicated to patch panels 14 copper 10G line cards per rack, with a maximum port line card density of 8 Next, 108 jumpers per rack And finally, 108 CAT6A UTP cables to connect it all (at 200 ft, that s 1000 lbs!) Cross-section of 12 x4 cable tray (to scale) 40% filled
Now a 10G Optical Link! Scenario: OM3 Fiber and 10G Electronics Scenario: 108 10G Optical Ports This will require 1U of rack space per rack dedicated to patch panels 3 optical 10G line cards per rack with a maximum port line card density of 48 Next, 18 MTP -to-single-fiber connector harnesses per rack And finally, 1 216-fiber cable to connect it all (at 200 ft, that s 40 lbs!) Cross section of 12 x4 cable tray (to scale) 2% filled
Let s Compare: 10G Copper UTP Network v. 10G Optical Network CAT6A UTP 10G Network Electronics operating power OM3 10G Network Electronics operating power 96,500 kw-hr per year 20,100 kw-hr per year $14,184 per year / rack $2,950 per year / rack Electronics cooling power Electronics cooling power 96,500 kw-hr per year 20,100 kw-hr per year $14,184 per year / rack $2,950 per year / rack Total power cost Total power cost $28,369 per year / rack $5,910 per year / rack Total CO2 emissions Total CO2 emissions 155 tons 32 tons 79% overall energy savings!
Ethernet What s Next? IEEE 802.3 HSSG Approved Motions 40 and 100 Gbps At least 100 m on OM3 multimode fiber At least 10 km on single-mode fiber At least 40 km on single-mode fiber (100G only) At least 10 m on copper cable assembly Key project dates Study group formed in July 2006 Project authorization in December 2007 Task force formed in January 2008 100G standard complete mid 2010
Ethernet What s Next? Recent history suggests that standards (and initial fiber sales) will lead actual port sales by ~3 years Given port sale historical trends, we can project initial applications ~2011 Most applications not expected until >2013 Ethernet and LAN Port Sales (MM units) 700 600 500 400 300 200 100 0 1985 1988 10Mb/s Standard 100Mb/s Standard 1Gb/s Standard 1991 1994 1997 2000 10Gb/s Standard Year 2003 2006 100Gb/s Standard 2009 2012 2015 '100Gbps' 10Gbps 1Gbps 100Mbps 10Mbps 10Mb-10Gb data from Dell Oro
Premises Cable Market Length Distribution of Cable Supplied to Customers 100 100% Relative Frequency 80 60 40 20 80% 60% 40% 20% Cumulative Frequency 0 0 50 100 150 200 250 Cable Length(m) 0% Length Distribution Cumulative Frequency
Ethernet What s Next? Several possible optical solutions are all currently being discussed in IEEE Parallel CWDM DSP Hybrid MTP Optical Connectivity
40G Ethernet Parallel Optics 12F MTP Interface
QSFP 40G Optical Transceiver Uses standard ribbon fiber cable with MTP Connector Source: Zarlink
100G Ethernet Parallel Optics 24F MTP Or Two 12F MTP Interface Source: Sumitomo
Fiber Skew Certain optical cables and terminations may not be suitable for parallel optics applications IEEE 40/100G skew requirement not defined to date. InfiniBand QDR 0.75ns skew requirement now exists CCS has performed skew testing demonstration compliance to 0.75 ns up to 300m Need to understand skew capability today for existing and future cable deployments Skew = difference in propagation time between lanes of a parallel transmission system. Δ propagation speed or distance Infiniband Skew Requirements:
SAN Fibre Channel Road Map Product Naming Throughput (MBps) Line Rate (GBaud) T11 Spec Technically Completed (Year) Market Availability (Year) Base2 Parallel Optics 1GFC 200 1.0625 1996 2GFC 400 2.125 2000 4GFC 800 4.25 2003 8GFC 1600 8.5 2006 16GFC 3200 17 2009 32GFC 6400 34 2012 64GFC 12800 68 2016 128GFC 25600 136 2020 1997 2001 2005 2008 2011 Market Demand Market Demand Market Demand
SAN Fibre Channel Road Map Product Naming Throughput (MBps) Line Rate (GBaud) T11 Spec Technically Completed (Year) Market Availability (Year) Base10 10GFC 2400 10.52 2003 2004 20GFC 4800 21.04 2007 2008 40GFC 9600 42.08 TBD Market Demand Parallel Optics 80GFC 19200 84.16 TBD Market Demand 160GFC 38400 168.32 TBD Market Demand
Fiber Channel over Ethernet (FCoE) Activity initiated at T11 Fiber Channel, April 2007 Encapsulate Fiber Channel Packet into a Ethernet Frame Lossless Packet Performance Supports utilization of low cost Ethernet electronics up to the SAN switch FCoE Server Adapter Card FCoE Gateway Line Card Designed to operate at 10G Large data centers focus Standard completion mid-2008 Commercial products 2009
InfiniBand Applications HPC Supercomputers Financial data center focus Electronic trading and algorithm modeling High BW with low latency Media Types Optical fiber Multimode and single-mode fiber MTP Connectivity Twin axial copper cable Factory-terminated only 15-20m distance capability Not adequate for structured wiring
InfiniBand Architecture: Server Area Network (HPC)
InfiniBand Data Rates Parallel Optics!!!
InfiniBand Electronics
Optical Media Converter Copper to optical media converter (OMC) Converts a power-enabled InfiniBand copper port to an optical port Uses standard ribbon fiber cable with MTP Connector Data rate specific (4X-SDR,DDR) Perfect match to fixed-port CX4 socket Emcore OMC Active optical cable (Zarlink, Intel) Integrated media converter Specified distances (10, 25, 50 100 m) Data rate specific (4X-SDR,DDR) Perfect match to fixed-port CX4 socket Zarlink
QSFP Optical Transceiver 2008 Hot pluggable 12 Fiber MTP Interface Data rate specific (4X SDR, DDR, QDR (10G, 20G, 40G)) Zarlink, Luxtera, QLOGIC and other manufacturers Source: Zarlink
MTP Connectivity Solutions in the Data Center Server Racks Server Main Distribution Area (MDA) Server Ports Switch Ports Storage Switch Racks Storage Ports
Ribbon Cable High-density ribbon design in a small-form-factor package 12-fiber ribbons Data center and LAN backbones MTP connectivity 12 fibers connectorized simultaneously Riser and plenum flame ratings Interlocking armor 96-Fiber Cable with eight MTP Connectors
MTP Connectorization Factory or field termination Terminate 12 fibers in one step with MTP connectors Expedites cable installation and MACs Minimizes errors Reduces congestion in patch panels Footprint similar to SC Connector
Ribbon Cable Field Termination No-Epoxy/No-Polish Connector TIA/EIA 604-5 (FOCIS) Installs in less than 4 minutes Fiber type OM1 62.5/125 µm (beige housing) OM2 50/125 µm (black housing) OM3 LOMMF (aqua housing) SMF OS2 (green housing)
MTP Connector Modules Used to break out the 12-fiber MTP connectors terminated on trunk cables into simplex or duplex style connectors SC, LC, ST compatible MDA cross-connect and EDA interconnect Support easy migration to parallel optics
Main Distribution Area Housing Back Panel PnP training4
Main Distribution Area Housing Front Panel LAN490
MTP Connector Harnesses Hardware interconnection to backbone cable Break out 12-fiber MTP connector into simplex or duplex connectors Transitions plug & play system MTP connector trunks straight into electronics Support easy migration to parallel optics
MTP Connector Harness SAN Director Termination
The Time Is NOW for Optical Connectivity Silicon and electronic industry focused on optical solutions 10GBASE-SR OM3 fiber optical connectivity is the choice solution when compared to 10GBASE-T copper connectivity OM3 fiber supports migration to established Fibre Channel, Ethernet, and InfiniBand roadmap high data rates MTP Connectivity Solutions available today for legacy serial and future parallel optics transmission schemes 1G, 10G, 16G 40G 100G
Contact Info Doug Coleman E-mail: doug.coleman@corning.com Phone: 828-901-5580 Fax: 828-901-5488 Address: 800 17th Street NW Hickory, NC 28601