100 GIGABIT ETHERNET TECHNOLOGY OVERVIEW & OUTLOOK Joerg Ammon <jammon@brocade.com> Netnod spring meeting 2012
Agenda Industry trends 100GE technology Outlook 2
Traffic growth NetNod:'Yearly' graph (1 Day Max) Graph source: NetNod Aggregate traffic graphs Stockholm (Max) 3
Bandwidth Capacity Demand in ISP Networks ISP bandwidth requirements are driven by increased broadband access through wired and wireless infrastructure to always on devices The killer app is not a single application, it is ubiquitous high bandwidth delivering on-demand personalized content Businesses and consumers want cheap, always on, high speed pipes for the applications of their choice on any device 4
Agenda Industry trends 100GE technology Outlook 5
Scale Beyond 10-Gigabit Ethernet Using Link aggregation R2 R5 R1 R4 R7 R3 R6 100 Gigabit Ethernet is available, but sometimes not (yet) applicable OC-768 POS for many providers is an unaffordable alternative ECMP with LAG is a more affordable way of scaling transit Load balancing algorithm efficiency is critical Path diversity is a must when ECMP and LAG are used Must transport multiple types of traffic L2, IPv4, IPv6, MPLS 6
Scale Beyond 10-Gigabit Ethernet Effective load Sharing on MPLS systems LSP Ingress PE Transit LSR Egress PE MPLS link L2 MPLS labels IP ETH payload L3/L4 L2/L3 At Ingress PE (packets entering a MPLS LSP): Load shares IP packets (IP/MPLS, L3VPN, IPv4/v6 in VPLS/VLL) using L2/L3/L4 headers At transit LSR (Checks first nibble after bottommost label): If 4/6, load shares using MPLS link L2/LSP Label/VC label/payload(l3/l4) headers Else, load shares using MPLS link L2/LSP Label/VC label/payload(l2/l3) headers At Egress PE (packets exiting a MPLS LSP): Load shares IP packets with no Ethernet header (IP/MPLS, L3VPN) using MPLS link L2/Label1/Label2/Payload(L3/L4) headers Load shares IP packets with Ethernet header (IPv4/v6 in VPLS/VLL) using MPLS link L2/Label1/Label2/Payload(L2/L3) headers 7
10, 40 and 100Gigabit Ethernet developments Port densities increase several hundred Gbps/slot for multiple ports 10 GbE prices continue to fall IEEE 802.3ba standard approved June 17, 2010 457 pages added to IEEE 802.3-2011 Shipping 1 st generation media, test equipment, router interfaces, and optical transport gear in 2010/2011 2 nd generation technology projects for both 40 and 100 GbE have started Expected on the market in 2012-2013 8
Current State of the Industry Fundamental 1 st generation technology constraints limits higher 100 GbE density and lower cost Electrical signaling inside the box 100 Gbps Attachment Unit Interface (CAUI) uses 10 x 10 Gbps Optical signaling outside the box 10x10 MSA: 10 x 10 Gbps 100GBASE-LR4 / 100GBASE-ER4: 4x25Gbps CFP module size and power consumption 145 mm long 10 18 in 2 Electrical 82 mm wide Optical 10 25 100 GbE CFP image courtesy of Finisar. 9
Current IEEE standards have a gap 100GBASE-SR10 supports up to 150 m on OM4 MMF 100GBASE-LR4 supports up to 10 km on SMF Members 100GBASE-LR4 100 GbE optics are very complex and expensive 10x10 MSA bridges the gap Support for 2 km, 10 km and 40 km on SMF Considerably more economical Eliminate expensive components Consume lesser power Network operator members! www.10x10msa.org 10
100Gigabit Ethernet in the media Production traffic (Atrato at AMS-IX) http://www.ams-ix.net/atrato-takes-the-lead-with-100gbe-ports-at-ams-ix/ SuperComputing 2011 100 G Networks (217km) Source: http://supercomputing.uvic.ca/ Memory to Memory Transfers showing 4 servers receiving data at an aggregate speed greater than 98 Gb/s (top right corner). We also see a total bi-directional data flow of 186 Gb/s 11
Pushing 100Gigabit of data 4 x SFP+ Media 4 x SFP+ Media 100GE CFP Media 100GE CFP Media PHY PHY PHY PHY 40 G NP 40 G NP 100 G NP 100 G NP True 100 Gbps elements 40 G TM 40 G TM 100 G TM 100 G TM Effective hashing Something to get the data across to another line card 12
Agenda Industry trends 100GE technology Outlook 13
100 Gbps Module Evolution Speeds vs Size Current Future 10 25 Electrical 10 25 25 Optical CFP CFP2 CXP QSFP28 Diagram courtesy of Molex. 14
IEEE Ethernet Standards Timelines 8 years between 10 GbE and 100 GbE standards IEEE needs to start immediately in order to finish a new Ethernet speed standard (400 GbE?) by 2016 400 GbE? Ethernet Speed 100 10 1 0,1 0,01 FE 3 GbE 10 GbE 4 8 100 GbE 6+? Year of Standards Release 15
What problems does 100 GbE solve? Enables network operators to deploy high capacity networks with Terabit-scale bandwidth Helps scale link aggregation (LAG) and equal cost multi-path (ECMP) Capacity Manageability Hashing Large flow distribution Has positive industry side effects Lower cost and higher density GbE and 10 GbE Higher bandwidth enables new applications we haven t imagined yet 16
Thank You
100 GbE Technology Reference 1 st Generation IEEE 1 st Generation 10x10 MSA 2 nd Generation IEEE Physical Layer Reach >1 m Backplane >5 m Copper Cable 7 m Copper Cable 100 m OM3/ OM4 MMF 100 m OM3, 150 m OM4 MMF <2? km SMF 2 km SMF 10 km SMF 40 km SMF Name 100GBASE -KR4 100GBASE -CR4 100GBASE -CR10 100GBASE -SR4 100GBASE -SR10 100GBASE -FR4 10x10-2k m 10x10-10k m 100GBASE -LR4 10x10-40k m 100GBASE -ER4 Standard Status Future IEEE 802.3bj Future IEEE 802.3bj 2010 IEEE 802.3ba Possible Future IEEE 2010 IEEE 802.3ba Possible Future IEEE March 2011 10x10 MSA August 2011 10x10 MSA 2010 IEEE 802.3ba August 2011 10x10 MSA 2010 IEEE 802.3ba Generation 2 nd 2 nd 1 st 2 nd 1 st 2 nd 1 st 1 st 1 st 1 st 1 st Electrical Signaling (Gbps) 4 x 25 4 x 25 10 x 10 4 x 25 10 x 10 4 x 25 10 x 10 10 x 10 10 x 10 10 x 10 10 x 10 Media Signaling (Gbps) 4 x 25 4 x 25 10 x 10 4 x 25 10 x 10 4 x 25 10 x 10 10 x 10 4 x 25 10 x 10 4 x 25 Media Type Backplane Twinax Twinax MPO MMF MPO MMF Duplex SMF Duplex SMF Duplex SMF Duplex SMF Duplex SMF Duplex SMF Media Module Backplane QSFP28, CFP2, CFP4 CXP QSFP28, CFP2, CFP4 CXP, CFP QSFP28, CFP2, CFP4 CFP CFP CFP CFP CFP Availability 2014 2014 2010 2013+ 2010 2013+ Q1 2011 Q3 2011 2010 (CFP2 in 2013+) Q3 2011 2012 18
82 mm wide 100 GbE CFP Modules C (100) Form-factor Pluggable New module optimized for 100 GbE long reach applications Used for 40GBASE-SR4, 40GBASE- LR4, 100GBASE-SR10, 100GBASE- LR4, 100GBASE-ER4, and 10x10 MSA Complex electrical and optical components need a large module Large module form factor and power consumption limits front panel density (larger than an iphone 4) 145 mm long 118 cm 2 14 mm high CFP image courtesy of Finisar. 19
Recent 100 GbE Developments 10x10 MSA finished several projects Up to 26 members including AMS-IX, Facebook and Google Initial 10x10-2km standard published in March, 2011 Additional 10x10-10km and 10x10-40km standards finished in August, 2011 2nd generation projects based on 4 x 25 Gbps electrical signaling have started New IEEE P802.3bj 100 Gb/s Backplane and Copper Cable Task Force was started in September, 2011 100GBASE-KR4: 4 x 25 Gbps over >1 m backplane 100GBASE-CR4: 4 x 25 Gbps over >5 m copper twinax cable http://www.ieee802.org/3/bj/ 20
Recent 100 GbE Developments New IEEE Next Generation 100Gb/s Optical Ethernet Study Group was started in July, 2011 100GBASE-SR4: 4 x 25 Gbps over 100 m OM3/OM4 MMF 100GBASE-FR4: 4 x 25 Gbps over <2? km SMF CAUI-4: electrical signaling to the CFP2 CPPI-4: electrical signaling to the QSFP28 and CFP4 QSFP28 and CFP2/4 will be competing for the highest front panel density in 2013+ http://www.ieee802.org/3/100gngoptx/index.html 21