Ethernet Evolution: The Path to 100 Gigabit Ethernet. John D Ambrosia - Scientist, Components Technology Chair, IEEE HSSG

Transcription

1 Ethernet Evolution: The Path to 100 Gigabit Ethernet John D Ambrosia - Scientist, Components Technology Chair, IEEE HSSG Version Oct 2006

2 Per IEEE-SA Standards Board Operations Manual, January 2005 At lectures, symposia, seminars, or educational courses, an individual presenting information on IEEE standards shall make it clear that his or her views should be considered the personal views of that individual rather than the formal position, explanation, or interpretation of the IEEE. 2

3 Acronym Cheat Sheet CFI Call for Interest DWDM Dense Wavelength Division Multiplexing EMI Electro-magnetic Interference Gbps Gigabit per Second HSSG Higher Speed Study Group ITU International Telecommunications Union IETF Internet Engineering Task Force JEDEC - Joint Electron Device Engineering Council MAC Media Access Control MDI Medium Dependent Interface MSA Multi Source Agreement OIF Optical Internetworking Forum PCS Physical Coding Sublayer PMA Physical Medium Attachment PMD Physical Medium Dependent PHY Physical Layer Device SERDES Serialize / De-serialize SMF / MMF Single Mode Fiber / Multi Mode Fiber Tbps Terabit per Second WIS WAN Interface Sublayer XGMII 10 Gigabit Media Indpendent Interface 3

4 Agenda The Ethernet EcoSystem IEEE Higher Speed Study Group Considerations for System Development 4

5 Agenda The Ethernet EcoSystem IEEE Higher Speed Study Group Considerations for System Development 5

6 The Ethernet Ecosystem Broadband Access Content Providers Broadband Access Internet Backbone Content Enterprise Internet Backbone Research Data Centers and Enterprise Internet exchange and Interconnection Points Research, Education and Government Facilities 6 IEEE HSSG Call-For-Interest, 7/2006

7 What is Driving the Need for Higher Speed Ethernet? Research network trends 10 GbE WANs Cluster and grid computing Data center trends Lots of GbE and 10 GbE servers ISP / IX trends Personalized content Consumer broadband 10 GbE peering Aggregating 10 GbE links with LAG is an interim solution Government/Research Data Center Service Provider 7

8 802.3ad Link Aggregation (LAG) It is being used Temporary fix for increased bandwidth demand Increased complexity Difficult to plan for capacity and traffic engineering Harder to manage & troubleshoot multiple physical links based on a single logical interface Cable & link management Uneven distribution of traffic Limitations in the standard Inefficient distribution of large flows Load balancing requires packet inspection or other knowledge 8 IEEE HSSG Call-For-Interest, 7/2006

9 Bandwidth and Growth Projections Consumer Broadband Access Broadband Cox: Access 10 GbE today, BW growth 50-75% per year for next 3 5 years Cisco: 10GbE today, Enterprise 40+ GbE (100 GbE preferred) in 5 years Corporate Data Centers and Enterprise Comcast: 4x10 GbE LAG today, 3X BW increase in 3 to 5 years Internet Backbone Level 3: 8x10 GbE LAG today, BW growth 15x in 5 years (~70%/year) Internet Backbone Internet Exchanges: Up to 8x10 GbE LAG today, BW growth 50-75% per year for next 3 5 years Content Providers Yahoo!: 4x10 GbE LAG today, BW doubling in <12 months LLNL: 4x10 GbE LAG and 500x10 GbE ports Content today, 10x speed requirement in 5 years on deployed ports ResearchESnet: 10 GbE today, 10 Gbps on 20+ links 5 years from now; 5-10 locations will require more than 40 Research, Education Gbps and Government Facilities 9 IEEE HSSG Call-For-Interest, 7/2006

10 Agenda The Ethernet EcoSystem IEEE Higher Speed Study Group Considerations for System Development 10

11 IEEE July, 2006 Plenary Higher Speed Study Group (HSSG) Call-For-Interest Led by John D Ambrosia / Joel Goergen, Force10 Broad Industry Support End Users System Vendors Component Vendors Higher Speed Study Group formation approved Sept John D Ambrosia, Force10, Confirmed as HSSG Chair 11

12 IEEE HSSG Reflector and Web To subscribe to the HSSG reflector, send an to: with the following in the body of the message: subscribe stds hssg <your first name> <your last name> end HSSG web page URL: 12

13 Developing a Standard Ideas From Industry Feasibility and Research Call for Interest Study Group Task Force Working Group Ballot Industry Pioneering 1 Year Higher Speed Ethernet is Here Ad Hoc Efforts CFI July 18, 2006 Sponsor Ballot 13 Standards Board Approval Publication IEEE ~4 Years

14 Normal Study Group Output Project Authorization Request 5 Criteria Broad Market Potential Compatibility Distinct Identity Technical Feasibility Economic Feasibility Project Objectives 14

15 Objectives Speed? Higher Speed Ethernet 15 Means faster MAC data rate Options Discussed 40 Gbps 80 Gbps 100 Gbps 120 Gbps Scaleable IEEE HSSG adopted 100 Gb/s (Nov 2006) MAC = Media Access Control MDI = Medium Dependent Interface PCS = Physical Coding Sublayer PMA = Physical Medium Attachment PMD = Physical Medium Dependent WIS = WAN Interface Sublayer XGMII = 10 Gigabit Media Independent Interface MAC Rate = 10G PHY Rate < 10G Higher Layers MAC Client MAC Control MAC Reconciliation Sublayer (RS) XGMII XGMII 64B/66B PCS 64B/66B PCS WIS PMA PMA PMD PMD MDI MDI MEDIUM MEDIUM 10GBASE-R 10GBASE-W

16 Objectives PHY Types and Reach? 10 Gb Ethernet Historical Perspective Fiber 10GBASE-SR / W 850 nm Serial 300 / 33 m 50 um / 62.5 um MMF 10GBASE-LRM 1310 nm Serial 220 m 50 um / 62.5 um MMF 10GBASE-LX nm WDM 300 m 50 um / 62.5 um MMF 10 km Single Mode Fiber 10GBASE-LR / W 1310 nm Serial 10 km Single Mode Fiber 10GBASE-ER / W 1550 nm Serial 40 km Single Mode Fiber Copper 10GBASE-CX4 4 lanes 15 m Twinaxial 10GBASE-T Twisted Pair 100 m UTP Backplane 10GBASE-KX4 4 lanes 1 m Improved FR-4 10GBASE-KR Serial 1 m Improved FR-4 16 To date, IEEE HSSG has adopted (Nov 2006) Support at least 10km on SMF. Support at least 100 meters on OM3 MMF

17 Summary of Adopted Objectives To date, IEEE HSSG has adopted: Support full-duplex operation only. Preserve the 802.3/Ethernet frame format at the MAC Client service interface. Preserve minimum and maximum FrameSize of current Std. Support a speed of 100 Gb/s at the MAC/PLS interface. Support at least 10km on SMF. Support at least 100 meters on OM3 MMF. All objectives adopted at IEEE 802 November 2006 Plenary 17

18 Existing Backbone 18 Ethernet Ecosystem 10 Gbps λ DWDM Broadband Access Internet Backbone Content Internet Backbone Research Enterprise 40 Gbps λ DWDM 10 Gbps λ DWDM wide deployment 40 Gbps λ DWDM Being evaluated Initial deployments Return on Investment? Interim solution? Rate Choice 100 Gbps maps into 10 Gb/s λ DWDM 120 Gbps maps into 40 Gb/s λ DWDM Ethernomics The Ecosystem is cost sensitive Choice of 120 vs 100 Gbps Faster? Wider? IEEE HSSG choose 100 Gbps

19 Future Backbone? Future Ethernet Ecosystem Broadband Access Enterprise 20+ Gbps λ DWDM? Internet Backbone Internet Backbone 100 Gbps λ DWDM? Research Content Ethernomics The Ecosystem is cost sensitive 20+ Gbps λ DWDM? Lower cost than 40 Gb/s Re-use of existing infrastructure? Return on Investment? 100 Gbps λ DWDM Return on Investment? Future capacity requirements 19

20 Thoughts Related to Backbone 40 Gbps λ DWDM Not widely deployed Interim solution to bandwidth crunch Support seen by Transport equipment providers Service providers Choice of 100 Gbps Service providers are evaluating Ethernet for transport Maps into existing deployed 10 Gbps λ DWDM Support for 100 Gbps λ DWDM seen by Transport equipment providers Service providers Recent decision by IEEE Future discussions and work needed 20

21 Other Bodies The Ethernet Alliance Optical Internetworking Forum 20Gbps to 25Gbps electrical interface project. 100Gbps to 160Gbps system interface project. ITU IETF JEDEC 21

22 Agenda The Ethernet EcoSystem IEEE Higher Speed Study Group Considerations for System Development 22

23 Chassis Slot Density Year System Introduced /7 in design now 2009 Slot density 40Gbps 60Gbps 120Gbps 400Gbps to 500Gbps Based on max back plane thickness of 300mils, 20TX and 20RX differential pipes. 23

24 Chassis Design 24 Chassis design issues: Backplane and channel signaling Lower system BER Connectors N+1 switch fabric Reduced EMI Clean power routing architecture Thermal and cooling Cable management All design aspects must also meet local regulatory standards

25 Backplane Considerations Slot Capacity Switching Capacity Performance Signal coding BER Impacts SerDes Design Design and technology drives scalability Advanced fiberglass materials Unique conductor layers Engineered trace geometries 25

26 Anatomy of a 100 Gbps Solution: Designing for EMI Compatibility Level in dbµv/m E300 Power Supply Radiated Emission for 30 MHz to 1 GHz EN Class A Electric Field S M M G Frequency in Hz EMI is Electromagnetic Interference, i.e. noise Too much noise interfere can cause bit errors Concerned about two types of EMI Conducted through power Radiated through air 26

27 Power Considerations End User Restrictions? Total system wattage? Input power quality not specified Higher speeds require lower noise 27

28 Thermal Management Considerations Cooling capacity per slot? Front to back filtered airflow for carrier deployments Cooling redundancy Heat can affect material performance which affects high-speed signaling performance 28

29 Line Card Components Ethernet PHYsical Layer Line drivers/receivers Encoders/decoders Timing Packet lookup memory Packet buffer memory Line Card CAM SRAM DRAM Switch Fabric 1 Media Interface PHY MAC Network Processor Fabric Interface Switch Fabric N Pluggable or fixed media interfaces 29 Ethernet Media Access Control Framing Addressing Error handling Flow control Packet lookup, classification and forwarding functions Interfaces to switch fabrics

30 High Speed Signaling Interfaces will use SERDES Short Reaches Long Reaches SERDES will replace parallel busing ASIC s need new higher speed SERDES 6.25 Gbps today 10 Gbps emerging Discrete higher speed SERDES for 20 Gb/s? Line Card SERDES CAM SRAM DRAM Switch Fabric 1 Media Interface PHY MAC Network Processor Fabric Interface 30 Switch Fabric N

31 Higher Speeds Drive Switch/Router Requirements Driving architectural requirements Massive hardware and software scalability >200 Gbps/slot switch fabric capacity Support for several thousand interfaces Multi-processor, distributed architectures Fast packet processing at line-rate 100 GbE is ~149 Mpps or 1 packet every 6.7 ns (10 GbE is only ~14.9 Mpps or 1 packet every 67 ns) 31

32 Higher Speeds Drive Density 100 Gbps Ethernet will benefit all Drives 10 GbE port density up and cost down Possible line-rate combinations 1 x 100 GbE port 10 x 10 GbE ports 100 x 1 GbE ports And even more oversubscribed port density The more things change the more they stay the same. 32

33 System Port Count Cycle * GE 336 Ports 672 Ports 1260 Ports 2240 Ports? 10 GE 28 Ports 56 Ports 224 Ports 560 Ports 2240 Ports 100 GE Standard In Development 56 Ports 33

34 Conclusions Aggregation is the Killer Application From carrier to the consumer Not talking desktop 100G Ethernet is coming! The more things change, the more they stay the same Lower speeds (1G and 10G) Higher density Integration and cost optimization Port density of 100G Higher scalability Integration and cost optimization 34

35 35 Thank You