100 Tb/s Aggregate Capacity Router using an Optical Switching Core Prof. Mohammed N. Islam Department of Electrical Engineering and Computer Science University of Michigan, Ann Arbor 1
OUTLINE I. Motivation and Router Background II. Broadcast & Select Optical Switching Core III. Sub-system Issues with B&S Architecture IV. Key Enabling Technologies (Components) V. Summary 2
100000% Relative preformance increase DWDM Link speed x2/8 months 10000% 1000% Router capacity x2.2/18 months Internet x2/yr Moore s law x2/18 m 100% DRAM access rate x1.1/18 m 1996 1998 2000 2002 3
Generic Router Architecture Data Hdr Header Processing Lookup Update IP Address Header Buffer Manager Data Hdr Address Address Table Table Buffer Buffer Memory Memory Data Hdr Header Processing Lookup Update IP Address Header Buffer Manager Data Hdr Address Address Table Table Buffer Buffer Memory Memory Data Hdr Data Hdr Header Processing Lookup Update IP Address Header Buffer Manager Address Address Table Table Buffer Buffer Memory Memory 4
First Generation Routers Shared Backplane CPU Route Table Buffer Memory CPU Memory Line Interface Line Interface MAC Line Interface MAC Line Interface MAC Typically <0.5Gb/s aggregate capacity Limitation: buffer memory 5
Second Generation Routers CPU Route Table Buffer Memory Line Card Line Card Line Card Buffer Memory Fwding Cache MAC Buffer Memory Fwding Cache MAC Buffer Memory Fwding Cache MAC Typically <5Gb/s aggregate capacity Limitation: bus interconnection 6
Third Generation Routers Switched Backplane Line Card CPU Card Line Card Line Interface CPU Memory Local Buffer Memory Routing Table Local Buffer Memory Fwding Table Fwding Table MAC MAC Typically <160Gb/s aggregate capacity Limitation: number of LC s in a rack 7
4 th Generation Routers/Switches Optics inside a router for the first time Optical links 100s of metres Switch Core Linecards 0.3-10Tb/s routers in development 8
Fundamental Problems We are designing for 100Tb/s next generation Fundamental Problems Switching Core» Electronic switch requires a rack for 2Tb/s switching core» Size, power dissipation and cost prohibitive for 100Tb/s Scheduler Algorithm» Algorithm typically grows as some power of N, as much as N^3» Takes more and more time to compute algorithm as N gets larger 9
100Tb/s Routers using DWDM? Reasonable bit rate: 40Gb/s Spectral efficiency: 4 b/s/hz (using sophisticated coding, super-fec, good filtering) Telecom window: 1400-1650nm (250nm) 100Tb/s = 40Gb/s x 2500 channels For channel spacing 12.5GHz (0.1nm), telecom window has 2500 channels Bottom line: doable, but tough 10
Goals of Design Low cost, low power dissipation, small size Lowest cost using tunable filters Capacitive devices Packaging or array of devices Combine interconnect with switching fabric Fiber and star couplers Functionality of OXC and Router Same DWDM on outside and inside Minimal impact on line cards Tunable filters in the common bay equipment Hide switching time of devices Use architectural tricks to use slower, cheaper devices 11
5 th Generation Routers/Switches Switching Fabric and Fiber Interconnection Combined Star Coupler Fiber Interconnect No Intermediate O/E/O for Interconnect Linecards 12
Broadcast & Select Switching Fabric Line Card l 1 1 2 Line Card Line Line Card l 2 1 2 Card Star Tunable Optical Filter (TOF) l n 1 Line Card Core 2 Line Card Tuning Schedule TOF 1 Tune TOF 1 Tx TOF 1 Tune TOF 1 Tx TOF 2 Tune TOF 2 Tx TOF 2 Tune TOF 2 Tx 13
Multi-Channel Fast Tunable Filters 0.55 Filter Array 0.55 14
Filters in Common Bay Equipment Optical Switch Power Supplies Midplane Control Boards 15
Advantages of B&S Switching Fabric Hitless reconfiguration and hide switching time Multicast & Broadcast traffic without copies OXC/Router functionality combined Passive, transparent switching fabric scalable LC s minimally impacted because tunable filters and circuits in common bay equipment RU Low-cost approach Power splitters/taps cheaper than WDM Transmitters can be low-cost EAM/DFB lasers All but fast tunable filters are off-the-shelf 16
Comparing with Tunable Lasers Significant space on Line Card! Switching time ~30-100nsec minimum External modulators required (~3-4GHz bw) Bottom Line: tunable filters much cheaper [ Lucent, IEEE PTL, July 2001 ] 17
Issues for B&S Switching Core Scalability using amplifiers Overcoming 1/N loss using WDM and amplifiers Emulating fast switching time using inexpensive devices Speed-up, aggregation, ping-ponging between filters Simplifying scheduler for large N Two stage switch using inexpensive switching fabric Continuum source for >100 channels of 40Gb/s One expensive laser and SC copies to >100nm wavelengths 18
Scalability Issues Inherently have a 1/N loss with Star Coupler For 2500 channels, loss of 34dB If use multiple filters per LC, then 3-6dB additional loss By way of reference, a typical optical link will have a budget of about 30dB (~22dB loss budget and another ~8dB for OADM/DC, etc) 19
Broadcast & Select Switching Fabric Line Card l 1 1 2 Line Card Line Line Card l 2 l n W D M 1 2 1 Card Tunable Optical Filter (TOF) Line Card 2 Line Card By-pass traffic Core By-pass traffic Tuning Schedule TOF 1 Tune TOF 1 Tx TOF 1 Tune TOF 1 Tx TOF 2 Tune TOF 2 Tx TOF 2 Tune TOF 2 Tx 20
Emulating fast switching using low-cost components 50-100ns devices cost << than sub-ns devices Speed-up 50 ns 50 ns 25 ns Aggregation and superframes 1 ms Ping-pong between different filters Rx Rx 21
Simplified Scheduler with 2- Stages Simple round-robin scheduler in two sections Middle stage may have additional memory to avoid mis-sequencing of packets 2x switching fabric, so fabric must be low cost! Questions to scheduler experts: Does the switching fabric need to reconfigure every cell cycle (in which case our approach is important) Are LC s needed in all three locations, or can we get away with two (in which case fabric is simplified) 22
Two-Stage Switch External Inputs 1 Internal Inputs 1 External Outputs 1 N Load Balancing N N First Round-Robin Second Round-Robin Switch gives 100% throughput for non-uniform, bursty traffic, without a scheduler or speedup! [Nick McKeown, Stanford University, Opticomm 2001] 23
Exemplary 2-stage Switching Fabrics LC#1 LC#1 LC#1 Input fibers LC#2 Star LC#2 Star LC#2 LC#N LC#N LC#N LC#1 LC#1 Power splitter LC#2 Star LC#2 Power combiner LC#N LC#N 24
Cost Estimate for 1Tb/s Core (2004) Assume 100 ch @ OC-192 Transmitter: $1.7k x 100 = 170K WDM: $0.1Kx100 = 10K Amplifier: = 50K 1xN splitter: $0.05Kx100 = 5K Tunable filter: $0.1k x 100 =10K Total: $245K Connectors, electronics, cases, software, extra Cost < $200K if we use broadcast & select without amplifier Tx Tx Tx Tx l 1 l 2 l 3 l N W D M 25
Laser Sources in Large Router Many LD s become expensive If channel spacing is close, then stabilizing wavelengths and maintaining channel spacing difficult and expensive For 40Gb/s (or 160Gb/s) per channel, light source can be expensive For many LD wavelengths, many part # s and have to match LD wavelength per LC With 40Gb/s sources and stabilization circuits, significant space on LC will be used 26
Supercontinuum (SC) Source One modelocked source and a common SC set-up Channel spacing set by passive WDM demux, which is used to carve out channels from SC For 40Gb/s (or muxed to 160Gb/s), only one expensive ML source required. SC copies to many wavelengths Each LC can have a modulator, but individual LD s are not required. If modulator broadband, few part # s ML laser and SC set-up will be in common bay equipment. Only modulator placed on LC BOTTOM LINE: SC less expensive for #l s >100 and 40Gb/s per channel or higher 27
Exemplary System ML 40 Gb/s source Common bay equipment Continuum Fiber TDM MUX WDM SC source can all be placed in common equipment bay LC#1 Mod l 1 Modulator placed on line card WDM can be replaced by power splitters and fixed or tunable filters LC#2 Mod l 2 LC#N Mod l N 28
SC Experimental Setup Mode Locked Ring Cavity EDFL Dt = 400 fs 15 db EDFA P 0 Pulse Chirping (2m SMF-28) SC Generation L (D) Diagnostics Optical Spectrum Analyzer Autocorrelator L: only 2 meters long 29
Exemplary SC Spectrum L = 2 [m] l o = 1539 [nm] P avg = 1100 [W] D Experimental Parameters = 1.13 [ps/nm-km] -20 db Bandwidth: 211 nm 0-10 -20 Optical Power [db] 1450 1500 1550 1600 1650 Wavelength [nm] 30
SC Spectral Flatness ±0.5 db maximum power fluctuation over 61 nm ± 0.1 db power fluctuation over 35 nm 35 nm ± 0.1 db 1485 1500 1515 ± 0.5 db 61 nm 2.5 0.0-2.5-5.0-7.5-10.0 1450 1475 1500 1525 1550 31
Coherence of Carved SC Spectrum Laser EDFA STD Fiber SC Fiber 1 Filtered Pulse t ~ 1 ps Compensated Pulse t ~ 0.5 ps OBF Compensation Fiber 2 3 Pulse width t FWHM 500 fsec can be carved from SC for high-speed TDM applications Intensity [AU] 1.0 0.8 0.6 0.4 0.2 0.0-10 Optical Power [dbm] -20-30 -40-50 -60-70 Autocorrelation -4-2 0 2 4 Time [psec] Spectrum 1450 1500 1550 1600 1650 1700 Wavelength [nm] 32
Timing Jitter of Source and Filtered SC Output Timing Jitter [psec] Time Jitter [sec] 1E-10 10 1E-11 1.0 1E-12 1E-13 0.1 1E-14 0.01 Carved Spectrum Laser Output Measurement Parameters 5 KHz Span 30 Hz Resolution 430 th Harmonic of laser fundamental 1E-15 10 100 1000 Frequency [Hz] Pulses carved from flat section of continuum have same timing jitter as the source Short SC fiber length minimizes additional timing jitter 33
Key Enabling Technologies (Components) Broadband Amplifiers Tunable Filters Surface Normal Modulators 34
All-Raman Broadband Amps Line fiber A B Gain fiber GFF Gain fiber C In Distributed Pump Module Lumped Pump Module Lumped Pump Module Out Very low MPI level Dispersion and slope compensation provided by the gain fiber Large gain & low NF over a 100nm continuous spectral window Demonstrated the transmission feasibility of 240 OC-192 channels over > 1500km SSMF 35
+24dBm Launch Power Flat input spectrum Output spectrum after 8 spans (DSE output) Line amplifier flat operation over 100nm demonstrated at +24dBm output power (corresponds to 0dBm/ch for 240 channels) 36
Electro-optic Tunable Filter Filter X-Section Optical cavity with electro-optic material Tune filter by voltage induced index changes of EO material Cavities 100 Substrate Mirrors -1 db BW -30 db BW Filter characteristics In-Band Ripple 1-Cavity Filter 25GHz 625 GHz <0.25 db 3-Cavity Filter 25 GHZ 100 GHz < 0.25 db Transmission (%) 80 60 40 20 0 1549.4 1549.6 1549.8 1550.0 1550.2 1550.4 1550.6 Wavelength (nm) 37
Pass-band shape vs. number of cavities Filter Type Fabry Perot Narrow Band Wide Band Filter Shape Lorentzian Square Square # Channels 1 channel 1 channel 4-16 channels # Cavities 1 cavity 3 cavities 8-10 cavities Application Channel Monitor OADM, MUX/DeMux OADM Transmission (%) 100 80 60 40 20 0 1549.5 1550.0 1550.5 Wavelength (nm) Transmission (%) 100 80 60 40 20 0 0 1549.4 1549.6 1549.8 1550.0 1550.2 1550.4 1550.6 Wavelength (nm) Transmission (db) -10-20 -30-40 -50-60 38 1548 1550 1552 1554 1556 1558 Wavelength (nm)
3-cavity filter tuning characteristics Transmission (db) 0-20 0.5 MV/cm 0.0 MV/cm -0.5 MV/cm -1.0 MV/cm ~15 nm Transmission (db) 0.00-0.03-0.06-40 1535 1540 1545 1550 1555 1560 1565 Wavelength (nm) 1560.00 1560.08 1560.16 Wavelength (nm) A tuning range of 15 nm is obtained with electric fields of 1 MV/cm to 0.5 MV/cm across each cavity The pass-band ripple increases near the extreme ends of the tuning range The transmission ripple remains less than 0.1 db in the 15 nm tuning range The group delay ripple increases by ~ 1 ps 39
Alternate Approach 1D MEMS simple piston up-down motion Combine optical functionality with well-known electro-static actuators Challenge: combining MEMS actuator growth with optical coating technology 40
FIMS Fast Interferometric MEMS Switch High speed achieved because Interferometer: maximum displacement l/4 Stress: make a tight guitar string 1D: simple 1D motion with simple control High reliability expected Small motion without hinges Electro-static actuators well-proven technology Low cost Simple packaging Standard processing steps with many devices on a wafer 41
Predicted Performance of FPI High speed while maintaining optical performance and low voltage Parameter Value Tuning Speed Tuning voltage -3 db bandwidth -30 db bandwidth Channel Selectivity Tuning range Finesse Insertion loss PDL 60 ns mechanical, 100 ns electrical 40 V max 0.1 nm 1.5 nm 12.5 Ghz (0.1nm) 100 nm 6300 1-3 db < 0.1 db 42
Surface Normal Modulator Transmission (db) 0-20 -40 -DV +DV FSR 1549.2 1549.6 1550.0 1550.4 1550.8 Wavelength (nm) p Tunable filter functions as a surface normal modulator Operates on single channel/frequency to encode data on cw light» Filter is tuned in and out of channel frequency band to create high and low signal states Sharp transition from high to low transmission» Does not require p phase shift for high contrast High-speed» Fast EO response + low capacitance Surface normal configuration is advantageous for building transmitter array Source array: Laser diode array, LED array or broadband light source with optical filter array Source array Surface-normal modulator array Data encoder 43
Summary Routers expected to be bottleneck in future systems 100Tb/s router project currently at UM Strawman system design to understand limiting technologies Limitations from switching fabric and scheduler Broadcast & Select architecture using DWDM technology Sub-system Issues of B&S Architecture Scalability using Broadband Amplifiers & WDM Hide switching time by ping-ponging between filters Two-stage switch for simplified scheduler Broadband continuum source to simplify transmitters Key Enabling Technologies Broadband Amplifiers are commercially available Fast tunable filter» Two approaches: Electro-optic thin films and 1-D MEMS Surface Normal Modulators can be made with fast filters 44