Positioning 40 and 100 GbE in data center inter-sw itch l ink ap p l ications and 40GbE PM D recom m endations Adam Carter, Cisco Al essan dro B arb ieri, Cisco 1

m Data Center inter-s w itc h l ink ap p l ic atio ns The adoption of 40GE as objectives for 8 02. 3 ba w il l address the server interconnect appl ication H ow ever, it is inevitabl e that the technol og y w il l ig rate into the nex t l ay ers of the netw ork and be adopted into the inter-sw itch l ink s w ithin the data center. The drivers for this m ig ration and the req u irem ents of these inter-sw itch l ink s w il l be discu ssed. 2

L is t o f s u p p o rters Dan Dove, P r oc u r ve N et w or k i ng b y H P C h r i s C ol e, F i ni s ar P et e A ns l ow, N or t el S t eve T r ow b r i d g e, A l c at el -L u c ent R i c k R ab i novi c h, T -P A C K E T S 3

M H M H M E th ernet arc h itec tu res in th e d ata c enter Prevalent Media T h ree-tier arc h itec tu re Switch T o p o lo g y et ro S i n g l e M o d e F i b er Core i g h D en si t y o d u l ar C h assi s M u l t i m o d e f i b er st ru ct u red cab l i n g 1 5 0 m @ 9 5 % co verag e ( ko l esar_ 0 1 _ 0 9 0 6. p d f ) D i s t ri b u t i on i g h D en si t y o d u l ar C h assi s ( E n d o f Ro w ) Rack server: U T P C o p p er B l ad e S erver: co p p er b ackp l an e T B S w i t c h e s Modular or lade A c c es s S erv ers T o p o f Rack ( 1 U ) E m b ed d ed B l ad e S w i t ch 1. A c c m g t 2. A c c m g ( t m t t b k c i. 3. I m p i t L 2 p g b p t i c Why do we need layers? om odat e hi h-densi y serv er dep loym ent s. om odat e net work rowt h easi er han a esh op olog y o ri ng a new rac or row om ng onli ne) rov e sc alab li y of net work rot oc ols net work desi n est rac e. 4

H H R el atio ns h ip b etw een s p eed and l ink d ens ity in th e th ree-tier DC arc h itec tu re S p eed i g hest C ore D i st ri b u t i on ( E nd of R ow) L owest A c c ess swi t c h ( T or / B lade S wi t c h) L owest i g hest # of li nk s The uplink speed at each switching tier is determined by two factors: 1. The downlink speed connecting to one layer below. 2. The number of downlinks being aggregated. 5

M M Ethernet speeds positioning in the data center for 1 G b E and 1 0 G b E optim iz ed access 1GbE Server Access T h ree-t i er a rch i t ect u re 10 GbE Server Access 4 0 / 1 0 0 G b E od u l ar C h as s i s 1 0 0 G b E 4 0 G b E 4 0 / 1 0 0 G b E 1 0 G b E b l ad e s w i t ch e s ( 1 6 s e r v e r s ) or 1 0 / 4 0 G b E ToR ( ~ 3 0-4 0 s e r v e r s ) 1 G b E Distribution od u l ar C h as s i s Top of Rack ( 1 U 4 8 por t s ) B l ad e S w i t ch ( 2 4-3 2 por t s ) 4 0 G b E b l ad e s w i t ch e s 4 0 / 1 0 0 G b E ToR 1 0 G b E F or p ower, si z e/ densi t y and c ost reasons 4 0 G i s li k ely t o dom i nat e ac c ess-t o-di st ri b u t i on swi t c h u p li nk s f or sev eral years af t er t he I E E E 8 0 2. 3 b a st andard wi ll b e rat i f i ed. D a t a V L A N 6

T h e trans itio n o f p o rt s p eed s in s w itc h es F or ex ample: 1 0 M to 1 0 0 M O n switches, as technology matures and costs decrease, the ratio of high speed to low speed ports alters. 12 0% High/low speed switch por t u n its r a tio 101% 100% 8 0% 100 M b p s / 10 M b p I t took 4 years for 1 0 0 M ports to match 1 0 M ports shipped. B eyond the 1 : 1 point, ratio takes off ( i.e. low speed v irtually disappears). This transition is driv en by technology cost. 6 0% 4 0% 2 0% 0% 19 9 5 19 9 6 19 9 7 19 9 8 19 9 9 Source: Dell O ro ( h i g h s p eed : low s p eed s h i p p ed p ort s ra t i o) 7

H igher-s peed Ethernet adoption dy nam serv ers and sw itches ics on M ass adoption of H igher S peed E thernet ( H S E ) on serv er ports is driv en by the av ailability of H S E on L A N on M otherboard ( L O M ). N I C -L O M m i x 1 00% 8 0% 6 0% 4 0% NETWORK INTERFACE CARD (NIC) L AN ON M OTH ERB OARD (L OM ) 2 0% 0% 2 001 2 002 2 003 2 004 2 005 A lthough H S E on serv er ports ev entually driv es H S E adoption on switch ports, the transition is not nearly as fast on the switch side! 1 0 0-1 0 0 0 s w i t ch p ort M i x 1 00% 9 0% 8 0% 7 0% 6 0% 2 0 0 2 y ea r of 1 G b E on L O M Total 100 Mbps Total 1000 Mbps Source: Dell O ro 5 0% 4 0% 3 0% 2 0% Note: Switch port data are not broken down by data center and wiring cl os et. 1 0% 0% 2 001 2 002 2 003 2 004 2 005 8

T he need for sol u tions optim iz ed for 1 G b E aggregation is here to stay w el l b ey ond 2 0 1 0 12 0% 100 M b p s / 10 M b p s 1000 M b p s / 100 M b p s 10 G b p s / 1000 M b p s 100% 101% 9 2 % F oreca s t 8 0% 6 0% High/low speed switch por t u n its r a tio 4 0% 2 0% 0% 19 9 5 19 9 6 19 9 7 19 9 8 19 9 9 2 000 2 001 2 002 2 003 2 004 2 005 2 006 2 007 2 008 2 009 2 010 2 011 4 % 1 0 0 M b / s St d 4 y ea rs ~ 1 :1 ra t i o Source: Dell O ro Q 3 0 7 ( h i g h s p eed : low s p eed s h i p p ed p ort s ra t i o) 1 0 0 0 M b / s St d 1 0 y ea rs ~ 1 :1 ra t i o Note: Switch port data are not broken down by data center and wiring cl os et. 1 0 G b / s St d 1 0 y ea rs ~ 1 :2 5 ra t i o 9

m 4 0 G P M F R u D ib er eq irem ents in th e Data Center D is tanc e and m ed ia req u irem ents f or 4 0 G O M -3 ri b b on O M -3 du p lex 1 0-4 0 G Aggregation O M -3 ri b b on O M -3 ri b b on 4 0 1 0 0 G I nf ras tr. ix C ore S i ng le m ode ~ 1 0 k m S i ng le m ode ~ 1 0 k m 1 0-4 0 G req u irem. 4 0 1 0 0 G Greenfield D ata C enter T y p e L eg a c y A multi-f ib e r P M D s h o uld r e p r e s e n t th e mo s t c o s t e f f e c tiv e in te r c o n n e c t s o lutio n f o r 4 0 G b E, h e n c e it is e x p e c te d th a t a 4 0 G P M D s up p o r tin g r ib b o n f ib e r s, w ill b e in s tr ume n ta l to d r iv e h ig h -v o lume a d o p tio n o f 4 0 G. 10

4 0 G R eq u irem ents S u m m ary Dr i ver s f or 4 0 G on O M -3 r i b b on: A ccess to distribution uplinks in greenfield data centers A ccess to distribution in legacy data center future-proofed for 1 0 0 G and/ or 4 0 / 1 0 0 mix ( blade switches v s. ToR switches) Dr i ver s f or 4 0 G on d u p l ex O M -3: A ccess to distribution in legacy data centers with cable infrastructure designed for 1 0 G be Dr i ver s f or 4 0 G on S M F : C ore links - for S M F we need a minimum of 1 0 km to be backward compatible with lower data rates. 11

R ec o m m end atio ns Now that 40 GbE has been adopted as part of the 8 02. 3 ba T ask F orc e, there i s a need to c onsi der i nter-swi tc h l i nk s appl i c ati ons at 40 GbE. O u r rec om m endati on i s that the 8 02. 3 ba T ask F orc e shou l d c onsi der the fol l owi ng : 1. T h e D C i n t e r -s w i t c h l i n k a p p l i c a t i o n s h o u l d b e c o n s i d e r e d t o b e p a r t o f t h e 4 0 G M M F r e a c h d i s c u s s i o n. 2. C o n s i d e r a s p a r t o f t h e 4 0 G M M F o b j e c t i v e b o t h a m u l t i -f i b e r a n d a d u p l e x f i b e r. 3. A d o p t i n g a n e w o b j e c t i v e f o r 4 0 G S M F w i t h m i n i m u m r e a c h o f 10 k m. 12