Data Center Ethernet - The Framework for Server I / O C on s ol i d at i on, V i rt u al i z at i on an d U n i fi ed Fab ri c Ray O H an l o n C i s c o E u r o p e TERENA Workshop Day 2 S e pt e m b e r 2 nd 1 0 : 1 5-1 0 : 4 5 1

Data Center N etw o rk i ng Env i ro nm ent Clustered and Grid apps are becoming more common M ulti-core CP U arch itectures and serv er v irtualiz ation is driv ing th e need f or more bandw idth per serv er N etw ork traf f ic is: Client to S erv er, S erv er to S erv er, S erv er to S torage and S torage to S torage H igh er dependency of A pplication perf ormance on netw perf ormance B andw idth and L atency matter T raf f ic ch aracteriz ation includes: S h ort and bursty messages: client to serv er and serv er to serv er & L ong steady messages: serv er to storage and storage to storage P ack et drop af f ects most traf f ic ty pes F lat N etw ork s are common 2 ork

T he Cas e f o r 1 0 G b E to the S erv er M ulti-core CP U arch itectures allow ing bigger and multiple w ork loads on th e same mach ine S erv er v irtualiz ation driv ing th e need f or more I / O bandw idth per serv er Grow ing need f or netw ork storage driv ing th e demand f or h igh er netw ork bandw idth to th e serv er 1 0 GE L A N on serv er M oth erboards ( L om ) beginning mid-2 0 0 8 (source: Broadcom ) 3

M ak i ng 1 0 G b E Co s t Ef f ec ti v e Mid 1980 s Mid 1990 s E a r l y 2 000 s L a t e 2 000 s SS FF P P + + tt o o SS FF P P + + 10Mb 100Mb SS 1G FF P P b+ + CC uu 10G b UTP Cat 3 UTP Cat 5 UTP Cat 5 S F P F i b e r X2-1 8 S F P + C u (BER better than 10 ) S F P + F i b e r C a t 6 / 7 T ec h n o lo g y S F P + C U C o p p e r Cable D i s t an c e P o w er ( eac h s i d e) T r an s c ei v er L at en c y ( li n k ) Twinax 1 0 m ~ 0. 1 W ~ 0. 1 µs S F P + U S R u l t r a sh o r t r e ac h M M O M 2 M M O M 3 1 0 m 1 0 0 m 1 W ~ 0 S F P + S R sh o r t r e ac h M M 6 2. 5 µm M M 5 0 µm 8 2 m 3 0 0 m 1 W ~ 0 1 0 G B A S E -T Cat6 Cat6a/ 7 Cat6a/ 7 5 5 m 1 0 0 m 3 0 m ~ 8 W ~ 8 W ~ 4 W 2. 5 µs 2. 5 µs 1. 5 µs 4

U ni f i ed F ab ri c Increased Efficiency, Simpler Operations MM gg mm t t NN et ww oo rr kk FF rr oo nt -E nd NN et ww oo rr kk BB aa cc kk uu pp NN et ww oo rr kk SS tt oo rr aa gg e NN et ww oo rr kk BB aa cc kk -E nd NN et ww oo rr kk Unified FF aa bb rr ic Unified FF aa bb rr ic aa nd I I // OO 5

T he Cas e f o r N etw o rk Co nv erg enc e R educe ov erall data pow er consumption by up to 8 %. E x tend th e lif ecy cle of current data center. W ire h osts once to connect to any netw ork - S A N, L A N, H P C. F aster rollout of new apps and serv ices. E v ery h ost w ill be able to mount any storage target. D riv e storage consolidation and improv e utiliz ation. R ack, R ow, and X -D ata Center V M portability become possible. 6

F C o v er Ethernet ( F Co E) FF CC oo EE M apping of F C F rames ov er E th ernet E nables F C to R un on a L ossless E th ernet N etw ork Ethernet F i bre C ha nnel T ra f f i c F ew F ew er Cables Both block I/O & Ethernet tra f f i c co-ex i st on sa m e ca ble er adapters needed O v erall less pow er I nteroperates w ith ex isting S A N s S ame M anagement model N o Gatew BB ee nn ee ff ii tt ss ay - S tateless 7

M M F Co E Enab l ers 1 0 G b ps Et he rn e t L ossl e ss Et he rn e t ( Dat a C e n t e r Et he rn e t ) at ches t he B 2 B cred it s u sed in Fib rechan n el t o pro v id e a lo ssless serv ice Et he rn e t j u m b o f ram e s ( 21 8 0 B yt e s) ax FC frame paylo ad = 2 1 1 2 b yt es E t hern et V 2 Frame, E t hert ype = FCo E Same as a physical FC frame Ethernet H ea d er F C o E H ea d er F C H ea d er F C P a y loa d C R C EO F F C S Co n t ro l in fo rmat io n : v ersio n, o rd ered set s (SO F, E O F) B W k g G r p w w w g / f c w w w c T.11 Committee FC-B -5 or in ou.t11.or oe.f oe.c om 8

O F Co E f o r Co nv erg ed N etw o rk s T od a y s M od el C onv erg ed N etw orks U si ng F C oe: T w o Serv er L O M E t hern et P o rt s T w o 1 G Q u ad -P o rt N I Cs T w o 4 G Sin g le P o rt H B A s T w o Serv er L O M E t hern et P o rt s T w o 1 0 G Sin g le P o rt Co n v erg ed N et w o rk A d apt ers (CN A ) V M o t io n B ac k u p V M N e t 1 B ac k u p V M N e t 2 B ac k u p V M N e t 3 B ac k u p V M o t io n B ac k u p V M N e t 1 B ac k u p V M N e t 2 B ac k u p V M N e t 3 B ac k u p F C S A N A S o u r c e : H P W e b sit e F C S A N A F C S A N B A d minist r at io n/ V M V M N e t 1 V M N e t 2 V M N e t 3 V M o t io n S e r v e r N I C R C A d minist r at io n/ V M S e r v e r N I C V M N e t 1 V M N e t 2 V M N e t 3 V M o t io n F C S A N B R C Ethernet L i nk F C o E T ra f f i c ther N etw o rk i ng T ra f f i c 9

V M A w are N etw o rk S erv i c es Network services d el ivered a t V M l evel g ra n u l a rity P rovid es im p roved V M p orta b il ity A b il ity to cu stom iz e n etwork ch a ra cteristics a t th e V M l evel a t wiresp eed S w i t c h P o r t I n d i v i d u al Co n t r o l an d E t h er n et P o li c i es S e r v e r V M 1 V M 2 V M 3 V M 4 V i r t u al I n t er f ac es w Ch ar ac t er i s t i c s i t h P h y s i c al I n t er f ac e 10

W hat i s Data Center Ethernet ( DCE)? D a ta C en ter E th ern et is a n a rch itectu ra l col l ection of E th ern et ex ten sion s d esig n ed to im p rove E th ern et n etworkin g a n d m a n a g em en t in th e D a ta C en ter. C isco is sh owin g in n ova tion wh il e workin g th rou g h th e sta n d a rd iz a tion p rocess with th ese ex ten sion s in op en sta n d a rd s f oru m s. 11

G M Data Center Ethernet F eatu res Ov e r v i e w F eature / S tandard P rio rit y Flo w Co n t ro l (P FC) IEEE 8 0 2. 1Q bb B enef it P ro v id es class o f serv ice flo w co n t ro l. A b ilit y t o su ppo rt st o rag e t raffic B an d w id t h M IEEE 8 0 2. 1Q Co n g est io n M IEEE 8 0 2. 1Q an ag emen t a z an ag emen t a u ro u pin g classes o f t raffic in t o Serv ice L an es Co S b ased E n han ced T ran smissio n E n d -t o -en d co n g est io n man ag emen t fo r L 2 n et w o rk D at a Cen t er B rid g in g E x chan g e P ro t o co l (D C BX ) L 2 M u lt ipat h fo r U n icast an d M u lt icast L o ssless E t hern et an ag emen t & A u t o -n eg o t iat io n pro t o co l fo r en han ced E t hern et capab ilit ies U t iliz e fu ll B i-sect io n al b an d w id t h acro ss mu lt iple act iv e pat hs. E limin at e span n in g t ree P ro v id es ab ilit y t o t ran spo rt v ario u s t raffic t ypes (e. g. St o rag e, R D M A ) 12

Data Center Ethernet F eatu res PP riority -B ased FF low Control (( PP FF C) Ena bles lossless F a bri cs f or ea ch cla ss of serv i ce P A U S E sent p er v i rtu a l la ne w hen bu f f ers li m i t ex ceed ed N etw ork resou rces a re p a rti ti oned betw een V L s ( E. g. i np u t bu f f er a nd ou tp u t q u eu e) T he sw i tch beha v i or i s neg oti a ble p er V L 13

Data Center Ethernet F eatu res EE nh anced TT ransmission SS election (( EE TT S S )) Ena bles Intelli g ent sha ri ng of ba nd w i d th betw een tra f f i c cla sses control of ba nd w i d th Bei ng S ta nd a rd i z ed i n IEEE 8 0 2. 1Q A lso know n a s P ri ori ty G rou p i ng a z 14

Data Center Ethernet F eatu res Congestion MM anagement M ov es cong esti on ou t of the core to a v oi d cong esti on sp rea d i ng A llow s End -to-end cong esti on m a na g em ent S ta nd a rd s tra ck i n 8 0 2. 1Q a u 15

Data Center Ethernet F eatu res DD ata Center BB ridging Capability ex ch ange PP rotocol Data Center Ethernet Data Center Ethernet H a nd sha ki ng N eg oti a ti on f or: C os BW M a na g em ent C la ss Ba sed F low C ontrol C ong esti on M a na g em ent ( BC N /Q C N ) A p p li ca ti on ( u ser_ p ri ori ty u sa g e) L og i ca l L i nk D ow n 16

M M M M P P Data Center Ethernet F eatu res LL ay er 22 MM ulti-p ath ing options 1 2 3 L A N L A N V ir tu a l S w itch L A N AC A AC B Active-Active vp C L 2 E C M AC A AC B L 2 E C M i m i S T P o U p l i B d g P o A l l o w s M u l p l A c v U p l i s S w i N o P L o o p s b y P i a M A C A d d s O y O P o C o m p l y T p a N H o p S w i El na tes n nk ri e rts ti e ti e nk tc h to etw rk rev ents nni ng res to nl ne rt etel ra ns rent to ex t tc h V i rtu a l S w i tc h reta i ns p hy s i c a l s w i tc hes i nd ep end ent c o ntro l a nd d a ta p l a nes V i rtu a l p o rt c ha nnel m ec ha ni s m i s tra ns p a rent to ho s ts o r s w i tc hes c o nnec ted to the v i rtu a l s w i tc h S T P a s f a i l -s a f e m ec ha ni s m to p rev ent l o o p s ev en i n the c a s e o f c o ntro l p l a ne f a i l u re U s I S I S b a s p o l o g y i m i S T P f m L 2 d o m a i P p a s o es ed to El na tes ro n ref erred th el ec ti n 17

S u m m ary A Unified fa b r ic is t h e end s t a t e net w o r k w h er e L A N, S A N, a nd I P C t r a ffic a r e c o nv er g ed o nt o a s ing l e net w o r k infr a s t r u c t u r e D a t a C ent er E t h er net is t h e fo u nda t io n fo r del iv er ing a Unified F a b r ic. D C E w il l a l s o a p p ea l t o c u s t o m er s t h a t a r e no t p l a nning fo r c o ns o l ida t ed I / O du e t o it s L 2 m u l t ip a t h ing c a p a b il it ies a nd it s l o s s l es s s er v ic e b eh a v io r D C E c o m p r is es a s et o f E t h er net enh a nc em ent s t a nda r ds t h a t a dh er e t o t h e em er g ing I E E E D a t a C ent er B r idg ing s p ec ific a t io n. h ttp: / / w w w. ieee8 0 2. org/ 1 / pages/ dcbridges. h tml 18

19

Del i v eri ng N etw o rk Co nv erg enc e to d ay S ta nd a rd s W i r e S p eed 1 0 G be S w i t c h i n g Cap ac i t y D at a Cen t er E t h er n et S c alabi li t y F i br e Ch an n el o v er E t h er n et Co n s o li d at i o n V M A w ar e N et w o r k i n g V i r t u ali z at i o n LAN S AN A S AN B E t h e r n e t L A N S A N A S A N B L A N LAN N 5 0 0 0 MAC A MAC B Active-Active N 5 0 0 0 N 5 0 0 0 A & B MAC A MAC B C MAC C E n d n o d e s Eco-S y stem 20

MM Implementing Unified Fabric with Nexus 5000 CC aa tt aa ll yy ss tt NN ex uu ss DD SS P re se rv e s Ex i st i n g L A N an d S A N I n v e st m e n t s L A N L A N S A N A S A N B P re se rv e s Ex i st i n g Du al S A N En v i ron m e n t s Cisco N ex us 5 0 0 0 C on sol i d at e d I / O S e rv e rs t o Ac c e ss L aye r Dat a C e n t e r Et he rn e t / F C oe L i n ks 1 0 G b E 1 0 G b E D a ta C en ter E th er n et F ib r e C h a n n el 1 0 G b E F C o E / D a ta C en ter E th er n et 21

D Enc ap s u l ati o n tec hno l o g i es is CS I F CoE I nf inib and O p e r a t i n g S y s t e m / A p p l i c a t i o n s S C S I L a y e r F C P* i S C S I F C P* F C P* F C P* S R P i S C S I F C IP i F C P i S E R T C P T C P T C P IP IP IP F C o E F i b rec ha nnel Ethernet a ta C enter Ethernet I nf i ni B a nd 1, 2, 4, ( 8 ), 1 0 G b p s 1, 1 0 G b p s ( 1 ), 1 0 G b p s 1 0, 2 0 G b p s * I nc l u d e s F C L ay e r 22