Roger D. Chamberlain, Ch'ng Shi Baw, Mark A. Franklin, Christopher Hackmann, Praveen Krishnamurthy, Abhijit Mahajan, and Michael Wrighton

Transcription

1 Evaluating th Prfrmanc f Phtnic Intrcnnctin Ntwrks Rgr D. Chambrlain, Ch'ng Shi Baw, Mark A. Franklin, Christphr Hackmann, Pravn Krishnamurthy, Abhijit Mahajan, and Michal Wrightn R.D. Chambrlain, Ch'ng Shi Baw, M.A. Franklin, C. Hackmann, P. Krishnamurthy, A. Mahajan, and M. Wrightn, Evaluating th Prfrmanc f Phtnic Intrcnnctin Ntwrks, in Prc. f th 35th Annual Simulatin Sympsium, April 2002, pp Cmputr and Cmmunicatins Rsarch Cntr Washingtn Univrsity Campus Bx 1115 On Brkings Dr. St. Luis, MO

2 Evaluating th Prfrmanc f Phtnic Intrcnnctin Ntwrks Rgr Chambrlain, Ch ng Shi Baw, Mark Franklin, Christphr Hackmann, Pravn Krishnamurthy, Abhijit Mahajan, and Michal Wrightn Cmputr and Cmmunicatins Rsarch Cntr Washingtn Univrsity, St. Luis, Missuri ABSTRACT This papr dscribs th dsign and us f th Intrcnnctin Ntwrk Simulatr (ICNS) framwrk. ICNS is a mdular, bjct-rintd simulatin systm that has bn dvlpd t invstigat prfrmanc issus in multiprcssr intrcnnctin ntwrks that xplit phtnic tchnlgy in thir dsign. W dscrib th ICNS infrastructur, prsnt tw distinct phtnic intrcnnctin ntwrks that hav bn mdld using ICNS, and giv prfrmanc rsults fr ach f ths ntwrks. 1 Intrductin With th advnt f ptical fibr, phtnic tchnlgy has bcm an indispnsabl cmpnnt in th dsign and dplymnt f th wrld s lng-distanc cmmunicatinsinfrastructur. Th bandwidth capacity f lng-distanc fibr links is nrmus, and th tchnical and cnmic advantagsf phtnic tchnlgy in thisarna ar indisputabl. What w havn t yt sn is th xplitatin f phtnic tchnlgy fr shrt-distanc cmmunicatins (.g., chip-t-chip and bard-t-bard links within a singl systm). Th rasnsfr thisar multifld. Gnrally, hwvr, althugh phtnic intrcnnctin ntwrkshav significantly incrasd bandwidth, th cmplxity and cst f such systms, cupld with th inability f prcssr intrfacs t cp with high phtnic rats, usually ngats any xpctd bandwidth advantags. It is a miscncptin that mrly rplacing an xisting lctrnic intrcnnct with an ptical fibr quivalnt will rsult in a viabl architctural dsign. T truly tak advantag f phtnic tchnlgy, th ttal systm dsign must b rthught with an undrstanding f th strngths and waknsss f phtnics. 1 Rsarch rprtd hrin is supprtd in part by NSF grant MIP and DARPA cntract DAAL01-98-C Th Intrcnnctin Ntwrk Simulatr (ICNS) framwrk wasdvlpd t hlp valuat candidat architctural altrnativsthat xplit phtnic tchnlgy in thir dsign. Our spcific intrst is th us f phtnics in th prcssr-t-prcssr intrcnnctin ntwrk that isan intgral part f any multicmputr systm. Th dsign f ICNS was nt limitd t thisapplicatin, hwvr, and w hav mdld bth multicmputr systms and switching fabrics fr intrnt rutrs. Th us f phtnic tchnlgisasbuilding blcks fr multicmputr intrcnnctsisnt, asyt, a wll-studid subjct. Phtnics psss strngths and waknsss diffrnt frm lctrnics frm an intrcnnctin ntwrk standpint. Thus, fundamntal dsign spac paramtrs such as slttd-tim vrsus asynchrnus transmissin, buffrd vrsus unbuffrd switching, packt-basd vrsus mssag-basd transprt, tc. nd t b rcnsidrd whn dsigning a phtnic intrcnnct. ICNS wasdvlpd using th MODSIM III languag in a mdular, bjct-rintd mannr. It isdsignd t b xtndd as nw architcturs ar prpsd and nw prfrmanc qustins ar raisd. Fr this rasn, th simulatin framwrk must b flxibl nugh t allw variusntwrk cmpnntswith varid charactristics t b mdld faithfully. Asapplicatinsdmand highr prfrmanc frm th intrcnnctin ntwrk, th dsign f th ntwrk will b mr clsly guidd by th spcific targtd applicatins. Hnc it is imprtant t cnsidr bth th applicatin and th intrcnnctin ntwrk tgthr in th dsign prcss. ICNS taks spcial car t nsur that applicatin-lvl simulatin mduls can b asily intgratd with th intrcnnct mduls. This allws applicatin issus t b fully xplrd in tandm with intrcnnct issus in th dsign prcss. At a vry high lvl, an intrcnnctin ntwrk

3 MssagObj MTMssagObj MHMssagObj MCMssagObj GMssagObj NtwrkObj NdObj OutPrtRcObj links links and switchs switch fabric links Figur 1: A Gnric Intrcnnctin Ntwrk. can b abstractd as a systm cmpsd f s that gnrat and cnsum mssags, and links and switchs that facilitat th transprtatin f mssags frm n t anthr. Figur 1 shws a gnric intrcnnctin ntwrk. T achiv th dsird flxibility and xtnsibility, a mdular, bjct-rintd apprach isadptd asth principl dsign and dvlpmnt mthdlgy fr th simulatr. Fr xampl, a cmpnnt that mdls a simpl buffrlss switching lmnt can b nhancd t mdl a switching lmnt with a simpl FIFO buffr. Th FIFO buffr cmpnnt can b asily xtndd t mdl a priritizd multiquu buffr. A switching lmnt with priritizd multiquu buffr can furthr b nhancd t mdl a switching lmnt with schduling capability. Th schdulr mdul can b asily mdifid t prfrm schduling using diffrnt plicis. W hav usd ICNS t mdl a pair f systms. Th first is th Gmini intrcnnct, a paralll phtnic and lctrnic ntwrk that utilizslithium nibat ptical switchs t cnstruct a circuit-switchd highbandwidth path in th switching fabric. Th scnd isa phtnic multiring intrcnnct, in which 2-D arraysf Vrtical Cavity Surfac Emitting Lasrs(VC- SELs) and phtdtctrs ar usd t prvid highbandwidth I/O t/frm CMOS chips. Th varity in phtnic tchnlgisusd, aswll asth distinct architctursthat rsult, pint t th flxibility f th ICNS framwrk. Sctin 2 dscribs th ICNS implmntatin nvirnmnt, as wll as th bas classs and basic bjct typsthat frm th cr f ICNS. Sctin 3 dscribs th mdl usd t simulat such bjcts as links, switchs, prcssing nds, tc. Sctin 4 dscribs th usag f th simulatr. Sctin 5 prvids a dscriptin f th architcturs simulatd t dat using ICNS, including a dscriptin f th phtnic cmpnntsthat ar nabling tchnlgy fr ths archi- LinkObj MCLinkObj SwitchNdObj GCPUObj TrminalNdObj GSwitchObj GSwitch2x2Obj GSwitch2x2CCObj MQBuffrObj GTrminalObj GTrminalVOQObj GTrminalVOQFQObj GGnratrObj GOPObj GOPObj Figur 2: ICNS partial class diagram. GOPVOQObj tcturs. It als givs sm prfrmanc rsults that hav bn drivd using ICNS. Sctin 6 cncluds and dscribs th futur plans fr ICNS. 2 ICNS Implmntatin Envirnmnt and Bas Classs 2.1 Th MODSIM III Implmntatin Envirnmnt ICNS isimplmntd using th MODSIM III languag dvlpd by th CACI PrductsCmpany. Th MODSIM III runtim nvirnmnt isprvidd in th frm f a shard runtim library. Th MODSIM III cmpilr taksin MODSIM III surc cd, cmpils it int C++ cd, and thn uss th systm C/C++ cmpilr and linkr t cmpil th C++ cd and link th rsulting bjct cd with th runtim library t mak an xcutabl prgram. MODSIM III isa mdular, blck structurd languag that supprts bjct rintd prgramming. Its runtim nvirnmnt prvidsan implicit vnt quu and th binding f vntst vnt handlrs. Simulatin tim prgrssin and vnt schduling ar implicitly handld by th runtim nvirnmnt. Using MODSIM III, ICNS isbuilt in an bjct-rintd fashin. Subclassing and bjct cmpsitin ar usd xtnsivly. This will b shwn in subsqunt sctins.

4 A B B is a subclass f A A B B aggrgats many instancs f A A B B aggrgats n instanc f A link surc addrss ntwrk surc addrss squnc numbr mssag typ tim stamp link dstinatin addrss ntwrk dstinatin addrss lngth channl Figur 3: Brif nmnclatur fr th symbls usd abvinfigur ICNS Bas Classs ICNS is primarily cmpsd f th fllwing bas classs: MssagObj NdObj NtwrkObj Of th thr classs listd abv, intractins btwn th MssagObj and th NdObj gnrat th basic vnts that mv th simulatin frward. Figur 2 shws a class diagram using OMT-ntatin. Th symblsusd in Figur 2 ar xplaind brifly in Figur 3. Only th classs that mr dirctly mdl particular ntwrk r multicmputr cmpnntsar shwn MssagObj MssagObj mdls mssags in th systm. Th bas MssagObj class allws surc and dstinatin addrsss, tim f cratin f th mssag, and th lngth f th mssag t b rcrdd. Thr ar als a basic st f intrfacs that facilitat intractins btwn mssags and NdObj drivativs. Th intractin mchanism will b dscribd latr in Sctin 2.3. MssagObj has bn xtndd t allw spcificatin f mssag typs, cmmunicatin channls bing usd, and link lvl surc/dstinatin addrsss via th MTMssagObj, MCMssagObj, and MHMssagObj subclasss rspctivly. Fr xampl, th GMssagObj subclassd frm all th MssagObj classs mntind can mdl th mssag shwn in Figur 4. MssagObj can als mdl cntrl s that hlp cntrl th ntwrk. This is dn by assigning rsrvd typ valus t mssags that th nds (.g., switchs and s) rcgniz as cntrl s. T allw th ICNS t b spcializd t th applicatin, MssagObj als has a rfrnc t a gnral bjct typ that may b f any class spcifid by th usr (shwn as in Figur 4). This allws th Figur 4: Th GMssagObj. usr t attach applicatin spcific infrmatin t a MssagObj. Als, in th vnt that a cmplx multilayrd transmissin prtcl is chsn, th rfrnc can b usd t lt n MssagObj ncapsulat anthr MssagObj. This nabls simulatin f layrd prtclswhr unitsar ncapsulatd at diffrnt prtcl layrs NdObj NdObj mdls anything that accpts a MssagObj and latr passs th MssagObj t sm thr ntity (such as anthr NdObj). Each instanc f a NdObj hasan ID and a st f intrfacsthat intracts with MssagObj. NdObj by itslf prvids n usful functin xcpt t mandat th minimal st f intrfacs that allws any MssagObj subclass t intract with any NdObj subclass in a cnsistnt mannr. Fr xampl, MssagObj nds t intract with switchs, links, and s. Sinc th classs that mdl switchs, links, and s ar subclasss f NdObj, a MssagObj can intract with any n f thm using th sam intrfacs. Rgardlss f whthr a MssagObj is intracting with a LinkObj, SwitchNdObj, r GCPUObj, th MssagObj uss th sam intrfacs dscribd in Sctin 2.3. Sinc ths classs ar all subclasss f NdObj as shwn in Figur 2, thy prvid MsagObj with a cnsistnt st f intrfacs NtwrkObj NtwrkObj isa cntainr bjct that hldsth NdObj s. NtwrkObj prvids a singl pint-fntry fr usr prgrams t accss a particular NdObj aswll askping track f th numbr f sand switchs in th systm. 2.3 Intractins btwn MssagObj and NdObj Th intrcnnctin ntwrk is usd t snd mssags. This is mdld as mssags bing passd frm

5 n nd t anthr in ICNS. Th fllwing dscribs hw MssagObj and NtObj intract. Whn a MssagObj is passd t a NdObj, it first asks if th NdObj can prcss it immdiatly. If th rply is ngativ, th MssagObj asks th NdObj t plac it in th NdObj sbuffr. Othrwis, th MssagObj asks th NdObj hw lng it will tak t prcss th MssagObj. Th MssagObj will wait that amunt f tim and thn ask th NdObj t finish up th prcssing. Whn askd by a MssagObj whthr th NdObj can prcss th MssagObj immdiatly, th NdObj givsa simpl ys r n answr. Whn askd t buffr a MssagObj, th NdObj can buffr th MssagObj as rqustd, r discard th MssagObj shuld th NdObj nt hav a buffr r hav a full buffr. Whn askd hw lng it will tak t prcss a MssagObj, th NdObj rturns a ral numbr rprsnting th amunt f tim ndd t prcss th NdObj. Fr xampl, if th NdObj cncrnd is a link, thn prcssing a MssagObj rquirs snding th MssagObj thrugh th link. Th prcssing tim is simply th link dlay. Whn askd t finish prcssing a MssagObj, th NdObj usually ithr discards th MssagObj r passs th MssagObj t anthr NdObj. 3 Dscriptin f Slctd Objcts in a Simulatd Intrcnnctin Ntwrk This sctin dscribs th frmulatin f spcific bjcts in an intrcnnctin ntwrk simulatd using ICNS. Th slctd bjcts ar as fllws: Links Trminals Mssag Gnratr Buffr Cntral Prcssing Unit (CPU) Switchs 3.1 Links Links simply accpt mssags frm a nd and pass th mssags t anthr nd. A simpl link has a fixd bandwidth, a dstinatin, and a prpagatin dlay paramtr. Th dstinatin f a link is usually a switch nd r a. A simpl link is mdld by th LinkObj and isdpictd in Figur 5. Th LinkObj, shwn in th middl lft f Figur 2, is a subclass f th NdObj. Th MCLinkObj, a subclass f LinkObj, xtnds th LinkObj t mdl multichannl links. MCLinkObj mssag ging in at tim t mssag lngth is l link with bandwidth BW and prpagatin dlay d Figur 5: A simpl link. mssag cming ut, ging t anthr nd at tim t + l/bw + d is suitabl fr mdling such bjcts as frquncy divisin r spatial divisin multiplxd links. Th multichannl link mdl isdpictd in Figur 6. mssags ging in channl 1, BW1, dlay d1 channl 2, BW2, dlay d2 channl 3, BW3, dlay d3 channl 4, BW4, dlay d4 channl m, BWm, dlay dm Figur 6: A multichannl link. Channl rrr can als b mdld by spcifying an rrr prbability paramtr. Evry tim a mssag ntrsa channl, it isdrppd r markd crruptd with th rrr prbability spcifid. 3.2 Trminals Trminals gnrat and cnsum mssags. A prcssing nd, fr xampl, can b mdld as a. Whn mdling prcssing applicatins th snsr banks can als b mdld as s (i.., s that gnrat but d nt cnsum mssags). Fr xampl, a mdling a prcssing nd wuld hav a cnstructin similar t that shwn in Figur 7 whil a mdling a snsr bank wuld lk lik that shwn in Figur 8. mssags arriv via input link buffr mdul (MQBuffObj) nw mssags gnratd as th rsult f prcssing ld mssags CPU Mdul (GCPUObj) buffr mdul (MQBuffObj) Figur 7: A prcssing nd mdl. mssags snt via utput link mssags cming ut

6 Mssag Gnratr (GGnratrObj) input link 1 buffr utput link 1 buffr buffr mdul (MQBuffObj) intput link 2 buffr utput link 2 mssags snt via utput link Figur 9: A 2 2switch. Figur 8: A snsr bank mdl Mssag Gnratin Th GGnratrObj isimplmntd t mdl a mssag gnratr. Mssags can b gnratd accrding t a Pissn prcss with usr spcifid rats. Mssag lngths can b xpnntially distributd r fixd. Supprt fr gnrating mssags accrding t thr statistical mdls (such as Gaussian, Gamma, Erlang, Wibull, tc.) can b asily addd. An imprtant class f applicatins, Spac Tim Adaptiv Prcssing (STAP), gnrat mssags that ar ithr vry larg (n th rdr f hundrdsf kilbyts(kb) r largr) r vry small (n th rdr f tns r hundrds f byts) [1]. This typ f traffic can als b simulatd using GGnratrObj. GGnratrObj is usd within a sinc nly s can gnrat mssags. Aftr a mssag has bn gnratd, it may nt b pssibl t snd th mssag immdiatly bcaus f ntwrk cntntin. In this cas, a mssag nds t b buffrd. Buffring is simulatd using th MQBuffObj dscribd in th fllwing sctin Buffr Buffring isan imprtant aspct f intrcnnctin ntwrk dsign. MQBuffObj implmnts a flxibl multi-quud buffr mdul s that varius buffring tchniqusand plicis(such aspriritizd quu, virtual utput quus, fair quuing, tc.) can b simulatd. MQBuffObj dsnt implmnt quuing plicis but mrly prvids basic nquu/dquu, stat rprting, and quu managmnt functinst thr classs that mak us f it. Fr xampl, GTrminalObj sutput prts(mdld using th GOPObj and GOPObj) us MQBuffObj as thir buffring mduls. Buffring plicy is implmntd in GOPObj and GOPObj. Whn a mssag arrivs at a switch nd, that mssag may nt b prcssd immdiatly bcaus sm thr mssags ar bing prcssd. In this cas, th incming mssag can b buffrd in MQBuffObj Cntral Prcssing Unit (CPU) T facilitat applicatin-lvl simulatin, th GCPUObj cla sisprvidd. GCPUObj accpts and prcsss mssags and kps such statistics as CPU utilizatin. MQBuffObj isusd by GCPUObj t buffr mssags that arriv fastr than it can prcss. GCPUObj may als gnrat nw mssags in rspns t mssags rcivd. GCPUObj can b subclassd t simulat spcific applicatins in cnjunctin with attaching applicatin spcific t th MssagObj. W nt that GCPUObj prfrmsapplicatin-lvl simulatin nly and ds nt simulat transmissin prtcls. Simulatin f transmissin prtcls is prfrmd by th nd itslf. 3.3 Switchs At th hart f th switching fabric f a ntwrk ar th switchs. Gnrally, an n m switch accpts an incming mssag frm any f th n inputs, dtrmins which f th m utputs th mssag shuld b snt t using th mssag s hadr infrmatin, and snds th mssag t th utput. Th GSwitchObj prvids a framwrk upn which a gnral n m switch capabl f prfrming packt- aswll ascircuit-switching can b built. Ruting a mssag t its utput dstinatin is a prcss that taks tim. In th vnt that mssag arrival utpacs th ruting prcss, th incming mssags ar buffrd. In additin, th rat at which mssags ar rutd t th utput may als utpac th rat at which mssags can actually b snt ut n th utput link. Mssags ar als buffrd in this cas. Again, MQBuffObj can b usd t maintain th buffrs. Figur 9 shws a 2 2 switch with a cntralizd input buffr and sparat utput buffrs. Th GSwitch2x2CCObj (a subclass f SwitchNdObj) implmnts a switch similar t that dpictd in Figur 9.

7 4 ICNS Usag Prvius sctins hav dscribd varius bjcts prvidd by ICNS that can b usd t simulat variuscmpnntsin an intrcnnctin ntwrk. Ths cmpnntsnd t b cnnctd tgthr t frm a ntwrk. ICNS prvidsa prcdur that can b usd t build a ntwrk using th availabl cmpnnts. Th prcdur, calld BuildGNtwrk, radsin a plain txt fil that dscribs th tplgy f th dsird ntwrk. Th prcdur instantiats cmpnnt bjctsasndd, and cnnctsthm accrding t th tplgy spcifid. 1 Each cmpnnt hasvariusparamtrsthat nd t b spcifid. Fr xampl, th BuildGNtwrk prcdur ndst b tld what typsf s, switchs, and links it shuld instantiat as it prcsss a tplgy dscriptr fil. Each nds t b tld at what rat it shuld gnrat mssags, what th mssags lngth distributin shuld b; what ar th link bandwidths, tc. ThusICNS prvidsth ParamObj class that rads in a list f paramtr valusfrm a txt fil. Th paramtr valusar thn usd by th BuildGNtwrk prcdur t cnfigur th rlvant cmpnntsasthy ar instantiatd. Th paramtr dscriptr fil is simply a list f paramtr nam-valu pairs. Using th ParamObj and BuildGNtwrk facilitis, a simpl tp lvl prgram can b built t accpt tw fil nams(n paramtr fil and n tplgy fil), us ParamObj t rgistr all paramtr valus (in th paramtr fil), thn us th BuildGNtwrk prcdur t build a ntwrk accrding t th tplgy fil, start simulatin, and finally, cllct and display statisticsfrm variuscmpnntsat th nd f simulatin. T supprt bth th vrificatin f th simulatin mdlsaswll asimprvd undrstanding f th pratin f mdld systms, a st f visualizatin tls hasbn dvlpd. Th visualizatin tlsar drivn frm a static tplgy dscriptin fil and trac drivd frm th simulatin xcutin. Th tplgy dscriptin dfins th structur f th ntwrk: bjcts, switching nd bjcts, quus within s and switching nds, and linksbtwn bjcts. Links(bth lctrical and ptical) frm cnnctinsbtwn th sand th switching nds. Within ach f th s and switching nds, th mssag quus ar rprsntd graphically. Diffrnt mssag typs (.g., stup, tardwn, ) ar rprsntd by distinct clrs. Whn 1 Usrs can build a ntwrk by dirctly manipulating th bjcts if th BuildGNtwrk prcdur prvidd prvs t rstrictiv t th particular applicatin. in us, links tak n th clr f th mssag typ in transit. Simulatr drivd trac ncapsulats th dynamic activity prsnt in th ntwrk. This rflcts th stat f th quus, links, and switchs, and thr cmpnnts. Th visualizatin tls ar implmntd in Java (primarily fr prtability rasns). 5 Phtnic Intrcnnctin Ntwrks Thr ar tw primary systm dsigns that th ICNS framwrk hasbn usd t mdl t dat. Each f ths tw systms rlis n a distinct phtnic tchnlgy, and asa rsult, th architctursar apprciably diffrnt frm n anthr. Th ability fr ICNS t b applid t bth f ths systms attsts t its flxibility. 5.1 Th Gmini Intrcnnct Th Gmini intrcnnct isan xprimntal implmntatin f a nvl prcssr-t-prcssr intrcnnctin ntwrk fr tightly-cupld multicmputrs[1, 3, 7, 8]. It includsan nd-t-nd ptical path (including switching f th ptical s) fr high-bandwidth, larg vlum mssag dlivry. Th ptical switching is accmplishd using LiNbO 3 lctrptical 2 2 switchs [11, 15]. In additin, Gmini includsan lctrical path (in paralll with th ptical path) that bth cntrlsth ptical path (i.., stup f th lctrptical switchs) and dlivrs lw-latncy, small vlum mssags. Th Gmini intrcnnct uss a Banyan tplgy. Althugh thisisa blcking ntwrk, it prvidsth minimum numbr f switching stags thrugh th ntwrk, and hasth additinal advantag that ach gs thrugh th sam numbr f switchs. An 8 8 Gmini ntwrk isillustratd in Figur 10. As can b sn in th figur, ach ptical switch has an assciatd lctrical cuntrpart. Du t th absnc f buffring in th ptical dmain, th ptical ntwrk iscircuit switchd. This implisthat it will prfrm wll fr larg vlum mssags, fr which th latncy assciatd with circuit stup and tardwn can b amrtizd vr a larg mssag insrtin tim. By cntrast, th lctrical ntwrk is packt switchd. Hr, th dsign can b ptimizd fr lw-latncy dlivry f small mssags (ithr mssags r cntrl mssags) that d nt hav significant bandwidth rquirmnts. Elctrptical 2 2 switching lmnts ar th ky dvicsin th fabricatin f th Gmini N N ptical path. [11, 15]. Ths switching lmnts rly n th lctrptic ffct (i.., th applicatin f an

8 Gmini Switch Fabric incming ptical mssags incming lctrical mssags and s CPU Mdul r A B lng r shrt r lng mssag gnratr shrt Elctrical Output Prt buffr Optical Output Prt utging lctrical mssags and s cntrl prcssr (prcss at lin spd) buffr utging ptical mssags Figur 12: Mdl fr a attachd t th ntwrk. lctrical cntrl and switch lctrical path ptical switch ptical path ruting functin mdul infrmsth cntrllr whr t frward a packt aswll ashw t cntrl itscmpanin ptical switch whn a path stup rqust is bing prcssd. Figur 10: An 8 8 Gmini ntwrk. input 0 OutputPrt 0 utput buffr lctric fild t an lctrptical matrial changsth rfractiv indx f th matrial). Th rsult is a 2 2 ptical switching lmnt whs stat is dtrmind by an lctrical cntrl. This is illustratd in Figur 11, which shws a switching lmnt in th passthrugh stat as wll as in th crssvr stat. input 1 input buffr ruting tabl r ruting functin cntrllr Output Prt 1 utput buffr utput 0 utput 1 lctrnic cntrl = pass thrugh lctrnic cntrl = crssvr ptical switch cntrllr t ptical switch ptical inputs ptical utputs ptical inputs ptical utputs Figur 13: Mdl fr th Gmini lctrical switch. Figur 11: Elctrptical switching lmnts. Th ICNS mdl f th Gmini intrcnnct isdscribd nxt. Th s cnnctd t th Gmini ntwrk ar mdld as gnral purps prcssrs with lctrical and ptical intrfacs. Figur 12 dpictsth mdl f a. Ntwrk cntrl s ar assumd t b prcssd at lin rat. Hnc thr isn input buffr fr th. Th has sparat utput buffrs fr mssags intndd fr diffrnt ntwrks. Th cntrllr markd A dispatchs incming packtsaccrding t packt typ. Th cntrllr markd B dispatchs utging traffic accrding t mssag typ and lngth. Figur 13 shws th mdl f a Gmini 2 2 lctrical switch. Th lctrical switch has a shard input buffr and sparat utput buffr at ach utput. A In th simulatin, th ptical and lctrical switchs ar mdld asn bjct. Th linksthat cnnct th switchs ar mdld as n link ntity with tw channls, n channl carris th lctrically switchd traffic, th thr th ptically switchd traffic. Th fllwing prfrmanc rsults cm frm discrt-vnt simulatins using ICNS. Figur 14 shws prfrmanc rsults (man mssag dlay vrsus ffrd lad) fr fur Gmini ntwrksusing bth abasicstup-tardwn prtcl and a virtual utput quuing prtcl. Th stup-tardwn prtcl uss th lctrical cntrl ntwrk t stablish a path in th ptical ntwrk, snds th via th ptical ntwrk, and thn tarsdwn th path. Th virtual utput quuing prtcl maintainssparat quus fr ach dstinatin at th surc, uss th lctrical cntrl ntwrk t attmpt t stablish a path t all dstinatins that hav mssags t b dlivrd, and

9 slcts n f th succssfully rsrvd paths t dlivr. Th paramtrs chsn fr th simulatd ntwrks ar such that th rati f avrag mssag lngth t cntrl lngth is 16K and th rati f ptical link bandwidth t lctrical link bandwidth is12. avrag dlay (s) Cmparisn f stup tardwn (ST) and VOQ prtcl (Exp. Dist. Lngths) 4x4 ST 8x8 ST 16x16 ST 32x32 ST 4x4 VOQ 8x8 VOQ 16x16 VOQ 32x32 VOQ ffrd lad Figur 14: Avrag ptical mssag dlay using th basic stup-tardwn prtcl and th virtual utput quuing prtcl [3]. Th fur ntwrks wr simulatd using th sam st f paramtrs. Mssags ar gnratd at ach accrding t an indpndnt and idntically distributd Pissn prcss. Thir dstinatins ar unifrmly distributd t all utputs and mssag lngthsar xpnntially distributd. In th figur, th lad axisisnrmalizd t th thrtical maximum thrughput. Whil thrughput is clarly limitd using th basic stup-tardwn prtcl, w s that th ptical ntwrk can prvid cls t 100% thrughput using th VOQ prtcl 2. At th sam tim th figur als shws that withut VOQ saturatin ccurs at an ffrd lad f btwn 30% and 50% capacity. Tabl 1 shws th lad xprincd by th cntrl s n th lctrical ntwrk. W s that fr th paramtrs chsn, snding multipl stup rqusts (asrquird by th VOQ prtcl) dsnt lad t significant cngstin in th lctrical ntwrk. This supprts th us f th lctrical ntwrk fr snd- 2 McKwn t al. hav prvn in [14] that 100% thrughput is achivabl in a nn-blcking, input-quud switch using a nn-fifo quuing schm such as VOQ assuming randm, hmgnus traffic. It rmains t b sn whthr such prfrmanc is achivabl in a blcking ntwrk such as th Banyan ntwrk usd in Gmini. ing shrt, latncy-snsitiv mssags (bth cntrl and ) withut incurring significant quuing dlays. Tabl 1: Elctrical ntwrk lad using th VOQ prtcl [3]. Ntwrk siz Elctrical ntwrk lad 4 4 < 0.6% 8 8 < 1.2% < 2.4% < 4.6% Th abv st f xampl prfrmanc simulatins illustrat th utility f ICNS fr valuating altrnativ quuing prtcls. In [7], th ICNS framwrk is usd t dmnstrat th imprtanc f a fairnss prtcl in th circuit-switchd ptical path as wll as prsnt prfrmanc rsults with th fairnss prtcl in plac. 5.2 A Phtnic Multiring Th phtnic multiring is a systm that xplits nw dvlpmntsin Vrtical Cavity Surfac Emitting Lasr (VCSEL) tchnlgy [2, 4, 6, 12]. Th nabling tchnlgy fr this systm is th availability f 2-dimnsinal arrays f VCSELs and dtctrs bndd t silicn circuitry [10]. Th unin f silicn prcssing with GaAs-basd ptlctrnics prvids a pwrful cmbinatin, significantly incrasing th cmmunicatinsbandwidth availabl ff-chip. Prttyp intrcnnctshav bn cnstructd with arraysf VCSELsand phtdtctrsn a singl chip [16]. In this systm, th VCSELs arrays and phtdid arrayswr flip-chip bndd t a CMOS chip using htrgnus intgratin tchniqus. Althugh th dmnstratin f [16] usd bulk ptics t dlivr light btwn ICs, ptical paths hav bn dsignd using bth rigid ptical links [5] (usful fr chipt-chip linksn a bard), and flxibl fibr imaging guids [9] (usful fr bard-t-bard links). Th availability f a larg numbr f VCSELdtctr pairs in th ptical intrcnnct suggsts th partitining f th ptical linksint stswith ach st bing assciatd with an individual channl (i.., spac-divisin multiplxing). Figur 15 illustrats th allcatin f VCSELsand dtctrsfr a fur channl systm utilizing arraysf ptical lmnts. Asshwn in th tp f th figur, n quartr f th lmnts ar usd fr ach channl. Each squar in th tp viw f Figur 15 cntainsa singl VCSEL r dtctr. If th individual lmnt cmmunicatsat

10 1 Gb/s, thisyilds16 2 /4 = 64 Gb/spr channl. Th sid viw f th figur illustrats (cncptually) hw tw adjacnt chipsmight cmmunicat. Tp Viw Sid Viw 16 Transmittr Rcivr Fr spac intrcnnctin ntwrk channl 1 channl 2 channl 3 channl Transmittr Rcivr Figur 15: Allcatin f VCSEL-dtctr pairst a fur channl systm VCSEL-dtctr arrays ar usd, with a 4 16 array allcatd t ach channl. Th phtnic tchnlgisusd in thisdsign ar mst cst-ffctiv whn usd with a fan-in and fanut f n and a tplgy mting thisfan-in/fan-ut gal isa ring. Whil thr ar many apprachst dvlping a ring basd intrcnnct, givn th vry high bandwidthsavailabl, th multiring [13] dsign f Figur 16 hasbn chsn utsid ring, fr xampl, is assciatd with nd 1 and th nxt-t-utsid ring is assciatd with nd 2. Th insid ring is assciatd with nd 4. With th multiring tplgy, ach ring can b thught f asa daisy chain trminating at th dstinatin nd. Thus, cmmunicatin btwn nd i and nd j rquirs that nd i snd its mssag n th ring which has nd j asth dstinatin. Th prfrmanc valuatin fr this systm xplrs th ability t rcnfigur th bandwidth assciatd with ach ring in th multiring by changing th numbr f VCSEL-dtctr pairs assciatd with ach channl (and hnc th input bandwidth availabl t ach dstinatin). Th applicatinsf intrst ar nsin which cmputatin and cmmunicatin altrnat with n anthr (i.., prcd in phass). Th prfrmanc rsults prsntd hr ar drivd frm 2 ral applicatins(synthtic aprtur radar (SAR) imag frmatin and a bamfrming applicatin) and 5 synthtic applicatins. Th synthtic applicatins hav frm 3 t 6 cmmunicatins phass. Thir cmmunicatins pattrns ar randmly chsn frm th fllwing st: bradcast, rduc, all-t-all, and pint-t-pint. Th flws and mssag sizs ar als randmly gnratd. An ICNS simulatin mdl was usd t valuat th prfrmanc f th rcnfigurabl phtnic multiring. An 8 nd systm was simulatd, with cmmunicatin traffic gnratd accrding t th 2 ral applicatins and 5 synthtic applicatins dscribd abv. Fr ach applicatin, th systm was simulatd twic. Th initial simulatin utilizd a unifrm bandwidth allcatin t ach ptntial flw (all surc-dstinatin pairs ar allcatd an qual fractin f th ttal bandwidth). In th scnd simulatin th intrcnnctin ntwrk wasrcnfigurd at th start f ach cmmunicatin phas t prvid an ptical bandwidth allcatin bst suitd t th cmmunicatin pattrn rquird by th algrithm in that phas. 3 Figur 16: Multiring. In th 4-nd xampl f Figur 16, ach f th fur rings is assciatd with a givn dstinatin nd. Th Figur 17 shws th man, minimum, and maximum spdup btaind fr ach typ f cmmunicatin pattrn, indpndnt f th applicatin in which it isfund. Th spdup isdfind asth rati f th cmmunicatin cmpltin tim with a unifrm bandwidth allcatin t th cmpltin tim with a rcnfigurd allcatin. Th prfrmanc imprvmnt is significant acrss th bard, indicating a clar bnfit t rcnfigurability in th intrcnnctin ntwrk.

11 Spdup P2P Bcast Rduc A2A Figur 17: Cmmunicatinsspdup with a rcnfigurabl multiring. [2] 6 Cnclusins and Futur Wrk In this dcumnt w hav prsntd th dsign f th intrcnnctin ntwrk simulatr, ICNS. W hav dscribd th classs that frm th cr f ICNS. W hav als shwn hw varius cmpnnts in a multicmputr intrcnnctin ntwrk can b mdld using th classs prvidd by ICNS. Th classs can b subclassd and xtndd t mdl cmpnnts with richr functins. ICNS has bn usd t study th Gmini ntwrk and a phtnic multiring ntwrk. W ar currntly xtnding th multiring ntwrk simulatin t includ dynamic rcnfigurability and xplring itsus asa switching fabric fr an intrnt rutr. Rfrncs [1] R. Chambrlain, M. Franklin, R. Krchnavk, and B. Baysal. Dsign f an ptically-intrcnnctd multicmputr. In Prc. f 5th Int l Cnf. n Massivly Paralll Prcssing Using Optical Intrcnnctins, pags , Jun [2] R. Chambrlain, M. Franklin, and P. Krishnamurthy. Prfrmanc valuatin f a rcnfigurabl, mbddd phtnic multiring intrcnnctin ntwrk. In Prc. f 5th High Prfrmanc Embddd Cmputing Wrkshp, Nvmbr [3] R.D. Chambrlain, M.A. Franklin, and Ch ng Shi Baw. Gmini: An ptical intrcnnctin ntwrk fr paralll prcssing. IEEE Trans. n Paralll and Distributd Systms, (inprss). [4] R.D. Chambrlain, M.A. Franklin, and A. Mahajan. VLSI phtnic ring intrcnnct fr mbddd multicmputrs: Architctur and prfrmanc. In Prc. f 14th Cnf. n Paralll and Distributd Cmputing Systms, August [5] M. Chataunuf t al. Dsign, implmntatin and charactrizatin f a 2-D bi-dirctinal frspac ptical link. In Prc. f Optics in Cmputing, pags , Jun [6] Ch ng Shi Baw, R.D. Chambrlain, and M.A. Franklin. Dsign f an intrcnnctin ntwrk using VLSI phtnics and fr-spac ptical tchnlgis. In Prc. f 6th Int l Cnf. n Paralll Intrcnncts, pags52 61, Octbr [7] Ch ng Shi Baw, R.D. Chambrlain, and M.A. Franklin. Fair schduling in an ptical intrcnnctin ntwrk. In Prc. f 7th Int l Symp. n Mdling, Analysis, and Simulatin f Cmputr and Tlcmmunicatins Systms, pags56 65, Octbr [8] Ch ng Shi Baw, R.D. Chambrlain, M.A. Franklin, and M.G. Wrightn. Th Gmini intrcnnct: Data path masurmnts and prfrmanc analysis. In Prc. f 6th Int l Cnf. n Paralll Intrcnncts, pags21 30, Octbr [9] H. Ksaka t al. A tw-dimnsinal ptical paralll transmissin using a vrtical-cavity surfac mitting lasr array mdul and an imag fibr. IEEE Phtn. Tch. Ltt., 9: , [10] Y. Li, E. Tw, and M. Hany, ds. Prc. n Shrt Distanc Optical Intrcnnctins in Digital Systms. IEEE, [11] Lucnt Tchnlgis. Guidd wav ptical switch prducts. Prliminary sht, [12] A. Mahajan, M.A. Franklin, and R.D. Chambrlain. Fairnss issus in an mbddd phtnic ring intrcnnct. In Prc.f4thHighPrfrmanc Embddd Cmputing Wrkshp, Sptmbr [13] M. Marsan t al. All-ptical WDM multi-rings with diffrntiatd QS. IEEE Cmmunicatins Magazin, pags58 66, Fbruary [14] N.W. McKwn, V. Anantharam, and J. Walrand. Achiving 100% thrughput in an inputquud switch. In Prc. f Infcm, March [15] E.J. Murphy, T.O. Murphy, R.W. Irvin, R. Grncavich, G.W. Davis, and G.W. Richards. Enhancd prfrmanc switch arrays fr ptical switching ntwrks. In Prc. f ECIO, April [16] D. Plant t al. A 256 channl bi-dirctinal ptical intrcnnct using VCSELsand phtdidsn CMOS. In Prc. f Optics in Cmputing, pags , Jun 2000.