ToappearinJ.ofParallelandDistributedProcessing. TheGeneralizedDimensionExchangeMethodforLoad Balancingink-aryn-cubesandVariants

Transcription

1 ToappearinJ.ofParallelandDistributedProcessing TheGeneralizedDimensionExchangeMethodforLoad Balancingink-aryn-cubesandVariants DepartmentofComputerScience,TheUniversityofHongKong,HongKong DepartmentofComputerScience,ShantouUniversity,P.R.China FrancisC.M.Lau Cheng-ZhongXu 1

2 changeparameterthatgovernsthesplittingofload Abstract TheGeneralizedDimensionExchange(GDE)methodis afullydistributedloadbalancingmethodthatoperates inarelaxationfashionformulticomputerswithadirect communicationnetwork.itisparameterizedbyanex- loadbalancing.anoptimalwouldleadtothefastest betweenapairofdirectlyconnectedprocessorsduring resultedintheoptimalforthebinaryn-cubes.inthis convergenceofthebalancingprocess.previousworkhas paper,wederivetheoptimal'sforthek-aryn-cube networkanditsvariants thering,thetorus,thechain, andthemesh.weestablishtherelationshipsbetween theoptimalconvergenceratesofthemethodwhenappliedtothesestructures,andconcludethatthegde relaxation-basedmethod,thediusionmethod. revealthesuperiorityofthegdemethodtoanother theoptimal'sdospeedupthegdebalancingproceduresignicantly.becauseofitssimplicity,themethod isreadilyimplementable.wereportontheimplemen- inwhichtheimprovementinperformanceduetogde balancingissubstantial. tationofthemethodintwodata-parallelcomputations Wefurthershowthroughstatisticalsimulationsthat methodfavorshighdimensionalk-aryn-cubes.wealso ing,k-aryn-cubenetworks,messagepassingmulticom- puters,circulantmatrices. Keywords.dimensionexchangemethod,loadbalanc- Method Correspondence:DrF.C.M.Lau,Departmentof ComputerScience,TheUniversityofHongKong,PokfulamRoad,HongKong.Fax:(852) , Runninghead.GeneralizedDimensionExchange 2

3 ListofSymbols D() Pij M() G=(V;E)systemgraph (G) G=(V;E)-colorgraph;isthechromaticindexofthegraphG diusionmatrix generalizeddimensionexchangematrix setofdistinctcolorpathsfromvertexitovertexj,withtypicalelementpij. degreeofthegraphg (M) Wt opt wti opt workloaddistributionattimet workloadofnodepiattimet optimaldiusionparameter optimalexchangeparameter (D) R1(M())asymptoticconvergencerateofthesequencefMtg convergencefactorofthegdemethod convergencefactorofthediusionmethod MCk MRk MMk1;k2;:::;knGDEmatrixofthecolormeshofsizek1k2kn MTk1;k2;:::;knGDEmatrixofthecolortorusofsizek1k2kn DCk GDEmatrixofthecolorchainofsizen GDEmatrixofthecolorringofsizen DTk1;k2;:::;kndiusionmatrixofthecolortorusofsizek1k2kn DRk DMk1;k2;:::;kndiusionmatrixofthecolormeshofsizek1k2kn improvementduetoremapping diusionmatrixofthechainofsizen diusionmatrixoftheringofsizen 3

4 1Introduction Weconsidertheproblemofdynamicloadbalancingin multicomputers.multicomputersareaclassofparallel machinesthatarecomposedofmanyautonomousprocessorsinterconnectedbyacommunicationnetwork[1]. Theprocessorsdonotshareanymemoryandtheycommunicateamongthemselvesviamessagepassing.From timetotime,theworkloadthatisspreadacrossthe processorsisfoundtobeinanunbalancedstate;load balancingistheninitiatedtobalancetheworkload.dimensionexchange(de)isoneofthefewdistributedload balancingmethodsthatoperateinarelaxationfashion forpoint-to-pointnetworks(adetailedsurveycanbe foundin[20]).withthedemethod,aninstanceof loadbalancingiscarriedoutasasequenceof\sweeps". Duringeachsweep,aprocessorcomparessuccessively itsworkloadwiththatofeachofitsnearestneighbors; followingeachsuchcomparison,anexchangeoperation isexecutedtoequalizetheworkloadbetweenthisnode andtheneighboringnodeconcerned.alternatively,insteadofexchangingworkloadson-the-y,theloadbalancingprocedurecanbedividedintotwophases:inthe rstphase,deisemployedtoworkouttherevisedload \indices"thatcorrespondtoabalancedstate;thenin thesecondphase,theactualloadmigrationswouldtake place.thismakesthemethodmoreapplicabletosituationsinwhichtheworkloadinvolveslargeamountsof data. TheDEmethodwasinitiallyintensivelystudiedin hypercube-structuredmulticomputers[14,15,5].since thesetofneighborsofaprocessorcorrespondexactlyto thedimensionsofthehypercube,asweepoftheiterativeprocessisequaltogoingthroughallthedimensions once.cybenkoprovedthatregardlessoftheorderin whichthesedimensionsareconsideredinasweep,this simpleloadbalancingmethodyieldsauniformdistributionfromanyinitialworkloaddistributionafterone sweep[5].healsorevealedthesuperiorityofthede methodtoanotherrelaxation-basedmethod,thediusionmethod[3,5],whenappliedtohypercubes.this theoreticalresultwassupportedinpartbytheexperimentcarriedoutbywillebeek-lemairandreeves[18]. TheDEmethodisnotlimitedtohypercubestructures.Hosseinietal.analyzedthemethodasapplied toarbitrarystructuresbasedonedge-coloringofundirectedgraphs[8].withedge-coloring,theedgesofa givengrapharecoloredwithsomeminimumnumberof colorssuchthatnotwoadjoiningedgesareofthesame color.a\dimension"isthendenedtobethecollectionofalledgesofthesamecolor.obviously,anndimensionalhypercubecanbecoloredwithaminimum ofncolors.duringeachiterationsweep,alldimensions (colors)areconsideredinturn.sincenotwoadjoining edgeshavethesamecolor,eachprocessorneedstodeal withonlyoneneighboratatimeduringasweep.clearly, foranarbitrarystructure,thedemethodcannolonger yieldauniformworkloaddistributioninasinglesweep. Nonetheless,Hosseinietal.showedthatgivenanyarbitrarystructure,theDEmethodconvergeseventuallyto auniformdistribution[8]. TheDEmethodischaracterizedby\equalsplitting" ofworkloadbetweenapairofneighboringprocessorsat everycomparison,whichwasshowntobeoptimalinhypercubestructuresbutnotnecessarilysoinotherstructuresthroughouranalysis[21].inthatpaper,wegeneralizedthedemethodbyaddinganexchangeparametertogoverntheamountofworkload(insteadofalways half)exchangedateverystep.thismethodiscalled thegeneralizeddimensionexchange(gde)method.we modeledthisgeneralizeddemethodusingamatrixiterativeapproachandderivedthenecessaryandsucient conditionforitsconvergence. Inthispaper,wecontinueouranalysisoftheGDE methodasappliedtothefamilyofk-aryn-cubeswhich includethering,thechain,thetorus,andthemesh.a k-aryn-cubeisastructurewithndimensions,knodes ineachdimensions[6,10].theringandthehypercubearespecialcasesofthek-aryn-cube.aringof knodesisak-ary1-cube,andann-dimensionalhypercubeisa2-aryn-cube.then-dimensionaltorusisa generalizationofthek-aryn-cube,whichallowsdierentnumbersofnodesindierentdimensions.takea ringandatorusandstripthemofalltheend-round connections,wegetachainandaringrespectively.we 4

5 limitourscopetothesestructuresbecausetheyarethe mostpopularchoicesoftopologiesincommercialparallel computers[10,13,16].examplesincludethehypercubestructuredintelipsc/860andncube/2,themeshstructuredintelparagon,inteltouchstonedelta,iwarp, andametek2010. Themaincontributionofthispaperisthederivation oftheoptimalexchangeparametersinclosedformfor thefamilyofk-aryn-cubes.theoptimalsolutionsfor thesestructuresareofconsiderablevaluebecausethere isthisrealneedinpracticalsituationsofchoosingan exchangeparameterthatwouldleadtothefastestconvergenceofthebalancingprocedure.apreviewofthese optimalparameterswithoutproofshasbeenincludedin ourpreviouspaper[21].asubsetoftheproofs,whichare basedoncirculantmatrixtheory[7],willbepresentedin thispaper.wecapitalizeonthemodelingpowerofcirculantmatrices,whichismostevidentincasesinwhich thestructuresconcernedcanberecursivelydened. Theotherimportantcontributionsofthispaperincludetheestablishmentoftherelationshipsbetweenthe convergenceratesofthesestructuresandaproofofthe superiorityofthegdemethodtothediusionmethod. Thelatteriswithrespecttotheconvergenceratesof thetwomethodswhenappliedtothefamilyofk-aryncubes.thematrixanalysisrevealstheasymptoticconvergenceratesbutshedslittlelightontheexactnumber ofsweepsneededforbalancing.therefore,weusestatisticalsimulationstoobtaintheactualnumberofsweeps requiredbygdebalancinginthesestructures.this numberturnsouttobeencouraginglysmallinallthe caseswetriedwhenusingtheoptimalparameterswederived.thisaddsalotofweighttothepracticalityofthe GDEmethod.Infact,themethodhasbeenemployedin theimplementationsoftworealisticdata-parallelcomputations,andtheimprovementovertheversionswithoutloadbalancingissubstantial.wegiveabriefreport ontheseimplementations. Therestofthepaperisorganizedasfollows.InSection2,wereviewtheGDEmethodanditsconvergence propertiesforthegeneralcase.insection3,weanalyze thegdemethodforthek-aryn-cubeanditsvariants, andderivetheiroptimalexchangeparameters.insection4,wemakeacomparisonbetweenthegdemethod andthediusionmethod.section5reportsontheresultsofastatisticalsimulationconcerningthenumberof iterationsweepsaswellasndingsfrompracticalimplementations.weconcludeinsection6withasummary oftheresultsanddiscussionoffurtherwork. 2TheGDEmethod Themodeloftheunderlyingsystemandcomputation assumedinthisstudyissimilartothatin[3,5,8,21]. Specically,themulticomputerweconsiderconsistsof anitesetofhomogeneousprocessorsinterconnectedas apoint-to-pointnetwork.thecommunicationlinksare bi-directionalandtheprocessorsinteractsynchronously withoneanother.werepresentsuchasystembyasimpleconnectedgraphg=(v;e),wherevisasetof processorslabeled1throughn,andevvisa setofedges.everyedge(i;j)2ecorrespondstothe communicationlinkbetweenprocessorsiandj.the underlyingparallelprogramisassumedtocomprisea largenumberofindependentprocesseswhicharethe basicunitsofworkload.oneormoreprocessesmay berunninginaprocessoratanytime.thetotalworkloadisassumedxed i.e.,noprocessesarecreatedor killed duringtheexecutionoftheloadbalancingprocedure.wequantifytheworkloaddistributionbyavector W=(w1;w2;:::;wN)T,wherewidenotestheworkload ofprocessoriwhichisintermsofthenumberofresiding processes.weassumethenumberofprocessesislarge sothattheworkloadofanodeisaninnitelydivisiblerealquantity.itisnotdiculttoseethatwithout thisassumption(resultinginthe\integerversion"which willbediscussedinsection5)ourresultsstillhold.the loadbalancingtaskistoredistributethesystemworkloadsuchthateachnodewouldendupwiththesame w=pwi=n,i=1;2;:::;n. Notethattheloadbalancingproblemresemblesin certainwaysanotherdistributeddecisionproblem,the agreementproblem[2].thislatterproblemrequiresthe nodesofasystemtoreachanagreementonacommon 5

6 scalarvalue,suchastheaverage,themaximum,orthe minimum,basedontheirownvalues.theloadbalancingproblem,however,requiresthenodesnotonlyto ecientmanner. justtheirworkloadsaccordinglyinanautomaticand reachanagreementontheaverageload,butalsotoadlems.thestaticworkloadassumptionisvalidincasestrictive,butyetitisapplicabletomanypracticalprob- wherethecomputationistemporarilysuspendedforload balancingandresumedafterloadbalancing.thisiswhy tuningtheeciencyoftheloadbalancingisoftoppriority.exampleofsuchcasescanbeeasilyfoundindy- Thecomputationmodeljustdescribedmightseemre- isdominatedbytheexecutiontime.thepracticalapplicabilityofiterativeloadbalancingwillbedemonstratenamicremappingofmulti-phasedata-parallelcomputations[12,11].theassumptionofindependentprocesses throughimplementationofdataparallelcomputations andcommunicationineachphaseandtheperformance isalsoreasonableinthiskindofcomputationsbecause theprocessingnodeswouldalternatebetweenexecution edgesofgaresupposedtobecoloredbeforehandwith insection5. theleastnumberofcolors(,say),andnotwoadjoining withintegersfrom1to,andrepresentthe-colorgraph edgesareassignedthesamecolor.weindexthecolors asg=(v;e),ofwhicheisasetof3-tuplesofthe form(i;j;c),(i;j;c)2eifandonlyifcisthechromatic TheGDEmethodisbasedonedge-coloringofG.The indexoftheedge(i;j)2e.figure1showsexamples ofcolorgraphsofringsandchains.thenumbersin parenthesesaretheassignedchromaticindices. Figure1:Examplesofcoloredgraphs 6 1 (3) (c) (a) (b) (d) i,theexchangeofworkloadwithaneighborjisexecuted goingthroughallthechromaticindicesonce thisiswhy sweepoftheiterativeprocess.asweepcorrespondsto weneedtousetheleastnumberofcolors.forprocessor loadwitheachofitsneighborsinturnaccordingtothe orderinwhichchromaticindicesareconsideredineach WiththeGDEmethod,aprocessorwouldexchange wherewiandwjarethecurrentworkloadsofprocessorsi andjrespectively,andistheexchangeparameterchosenbeforehandforthegivennetwork.notethatwhen in[5,8,15]. =1=2,theGDEmethodisreducedtotheDEmethod wi=(1?)wi+wj calworkloadofprocessoriatsweept.thentheoverall workloaddistributionatsweeptisdenotedbythevectorwt=(wt1;wt2;:::;wtn)t.supposew0istheinitial Fora-colorgraph,asweepoftheGDEalgorithm index,t=0;1;2;3;:::,andwti(1in)bethelo- comprisesstepswhichwillcoveralltheneighborsof everynodeforworkloadexchange.lettbethesweep distributioninthesystematsweeptcanbemodeledby wherem()iscalledthegdematrixofthe-color workloaddistribution.thenthechangeoftheworkload graphg,andm()=m()m?1():::m1(). theequationwt+1=m()wt loaddistributionofthesystematstepcofsweept. EachMc()(1c)reectsthechangeofthework- Tomakedynamicloadbalancingwork,therearetwo R1(M()). latterisreectedbytheasymptoticconvergencerate, atitsterminated,load-balancedstate.theformerconcernstheconvergenceofthesequencefmt()g,andthe mainissues.oneistheterminationcondition;theother iseciency,i.e.,thetimeneededforthesystemtoarrive andsucientconditionfortheterminationofthegde 1,itcanthenbeshownthat0<<1isanecessary stochasticwhen01andprimitivewhen0<< GiventhefactthatM()isnon-negativeanddoubly

7 methodfromanyinitialloaddistribution(theorem3.1 of[21]). Regardingtheconvergencerate,theeigenvaluesof M()playafundamentalrole.Letj(M())(1 jn)betheeigenvaluesofm(),and(m())and (M())bethedominantandsubdominanteigenvaluesrespectivelyofM()inmodulus.Since(M()) isuniqueandequalto1,itfollowsthatr1(m())=?ln(m()).(m())isalsoreferredtoastheconvergencefactorofthegdemethodinthecorresponding -colorgraph.thus,thetaskhereistochooseaso that(m())isascloseto0aspossible,i.e.,r1(m()) aslargeaspossible. Tondtheminimum(M()),wehavetorstconstructtheGDEmatrixM().Therepresentationof eachelementmijinm()isbasedontheconceptof colorpath.acolorpathoflengthlfromvertexito vertexjisasequenceofedgesoftheform (i=i0;i1;c1);(i1;i2;c2);:::;(il?1;il=j;cl) whereallintermediateverticesis(1sl?1)aredistinctandc1>c2>:::>cl1.itindicatesthat processoriwillreceivesomeworkloadfromprocessorj alongthepathinaniterationsweep.twocolorpaths fromitojingaresaidtobedistinctiftheirintermediateverticesdonotcoincideatall.allthedistinct colorpathsfromitojcompriseasetpij.sincethe computationformulasfortheelementsofm()willbe referredtofrequentlyintheremainderofthispaper,we reproducebelowthelemmathatdenesthem.examplesofgdematricescanbefoundinlatersectionsof thispaper. Lemma2.1(Lemma3.2of[21])LetM()betheGDE matrixofa-colorgraphg.if0<<1,thenfor 1i;jN mij=pp2pij((1?)rplp) i6=j mii=(1?)(i)+pp2pii((1?)rplp) wherelpisthelengthofthecolorpathp2pijofthe form(i=i0;i1;c1);(i1;i2;c2);:::;(ilp?1;ilp=j;clp); andrp=plp s=0ns,wheren0isthenumberofincident edgesofiwhosechromaticindexislargerthanc1;nlpis thenumberofincidentedgesofjwhosecolornumberis smallerthanclp;andns(1slp?1)isthenumber ofincidentedgesofiswhosechromaticindexislarger thancs+1andsmallerthancs. Ourobjectiveistodeterminetheoptimalexchange parameteroptforagivengdematrix,whichwould minimize(m())andmaximizetheconvergencerate. Forarbitrarynetworks,thisisanopenprobleminmatrix theory[17].forsomenetworkswitharegulartopology, however,itispossibletoanalyzeexactlytheeectof ontheconvergencerateaswellastoderivetheoptimal exchangeparameter.in[21],weprovedthat=1=2 (equalsplittingofworkloadbetweenapairofnearest neighbors),whichwas\built-in"inthedemethod,is indeedtheoptimalchoiceforcertainstructures(thehypercube,forexample).forotherstructures,suchasthe k-aryn-cube,however,=1=2isnottheoptimalchoice, aswewillshowinthenextsection. 3Analysisofthek-aryn-cube andvariants Webeginwiththeringstructure,i.e.,ak-ary1-cube, andthengeneralizeittothen-dimensionalk1k2 kntorus.theanalysisofthetorusdependsonthe analysisoftheringastheformercanbetreatedasan assemblyofrings.themainresultforthek-aryn-cube followstriviallyfromtheresultsforthetorus. Themodelingtoolweusefortheanalysisisaspecialkindofmatricescalledblockcirculantmatrices.It happensthatthegdematricesofthe\even"casesof theabovestructures evennumberofnodesinevery dimension areblockcirculantmatrices.weconcentrateontheseevencasesfortheremainderofthispaper,andmaketheremarkherethattheresultsforthe evencasesshouldbeapplicable(approximately)tothe non-evencases.wegivesimulationresultsandasimple argumentinsection5tosupportthis.nevertheless,the analysisofthenon-evencasesshouldstillbeaninterest- 7

8 ingtheoreticalproblemtotackle. culantmatrices[7]. whichcanbeeasilyderivedbasedonthetheoryofcir- thefollowingtwolemmasconcerningblockcirculantmatricesanddirectproductsofmatrices.weomittheproofs Forthesubsequentanalysis,weneedtomakeuseof Ifr=1,ablockcirculantmatrixdegeneratestoacirculantmatrix. Lemma3.1LetmatrixA=(A1;A2;:::;Am).Then A2A3:::A11CA AmA1::::::Am?1 Thenablockcirculantmatrixisamatrixoftheform LetA1;A2;:::;Ambesquarematricesoforderr. theeigenvaluesofthematrixaarethoseofmatrices matrixa=(a1;a2),thentheeigenvaluesofaare thoseofa1+a2,togetherwiththoseofa1?a2. A1+!jA2+:::+!j(m?1)Am,j=0;1;:::;m?1,where matrixofordermndenedby respectively.thenthedirectproductofaandbisa!j=cos2j LetAandBbesquarematricesofordermandn, m+isin2j m,i=p?1.inparticular,if Lemma3.2LetA,Bbesquarematricesoforderm j=1;2;:::;n.thentheeigenvaluesofabare i(a)j(b). andnwitheigenvaluesi(a)andj(b),i=1;2;:::;m, a1;0ba1;1ba1;m?1b am?1;0bam?1;1bam?1;m?1b1ca Forsimplicityofnotation,welet U= V= Q= Q2!= Q1 U2!= U1 V1 V2!= 1? (1?)2 (1?)(1?)2 (1?)2(1?)!; 1?!: 2(1?)!; 8 3.1Theringstructure Theringisak-ary1-cubestructurewhosenodeswelabel 1throughk.Anevenringcanbecoloredwith2colors, asinfigure1(b).thiscoloringisunique i.e.,there isonlyonewayofcoloringtheedgeswithoutrespectto thepermutationoflabels.thegdematrixofaneven Lemma3.3LetGRkbeacolorringofevenorderk, theform0b@v2 ringisinblockcirculantform,asfollows. andmrkbeitsgdematrix.thenmrkisamatrixof V1 UVUṾ... UV U2 Proof.TheproofisbyinductionontheorderofGRk. (3).WeneedtoshowthatiftheGDEmatrixMR2m First,itiseasytoverifythatMR4isintheformof U1 1 CAkk (3) ofgr2m,m2,isintheformof(3),thenthegde vertexlabeled2mingr2misrelabeled2m+2,andthe withtwoextravertices,asillustratedinfigure2.the matrixmr2m+2ofgr2m+2isnecessarilyinthatform aswell.let'sviewgr2m+2asanexpansionofgr2m newlyaddedverticesarelabeled2mand2m+1.then, accordingtothecomputationformulasinlemma2.1, weobtainthefollowing. Forj=2m+1;2m+2, Figure2:AnexpansionoftheringGR2m (MR2m+2)1;j?2=0; (MR2m+2)1;j=(MR2m)1;j?2; (MR2m+2)2m+2;j=(MR2m)2m;j?2; m 2m 2m-1 2m-2 m m 2m-1 2m+2 GR 2m GR 2m+2 2m+1 2m 2m-2 m+1

9 forj=1;2,(mr2m+2)2m+2;j=(mr2m)2m;j; (MR2m+2)2m;j=0; fori=2m;2m+1;2m?1j2m+2, (MR2m+2)i;j=(MR2m)i?2;j?2: Hence,MR2m+2,andthereforeMRk,areintheform of(3).2 Asanexample,theGDEmatrixoftheringoforder 8,MR8,isasinFigure3. GiventhisparticularstructureofthematrixMRk,we canthenderivetheoptimalexchangeparameteranddeterminetheeectoftheringorderkontheconvergence rate. Theorem3.1LetGRkbeacolorringofevenorderk,MRkbetheGDEmatrixofGRk,andk= 2m.Thentheoptimalexchangeparameteropt(MRk) isequalto1 1+sin(=m);andforagiven,R1(MRk) R1(MRk?2). Proof.ConsidertheparticularformofMRkasshownin Lemma3.3.ItiseasytoseethatMRkcanberepresented byablockcirculantmatrix(a1;a2;0;:::;0;am), wherea1= (1?)2(1?) (1?)(1?)2!; A2= 00 (1?)2!; Am= 2(1?) 00!: Andforj=0;1;:::;m?1, A1+!jA2+!j(m?1)Am= (1?)2+!m?j2(1?)+!m?j(1?) (1?)+!j(1?)(1?)2+!j2! because!j(m?1)=!m?j.fromlemma3.1,theeigenvaluesofthematrixmrkaretherootsoftheequation 2?2?(1?)2+2cos(2j=m)+(1?2)2=0 Thatis, =(1?)2+2cos(2j r(1+cos(2jm) m))((1+cos(2j m))2?4+2); wherej=0;1;:::;m?1. Clearly,thedominanteigenvalueofMRkinmodulus, (MRk),isequalto1,andwhenk6=4,thesubdominant eigenvalueofmrkinmodulus,(mrk),isequalto 8><>:2?1 if2?p2(1?e) 1+e<1 (1?)2+2e+p(1+e)((1+e)2?4+2) if0<2?p2(1?e) 1+e wheree=cos(2=m).therefore,foragivenringof ordern=2m,n6=4,itsoptimalexchangeparameteris asfollows opt(mrk)=2?p2(1?cos(2=m)) 1+cos(2=m)= 1 1+sin(=m): Whenk=4,wehaveabinary2-cube(two-dimensional hypercube)forwhichopt(mr4)=1=2and(mr4)= 0,whichisinagreementwithpreviousresultsforthe hypercube[5].moreover,(mrk)increaseswithmfor agiven.hence,r1(mrk)decreasesasmincreases. Thatis,foragiven,R1(MRk)R1(MRk?2).2 Theabovetheoremsaysthatforagiven,themore verticesanevenringhas,thesloweritsconvergencerate. Italsogivestheformulafortheoptimalforanyeven ring,whichcanbeusedinpracticetocomputetheexact optimalvaluefortheexchangeparameterforagiven ring.forexample, opt(mr16)=2=(2+q2?p2)0:723: 3.2Thetorusstructure Thek-aryn-cubeisaspecialcaseofthen-dimensional k1k2kntorus.inthiscase,k1=k2== kn=k.thegeneralcaseappearstobemoreinterestingintermsofitsanalysis.werstconsiderthetwodimensionalk1k2toruswithevennumberofnodesin bothdimensions.itcanbeviewedasacollectionofverticalandhorizontalevenrings(seefigure4),andhence 9

10 (1?)(1?)2(1?)2 2(1?)0 2(1?) (1?)2 2(1?)(1?)2(1?)0 0 0(1?)(1?)2(1?)2 0 2(1?)(1?)2(1?)0 0(1?)(1?)2(1?)2 2(1?)(1?)2(1?) Figure3:TheGDEmatrixoftheringoforder8,MR CA: let mensionexchangeoperatorisconcerned.therefore,we resultsintheprevioussectionfortheringcanbeapplied totheanalysishere.tohandlethedegeneratecaseofk1 isequivalenttoachainoftwonodesasfarasthedi- order2.thereasonforthisisthataringoftwonodes ork2equalto2,weusethegdematrixforachainoftorusiscoloredintherow-majorway.notethatwith MR2= 1? 1?!: thiscoloring,allhorizontaledgesaresmallerthanthe Lemma3.4LetMTk1;k2betheGDEmatrixofak1 becauseoftheparticularstructureofthegdematrix verticaledgesinchromaticindices.wewillsoonseethat MRk1. asrevealedbythefollowinglemma,thesetwocolorings Proof.Givenanorderedpairofvertices<i;j>in k2evencolortorusgtk1;k2.thenmtk1;k2=mrk2 havethesameeectontheconvergencerate. GTk1;k2, 1.ifbothiandjareinthesamehorizontalring,i.e., bi=k1c=bj=k1c,thenbecauseofthecoloringwe majorlabeling.similarly,therearetwowaysofcolor- ingtheedges:row-majorandcolumn-majorcoloring, =(MRk2)bi=k1c;bj=k1c(MRk1)imodk1;jmodk1 (1?)2(MRk1)imodk1;jmodk1 ifk2=2 otherwise matrixofanetworkisinvariantforanypermutationof labeling.asthespectrumofeigenvaluesofthegde thenodelabels,wearbitrarilychoosethesnake-likerow- nodesofthetorus:row-majorandsnake-likerow-major Figure4showsthetwocommonwaysoflabelingthe Figure4:Colortoriof44 useinwhichallhorizontaledgesaresmallerthan theverticaledgesinchromaticindices, (MTk1;k2)i;j =8><>:(1?)(MRk1)imodk1;jmodk1 asshowninthegure.wearbitrarilyassumethatthe2.ifiandjareindierenthorizontalrings,i.e., 10andthereexisttwocolorpathspi1jandpii1fromi1 bi=k1c6=bj=k1c,thenthereisacolorpathfrom tojandfromitoi1,respectively,asinillustrated suchthati1modk1=imodk1,bi1=k1c=bj=k1c, itoj,saypij,ifandonlyifthereexistsavertexi1 (3) (3) (3) (3) 7 (3) (3) snake-like row-major node-labeling row-major edge-coloring (a) (3) (3) 13 (3) (3) (3) (3) (3) (3) (3) 6 14 (3) (b) row-major node-labeling column-major edge-coloring

11 i 1 (3) (3) (3) (3) j i Figure5:Illustrationofthecolorpathfromvertexito vertexj infigure5.letl1andl2bethelengthsofpi1jand pii1,respectively;r1bethesumofthenumber ofincidenthorizontaledgesofi1thatarebetween, inchromaticindices,theincidentedgesofi1along thepathpij,andthenumberofincidenthorizontaledgesofi1whosechromaticindexislessthan thatofthelastedgeinpij;andr2bethesumof thenumberofincidentverticaledgesofi1thatare between,inchromaticindices,theincidentedgesof i1alongpij,andthenumberofincidentverticaledgesofi1whosechromaticindexislargerthan thatoftherstedgeinpij.then,accordingtothe computationformulasinlemma2.1,wehave (MTk1;k2)i;j =l1+l2(1?)r1+r2 =l2(1?)r2l1(1?)r1 =(MRk2)bi=k1c;bi1=k1c(MRk1)i1modk1;jmodk1 =(MRk2)bi=k1c;bj=k1c(MRk1)imodk1;jmodk1 Byreferringtothedenitionofdirectproductin Lemma3.2,thelemmaisproved.2 Ifinsteadweusecolumn-majorcoloring,thenallhorizontaledgesarelargerthantheverticaledgesinchromaticindices.Byfollowingtheabovesteps,however,we wouldndthattheresultinggdematrixisthesameas MTk1;k2.Wecontinuetoassumerow-majorcoloringin thefollowingdiscussion. WenowturntotheconvergencerateofGDEinthe torus,andseehowitisrelatedtotheconvergencerate inthering. Theorem3.2LetMTk1;k2betheGDEmatrixofa k1k2evencolortorusgtk1;k2.then,theoptimalexchangeparameteropt(mtk1;k2)isequaltoopt(mrk); andforagiven,r1(mtk1;k2)=r1(mrk),where k=maxfk1;k2g. Proof.FromLemma3.2,itisclearthat (MTk1;k2)=maxf(MRk1);(MRk2)g because(mrk1)=(mrk2)=1.moreover,fromtheorem3.1,(mrk1)(mrk2)ifandonlyifk1k2. Hence,(MTk1;k2)=(MRk),wherek=maxfk1;k2g; and(mtk1;k2)isminimizedwhen=opt(mrk).in otherwords,r1(mtk1;k2)=r1(mrk)andbothmrk andmtk1;k2havethesameoptimalexchangeparameter.2 Asaresult,onecancomputetheoptimalexchangeparameteropt(MTk1;k2)usingtheformulainTheorem3.1. Moreover,theabovetheoremshowsthattheconvergence rateintwo-dimensionaltorusstructuresdependsonly onthelargerdimension.forexample,thetorigt16;j, j=2;4;:::;16,allhavethesameconvergenceratefora givenandsharethesameoptimalexchangeparameter opt(mt16;j)=2=(2+p2?p2)0:723: Theresultsforthetwo-dimensionaltorusshownabove canbegeneralizedtomulti-dimensionalcases.consider ak1k2:::kneventorusandassumethatthisndimensionaltorusiscoloredinawaysimilartothatfor thetwo-dimensionaltorus.then,itsgdematrixcan beexpressedintermsofdirectproductsofcolorrings. Lemma3.5LetMTk1;k2;:::;knbetheGDEmatrixof ann-dimensionalk1k2knevencolortorus GTk1;k2;:::;kn.Then,MTk1;k2;:::;kn=MRk1MRk2 MRkn: Weomitthetediousproofherewhichisbasedoninductiononthedimensionn.Fromthislemma,thefollowing resultisimmediatelyinorder. Theorem3.3LetMTk1;k2;:::;knbetheGDEmatrixof ann-dimensionalk1k2knevencolortorus GTk1;k2;:::;kn.Then,theoptimalexchangeparameter 11

12 opt(mtk1;k2;:::;kn)isequaltoopt(mrk);wherek= max1jnfkjg;andforagiven,r1(mtk1;k2;:::;kn)= R1(MRk): Sinceak-aryn-cubeisaspecialcaseofanndimensionaltorus,wehavethefollowingmajorresult forthek-aryn-cube. Corollary3.1LetMTk;nbetheGDEmatrixofacolor k-aryn-cube,keven.thentheoptimalexchangeparameteropt(mtk;n)isequaltoopt(mrk);andforagiven,r1(mtk;n)=r1(mrk): 3.3Summaryoftheoreticalresults Thelasttheoremanditscorollaryintheprevioussectionequatetheconvergenceratesofthering,thetorus andthek-aryn-cube,thereforeleadingtothisimportantconclusion:givenaxednumberofnodes,thebest waytoconnectthemasfarasgdeloadbalancingis concernedisasak-aryn-cube.thenfromtheabove analysis,wendthatthesmallerthevalueofk,thebettertheconvergencerate;hence,thebinaryn-cubeisthe bestchoice,whichtakesexactlyonesweeptobalance theworkload.however,asdallypointsoutin[6],there areotherpracticalreasonsforwhichak-aryn-cubewith abiggerkispreferable. Theanalysistechniqueasexempliedintheabovesectioncanalsobeappliedtothechainandthemeshwhich canbeviewedasvariantsoftheringandthetorusby deletingtheend-roundconnections.thedetailedproofs forthefollowingtwotheoremscanbefoundin[19]. Theorem3.4LetGCkbeacolorchainofevenorderk, andmckbeitsgdematrix.thentheoptimalexchange parameteropt(mck)isequaltoopt(mr2k);andfora given,r1(mck)=r1(mr2k): Theorem3.5LetGMk1;k2;:::;knbeann-dimensional evencolormesh,andmmk1;k2;:::;knbeitsgde matrix.thentheoptimalexchangeparameter opt(mmk1;k2;:::;kn)isequaltoopt(mrk);andfora given,r1(mmk1;k2;:::;kn)=r1(mck);wherek= max1jnfkjg. Basedonthesetheorems,theoptimalexchangeparametersforevenchainsandmeshescanbeeasilyobtained. Forexample,forj=2;4;6;8, opt(mm8;j)=opt(mc8)=opt(mr16)0:723: Thesetheoremsalsoshowthattheconvergencerateofa meshdependsonlyonitslargestdimension.now,based onthesetheoremsandthoseintheprevioussection,we canestablishtherelationshipsbetweentheconvergence ratesofthering,thetorus,thechainandthemesh.here isasummaryoftheresults.notethattheresultsforthe k-aryn-cubeareimplicitintheresultsforthetorus. 1.Supposeeachki,1in,iseven,andk= maxfki;1ing.then, opt(mr2k)=opt(mt2k1;2k2;:::;2kn) =opt(mck)=opt(mmk1;k2;:::;kn); whichisequalto1 1+sin(=k). 2.Forevenringsandchainsandagiven,themore verticesthestructurehas,theslowertheconvergencerate. 3.Fortoriandmeshesofevenorder(evennumberof nodesineachdimension),theconvergenceratedependsonlyonthelargestdimension. 4.Theconvergenceratesofthesefourstructuresare relatedasfollows. R1(MR2k)=R1(MT2k1;2k2;:::;2kn) =R1(MCk)=R1(MMk1;k2;:::;kn) whereeachki;1in,iseven,andk= maxfki;1ing. 4Comparisonwiththediusion method Wehavederivedtheoptimalexchangeparametersfor leadingtothefastestasymptoticconvergencerateinthe k-aryn-cubeanditsvariants.here,wewouldliketo 12

13 comparethegdemethodwithanotherrelaxation-based method,thediusionmethod[3,5,23].themeasureof interestisstilltheconvergencerate. Withthediusionmethod,aprocessorwouldinteract withallitsneighborssimultaneouslyateachstep.for processori,thechangeofworkloadinaprocessoriis executedaswi=wi+x j2a(i)(wj?wi) wherea(i)isthesetofnearestneighborsandisthe diusionparameterwhichdeterminestheportionofexcessofworkloadtobediusedaway.asawhole,the changeoftheworkloaddistributionatsteptismodeled bytheequationwt+1=d()wt (5) whered()isthediusionmatrix1,asgivenin[5]. Theeciencyofthediusionmethodisreectedby theasymptoticconvergencerate,r1(d()),whichis equalto?ln(d()).(d())isthesubdominant eigenvalueofd(),andisalsoreferredtoastheconvergencefactorofthediusionmethod.wehavepreviouslyderivedtheoptimaldiusionparametersforthekaryn-cubeanditsvariants[23].table1summarizesthe optimalparametervaluesandtheircorrespondingconvergencefactors.forcomparison,theresultsofthegde methodarealsoincluded.clearly,(m())<(d()) inbothtorusandmeshstructures. Inmulticomputers,therearetwobasiccommunication models.oneistheserialcommunicationmodelwhich restrictsanodetocommunicatingwithatmostonenearestneighboratatime;theotheristheparallelcommunicationmodelwhichallowsanodetocommunicate withallitsnearestneighborssimultaneously.clearly, theserialcommunicationmodelfavorsthedimensionexchangemethodandtheparallelmodelfavorsthediusionmethod.recallthatthegdematrixm()reects acompletesweep i.e.,consecutiveiterationsteps, eachofwhichinvolvesani/ocommunicationatanode. 1Thesameisusedfortheentirenetwork.Itispossibletouse dierent'sfordierentedgesofthenetwork,asdiscussedin[5]. Ontheotherhand,thediusionmatrixD()reectsa singleiterationstepwhichinvolves(g)i/ocommunicationsatanode,where(g)isthemaximumdegree ofthenodes.hence,r1(m(opt))>r1(d(opt))in theserialcommunicationmodel. Intheparallelcommunicationmodel,anndimensionaltorusormeshcanbecoloredby2ncolors, andhenceadimensionexchangesweepwouldtakeas muchtimeasthatfor2ndiusionsteps;these2ndiffusionstepsinfactcorrespondtothematrixd2n.we shouldthereforecompare(d2n(opt))with(m(opt)). Sincethediusionmatrixissymmetric,itfollowsthat (D2n(opt))=2n(D(opt)).Thus,inthecaseofthe torus, (D2n(opt))=( 4n 2n+1?cos(2=k)?1)2n (4 3?cos(2=k)?1)2 >(M(opt)); andinthecaseofthemesh, (D2n(opt))=(1?1n+1ncos(=k))2n (cos(=k))2 >(M(opt)): Wehavethusprovedthefollowing,whichisvalidfor boththeserialandtheparallelcommunicationmodels. Theorem4.1TheGDEmethodconvergesasymptoticallyfasterthanthediusionmethodwhenappliedto thek-aryn-cubeanditsvariants. In[5],Cybenkocomparedtheecienciesofthesetwo methodswhenappliedtothebinaryn-cubestructure, andrevealedthesuperiorityofthedimensionexchange methodinbothcommunicationmodels.theabovetheoremextendshisresulttothefamilyofk-aryn-cube structures. 13

14 torusandmesh;kiiseven,andk=maxfkig;i=0;1;:::;n. Table1:ComparisonbetweentheconvergencefactorsoftheGDEmethodandthediusionmethodink1k2kn 5Simulationandpracticalim- 1+sin(2=k) 1+sin(=k) opt 1 1+sin(2=k)?1 1+sin(=k)?1 (M(opt)) 2 2n+1?cos(2=k) opt 1Diusionmethod 1?1n+1ncos(=k) 2n+1?cos(2=k)?1 (D(opt)) plementations 4n Forpracticalapplications,itwouldbeofconsiderable iterationsweeps,theexperimentsrevealinmeasurable ticalsimulationexperimentsonanumberoftestcases. Inadditiontogivingusinformationonthenumberof termstheeciencygainsduetotheoptimalexchange GDEproceduretobalancethesystem'sloadcanbeobtainedorestimated.Tothisend,weconductedstatis- valueifthenumberofiterationsweepsrequiredbytheitsworkloadaccordingtotherevisedformula Asdiscussedin[8],theintegerversionoftheoriginalDE wi=(d(1?)wi+wjeifwiwj b(1?)wi+wjcotherwise (6) theoreticalresultsconcerningtheequivalenceofthevariousconvergenceratesandtheoptimalityofthederived grainparallelismswhicharemorerealisticandmore commoninpracticalparallelcomputingenvironments, onecantreattheworkloadsoftheprocessorsmorecon- Inthetheoreticalanalysis,werepresenttheworkloadloadbalancingproceduretoendwithavarianceofsome method(i.e.,=1=2)isjustaperturbationofitsreal parameters.thesimulationresultsalsoconrmedour parametervalues. ofaprocessorbyarealnumber,whichisreasonableundertheassumptionofverynegrainparallelismasexhibitedbythecomputation.tocovermediumandlarge thresholdvalue(inworkloadunits)betweenneighboring conclusion. counterpartandwillconvergetoanearlybalancedstate. Applyingtheperturbationtheorytotherealversionof ourgdemethodverbatim,wecancometoasimilar processors.thisthresholdvaluecanbetunedtosatisfactoryperformanceoftheprocedureinpractice,as illustratedin[9].inalloursimulationexperiments,this Becauseoftheuseofintegerworkloads,weallowthe simulationexperiments.allweneedtodoistomodify stripoftheoceanforwhichtheprocessorisresponsible. WeusedtheintegerversionoftheGDEmethodinour venientlyasnon-negativeintegers,asisdonein[8].forvalueissettooneworkloadunitwhichisclosesttototal theexchangeoperatorofeq.insection2.during tegerwhichcorrespondstothenumberofshesinthe willdiscussshortly,theworkloadofaprocessorisanin- example,inthewatorsimulationexperimentwhichwebalancing.then,itisclearthat0:5<1becausea pairofneighboringprocessorswithavarianceofmore thanoneworkloadunitwouldnotbalancetheirworkloadsanymorewhen<0:5.sincethetermination mechanismforglobalterminationdetectiontothesimu- conditionofaprocessorisratherlocalized,weadda lation[22].weexcludetherathersmalldelayfortermi- ectspurelytheeciencyofthegdemethod.this results,andsothenumberofsweepsreportedbelowre- exchangewithaneighborj,processoriwouldupdatenationdetectionusingthismechanisminoursimulation 14helptheusersofthismethodtosetsuchalimit. beforehand.infact,oursimulationresultsbelowcan terminationdetectionmightnotbenecessaryifwecan setalimitonthenumberofsweepsforagivenstructure

15 Table2:ExpectednumberofsweepsE(NS)forRing16,Chain8,16-ary2-cube,andMesh (M())Ring1616-ary2-cubeChain8Mesh Thenumberofiterationsweeps,denotedbyNS,isexpectedtodependuponsuchfactorsastheexchangeparameter,theinitialworkloaddistribution,thetopology andthesizeoftheunderlyingsystemstructure.theinitialworkloaddistributionisarandomvector,eachelementofwhichisdrawnindependentlyfromanidenticalopt=0:723.italsoappearsthattheabsolutevalues bound.themeanworkloadaprocessorgets(i.e.,the workloadaprocessorgetsisdeterminedbythedistributionmean.table2displaystheexpectednumbersof expectedworkload)isthusequaltob.theamountof uniformdistributionin[0;2b],wherebisaprescribedtheoptimumpoint,whichisinlinewithourtheoretical resultsontheequivalenceoftheconvergencerates. turesareveryclosetoeachother,especiallywhennear 0.8,whichisinagreementwiththetheoreticalresultof 5.1Numberofiterationsweeps 8.25 sweepsgeneratedbytheexperimentsforthestructures 0.5to0.9instepsof0.05.Eachdatapointistheaverageof100runs,eachusingadierentrandominitial valuesofrisesanddropswiththevalueof(m()), loaddistribution.thesecondcolumninthetableshows ofthevariouscases.fromthetable,itisclearthatture.thisreallyputsforththegdemethodasaprac- 2-cube!).Wealsotrieddierentnumbersofprocessors ameanof128unitsperprocessor,andvariesfrom ofring16,chain8,16-ary2-cube(i.e.,torus1616), borhoodof8sweeps(evenfora256-processor16-ary optimalsweepnumbersarerathersmall intheneigh- oftheexpectednumberofsweepsforthevariousstruc- Furthermore,itismostencouragingtoseethatthe andmesh84.theinitialworkloaddistributionhas theexpectednumberofsweepsineachcasefordierent theconvergencefactors,(m()),ofthegdematrices ticalmethodforloadbalancinginrealmulticomputers. portionaltothenumberofprocessorsforthechain,and foreachkindoftopology.theresultsforchainsofup showsthattheoptimalnumberofsweepsislinearlypro- hencetothedimensionorderkofthek-aryn-cubestruc- to128processorsaredepictedinfigure6.thegure andthattheoptimalexchangeparameteroptofeachnoticethattheconvergenceratesinthetheoretical caseisnotequalto0.5,butsomewherebetween0.7andanalysisareintermsofsweepsovertime.asweepof thegdemethodmayinvolvedierentnumbersofnear- k-aryn-cube(keven),asweepcomprises2ncommunicationsteps(whenn>2)ornsteps(whenn=2).thus, 15estneighborcommunicationsinvariousstructures.Ina foragivennumberofprocessors,ahigherdimensional

16 E(NS) N Figure6:ExpectednumberofsweepsE(NS)usingopt forchainsofvarioussizes k-aryn-cube,eventhoughittakesfewersweepstobalancetheload,requiresmorecommunicationstepswithin asweepinreality.however,fromfigure6,wepointout thattheminimalnumberofsweepsnecessaryforconvergencewoulddecreaseatalogarithmicratewiththe increaseinthenumberofdimensions;thisisbecause thedimensionorderdecreasesatthesamerateasthe increaseinthenumberofdimensionsforagivennumberofprocessors.asanexample,consideraclusterof 4096processors,whichcanbeorganizedasastructureof 64-ary2-cube,16-ary3-cube,8-ary4-cube,or2-ary12 -cube.theminimalsweepnumbersforthesestructures areabout35,8,4,and1sweep,respectively.sincethe numberofcommunicationstepswithinasweepwould onlydoublewitheveryaddeddimension,itisjustied tomaintainthatthegdemethodismosteectivein highdimensionalk-aryn-cubes(inparticular,thebinary n-cubeandthe4-aryn-cube). 5.2Thenon-evencases Inadditiontotheevencases,wealsosimulatedafew non-evencasesforwhichthetheoreticalanalysishasnot beenabletodealwithbecauseoftheirstructuraldifferencesfromtheevencases.weapproximatedtheir optimalexchangeparametersusingtheformulasforthe evencases.table3summarizesthesimulationresults. Thenumbersinparenthesesinthebottomrowofthe tablearetheapproximatedoptimalparameterandthe resultingnumberofsweepsrespectively.inlinewith ourremarksbefore,theoddornon-evencasesdonot behavedierentlyfromtheevencasesintermsoftheir convergencepatternandtheoptimalvaluesfortheexchangeparameter,whichcanbeseenbycomparingtable3withtable2.hence,itisreasonabletoconclude thattheresultsfortheevencasescanbeapplied,asa closeapproximation,tothenon-evencases. Notethatanoddringhastobecoloredusingthree colors,asinfigure1(a).thatis,asweepofthegde methodinanoddringcomprisesthreecommunication steps.asonlytwoprocessorsareinvolvedinworkload exchangeinthethirdcommunicationstep,itseemsthat muchcommunicationbandwidthmaybewasted.however,acloseexaminationofthecommunicationpattern ofgdebalancingintheoddring,asshowninfigure7, wouldrevealthatthethirdchromaticindexwouldaccountforonlyasmallfractionofthetotalcommunicationoverheadincurredinthebalancingprocess.in t t t t t 1 2 (3) Figure7:CommunicationpatternoftheGDEmethod inaringof5nodes thegure,thevebigdotsinthecentrerepresentprocessorsthatareconnectedasaring.weattachatime axisttoeachprocessorforeasyviewing.eachdotted double-arrowrepresentsacommunicationstepbetweena 16

17 Table3:ExpectednumberofsweepsE(NS)forRing15,Chain7,Torus165andMesh Ring15Torus165Chain7Mesh (0.711,9.75)(0.723,8.70)(0.697,8.23)(0.723,8.05) pairofnearestneighborsfortheexchangeoftheirworkloads.attimet=1,allprocessorsexceptthefthare becomesidleattimet=2.whiletherstandfth involvedincommunication.then,therstprocessor processorsarebusyexecutingthethirdcommunication sweep.atthistime,onlythesecondprocessorisinthe idlestate.continueon,weseethatthegdeprocedure stepattimet=3,thethirdandfourthprocessorsare alreadyexecutingtherstcommunicationofthenext5.3improvementsduetotheoptimal ringofvenodescostsonlyoneextracommunication inanoddringof2k+1nodescostsonecommunication iterationsweeps.thisexamplecanbegeneralizedtoan oddringofarbitrarysize.ingeneral,thegdebalancing cationsteps.inotherwords,gdebalancinginanodd stepmorethananevenringofcomparablesizefortwo nishestwocompleteiterationsweepsinvecommuni-ofourgdemethodovertheoriginaldemethod,we thechoiceof=1=2whichisusedintheoriginalde method.tofurtherexamineandquantifythebenets parameters deneametricformeasuringimprovements,denoted: Itisclearfromthesimulationresultsthattheoptimalexchangeparameteroptyields(much)betterresultsthan betweentheringandtheotherstructuresinprevious ancinginnon-evenandevencasesofthesestructures. stepmorethanthatintheevenringofcomparablesize forkiterationsweeps.basedontheequivalenceresultswherenshandnsoptaretheexpectednumbersof =NSh?NSopt NSh100% sections,wecanconcludethatthereisanegligiblysmallspectively.theimprovementreectsthesuperiorityof sweepsfromusing=1=2andtheoptimaloptre- dierencebetweentheecienciesofoptimalgdebal-originalone(de).tables4{7showtheresultsfordierentstructures;foreachstructure,dierentsizesofthe structureanddierentworkloadsperprocessorareconnicantunderheavyloads.forexample,inaringwitcreasesastheaverageworkloadperprocessorincreasessidered. 64nodes,ifeachprocessorisloadedwith10units,then NSh=7:39,NSopt=6:91and=6:5%;ifeachprocessorisloadedwith10,000units,thenNSh=543, Theseresultssuggestthattheimprovement()in- theoptimaldimensionexchangemethod(gde)overthe Itisnotsoevidentunderlightloadsbutbecomessig- 17

18 Table4:Improvements(%)forringsofvarioussizesand workloadsmeanworkloadperprocessor Table6:Improvements(%)forchainsofvarioussizes andworkloadsmeanworkloadperprocessor Size Size andworkloadsmeanworkloadperprocessor Table7:Improvements(%)formeshesofvarioussizes Size Table5:Improvements(%)fortoriofvarioussizesand workloads NSopt=50,and=90:8%.Thesimulationresultsalso Size Meanworkloadperprocessor astructurewhenworkloadisheavy i.e.,thelargerthe showthattheimprovementisproportionaltothesizeof system,thebettertheperformanceofourgdemethod usingoptimalexchangeparameters.insummary,dynamicloadbalancingusingdimensionexchangedoes Thesimulationexperimentsshedlightonthenumberof eter. 5.4Practicalimplementations iterationsweeps,butignoredtheoverheadthatmightbe benetsubstantiallyfromtheoptimalexchangeparamtherdidtheytellusanythingabouttheexpectedper- incurredinactualimplementationofthemethod.nei- 18dataparallelcomputations.Itturnsoutthattheover- plementeditasthedynamicloadbalancerwithintwo cations.wethereforetookthegdemethodandim- formancegainwhenthemethodisusedinrealappli-

19 headduetotheperiodicexecutionofthegdemethod isverysmallbutthegaininperformancethroughthe balancingovertheversionswithoutgdebalancingis substantial.indataparallelcomputations,thecomputationalrequirementassociatedwitheachportionofa problemdomainmaychangeasthecomputationproceeds.toreducethepenaltyofloadimbalances,aneffectivewayistoperiodically\remap"(re-decompose) theproblemdomainontotheprocessors;thegoalofthis remappingistotrytocreateabalancedworkloadacross theprocessorsforthenextphaseofthecomputation[11]. ToallowustoevaluatetheGDEmethodaswellasto studygde-basedremapping,weimplementedtwomajorapplicationsinamulticomputer(atransputerarray): thewatorsimulationandtheparallelthinningofimages.theremappingmechanismintheseapplications comprisestwocomponents:thedecisionmakerandthe workloadadjuster.thedecisionmakerusesthegde methodtodrivetheprocessorsintoaconsensusonthe uniformworkloaddistribution;thentheworkloadadjusterwouldcarryouttheactualworkloadredistribution accordingtothedecisionsjustmade.thisremapping mechanismisinvokedperiodically.inthewatorsimulationofa256256toroidaloceanonan16-transputer ring-structurednetwork,itisfoundthatfrequentremapping(onceeverytwosimulationsteps)leadstoa10{ 20%improvementonthetotalsimulationtime(for100 stepsofsimulatingtheocean).inparallelthinningof a128128image(apopularimageofaman'sbody) onan8-transputerchain-structurednetwork,frequent remappingusinggdeyieldsaperformancegainof10% onthinningtimeeventhoughthetestimagedoesnot favorremapping.detailsofanddiscussionaboutthese experimentscanbefoundin[19].theseresultshaveled ustobelievethatthegdemethodwiththeoptimal exchangeparametersisaviabletoolfordynamicload balancinginpracticalimplementationsofdata-parallel computations. 6Concludingremarks WehaveanalyzedtheGDEmethodfordynamicload balancingasappliedtothek-aryn-cubeandits variants thering,thetorus,thechain,andthemesh. Wehavederivedtheoptimalexchangeparametersin closedform,whichmaximizetheconvergenceratesof GDEbalancinginthesestructures.Wehaveshownthat thereexistscloserelationshipsbetweentheirconvergence rates,andconcludedthatthegdemethodfavorshighdimensionalk-aryn-cubesforagivennumberofprocessors.wehavealsorevealedthesuperiorityofthegde methodtothediusionmethodwhenbothareapplied tothesestructures.throughstatisticalsimulationexperiments,wehaveshownthattheeciency(interms ofnumberofstepstoconvergence)ofusingtheoptimal exchangeparametersissignicantlybetterthanthatof thenon-optimalcasessuchastheoriginaldemethod. Thispaperhasanalyzedtheoreticallyonlytheeven casesofthevarioustopologies.thisisduetoourusing matrixpartitioningandcirculantmatricesfortheanalysis.itisconceivablehoweverthattheoddcaseswould behavemoreorlessthesameastheirevencounterparts especiallywhenthenumberofnodesislarge.wefound thistobetrueforthenon-evencaseswesimulated.we alsopresentedanargumentthatsuggestedthatthedifferencebetweenthetwointermsofeciencyshouldbe negligiblysmall.nevertheless,ndingadierentmathematicaltooltoanalyzealsotheoddcaseswouldbean interestingtheoreticalpursuit.inaddition,afterhaving dealtwithsomeofthemostcommonregularstructures, itisnaturaltothinkofarbitrarystructures.unfortunately,thederivationoftheoptimalexchangeparameter forarbitrarystructuresrequiresasolutiontotheproblemofspecifying,inanalyticalform,thedependenceof thesubdominanteigenvalueinmodulusofamatrixon thematrixelements.thisisstillopeninmathematics[4]. Acknowledgements.Wethankstheanonymous refereesfortheirconstructivecomments. 19

20 References [1]W.C.AthasandC.L.Seitz.Multicomputers:message{passingconcurrentcomputers.IEEE Computer,21(8):9{24,August1988. [2]D.P.BertsekasandJ.N.Tsitsiklis.Parallelanddis- tributedcomputation:numericalmethods.prentice- HallInc.,1989. [3]J.B.Boillat.Loadbalancingandpoissonequation inagraph.concurrency:practiceandexperience, 2:289{313,December1990. [4]A.BroderandE.Shamir.Onthesecondeigenvalue ofrandomregulargraphs.inproc.of28th.ieee FoundationsofComputerScience,pages286{294, [5]G.Cybenko.Loadbalancingfordistributedmemorymultiprocessors.JournalofParallelandDistributedComputing,7:279{301,1989. [6]W.J.Dally.Performanceanalysisofk-aryn-cube interconnectionnetworks.ieeetransactionson Computers,39(6):775{785,June1990. [7]P.J.Davis.Circulantmatrices.JohnWileyand SonsInc.,1979. [8]S.H.Hosseini,B.Litow,M.Malkawi,J.Mcpherson,andK.Vairavan.Analysisofagraphcoloring baseddistributedloadbalancingalgorithm.journal ofparallelanddistributedcomputing,10:160{166, [9]R.LulingandB.Monien.Loadbalancingfordistributedbranchandboundalgorithm.InProceedingsof6thInternationalParallelProcessingSymposium,pages543{5448,March1992. [10]L.M.NiandP.K.McKinley.Asurveyofwormholeroutingtechniquesindirectnetworks.IEEE Computer,26:62{76,February1993. [11]D.M.NicolandP.F.Reynolds.Optimaldynamicremappingofdataparallelcomputation. IEEETrans.onComputers,39:206{219,February1990. [12]D.M.NicolandJ.H.Saltz.Dynamicremappingofparallelcomputationswithvaryingresourcedemands.IEEETransactionsonComputers, 37(9):1073{1087,September1988. [13]G.RamanathanandJ.Oren.Surveyofcommercialparallelmachines.ACMComputerArchitecture News,21(3):13{33,June1993. [14]S.Ranka,Y.Won,andS.Sahni.Programminga hypercubemulticomputer.ieeesoftware,5:69{77, September1988. [15]Y.ShihandJ.Fier.Hypercubesystemsandkey applications.ink.hwangandd.degroot,editors, ParallelProcessingforSupercomputersandArtical Intelligence,pages203{243.McGraw-HillPublishingCo.,1989. [16]G.Trew,A.andWilson.Past,PresentandParallel: ASurveyofAvailableParallelComputerSystems. Springer{Verlag,1991. [17]R.S.Varga.Matrixiterativeanalysis.Prentice- Hall,1962. [18]M.Willebeek-LeMairandA.P.Reeves.Localvs. globalstrategiesfordynamicloadbalancing.in ProceedingsofInternationalConferenceonParallelProcessing,volume1,pages569{570,1990. [19]C.-Z.Xu.Iterativemethodsfordynamicloadbalancinginmulticomputers.PhDthesis,Dept.of ComputerScience,TheUniversityofHongKong, [20]C.-Z.XuandF.C.M.Lau.Iterativedynamicload balancinginmulticomputers.journalofoperationalresearchsociety.toappear.(availableas Tech.ReportTR-92-09,Dept.ofComputerScience, TheUniv.ofHongKong,Sep.1992). [21]C.-Z.XuandF.C.M.Lau.Analysisofthegeneralizeddimensionexchangemethodfordynamicload balancing.journalofparallelanddistributedcomputing,16:385{393,december

21 [23]C.-Z.XuandF.C.M.Lau.Optimalparametersfor [22]C.-Z.XuandF.C.M.Lau.Terminationdetection loadbalancingusingthediusionmethodinthekaryn-cubenetworks.informationprocessingletters,47(5):181{187,september1993.(alongerversionappearsastech.reporttr-93-03,dept.otributedprocessing,pages196{203,december1992ingsof4thieeesymposiumonparallelanddis- forlooselysynchronizedcomputations.inproceed- ComputerScience,TheUniv.ofHongKong,Mar. 1993). 21