ParallelDynamicLoad-BalancingforAdaptiveDistributive MemoryPDESolvers. NasirTouheed by

Transcription

1 ParallelDynamicLoad-BalancingforAdaptiveDistributive MemoryPDESolvers NasirTouheed by Submittedinaccordancewiththerequirements forthedegreeofdoctorofphilosophy SchoolofComputerStudies TheUniversityofLeeds Thecandidateconrmsthattheworksubmittedishisownandthatappropriate September1998 credithasbeengivenwherereferencehasbeenmadetotheworkofothers.

2 withtheparalleladaptivesolutionofpartialdierentialequations(pdes).weare Thisthesisisconcernedwiththeissueofdynamicload-balancinginconnection Abstract ii schemesonunstructuredgridsandweassumethatgeometricparallelismisused, wherebytheniteelementornitevolumegridsarepartitionedacrosstheavailable parallelprocessors.forparalleleciencyitisnecessarytomaintainawellbalanced partitionandtoattempttokeepcommunicationoverheadsaslowaspossible.when interestedinparallelsolutionsbaseduponeitherniteelementornitevolume anddierentkindsofparallelmachinesarementioned.theniteelementmethodis adaptivityoccurshoweveragivenpartitionmaydeteriorateinqualityandsoitmust alsointroducedanditsparallelimplementationisdiscussedinsomedetail:leading workinthiseld.inchapteroneabriefhistoryofparallelcomputersispresented bemodieddynamically.thisistheproblemthatweconsiderinthiswork. tothederivationofastaticload-balancingproblem.anumberofimportantstatic Chaptersoneandtwooutlinetheprobleminmoredetailandreviewexisting descriptionofsomeerrorindicatorsandcommontechniquesformeshadaptivity. Itisshownhowthisadaptivitymayleadtoaloadimbalanceamongtheavailable loadbalancingalgorithmsarethendiscussed.chaptertwocommenceswithabrief processorsofaparallelmachine.wethendiscusssomewaysinwhichthestatic load-balancingalgorithmsofchapteronecanbemodiedandusedinthecontext ofdynamicload-balancing.theprosandconsofthesestrategiesarediscussedand thennallysomespecicdynamicload-balancingalgorithmsareintroducedand dertaken.inthispreliminary(sequential)versionthedualgraphofanexisting algorithmareoutlinedandanumberofpreliminarynumericalexperimentsareun- discussed. partitionedcomputationalmeshisrepartionedamongthesamenumberofprocessorssothataftertherepartitioningstepeachprocessorhasanapproximateequal anotherarerelativelysmall. tationofthisnewalgorithmandmakingcomparisonwithexistingtechniques.in loadandthenumberofedgesofthisdualgraphwhichcrossfromoneprocessorto Chapterfourthealgorithmisimplementedfora2-dadaptiveniteelementsolver uponanumberofgeneralisationsofexistingalgorithms.thedetailsofthenew InChapterthreeanewdynamicload-balancingalgorithmisproposedbased Theremainderofthethesisisconcernedwiththepracticalparallelimplemen-

3 forsteady-stateproblems,andinchaptervethegeneralityoftheimplementation isenhancedandthealgorithmisappliedinconjunctionwitha3-dadaptivenite volumesolverforunsteadyproblems.inthissituationfrequentrepartitioningof iii whichinvolveverynon-uniformrenement. theworkofthisthesis.thesecomparisonsareveryfavourableforcertainproblems gorithmdetailedhereagainstnewsoftwarethatwasdevelopedsimultaneouslywith themeshisrequired.inthischapterperformancecomparisonsaremadefortheal- andansgiorigin2000.forthepurposesofnumericalcomparisonsalltimings ansgipowerchallenge,dierentworkstationnetworks(sgiindysandsgio2s), codehasbeentestedbymakinguseofavarietyofplatforms,includingacrayt3d, CusingMPIversion1.1(whereapplicable).ThePortabilityoftheload-balancing AllsoftwareimplementationsdescribedinthisthesishavebeencodedinANSI quotedinthisthesisareforthesgiorigin2000unlessotherwisestated.

4 encouragementthroughoutthecourseofthisresearch.notonlywasheveryhelpful IwouldliketothankmysupervisorDr.PeterJimackforhisguidanceand Acknowledgements iv inguidingmethroughoutmystayatleeds,buthewasalsoverypatientwhenit theirhelpfuladviceanddiscussionsduringthistime. ThanksalsotoDr.MartinBerzins,Dr.DavidHodgsonandDr.PaulSelwoodfor cametocorrectingmypoorlydraftedchaptersasregardstotheenglishlanguage. madraqasim,fazilahharon,zahidhussain,jaw-shyongjan,sharifullahkhan, RashidMahmood,SarfrazAhmadNadeem,AllahNawaz,ProfessorMuhammad werealwayshappytohelpme. ThanksarealsoduetomycolleaguesIdreesAhmad,SyedShafaatAli,Muham- MythanksalsogototheGeneralOceandSupportstaoftheSchoolwho UniversityofEdinburghforallowingmetousetheirparallelcomputingfacilities, necessarilyrelatedtotheresearch. Abdul-RaufQuraishi,ShujaMuhammadQuraishiandAlexTsaiformattersnot helpandsupportintheearlydaysofmystayinyork. includingthecrayt3d,andtothankdr.alanwoodofuniversityofyork,forhis IwouldalsoliketoacknowledgetheEdinburghParallelComputingCentreatthe Iespeciallywishtothankmyparentsandsisters,brother-in-laws,mother-in-law andfather-in-lawfortheirencouragement.mytwodaughtersmaryamandsidrah intheukhavebeenagreatencouragementthroughouttheperiodofthisresearch. (whowasamuchneededandwelcomeadditiontoourfamilyinthemiddleofthis MembersofmyextendedfamilyinPakistanandofmyimmediatefamilyhere project)havebeenmostpatientwhileinishedthistask.thisprojectwouldnot formsofstudy-leave,cotsandorsawardsrespectively. PakistanandtheCommitteeofVice-ChancellorsandPrincipalsoftheUniversities oftheunitedkingdomforsupportingmenanciallythroughoutmyresearchinthe havebeencompletedwithouttheconstantloveandsupportofmywife,shagufta. AttheveryendIwouldliketothankTheAlmighty,forthemuchneededcourage Finally,mythanksalsogototheUniversityofKarachi,theGovernmentof theproject. andstrengthwhichhegrantedmeatthisrelativelyoldageofmylifetonalise

5 Contents 1Introduction 1.2ComparisonBetweenSIMDandMIMDComputers:::::::::7 1.1IntroductiontoParallelComputers:::::::::::::::::: GeneralMIMDSystems:::::::::::::::::::: SIMDSystems::::::::::::::::::::::::: FiniteElementMethodsforEllipticPDEs::::::::::::::9 1.4Time-DependentProblems:TheLinearDiusionEquation:::::15 1.5ParallelFiniteElementandLoad-Balancing::::::::::::: PiecewiseLinearFiniteElements:::::::::::::::11 1.6RecursiveGraphPartitioningHeuristics::::::::::::::: TheMethodofLines:::::::::::::::::::::: AlgorithmicDetails::::::::::::::::::::::: RecursiveCoordinateBisection(RCB):::::::::::: ModiedRecursiveGraphBisection(MRGB):::::::: RecursiveInertialBisection(RIB)::::::::::::::: RecursiveGraphBisection(RGB)::::::::::::::: RecursiveSpectralBisection(RSB)::::::::::::::22 1.8OtherGraphPartitioningTechniques:::::::::::::::::25 1.7MultisectionalGraphPartitioningHeuristics::::::::::::: RecursiveNodeClusterBisection(RNCB):::::::::: KernighanandLinTypeAlgorithms::::::::::::: GreedyAlgorithm(GR):::::::::::::::::::: StripwiseMethods::::::::::::::::::::::: MultidimensionalSpectralGraphPartitioning:::::::: StateoftheArtSoftwareToolsforGraphPartitioning:::26 v

6 2AdaptivityandDynamicLoadBalancing CONTENTS 2.1SpatialErrorIndicators::::::::::::::::::::::::30 2.2DierentTypesofRenements::::::::::::::::::::31 29 vi 2.4DiusionAlgorithms::::::::::::::::::::::::::39 2.3RelationBetweenAdaptivityandDynamicLoadBalancing::::: RegenerationSchemes::::::::::::::::::::: BasicDiusionMethod::::::::::::::::::::: LocalMeshAdaptationSchemes:HierarchicalRenement: AMulti-LevelDiusionMethod:::::::::::::::: GeneralisationsofStaticAlgorithms:::::::::::::36 2.6TwoParallelMultilevelAlgorithms::::::::::::::::::45 2.5MinimisingDataMigration:::::::::::::::::::::: ParMETIS:::::::::::::::::::::::::::: ParJOSTLE::::::::::::::::::::::::::: DimensionExchangeMethod:::::::::::::::::42 3ANewDynamicLoadBalancer 3.1MotivationoftheAlgorithm::::::::::::::::::::::51 2.7TwoFurtherParadigms::::::::::::::::::::::::47 3.2DescriptionoftheAlgorithm:::::::::::::::::::::: AlgorithmofVidwansetal.:::::::::::::::::: AlgorithmofOliker&Biswas::::::::::::::::: LocalMigration::::::::::::::::::::::::: GroupBalancing:::::::::::::::::::::::: FurtherRenementoftheAlgorithm:LocallyImprovingthePartitionQuality:::::::::::::::::::::::::::::::56 4ParallelApplicationoftheDynamicLoadBalancerin2-d 3.5Examples::::::::::::::::::::::::::::::::60 3.6Conclusions:::::::::::::::::::::::::::::::73 3.4GlobalLoad-BalancingStrategy:DivideandConquerApproach::58 4.2AParallelDynamicLoad-BalancingAlgorithm::::::::::::80 4.1Introduction::::::::::::::::::::::::::::::: GroupBalancing::::::::::::::::::::::::81 77

7 CONTENTS 4.3DiscussionoftheAlgorithm:::::::::::::::::::::: LocalMigration::::::::::::::::::::::::: DivideandConquerandParallelImplementation::::::84 vii 4.4DescriptionofRelatedDataStructuresAssociatedWiththeRedistributionoftheMesh:::::::::::::::::::::::::: ActivityofType2Processors:UnpackingtheLoad::::: ActivityofType1Processors:PackingtheLoad::::::88 4.5DierentIssuesandRelatedFunctionsUsedintheMainAlgorithm 4.3.3ActivityofType3Processors:ThirdPartyAdjustment::88 ByProcessorsofType1:::::::::::::::::::::::: HandlingofVertices::::::::::::::::::::::93 4.7DierentIssuesWhichareRelatedWithProcessorsofType3::: DierentIssuesWhichareRelatedWithProcessorsofType2::: insertion():::::::::::::::::::::::::::: deletion():::::::::::::::::::::::::::: HandlingofEdges:::::::::::::::::::::::95 4.9SomeExamples::::::::::::::::::::::::::::: UseofMessagePassingInterface(MPI):::::::::::::::: Discussion:::::::::::::::::::::::::::::::: AlternativeAlgorithms::::::::::::::::::::: ComparativeResults::::::::::::::::::::::105 5ParallelApplicationoftheDynamicLoadBalancerin3-d 5.1Introduction::::::::::::::::::::::::::::::: Conclusions::::::::::::::::::::::::::::::: DiscussionII:::::::::::::::::::::::::: DiscussionI::::::::::::::::::::::::::: AParallelAdaptiveFlowSolver:::::::::::::::::::: AParallelFiniteVolumeSolver:::::::::::::::: AParallelAdaptiveAlgorithm:::::::::::::::: ApplicationoftheParallelDynamicLoad-BalancingAlgorithm:: DynamicLoadBalancing:::::::::::::::::::::::: CalculationofWPCG::::::::::::::::::::: UseofTokens::::::::::::::::::::::::::134

8 CONTENTS 5.5ComputationalResults::::::::::::::::::::::::: UseofGlobalCommunication::::::::::::::::: NoColouring::::::::::::::::::::::::::135 viii 5.6Discussion:::::::::::::::::::::::::::::::: Examples::::::::::::::::::::::::::::138 6ConclusionandFutureAreasofResearch 5.7InvestigationintoScalabilityoftheAlgorithm:::::::::::: Conclusions::::::::::::::::::::::::::::::: DiscussionII:::::::::::::::::::::::::: DiscussionI::::::::::::::::::::::::::: SummaryofThesis::::::::::::::::::::::::::: PossibleExtensionstotheResearch::::::::::::::::::

9 ListofFigures 1.5Two-dimensionaltorus.:::::::::::::::::::::::::8 1.4(a)Two-dimensionalmesh,(b)three-dimensionalmesh.:::::::8 1.2Aringofprocessors.::::::::::::::::::::::::::6 1.3Hypercubesof(a)dimension1,(b)dimension2and(c)dimension Alineararrayofprocessors.::::::::::::::::::::::6 2.4ThematrixA.::::::::::::::::::::::::::::::44 2.3Dimensionexchangemethod.:::::::::::::::::::::43 2.1Diusionmethod.::::::::::::::::::::::::::::41 2.2Multi-leveldiusionmethod.::::::::::::::::::::::42 1.6EntriesoftheLaplacianmatrix.::::::::::::::::::::23 3.1CalculationofSender,ReceiverandMigtot:::::::::::::::54 3.6Group-balancingalgorithm:versiontwoofloadbalancingofthetwo 3.5Analgorithmforreningthepartitionsbetweenapairofprocessors Thecalculationofgain.:::::::::::::::::::::::::55 3.4Initialversionofloadbalancingofthetwogroups.::::::::::57 3.3Updationofgaindensitiesandedgescutbetweentheprocessors.::56 3.8ThecoarsemeshofExample1.::::::::::::::::::::63 3.7Adivide&conquertypedynamicload-balancingalgorithm.::::61 3.9ThecoarsemeshofExample2.:::::::::::::::::::: Thecoarse\Texas"meshofExample4.:::::::::::::::69 groups.::::::::::::::::::::::::::::::::::59 4.1Updatingthegains.::::::::::::::::::::::::::: Adaptedmeshafter240time-stepsforExample5.:::::::::: Coarsemeshof5184elementsadaptedtoinitialshockconditionfor Example5.:::::::::::::::::::::::::::::::72 ix

10 LISTOFFIGURES 4.3Paralleldynamicload-balancingalgorithm.::::::::::::::86 4.2Loadbalancingofthetwogroups.:::::::::::::::::::85 4.4Thearraynonodessuballwhichcanaccommodateninecoarseelements.92 x 4.5ThefunctionShared().:::::::::::::::::::::::::94 4.8ThefunctionChangenbhd2().::::::::::::::::::::: ThefunctionEdgeChange().::::::::::::::::::::::99 4.6ThefunctionShared2().::::::::::::::::::::::::94 4.7ThefunctionChangenbhd().::::::::::::::::::::::96 4.9ThefunctionChangenbhd3().::::::::::::::::::::: ThefunctionDirichEdgeChange().::::::::::::::::::: ThecoarsemeshofExample3.:::::::::::::::::::: ThepartialviewofthecoarsemeshofExample4.:::::::::: Regularrenementdissectinginteriordiagonal:::::::::::: Meshdata-structuresinTETRAD:::::::::::::::::: Scalabilitycomparisonusingare-balancingtoleranceof5%forExample1(whereTime=RedTime+SolTime).:::::::::::: CalculationofarowoftheweightedLaplacianmatrix.::::::: Greenrenementbytheadditionofaninteriornode:::::::: Calculationofweightsofverticesandedgesoftheweighteddualgraph Scalabilitycomparisonusingare-balancingtoleranceof15%forExample1(whereTime=RedTime+SolTime).:::::::::::: Scalabilitycomparisonusingare-balancingtoleranceof10%forEx- 5.11Scalabilitycomparisonusingare-balancingtoleranceof15%forExample2(whereTime=RedTime+SolTime).::::::::::::15ample2(whereTime=RedTime+SolTime).:::::::::::: Scalabilitycomparisonusingare-balancingtoleranceof5%forEx- 5.10Scalabilitycomparisonusingare-balancingtoleranceof10%forEx- 5.12Scalabilitycomparisonusingare-balancingtoleranceof15%forExample1(whereTime=RedTime+0.2*SolTime).::::::::: Scalabilitycomparisonusingare-balancingtoleranceof15%forExample1(whereTime=RedTime+5*SolTime).::::::::::157

11 LISTOFFIGURES 5.15Scalabilitycomparisonusingare-balancingtoleranceof15%forExample2(whereTime=RedTime+0.2*SolTime).:::::::::15ample2(whereTime=RedTime+25*SolTime).::::::::: Scalabilitycomparisonusingare-balancingtoleranceof15%forExample1(whereTime=RedTime+25*SolTime).:::::::::157 xi 5.17Scalabilitycomparisonusingare-balancingtoleranceof15%forEx- 5.16Scalabilitycomparisonusingare-balancingtoleranceof15%forExample2(whereTime=RedTime+5*SolTime).::::::::::158

12 ListofTables 3.1Partitiongeneratedinparallelon8processorsalongwithournal 3.2SummaryofresultswhentheNew,Vidwansetal.,ChacoandJOS- 3.3Partitiongeneratedinparallelon8processorsalongwithournal partitionsforexample1.::::::::::::::::::::::::64 3.4SummaryofresultswhentheNew,Vidwansetal.,ChacoandJOS- TLEalgorithmsareappliedtotheinitialpartition(seeTable3.1)of Example1.:::::::::::::::::::::::::::::::64 3.5Partitiongeneratedinparallelon8processorsalongwithournal partitionsforexample2.::::::::::::::::::::::::66 3.6SummaryofresultswhentheNew,Vidwansetal.,ChacoandJOS- TLEalgorithmsareappliedtotheinitialpartition(seeTable3.3)of Example2.:::::::::::::::::::::::::::::::66 3.7Partitiongeneratedinparallelon16processorsalongwithournal partitionsforexample3.::::::::::::::::::::::::68 3.8SummaryofresultswhentheNew,Vidwansetal.andJOSTLEalgorithmsareappliedtotheinitialpartition(seeTable3.7)ofExample partitionsforexample4.::::::::::::::::::::::::70 TLEalgorithmsareappliedtotheinitialpartition(seeTable3.5)of Example3.::::::::::::::::::::::::::::::: SummaryofresultswhentheNew,Vidwansetal.,ChacoandJOS- 3.9Initialandnalpartitions(producedbytheNewalgorithm)forExample5.:::::::::::::::::::::::::::::::::73 4.:::::::::::::::::::::::::::::::::::::70 TLEalgorithmsareappliedtotheinitialpartition(seeTable3.9)of Example5.:::::::::::::::::::::::::::::::73 xii

13 LISTOFTABLES 3.12SummaryofresultswhentheNew,Vidwansetal.,ChacoandJOS- 3.11Initialandnalpartitions(producedbytheNewalgorithm)forExample6.:::::::::::::::::::::::::::::::::74 xiii 4.1DataforthepartitionsofExample1(involvingparallelmeshgenerationandrepartitioningon2processors).:::::::::::::::107 TLEalgorithmsareappliedtotheinitialpartition(seeTable3.11) 4.2DataforthepartitionsofExample2(involvingparallelmeshgenerationandrepartitioningon4processors).:::::::::::::::107 ofexample6.::::::::::::::::::::::::::::::74 4.3DataforthepartitionsofExample3(involvingparallelmeshgenerationandrepartitioningon4processors).::::::::::::::: DataforthepartitionsofExample4(involvingparallelmeshgenerationandrepartitioningon4processors).::::::::::::::: DataforthepartitionsofExample5(involvingparallelmeshgenerationandrepartitioningon2processors).::::::::::::::: DataforthepartitionsofExample6(involvingparallelmeshgenerationandrepartitioningon4processors).::::::::::::::: Comparisonofdynamicload-balancingresultsusingfouralgorithms 4.8DataforthepartitionsofExample7(involvingparallelmeshgenerationandrepartitioningon8processors).::::::::::::::: DataforthepartitionsofExample8(involvingparallelmeshgenerationandrepartitioningon16processors).::::::::::::::115 forexamples1to6.:::::::::::::::::::::::::: DataforthepartitionsofExample9(involvingparallelmeshgenerationandrepartitioningon8processors).::::::::::::::: DataforthepartitionsofExample10(involvingparallelmeshgenerationandrepartitioningon16processors).::::::::::::: DataforthepartitionsofExample11(involvingparallelmeshgenerationandrepartitioningon8processors).::::::::::::::11erationandrepartitioningon16processors).::::::::::::: DataforthepartitionsofExample12(involvingparallelmeshgen- 4.14Comparisonofdynamicload-balancingresultsusingfouralgorithms forexamples7to12.::::::::::::::::::::::::::120

14 LISTOFTABLES 5.2Solutiontimes,redistributiontimes,totalmigrationweightsandmi- 5.1Somepartition-qualitymetricsimmediatelybeforeandafterasingle re-balancingstepforexample1.::::::::::::::::::::141 xiv 5.3Solutiontimes,redistributiontimes,totalmigrationweightsandmi- 5.4Solutiontimes,redistributiontimes,totalmigrationweightsandmigrationfrequenciesfor300time-stepsusingare-balancingtolerance of10%forexample1.:::::::::::::::::::::::::143 of5%forexample1.:::::::::::::::::::::::::: Solutiontimes,redistributiontimes,totalmigrationweightsandmi- 5.5Somepartition-qualitymetricsimmediatelybeforeandafterasingle re-balancingstepforexample2.::::::::::::::::::::146 of15%forexample1.::::::::::::::::::::::::: Solutiontimes,redistributiontimes,totalmigrationweightsandmi- 5.8Solutiontimes,redistributiontimes,totalmigrationweightsandmigrationfrequenciesfor300time-stepsusingare-balancingtolerance of10%forexample2.:::::::::::::::::::::::::148 of5%forexample2.::::::::::::::::::::::::::147 of15%forexample2.:::::::::::::::::::::::::149

15 Chapter1 Sincethemiddleofthecurrentcentury,breakthroughsincomputertechnologyhave Introduction madeatremendousimpactonnumericalmethodsingeneralandthenumerical solutionsofpartialdierentialequationsinparticular.duringtheinfancyperiod andhasasmuchmemoryasispossible(oraordable).theprocessoriscommonly ofcomputerstheywereserialinnature.thismeansthattheywerebuiltusing knownasthecentralprocessingunit(cpu)andisfurtherdividedintoacontrol thevonneumannparadigm:withasingleprocessorwhichrunsasfastaspossible unitandanarithmetic-logicunit(alu).thememorystoresbothinstructionsand data.thecontrolunitdirectstheexecutionofprograms,andthealucarries outthecalculationscalledforintheprogram.whentheyarebeingusedbythe program,instructionsanddataarestoredinveryfastmemorylocations,called registers.asfastmemoryisquiteexpensive,therearerelativelyfewregisters. carryingoutthreetrillioncopiesofdatabetweenmemoryandregisterspersecond isthatofthespeedoflight,soinordertobuildacomputerwhichiscapableof say,onehastoteach32-bitwordintoasquarewithsidelengthof10?10meters example,themaximumspeedatwhichthedatacantravelfrommemorytocpu Theperformanceofsuchcomputersisclearlylimitedbyphysicallaws.For (thisisapproximatelyequaltothesizeofarelativelysmallatom).thisissimply theuseofcachememory-whichisimplementedonthesamechipasthecpu.the ideabehindcacheistheobservationthatprogramstendtoaccessbothdataand takenbythedatawhiletravellingfrommemorytoregisters.thisisachievedby notpossible-see[79]fordetails. Inordertospeedupthemachine,onepossibilityistoreducethetransfertime 1

16 CHAPTER1.INTRODUCTION instructionssequentially.hence,ifwestoreasmallblockofdataandasmallblock ofinstructionsinfastmemory(cache),mostoftheprogram'smemoryaccesseswill usethiscachememoryratherthantheslowermainmemory.thismemorywill 2 outsidethechip. beslowerthanregistersbutitwillbefasterthanthemainmemoryimplemented thistrendcontinueduntilthe1980s.atthistimeanewdesignphilosophycalledthe ofhigh-levellanguagesaswellasthecomplexfunctionsofoperatingsystems,and dressingmodesandothermechanisms.thesenewfeaturesallowedtheexecution usedanincreasedchipareatointroducenewandsophisticatedinstructions,ad- Duringtheinitialstagesinthedevelopmentofmicrochips,designerstypically reducedinstructionsetcomputer(risc)emerged.theriscsupportersarguethat allthesenewinstructionscomplicatethedesignofthecontrolunit,slowingdown theexecutionofbasicoperations.asimpleinstructionsetallows,inprinciple,a byriscsupportersisthatthesimplicationofthecontrolunithelpstosavechip bemorethancompensatedforbytheincreasedspeed.anotheradvantageclaimed areaforthecontrolimplementation.thiscanbeusedtoimplementspecialfeatures simple,fastimplementation,sothelargernumberofinstructionsthatisrequirecan intheoperatingunit,aimedatimprovingtheexecutionspeed.therearemany therewereandstillareimportantclassesofprobleminscienceandengineering variantsofriscprocessors,amongthemareberkeleyrisc,microprocessorswithoutinterlockedpipestages(mips)andtheinmostransputer(seechapter10of [20]). whichpractitionershavenotbeenabletosolvesuccessfully.forexample,toattack the\grandchallenges"([17])monthsorevenyearsareneededbythebestofthese computers.agrandchallengeisafundamentalprobleminscienceorengineering Evenafteralltheseadvancementsinthedevelopmentofthecomputerindustry thathasabroadeconomicandscienticimpact,andwhosesolutioncouldbeadvancedbyapplyinghigh-performancecomputingtechniquesandresources([67]). Manyoftheseproblemsarebasicallylargecomputationaluiddynamicsproblems whichcanbemodelledbyasetofpartialdierentialequations(pdes). asmentionedin[26].thisisthesimulationofathree-dimensional,fullyresolved, icallyweconsiderheretwoexamples.therstoneisstudiedbycaseetal.([16]) turbulentowasmightoccurinthedesignofaportionofashiphull.theprimary Tohaveanideaofthecomputingrequirementstosolvesuchproblemsnumer-

17 CHAPTER1.INTRODUCTION parameterforcharacterisingtheturbulentuidowisthedimensionlessquantity smallwavenumberintheowoneneedsr9=4meshpoints([21]).thatisn=109 berofabout104orgreater.inordertofullyresolveimportantdisturbancesofa knownasthereynoldsnumber(r).suchasimulationwouldhaveareynoldsnum- 3 meshpointsforeachtimestep.eachmeshpointhasonepressuretermandthree velocitytermsforboththecurrentandtheimmediatepasttimestep.thisisa totalof8109scalarvariables.iftemperatureorotherparametersmustalsobe numberofarithmeticoperationsvarieswidely,dependinguponthesolutionmethod maintainedforeachpoint,thenabout1010wordsofdatamemoryarerequired.the employed.oneecientapproachthattakesadvantageoftheproblemgeometry, wanttopredicttheweatheroveranareaof milesfortwo-dayperiod hasbeenestimatedtorequireonlyabout500additionsand300multiplicationsper gridpoint.thisleadstoanoperationscountof1012operationspersingletime step(see[26]fordetails). iftheareaisbeingmodeleduptoaheightof11milesandonewishestopartition andtheparametersneedtocomputedonceeveryhalfhour.asmentionedin[67], this cubicmiledomainintosegmentsofsize Thesecondproblemisthatofmodelingandforecastingofweather.Supposewe memory.itisalsoestimatedin[67]thatforthispredictionthetotalnumberof operationsis1015. thentherewouldbe1011dierentsegments.soweneedatleast1011wordsofdata 108operations/second. wasnotevenclosetohavingenoughcapability([84])toperformthesecalculations. It'sprimarymemorywaslimitedto106words,andtheexecutionratewasabout Hencebythistimeitwasclearthatnew,morepowerfulcomputersystemswould Theworld'smostpowerfulcomputerofthemid70'swastheCRAY-1,which beguntostartapproachingitsphysicallimits,thecommunityhadnochoicebutto beneededtosolvethisclassofproblems.sincethesingleprocessormachineshad Parallelcomputersperformtheircalculationsbyexecutingdierentcomputational consideralternativeparadigmssuchasparallelmachines. tasksonanumberofprocessorsconcurrently.theprocessorswithinaparallel 1.1IntroductiontoParallelComputers

18 CHAPTER1.INTRODUCTION Thisexchangeofinformationoccurseitherintheformofexplicitmessagessentby computergenerallyexchangeinformationduringtheexecutionoftheparallelcode. oneprocessortoanotherordierentparallelprocessorssharingaspeciedcommonmemoryresourcewithintheparallelcomputer.theparallelload-balancing In1966MichaelFlynn([33])classiedsystemsaccordingtothenumberofin- 4 structionstreamsandthenumberofdatastreams.thetwoimportantsystems are:simd-singleinstructionstream,multipledatastream, algorithms,proposedinthisthesis,workverywellontheseparadigms. computingarchitecture SIMDSystems MIMD-MultipleInstructionstream,MultipleDatastream. SuchasystemhasasingleCPUdevotedtoexclusivelytocontrol,andalarge Thissectionprovidesabriefintroductiontotheseimportantclassesofparallel collectionofsubordinateprocessors,eachhavingonlyalus,andtheirown(small amountof)memory.duringeachinstructioncycle,thecontrolprocessorbroadcasts aninstructiontoallofthesubordinateprocessors,andeachofthesubordinate processorseitherexecutestheinstructionorisidle. tionmachinesthatwereproducedbythinkingmachines.thecm-2hadupto 65,3561-bitprocessorsandupto8billionbytesofmemory.Masparalsoproduced SIMDmachines.TheMP-2hasupto16,38432-bitALUsandupto4billionbytes ofmemory. ThemostfamousexamplesofSIMDmachinesaretheCM-1andCM-2connec GeneralMIMDSystems ThekeydierencebetweenMIMDandSIMDsystemsisthatwithMIMDsystems, theprocessorsareautonomous:eachprocessorisafull-edgedcpuwithbotha controlunitandanalu.thuseachprocessorsiscapableofexecutingitsown programatitsownpace.theworldofmimdsystemsisdividedintosharedmemoryanddistributed-memorysystems.

19 CHAPTER1.INTRODUCTION Shared-MemoryMIMD Agenericshared-memorymachineconsistsofacollectionofprocessorsandmemory modulesinterconnectedbyanetwork.eachprocessorhasaccesstotheentire 5 advantageofbeingveryrapid(inprinciple)andisgenerallysimplertoprogram. addressspaceofthememorymodules.sothatanydatastoredintheshared However,itsmaindrawbackisthattherecanbeseriousdelays(contentiontime) memoryiscommonto,andcanbeaccessedby,anyoftheprocessors.thishasthe ofabus,thesearchitecturesdonotscaletolargenumberofprocessors:thelargest time.thesimplestnetworkconnectionisbusbased.duetothelimitedbandwidth ifmorethanoneprocessorwantstousethesamelocationinmemoryatthesame congurationofthecurrentlypopularsgichallengexlhasonly36processors. RecentlySiliconGraphics,Inc.hasdesignedandmanufacturedtheOrigin2000 network.forexamplethebasicunitoftheconvexspp1200isa55crossbar shared-memoryarchitecturesrelyonsometypeofswitch-basedinterconnection utilisesscalableshared-memorymultiprocessing(s2mp)architecture.mostother R10000processorswithapeakperformanceof400Mopeach.Thecomputer computer.thebasicbuildingblockoftheoriginisanodebuiltupontwomips processorsareconnecteddirectlyorindirectlybymeansofcommunicationwires. switch. Distributed-MemoryMIMD Indistributed-memorysystems,eachprocessorhasitsownprivatememory.These Fromtheperformanceandprogrammingpointofviewtheidealinterconnection cost)ofsuchanetworkmakesitimpracticaltoconstructsuchamachinewithmore networkisafullyconnectednetwork,inwhicheachprocessorisdirectlyconnected toeveryotherprocessor.unfortunately,theexponentialgrowthinthesize(and thanafewprocessors.attheoppositeextremefromafullyconnectednetworkis immediatelyadjacentneighbouringprocessors(seefigure1.1).aringisaslightly alineararray:astaticnetworkinwhichallbuttwooftheprocessorshavetwo morepowerfulnetwork.thisisjustalineararrayinwhich\terminal"processors onlyadditionalcostisthecostofp-1orpwiresforanetworkofpprocessors. havebeenjoined(seefigure1.2).thesenetworksarerelativelyinexpensive;the Moreoveritisverycheaptoupgradethenetwork-toaddoneprocessorweonly

20 CHAPTER1.INTRODUCTION 6 Figure1.1:Alineararrayofprocessors. needoneextrawire.therearetwoprincipaldrawbacks: iftwoprocessorsarecommunicating,it'sverylikelythatthiswillprevent otherprocessorswhicharealsoattemptingtocommunicatefromdoingso, Figure1.2:Aringofprocessors. networkthatgivesagoodbalancebetweenthehighcostandhighspeedofthe inalineararraytwoprocessorsthatareattemptingtocommunicatemayhave fullyconnectednetworkandthelowcostbutpoorperformanceofthelineararray Inbetweenthetwoextremesahypercubeisapracticalstaticinterconnection necessarytoforwardthemessagealongasmanyasp/2wires. toforwardthemessagealongasmanyasp-1wires,andinaringitmaybe taketwohypercubesofdimensiond?1andjointhecorrespondingprocessorswith ofasingleprocessor.inordertoconstructahypercubeofdimensiond>0,we communicationwires(seefigure1.3).itisclearthatahypercubeofdimension orring.hypercubesaredenedinductively:adimension0hypercubeconsists

21 CHAPTER1.INTRODUCTION 7 theshortestpaththenthemaximumnumberofwiresamessagehastotravelis d.thisismuchfewthanforthelineararrayofring.theprincipaldrawbackto dwillconsistof2dprocessors.itisalsoclearthatinahypercubeofdimension deachprocessorisdirectlyconnectedtodotherprocessorsandthatifwefollow Figure1.3:Hypercubesof(a)dimension1,(b)dimension2and(c)dimension3. thehypercubeisthatitisnoteasytoupgradethesystem:eachtimewewish toincreasethemachinesize,wemustdoublethenumberofprocessorsandadd hypercube(anncube10with1024processors). anewwiretoeachprocessor.therst\massivelyparallel"mimdsystemwasa fromthen-dimensionalmeshbyadding\wrap-around"wirestotheprocessorson theborder.asfarasupgradingisconcernedmeshesandtoriarebetterthan arraysandrings,respectively.observethatann-dimensionaltoruscanbeobtained (seefigures1.4and1.5),whicharesimplyhigherdimensionalanaloguesoflinear Intermediatebetweenhypercubesandlineararraysarethemeshesandtori hypercubes(althoughnotasgoodaslineararraysandrings).forexample,if onewishestoincreasethesizeofaqqmesh,onesimplyaddsaq1mesh andqwires.meshesandtoriarecurrentlyquitepopular.theintelparagonis tori. 1.2ComparisonBetweenSIMDandMIMDCom- atwo-dimensionalmesh,andthecrayt3dandt3eareboththree-dimensional SIMDcomputersrequirelesshardwareandlessmemorythanMIMDcomputers In[67],Kumaretal.discusstheprosandconsofSIMDandMIMDcomputers.

22 CHAPTER1.INTRODUCTION 8 Figure1.4:(a)Two-dimensionalmesh,(b)three-dimensionalmesh. Figure1.5:Two-dimensionaltorus.

23 CHAPTER1.INTRODUCTION operatingsystemateachprocessor.simdcomputersarenaturallysuitedfordata- becausetheyhaveonlyoneglobalcontrolunitandonlyonecopyoftheprogram needstobestored.ontheotherhand,mimdcomputersstoretheprogramand9 ecutedonalargedataset(whichisthecaseintheeldofimageprocessingfor example). parallelprograms;thatis,programsinwhichthesamesetofinstructionsareex- oftime.data-parallelprogramsinwhichsignicantpartsofthecomputationare executedierentinstructionsinthesameclockcycle,soifaprogramhasmany tionals,itisentirelypossiblethatmanyprocessorswillremainidleforlongperiods conditionalbranchesorlongsegmentsofcodewhoseexecutiondependsoncondi- AcleardisadvantageofSIMDcomputersisthatdierentprocessorscannot containedinconditionalstatementsarethereforebettersuitedtomimdcomputers thantosimdcomputers. processorshasitsowncontrolunit.itmayseemthatthecostofeachprocessor mustbehigherthanthecostofasimdprocessor.however,itispossibleto usegeneral-purposemicroprocessorsasprocessingunitsinmimdcomputers.in contrast,thecpuusedinsimdcomputershastobespeciallydesigned.hence, IndividualprocessorsinanMIMDcomputeraremorecomplex,becauseeach duetoeconomiesofscale,processorsinmimdcomputersmaybebothcheaperand morepowerfulthanprocessorsinsimdcomputers. 1.3FiniteElementMethodsforEllipticPDEs Taylorseriesandinvolvethevaluesofthesolutionatneighbouringpointsinthe replacedbydierencequotients.thedierenceoperatorsareusuallyderivedfrom Inthenitedierenceapproximation,thederivativesinadierentialequationare equationsarethenitedierence,theniteelementandthenitevolumemethods. Probablythethreemostpopularnumericaltechniquesforsolvingpartialdierential domain.aftertakingtheboundaryconditionsintoaccount,a(sparse)system ofalgebraicsimultaneousequationsisobtainedandcanbesolvedforthenodal unknowns. toimplementonregulardomains.unfortunatelythismethodisdiculttoapply forsystemswithirregulargeometriesand/orunusualboundaryconditions. Thenitedierencesmethod(FDM)iseasytounderstandandstraightforward

24 CHAPTER1.INTRODUCTION forsuchsystems.incontrasttonitedierencetechniques,theniteelement methoddividesthesolutiondomainintosimplyshapedregionsor\elements".an Theniteelementmethod(FEM)providesanalternativethatisbettersuited 10 approximatesolutionforthepdecanbedevelopedforeachoftheseelements.the totalsolutionisthengeneratedbylinkingtogetheror\assembling"theindividual solutionstakingcaretoensurecontinuityattheinterelementboundaries.thusthe Inthisschemethesolutionisrepresentedasaseriesofpiecewiseconstantelements. elements(controlvolumes).foreachcontrolvolumetheareaintegralisconverted PDEisapproximatelysatisedinapiecewisefashion(seebelow). intoalineintegraloveritsedgesandthenumericaluxattheboundariesalso ThediscretisedformofthePDEisfoundbyintegratingtheequationoverthe Thenitevolumemethod(FVM)mayalsobeappliedonunstructuredmeshes. calculated. thisthesis.theinterestedreadercanconsultthebooksofjohnson([58])andstrang &Fix([95]).Howeverwedescribethemethodbrieyinthecaseofaparticular PDE;Poisson'sequationin2dimensions: Acomprehensivedescriptionoftheniteelementmethodisbeyondthescopeof Forclarityweassumethefollowingboundaryconditionsareimposed: Notethatthisequationisalinearsecondorderpartialdierentialequationwhich forx2<2: (1.1) condition. arisesinalargenumberofphysicalsituations(e.g.owofanidealuid). equation(1.1)byatestfunctionwandintegrateovertoget, Werstderivetheweakformoftheequation(1.1).Todothiswemultiplythe Byusingthedivergencetheoremweget, -Rwr2udx=Rwfdx.

25 rstderivativesaresquareintegrableinandwhicharezeroeverywhereon?1) CHAPTER1.INTRODUCTION 11 Forsimplicitywewillassumeg0inwhichcasetheexpressionsimpliesstill further: thenaboveintegralformreducesto, Rru:rwdx?R?2gwds=Rwfdx.?1.TheaboveintegralformthenleadstothefollowingweakformofthePoisson's NowletH1E()bethespaceofallthosefunctionswhoserstderivativesaresquare integrableinandwhichsatisfythedirichletboundaryconditioneverywhereon Rru:rwdx=Rwfdx. PDE. forallw2h10(). Therestofthissectionconsiderstheniteelementapproximationtothesolution Findu2H1E()suchthat ofthisweakform. Zru:rwdx=Zwfdx; (1.2) 1.3.1PiecewiseLinearFiniteElements Theveryrststepintheapproximationofubytheniteelementmethodisto thissectionwewillassumethesearetriangles).thisisalwayspossibleprovided dividethedomainintoalargenumberofsmallnon-overlappingsubdomains(in thedomainusingisoparametricniteelements(see[19]). approachtothecurveboundary(see[95]fordetails).anotherwayistotriangulate thatisitselfapolygon(i.e.therearenocurvedboundaries).therearemethods meansofasetoflinesegmentsinsuchawaythatinthelimittheselinesegments tohandlecurvedboundaries.onewayistoapproximatethecurveboundaryby from1ton=nb+ne(wherenbisthenumberofverticesintheinteriorofthe domainorontheneumannboundary,?2,andneisthenumberofverticeson Letussupposethatthevertices(nodes)ofthetriangleshavebeennumbered

26 CHAPTER1.INTRODUCTION choosetherstdegreepolynomialshere(polynomialsofdegreezerocannotbe thedirichletboundary,?1).oneachtriangleuisapproximatedbyalowdegree polynomial.althoughanydegreepolynomialscanbeselectedforsimplicitywe 12 functionsarelinearoneachtriangleandsatisfypj(x)=1ifxisthepositionvector Nowwecanwriteu(anapproximationstou)intermsofthesebasisfunctionsas, ofthenodejandpj(x)=0ifxisthepositionvectorofanyoftheothernodes. usedsincewerequirethederivativestobesquareintegrable). Wenextdenesimple\basis"functionsPi(x)forallnodesifrom1toN.These Dirichletboundarycondition,u=uE,fori=nB+1,...,nB+nE.(Notethat, whereaiareunknown(tobedetermined)fori=1,...,nb,andaregivenbythe duetoourchoiceofbasisfunctions,aiisthevalueofuwhenevaluatedattheith nodeofthemesh).ifwesubstitutethevalueofufromequation(1.3)foruand u=nxi=1aipi(x); (1.3) elementequationsisgivenby: equationsfortheunknownsa1;:::;anb.thissystem,knownasthegalerkinnite replacewbypj(x),forj=1,...,nb,inequation(1.2)wethengetasystemofnb Typicallythisiswritteninmatrixformas nbxi=1aizrpi(x):rpj(x)dx=zpj(x)f(x)dx?nx i=nb+1aizrpi(x):rpj(x)dx; Ka=f; forj=1,...,nb. (1.5) (1.4) Kji=RrPj(x):rPi(x)dx)andaisavectoroftheunknownsa1;:::;anB. wherekisreferredtoasthe\globalstinessmatrix"(whoseentriesaregivenby isthattheentrykjiofthematrixkwillalwaysbezeroiftheverticesnumberedj 1.3.2AlgorithmicDetails Havingderivedtheniteelementequations(1.5)wenowdiscusshowthematrixK andthevectorfcanbeobtainedsystematically.themostimportantpointtonote andiarenotconnectedbyanedgeofthemesh.thisisbecausethedotproductof rpj(x)andrpi(x)willbezerooneverytriangularelementinsuchacase.since

27 CHAPTER1.INTRODUCTION \sparsematrix". thismeansthatmostoftheentriesofkwillalwaysbezero,werefertothisasa SupposethattheniteelementmeshconsistsofEtriangularelementse(e= 13 for(j=1;jn;j++) Hencewemayusethefollowingpseudo-codetocalculateK: 1,...,E).TheneachentryofKmaybeobtainedfromthefollowingformula: for(i=1;in;i++)f Kji=RrPj(x):rPi(x)dx==PEe=1RerPj(x):rPi(x)dx: K(j,i)=0 for(j=1;jn;j++) Theorderoftheloopscaneasilybere-arranged: g. for(e=1;ee;e++) for(i=1;in;i++) K(j,i)=K(j,i)+RerPj(x):rPi(x)dx for(e=1;ee;e++) for(j=1;jn;j++) K(j,i)=0 NowwecanmakeuseofthesparsitycausedbythelocalnatureofP1;:::;PN: for(j=1;jn;j++) for(i=1;in;i++) for(i=1;in;i++) K(j,i)=0 K(j,i)=K(j,i)+RerPj(x):rPi(x)dx: for(e=1;ee;e++) for(j=1;j3;j++)f j=numberofnodewhichisj-thvertexofelemente g. for(i=1;i3;i++)f gk(j,i)=k(j,i)+rerpj(x):rpi(x)dx i=numberofnodewhichisi-thvertexofelemente Atthispointwecanmakethefollowingobservations.

28 CHAPTER1.INTRODUCTION Itisnecessarytonumbertheverticesofeachelement,e,ofthetriangulation of;1,2,3.also,anintegerarray,\icon"say,needstobesetupwhichstores thenodenumberofeachvertexofeachelement. 14 Itisalsonecessarytostoreanarrayofthepositionvectors,sjsay,ofthe mentpseudo-codeshouldnowlooksomethinglike: Now,ifweassumethatthenEnodeson?1arenumberedlast,thentheniteele- Asimilararrangementcanbemadeinordertocalculatef,where fj=rf(x)pj(x)dx=pee=1ref(x)pj(x)dx: verticesofthemesh. for(j=1;jnb;j++)f f(j)=0 for(i=1;inb;i++) for(e=1;ee;e++) gfor(j=1;jne;j++) a(nb+j)=ue(s(nb+j;1);s(nb+j;2)) K(j,i)=0 for(j=1;j3;j++)f j=icon(e,j) if(jnb)f f(j)=f(j)+ref(x)pj(x)dx for(i=1;i3;i++)f i=icon(e,i) if(inb) ggelsef(j)=f(j)-a(i)rerpj(x):rpi(x)dx K(j,i)=K(j,i)+RerPj(x):rPi(x)dx Solvethesystem:Ka=f: gtheparallelgenerationandsolutionofthissystemwillbediscussedinx1.5.

29 CHAPTER1.INTRODUCTION 1.4Time-DependentProblems:TheLinearDiffusionEquation 15 Wenowmoveontoconsiderhowwemaygeneralisetheabovetheorytodealwitha lineartime-dependentdierentialequation.thesimplestparabolictime-dependent dierentialequationisthelineardiusionequation: andsomeboundaryconditions,suchas u(x;0)=u0(x)forallx2; (1.6) andthelaplacianoperator,r2,isassumedtoapplyonlytothesespatialvariables Notethatinaboveallthespatialvariableshavebeengroupedtogetherasx notthesame. variabletshouldnotbethoughtofasbeing\justanotherindependentvariable", likexandysay,becausetheboundaryconditionsassociatedwiththisvariableare wouldliketocomputethesolutionforarbitraryvaluesoftwhicharelessthant theothervariableswherewegenerallyknowaboutthebehaviourofthesolution (wehavenoideaaboutthebehaviourofthesolutionattimet).thisdiersfrom throughouttheboundaryofthespatialdomain. Asfaras`t'isconcernedweonlyknowthesolutionattheboundaryt=0and whichtreatsthespatialvariablesandtimevariableindependently.fortunatelythe methodoflinesexactlydoesthesame TheMethodofLines Keepinginmindthespecialnatureofthevariable`t'weneedapracticalmethod dierentialequations(odes),byonlydiscretisinginspaceintherstinstance. ThisisageneralmethodwhichreducesasystemofPDEstoasystemofordinary

30 CHAPTER1.INTRODUCTION ofspatialdiscretisations(e.g.niteelementornitevolume)tobeusedwithany standardodesolver(see[9],forexample).weattempttofollowthisapproach Thespatialandtemporaldiscretisationarethusindependent,allowingavariety 16 Thisyieldsthefollowingsystemofequations, toobtaintransientsolutionsusingtheniteelementmethodpresentedinprevious section. multiplyequation(1.6)byatestfunction,pj(x),whichhasnotimedependence. Thismeansweonlytriangulatethespatialpartofthedomain,andthenwe andmakinguseofthedivergencetheoremasbeforethisbecomes, Intwodimensionswemayagaindividethedomain,,intotrianglesandnumber theverticesofthesetrianglesfrom1ton=nb+newherenbandneareas ofthemesh.sinceweareinterestedinatime-dependentniteelementsolutionwe (1.7) seekanapproximation,u(x;t),tothetruesolution,u(x,t),oftheform ofnbequationsfornbunknowns(inthiscasea1(t);:::;anb(t)).thissystemis Dirichletboundarycondition,u=uE,fori=nB+1,...,nB+nE. whereai(t)areunknown(tobedetermined)fori=1,...,nb,andaregivenbythe Now,replacingubyuinequations(1.7)forj=1,...,nBweagainobtainasystem u=nxi=1ai(t)pi(x); givenbynbxi=1dai Asbeforewemayexpressthisinmatrixnotation,inwhichcaseitbecomes, dtzpipjdx=?nbxi=1aizrpi:rpjdx+z?2gpjds+zfpjdx? i=nb+1aizrpi:rpjdx?nb+ne nb+ne X Mda dt=?ka+f(t): i=nb+1dai XdtZPiPjdx: (1.8)

31 Mji=RPiPjdx).Inthiscasethevectorfdependsupontthroughthepossible CHAPTER1.INTRODUCTION andthematrixmisknownasthe\galerkinmassmatrix"(withentriesgivenby Again,Kisthe\globalstinessmatrix"(whoseentriesaregivenbyKji=RrPi:rPjdx) 17 dependenceofthefunctionfinequation(1.6)upont,orthepossibledependence ofthedirichletboundaryconditionupont(throughthefunctionue). in[9])fordealingwithequationssuchastheseinanecientmanner(i.e.using system,itisasystemofnbordinarydierentialequationsforwhichwecaneasily manystandardtechniques(e.g.thesoftwarepackagesprintwhichisdescribed obtaininitialvaluesfortheunknownsai(t)(fromthefunctionu0(x)).thereare Itshouldbenoticedthatthesystemofequations,(1.8),isnotanalgebraic localerrorthroughadaptivetime-stepping).nevertheless,ateachtimestepanite elementcalculationssimilartothatdescribedinx1.3mustbeundertaken. memoryandspeedofaserialmachinestarttobecomeaserialbottleneck.also thesystemofequations(1.5)canbeeasilyandquicklysolvedonaserialmachine. Butwhenthenumberofdegreesoffreedomisinexcessofamillionorsothenthe Forasmallproblemwherethenumberofdegreesoffreedomisjustafewthousand 1.5ParallelFiniteElementandLoad-Balancing Byusingsuchamachinenotonlycanwehopetosolvelargerproblems(e.g.in forsomeapplicationswherethesizeoftheproblemisnotsobigthetimetaken structuralmechanics)butwecanalsohopetosolvethemmorequickly. byaserialmachinemaystillbeverylarge(fornon-linearproblemsforexample, wheretheiterativemethodsforsolvingthecorrespondingsystem(1.5)arequite sparsesystemofequations(1.5)inparallel.letussupposethedomainhasbeen expensive).inthesecasesapromisingwayforwardistouseaparallelarchitecture. dividedintonsubdomains1,2,...,nandtheithsubdomainihasbeen ontheinterfacebetweenthesubdomainsarelabelledaandtheunknownsinside assignedtotheithprocessorofaparallelmachine.letusassumethattheunknowns Intherestofthissectionwediscussamethodforassemblingandsolvingthe writtenintheform, a1thenina2,a3,...,anandlastlyinathenthesystemofequations(1.5)canbe eachsubdomainsarelabelleda1,a2,...,an.ifwerstnumbertheunknownsin

32 CHAPTER1.INTRODUCTION 264 A 1A2: C2 C1 18 B1B2::BnA: AnCn : a a2 a : = 2 64fn f2 f f : 1 whereai,bi,ciandaarethemselvesusuallysparse.itisclearfromthedenition ofthebasisfunctionspjthatfi,ai,biandciaretotallyhousedbytheithprocess 3 75; (1.9) distributedacrossdierentprocessors.eachprocessorcancomputeandassemble itsowncontributiontothem,independently,storingthemintheblocksfiandai say(sothatf=f1+f2+...+fnanda=a1+a2+...+an). andhencecanbeassembledindependentofeachotherinparallel.butfandaare form: Inordertosolvethesystemofequations(1.9)werstwriteitincomponent Ifwesubstitutethevalueofaifromequation(1.10)in(1.11)wegetthefollowing equation: Aiai+Cia=fi;i=1;2;:::;n; XiBiai+Aa=f: i(fi?cia)+aa=f; (1.11) IfwedeneAsbytheequation, Onsimplicationthisreducesto, (A?XiBiA?1 ici)a=f?xibia?1 ifi: (1.13) (1.12) thentheequation(1.13)cansimplybewrittenas, Asa=f?XiBiA?1 As=A?XiBiA?1 ici; ifi: (1.14) parallelmachinesbecauseeachsysteminequation(1.10)isentirelyindependent Ifequation(1.15)isthensolvedforathenthiscanbesubstitutedintoequation (1.10)andsolvedforaiforalli.Thisapproachisidealfordistributedmemory

33 CHAPTER1.INTRODUCTION tosolveequation(1.15)thenitisnotnecessarytoexplicitlyformthematrixas aniterativemethod,suchastheconjugategradient(cg)algorithm([40]),isused andmaythereforebesolvedinparallelwiththeotherswhenrequired.moreover,if 19 sowehave wherepisthedirectionvectorobtainedfromtheresidualofthekthiteratesofa, of(1.14).themainstepinvolvedisthematrixvectormultiplicationofw=asp multiplicationandsubdomainsolves(somelocalcommunicationisalsorequired betweenprocessorssharinginterpartitionboundaryvertices). Fromequation(1.16)itisclearthatwcanbeobtainedusingonlymatrix-vector Fromabovediscussionitisclearthatthecommunicationoverheadisproportionaltothenumberofverticesontheinterpartitionboundary,henceoneshould w=ap?xibi(a?1 i(cip)): processorswillbeidlewhileothersarestillbusysolvingtheirsystems). eachsubdomainwilltrytosolvetheequation(1.10)inparallel,henceitisdesirable thatthenumberofunknownsineachofaiisapproximatelysame(otherwisesome trytokeepthisboundaryassmallaspossible.alsooncethevectoraisknown havetwomainfeatures, eachprocessorshouldstoreapproximatelythesamenumberofverticesor Hencethedecompositionoftheelementsofthemeshintosubdomainsshould numberofverticeswhichlieontheboundarybetweentheprocessorsshould elements(toensureequalload), andthatapairofnodesisconnectedbyanedgeonlyifthecorrespondingelements areneighboursofeachother,thenaboveproblembecomesaspecialcaseofamore Thedualgraphofagivenmeshisobtainedbyreplacingeachelementbyanode, Inordertoachievetheabovewerstdenethedualgraphofagivenmesh. bekeptlow. beanundirectedgraphwherenisthesetofnodeswithknknodesandeisthe setofedgeswithkekedges,partitionnintonsubsets,n1,n2,...,nnsuchthat generalproblem,namelythegraphpartitioningproblem. Ni\Nj=;fori6=j,kNik=kNk/nandSiNi=N,andthenumberofedges Then-waygraphpartitioningproblemisdenedasfollows:LetG=G(N,E)

34 obtaina2-waypartitionofn,andthenwefurthersubdivideeachpartusing2- partitionproblemismostfrequentlysolvedbyrecursivebisection.thatis,werst CHAPTER1.INTRODUCTION ofewhoseincidentverticesbelongtodierentsubsetsisminimised.then-way 20 of2-waypartitionsorbisections. problemofperformingan-waypartitionisreducedtothatofperformingasequence waypartitions.afterlognphases,graphgispartitionedintonparts.thus,the,isnotsolvableinpolynomialtime.itisinfactannp-hardproblem([22,36,68]). Neverthelessthereareheuristicapproacheswhichperformwellinmostcases.In thenextfewsectionswereviewsomeofthemoreimportantoftheseheuristics. Unfortunatelythisproblem,whichiswell-knowninthegraphtheoryliterature 1.6RecursiveGraphPartitioningHeuristics concentrateonbisectingthegraphsubjecttotheloadbalancingandcut-weight Forthesakeofsimplicity(asmentionedabove),manygraphpartitioningheuristics nature,butthecorrespondingdisadvantageisthatthetotalnumberofsubdomains approachisthatitiseasytoimplementinparallelbecauseofthedivideandconquer canbeappliedrecursivelyontherecentsubdomains.themainadvantageofthis imisationconstraints.whenmorethantwosubdomainsarerequired,theprocedure (thenumberofedgesontheinter-partitionboundaryiscalledthecut-weight)min- thusproducedmustbeapowerof RecursiveCoordinateBisection(RCB) twoorthree-dimensionalcoordinatesavailableforthenodes.asimplebisection LetG=G(N,E)beagivenundirectedgraph.Wemustalsoassumethatthereare strategy,duetosimon([90]),whichisaslightgeneralisationofanearliermethod maininghalfareassignedtotheothersubdomain. usedbywilliamsin[114],forthegraphgistodeterminethecoordinatedirection ofthelongestexpansionofthedomain.withoutanylossofgenerality,assumethat Halfofthenodeswithsmallx-coordinateareassignedtoonesubdomain,there- thisisthex-direction.thenallnodesaresortedwithrespecttotheirx-coordinate. doesnottakeadvantageoftheconnectivityinformationgivenbythegraph.itis Althougheasytoprogram,theprincipaldrawbackofRCBisthatthemethod

35 CHAPTER1.INTRODUCTION methodisnotgenerallysuitableforourpurpose. thereforeunlikelythattheresultingpartitionwillhavealowcut-weightandsothis RecursiveInertialBisection(RIB) ThismethodisageneralisationofRCBtechniqueswhichisdescribedin[28,75] eithersideofit).thisprocedureisthenrepeatedrecursivelyforeachsubdomain. makingacutwhichisorthogonaltothisaxis(withapproximatelyequalweightson ofinertiaforthesepointmassesisthencalculatedandthedomainisbisectedby locatedatthecentroidoftheircorrespondinginitialelement.theprincipalaxis forexample.here,theverticesofthedualgraphareconsideredaspointmasses HeretheideaistousethegraphdistanceasopposedtoEuclideandistanceusedin arelativelyhighcut-weight([28]) RecursiveGraphBisection(RGB) Thismethodisextremelyfast,butliketheRCBitalsoproducespartitionswith d(ni,nj)=numberofedgesintheshortestpathconnectingniandnj. x1.6.1.recallthatthegraphdistancebetweenthetwonodesniandnjisgivenby whichareclosetothisextremenodeareplacedinonesubdomainandtheremaining halfareplacedintheothersubdomain. nodesaccordingtotheirdistancefromoneoftheextremenodes.halfthevertices thepseudo-diameter)ofthegraph(seegeorgeandliu([37]))andthensortthe Herethestartingpointistondthediameter(or,sincethisisexpensivetond, thatatleastoneofthetwosubdomainsisconnected.butitisstillpossiblethat theothersubdomainmaynotbeconnected.hencewemayendupwithasituation inwhichnotallofthesubdomainsareconnected. Ifwestartoutwithaconnectedgraphthenbyconstructionitisguaranteed 1.6.4ModiedRecursiveGraphBisection(MRGB) In[50]HodgsonandJimackpresenttheirowngraphbisectionmethodMRGB.This byattemptingtoproducesubdomainswhichareallsimplyconnected. methodisamodicationofthergbmethod,whichtriestoimproveontheoriginal InMRGBeachbisectionbeginsbyndingtwoapproximatelyextremalnodes