SPE Copyright 2001, Society of Petroleum Engineers Inc.

Transcription

1 SPE Parallel Computig Techiques for Large-Scale Reservoir Simulatio of Multi- Compoet ad Multiphase Fluid Flow K. Zhag, Y. S. Wu, SPE, C. Dig, K. Pruess, SPE, ad E. Elmroth, Lawrece Berkeley Natioal Laboratory Copyright 2001, Society of Petroleum Egieers Ic. This paper was prepared for presetatio at the SPE Reservoir Simulatio Symposium held i Housto, Texas, February This paper was selected for presetatio by a SPE Program Committee followig review of iformatio cotaied i a abstract submitted by the author(s). Cotets of the paper, as preseted, have ot bee reviewed by the Society of Petroleum Egieers ad are subject to correctio by the author(s). The material, as preseted, does ot ecessarily reflect ay positio of the Society of Petroleum Egieers, its officers, or members. Papers preseted at SPE meetigs are subject to publicatio review by Editorial Committees of the Society of Petroleum Egieers. Electroic reproductio, distributio, or storage of ay part of this paper for commercial purposes without the writte coset of the Society of Petroleum Egieers is prohibited. Permissio to reproduce i prit is restricted to a abstract of ot more tha 300 words; illustratios may ot be copied. The abstract must cotai cospicuous ackowledgmet of where ad by whom the paper was preseted. Write Libraria, SPE, P.O. Box , Richardso, TX , U.S.A., fax Abstract Massively parallel computig techiques ca overcome limitatios of problem size ad space resolutio for reservoir simulatio o sigle-processor machie. This paper reports o our work to parallelize a widely used umerical simulator, kow as TOUGH2, for oisothermal flows of multicompoet, multiphase fluids i three-dimesioal porous ad fractured media. We have implemeted the TOUGH2 package o a Cray T3E-900, a distributed-memory massively parallel computer with 695 processors. For the simulatio of largescale multicompoet, multiphase fluid flow, the requiremets for computer memory ad computig time are extesive. Because of the limitatio of computer memory i each PE (processig elemet), we distribute ot oly computig time but also the memory requiremet to differet PEs. I this study, the METIS software package for partitioig ustructured graph ad meshes is adopted for domai partitioig, ad the Aztec liear solver package is used for solvig liear equatio systems. The efficiecy of the code is ivestigated through the modelig of a three-dimesioal variably saturated flow problem, which ivolves more tha oe millio gridblocks. The executio time ad speedup are evaluated through comparig the performace of differet umbers of processors. The results idicate that the parallel code ca sigificatly improve capacity ad efficiecy for large-scale simulatios. Itroductio TOUGH2 1, 2 is a geeral-purpose umerical simulatio program for multi-dimesioal, multiphase, multicompoet heat ad fluid flows i porous ad fractured media. The code is writte i stadard ANSI FORTRAN 77. Sice its release i 1991, the program has bee used worldwide i geothermal reservoir egieerig, uclear waste isolatio, evirometal assessmet ad remediatio, ad modelig flow ad trasport i variably saturated media. The umerical scheme of the TOUGH2 code is based o the itegral fiite differece (IFD) method. The coservatio equatios ivolvig mass of air, water, chemical compoets ad thermal eergy are discretized i space usig the IFD method. Time is discretized fully implicitly usig a first-order backward fiite differece scheme. The discretized oliear system of fiite differece equatios for mass ad eergy balaces are solved simultaeously usig the Newto/Raphso iterative scheme. For the basic versio (i.e., sigle CPU), the code is equipped with both direct ad iterative solvers. 3 The developmet of parallel computers has made it possible to coduct large-scale reservoir simulatios. I the past decade, the total umber of gridblocks used i a typical reservoir simulatio icreased from thousads to millios. 4 Oe of the most popular parallel computer architectures is the distributed-memory machie, the massively parallel processor (MPP) computer, which ca be made up of hudreds to thousads of processors. Elmroth et al. 5 developed a parallel prototype scheme for the TOUGH2 code ad implemeted the computig time distributio o MPP computer. Their ivestigatio idicates that a parallel code ca dramatically ehace computatioal efficiecy. The preset work presets the further progress i reducig memory requiremet ad improvig computatio efficiecy, icludig the optimizatio for solvig extremely large reservoir simulatio problems. The parallelizatio of the TOUGH2 code was implemeted o a Cray T3E-900, a MPP computer. The parallel code was developed from the origial TOUGH2 code by itroducig the message-passig iterface (MPI) library. 6 MPI is a stadard procedure for message passig that allows data trasfer from oe processor to aother. The parallel implemetatio first partitios a ustructured simulatio domai usig the METIS graph partitioig programs. 7 The spatially discretized oliear equatios describig the flow system are the set up for each partitioed part at each time step. These equatios are solved with the Newto iteratio method. I each Newto step, a osymmetric liear equatio system is formed for each part of the domai ad is the solved usig a precoditioed

2 2 K. ZHANG, Y. S. WU, SPE, C. DING, K. PRUESS, SPE, AND E. ELMROTH SPE iterative solver selected from the Aztec liear solver package. 8 Durig each Newto iteratio, the liearized equatio systems must be updated with the updatig i primary variables. Updatig the left-had side Jacobia matrix requires commuicatio betwee differet processors for data exchage across the partitioig borders. By distributig the computatio time ad memory requiremet to processors, the parallel TOUGH2 code allows more accurate represetatio of reservoirs because of its ability to iclude more detailed iformatio o a refied grid system. The sigificat ehacemet o computatioal efficiecy i the parallel TOUGH2 code is demostrated through modelig of a field flow problem. The code has bee used to develop a three-dimesioal (3-D) model of multiphase fluid flow i variably saturated fractured rocks. The 3-D model uses more tha 10 6 gridblocks ad coectios (iterfaces) to represet the usaturated zoe of the highly heterogeeous, fractured tuffs of Yucca Moutai, Nevada, a potetial udergroud repository for high-level radioactive wastes. Numerical simulatio of the usaturated zoe flow system at Yucca Moutai has become a stadard tool i sitecharacterizatio ivestigatio. 9 However, the 3-D, site-scale usaturated flow models, developed sice the early 1990s, 10 i geeral use very coarse umerical grids primarily because of limitatio i computatioal capacity. I this paper, we discuss the mai issues addressed i the implemetig TOUGH2 o the massively parallel T3E-900 machie. We the preset a example problem for usaturated flow at the Yucca Moutai site, Nevada, which has more tha 1 millio grid blocks. This problem is used to evaluate speedup from code parallelizatio, ad to cofirm the solutio accuracy of the massively parallel code by compariso with results from a sigle-processor machie. Parallel Implemetatio As discussed above, the TOUGH2 code usig a IFD method 11, 12 solves mass ad eergy balace equatios of fluid ad heat flow i a multiphase, multicompoet system. The IFD approach avoids ay referece to a global system of coordiates ad thus offers the advatages of beig applicable to regular or irregular discretizatio i multiple dimesios. However, the flexibility i IFD formatio griddig makes a model grid that itrisically ustructured, which must be take ito accout by a parallelizatio scheme. I the basic versio of the TOUGH2 code, the discretizatio i space ad time usig the IFD leads to a set of strogly coupled oliear algebraic equatios, which is liearized by the Newto method. Withi each Newto iteratio, the Jacobia matrix is first calculated by umerical differetiatio, the resultig system of liear equatios the solved usig a iterative liear solver with precoditioig. Time steps ca be automatically adjusted (icreased or decreased) durig a simulatio ru, depedig o the covergece rate of the iteratio process. For a TOUGH2 simulatio, the most time-cosumig steps of the executio cosist of two parts: (1) solvig the liear system of equatios ad (2) assemblig the Jacobia matrix. Cosequetly, oe of the most importat aims of the parallel TOUGH2 code is to distribute computig time for these two parts. The mai schemes implemeted i the parallel code iclude grid partitioig, grid reorderig, optimizig data iput, assembly of the Jacobia matrix, ad solvig the liear system. The first stage of the work was summarized by Elmroth et al. 5 The followig sectios give a overview of the most importat parallel implemetatio procedures. Grid Partitioig ad Gridblock Reorderig. Efficiet ad effective methods for partitioig ustructured grid domais are critical for successful parallel computig schemes. Largescale umerical simulatios o parallel computers require the distributio of gridblocks to differet processig elemets. This distributio must be carried out such that the umber of gridblocks assiged to each PE is the same ad the umber of adjacet blocks duplicated ad copied to each PEs is miimized. The goal of the first coditio is to balace the computatio efforts amog the PEs; the goal of the secod coditio is to miimize the time-cosumig commuicatio resultig from the placemet of adjacet blocks to differet processors. I a TOUGH2 simulatio, a model domai is represeted by a set of gridblocks (elemets), ad the iterfaces betwee every two gridblocks are represeted by coectios. The etire coectio system of gridblocks is defied through iput data. From the coectio iformatio, a adjacecy matrix ca be costructed. The adjacecy structure of the model meshes is stored usig a compressed storage format (CSR). I this format, the adjacecy structure of a domai with gridblocks ad m coectios is represeted usig two arrays, xadj ad adj. The xadj array has a size of +1 whereas the adj array has a size of 2m. The adjacecy structure of the model grids is stored i a compressed format which ca be described as follows. Assumig that elemet umberig starts from 1, the the adjacecy list of elemet i is stored i a array adj, startig at idex xadj( i) ad edig at idex xadj(i+1)-1. That is, for each elemet i, its adjacecy list is stored i cosecutive locatios i the array adj, ad the array xadj is used to poit to where it begis ad where it eds. Figure 1a shows the coectio of a 12-elemets domai ad Figure 1b illustrates its correspodig CSR format arrays. We use three partitioig algorithms implemeted i the METIS package versio The three algorithms are here deoted the K-way, the VK-way, ad the Recursive partitioig algorithm. K-way is used for partitioig a graph ito a large umber of partitios (greater tha 8). The objective of this algorithm is to miimize the umber of edges that straddle differet partitios. If a small umber of partitios are desired, the Recursive partitioig method, a recursive bisectio algorithm, should be used. VK-way is a modificatio of K-way ad its objective is to miimize the

3 PARALLEL COMPUTING TECHNIQUES FOR LARGE-SCALE RESERVOIR SIMULATION SPE OF MULTI-COMPONENT AND MULTIPHASE FLUID FLOW 3 total commuicatio volume. Both K-way ad VK-way are multilevel partitioig algorithms. Figure 1a shows a scheme of partitioig a sample domai ito three parts. Gridblocks are assiged to particular processors through partitioig methods ad reordered by each processor to a local orderig. Elemets correspodig to these blocks are explicitly stored o the processor ad are defied by a set of idices referred to as the processor s update set. The update set is further divided ito two subsets: iteral ad border. Vector elemets of the iteral set are updated usig oly iformatio o the curret processor. The border set cosists of blocks with at least oe edge to a block assiged to aother processor. The border set icludes blocks that would require values from other processors to be updated. The set of blocks that are ot i the curret processor, but eeded to update compoets i the border set, is referred to as a exteral set. Table 1 shows the partitioig results ad oe of the local umberig schemes for the sample problem preseted i Figure 1a. The local umberig of gridblocks is doe to facilitate the commuicatio betwee processors. The umberig sequece is iteral blocks followed by border blocks ad fially by the exteral set. I additio, all exteral blocks from the same processor are i cosecutive order. Similar to vectors, a subset of matrix with o-zero etries is stored o each processor, I particular, each processor stores oly those rows, that correspod to its update set. These rows form a submatrix whose etries correspod to variables of both the update set ad the exteral set defied o this processor. Iput Data Orgaizatio. The iput data for reservoir simulatios iclude hydrogeologic parameters ad costitutive relatios of porous media, such as absolute ad relative permeability, porosity, capillary pressure, thermophysical properties of fluid ad rock, as well as iitial ad boudary coditios of the system. I additio, a umerical code requires specificatio of space-discretized geometric iformatio (grid) ad various program optios such as computatioal parameters ad time-steppig iformatio. For a typical, large-scale, three-dimesioal model, computer memory of several gigabytes is geerally required. Therefore, the eed arises to distribute the memory requiremet to all processors. Each processor has a limited space of memory available. To make efficiet use of the memory of each processor, the iput data files of the TOUGH2 code are orgaized i sequetial format. There are two groups of large data blocks withi a TOUGH2 mesh file: oe with dimesios equal to the umber of grid blocks, the other with dimesios equal to the umber of coectios (iterfaces). Large data blocks are read oe by oe through a temporary full-size array ad the distributed to PEs oe by oe. This method avoids storig all iput data i a sigle processor ad greatly ehaces the I/O efficiecy. The I/O efficiecy is further improved by storig the iput data i biary files. The data iput procedures ca be schematically outlied as follows: I PE0: Ope a data file Read first parameter for all blocks (total NEL blocks) ito array Temp(NEL) Do i=1,totalpes Call MPI_SEND( ) to sed the appropriate part of Temp(NEL) to PEi. Ed do Read secod parameter for all blocks ito array Temp(NEL) Do i=1,totalpes Call MPI_SEND( ) to sed the appropriate part of Temp(NEL) to PEi. Ed do. Repeat for all parameters that eed to be read from data file for all gridblocks. Read first parameter for all coectios (NCON) ito array Temp(NCON) Do i=1,totalpes Call MPI_SEND( ) to sed the appropriate part of Temp(NCON) to PEi. Ed do Read secod parameter all coectios ito Temp(NCON) Do i=1,totalpes Call MPI_SEND( ) to sed the appropriate part of Temp(NCON) to PEi. Ed do.. Repeat for all parameters that eed to be read from data file for all coectios. Close data file. I PE1, PE2,, PE: Allocate required memory space for curret PE. Call MPI_RECV( ) to receive the part of data that belogs to curret PE from PE0. Certai parts of the parallel code require full-coectio iformatio, such as for domai partitioig ad localcoectio idex searchig. These parts ca be the bottleeck of memory requiremet for solvig a large problem. Sice the full-coectio iformatio is used oly oce at the begiig of a simulatio, it may be better hadled i a preprocessig procedure. Assembly ad Solutio of Liear Equatio Systems. The discrete mass ad eergy balace equatios solved by the TOUGH2 code ca be writte i residual form: 1,2

4 4 K. ZHANG, Y. S. WU, SPE, C. DING, K. PRUESS, SPE, AND E. ELMROTH SPE R κ ( x t { V t+ 1 m ) = M A m F κ κ m ( x ( x t+ 1 t+ 1 ) M κ ) + V q ( x t κ, t+ 1 ) }...(1) where the vector x t cosists of primary variables at time t, κ R is the residual of compoet κ for block, M deotes mass per uit volume for a compoet, V is the volume of the block, q deotes siks ad sources, tdeotes curret time step size, t+1 deotes the curret time, A m is the iterface area betwee blocks ad m, ad F m is the flow betwee them. Equatio (1) is solved usig Newto-Raphso iteratio method, leadig to i R κ, t+ 1 + i p+ i p xi p κ, t 1 ( x, 1 x, ) = R ( x, i p ) (2) where x i,p represets the value of ith primary variable at pth iteratio step. The Jacobia matrix as well as the right-had side of (2) eeds to be recalculated at each Newto iteratio. The computatioal efforts are extesive for a large simulatio problem. I the parallel code, the assembly of liear equatio system (2) is shared by all the processors. Each processor is resposible for computig the rows of the Jacobia matrix that correspod to blocks i the processor s update set. Computatio of the elemets i the Jacobia matrix is performed i two parts. The first part cosists of computatios relatig to idividual blocks. Such calculatios are carried out usig the iformatio stored o curret processor ad commuicatios to other processors are ot ecessary. The secod part icludes all computatios relatig to the coectios. The elemets i the border set eed iformatio from the exteral set, which requires commuicatio betwee eighbor processors. Before performig these computatios, a exchage of relevat variables is required. For the elemets correspodig to border set blocks, oe processor seds these elemets to differet but related processors, which receive these elemets as exteral blocks. The Jacobia matrix for local gridblocks i each processor is stored i the distributed variable block row (DVBR) format, 8 a geeralizatio of the VBR format. All matrix blocks are stored row-wise, with the diagoal blocks stored first i each block row. Scalar elemets of each matrix block are stored i colum major order. The data structure cosists of a real vector ad five iteger vectors, formig the Jacobia matrix. The detail explaatio for the DVBR data format ca be foud from referece 8. The fial, local liear equatio systems are solved by usig the Aztec liear solver package 8. We ca select differet solvers ad precoditioers from the package. The available solvers iclude cojugate gradiet, restarted geeralized miimal residual, cojugate gradiet squared, trasposed-free quasi-miimal residual, ad bi-cojugate gradiet with stabilizatio methods. The results preseted i this paper have bee obtaied usig the stabilized bi cojugate gradiet method with block Jacobia scalig ad a domai decompositio precoditioer (additive Schwarz). I block Jacobia scalig, the block size correspods to the VBR blocks, which are determied by the equatio umber of each gridblock. Detailed discussios o precoditioig ad scalig scheme were preseted by Elmroth et al. 5 Durig a simulatio, the time steps are automatically adjusted (icreased or reduced), depedig o the covergece rate of the iteratio process i the curret step. I the parallel versio code, the time-step size foud i the first processor (master processor, amed PE0) is applied to all processors. The covergece rates may be differet i differet processors. Oly whe all processors reach stoppig criteria will the time march to the ext step. Fial solutios are derived from all processors ad trasferred to master processor for output. Results for the coectios that cross the boudary of two differet processors are obtaied by averagig the solutios from the two processors. Data Exchage Betwee Processors. Data commuicatio betwee processors is a essetial compoet of the parallel TOUGH2 code. Although each processor solves the liearized equatios of the local blocks idepedetly, commuicatio betwee eighborig processors is ecessary to update ad solve the etire equatio system. The data exchage betwee processors is implemeted through the EXCHEXTERNAL subroutie. Whe this subroutie is called by all processors, a exchage of vector elemets correspodig to the exteral set of the gridblocks will be performed. Durig time steppig or a Newto iteratio, a exchage of exteral variables is also required for the vectors cotaiig the secodary variables ad the primary variables. Detailed discussio of the implemetatio of data exchage ca be foud i Elmroth et al. 5 Program Structure. The parallel versio of TOUGH2 has almost the same program structure as the origial versio of the software, but solves a problem usig multiple processors. We itroduce dyamic memory maagemet, modules, array operatios, matrix maipulatio, ad other FORTRAN 90 features to the parallel code. MPI is used for message passig. Aother importat modificatio to the origial serial code is i the subroutie of time-step loopig. This subroutie provides the geeral cotrol of problem iitializatio, grid partitioig, data distributio, memory-requiremet balacig amog all processors, time steppig, ad output. All data iput ad output are carried out through the master processor. The most time-cosumig efforts, such as assemblig the Jacobia matrix, updatig thermophysical parameters, ad solvig the liear equatio systems, are distributed to all processors. The memory requiremets are also distributed to all processors.

5 PARALLEL COMPUTING TECHNIQUES FOR LARGE-SCALE RESERVOIR SIMULATION SPE OF MULTI-COMPONENT AND MULTIPHASE FLUID FLOW 5 Distributio of computig time ad memory requiremets is essetial for achievig a capacity for solvig large-scale field problems. Figure 2 gives a abbreviated overview of the program flow chart. r b (3) Applicatio O Yucca Moutai Problem Performace of the parallel code was evaluated ad demostrated through a three-dimesioal flow simulatio of the usaturated zoe at Yucca Moutai, Nevada. The problem is based o the site-scale model developed for ivestigatios of the usaturated zoe at Yucca Moutai, Nevada. 9,10 It cocers usaturated flow through fractured rock usig a 3-D, ustructured grid ad a dual permeability coceptualizatio for hadlig fracture-matrix iteractios. The usaturated zoe of Yucca Moutai is beig ivestigated as a potetial subsurface repository for storage of high-level radioactive wastes. The model domai of the usaturated zoe ecompasses approximately 40 km 2 of the Yucca Moutai area, is betwee 500 ad 700 m thick, ad overlies a relatively flat water table. The 3-D model domai as well as a 3-D irregular umerical grid used for this example is show for a pla view i Figure 3. The model grid uses relatively refied griddig i the middle, repository area, ad icludes several early vertical faults. The grid has about 9,800 blocks per layer for fracture ad matrix cotiua, respectively, ad about 60 computatioal grid layers i the vertical directio, resultig i a total of 1,075,522 gridblocks ad 4,047,209 coectios. A distributed-memory Cray T3E-900 computer equipped with 695 processors has bee used for the simulatio. Each processor has about 244 MB available memory ad is capable of performig 900 millio floatig operatios per secod (MFLOPS). The groud surface is take as the top model boudary, ad the water table is regarded as the bottom boudary. Both top ad bottom boudaries of the model are assumed Dirichlet-type coditios. I additio, o the top boudary, a spatially varyig ifiltratio is applied to describe the et water recharge, with a average ifiltratio rate of 4.6 mm/yr over the model domai. 10 The properties used for rock matrix ad fractures for the dual permeability model, icludig twophase flow parameters of fractures ad matrix, were estimated based o field tests ad model calibratio efforts, as summarized i Wu et al. 9 The liear equatio system arisig from the Newto iteratio of the Yucca Moutai problem is solved by the stabilized bi-cojugate gradiet method. A domai decompositio-based precoditioer with ILUT icomplete LU factorizatio has bee selected for precoditioig, ad the K-way partitioig algorithm has bee selected for partitioig the problem domai. The stoppig criteria used for the iterative liear solver is 2 where. = (1/ ) i = r 2 1 i, is the total umber of ukows, ad r ad b are the residual ad right-had side, respectively. Two types of tests were ru (1) to examie the accuracy of the parallel code, ad (2) to evaluate the code performace ad parallelizatio gais for differet umbers of processors. The first test simulates the flow system to steady state. The simulatio results for steady state flux through the repository ad bottom layer are compared to results previously obtaied from simulatios o a sigle-processor machie. The secod test used differet umbers of processors to simulate the usaturated flow system for 200 time steps. Steady State Test. The test problem was desiged to test the accuracy of solutios. We have verified the modelig results from the parallel code by comparig the solutios for a smaller grid model usig a oe-dimesioal vertical colum. The solutios for the smaller problem were obtaied usig the origial, sigle-cpu versio ad the parallel versio of the TOUGH2 code. The test preseted here provides a further verificatio of the code for large-scale simulatios. The 3-D test problem was ru o 64 processors for 3,684 time steps to reach steady state, recogized whe the fluxes goig ito ad leavig the flow system are equal (withi a arrow differece). Because of the time limitatio of the computer batch system, the whole simulatio is divided ito five stages. Each stage rus about 700 time steps i less tha four hours. The legth of a total simulatio time is about years whe steady state is obtaied. The percolatio flux through the repository horizo ad below is oe of the most importat factors cosidered i evaluatio of repository performace. Figures 4 ad 5 show the flux distributios alog the repository horizo ad at the bottom of the simulatio domai (the water table). The dark color idicates higher values of percolatio fluxes. The flux is defied i the figures as total mass flux through both fractures ad matrix. Compariso of the simulatio results (Figures 4 ad 5) agaist those usig coarse-grid models 10 idicates that the refied-grid model produces results with much higher resolutio ad more accurate flow distributios at the repository level as well as the water table. I particular, the curret, refied model predicts more sigificat lateral flow i the upper part of the usaturated zoe, above the repository horizo, due to usig fier vertical grid spacigs i these layers. These modelig results will have direct impact o assessig repository performace. Further simulatio results will be reported elsewhere. Performace Test. I the secod test, the problem was solved usig 32, 64, 128, 256, ad 512 processors, respectively.

6 6 K. ZHANG, Y. S. WU, SPE, C. DING, K. PRUESS, SPE, AND E. ELMROTH SPE Because of the automatic time-step adjustmet, based eve o the same covergece rate of the iteratio process, the legth of simulatio times over 200 time steps usig differet umbers of processors may be differet. However, the computatioal targets are similar, ad comparig the performace of differet umbers of processors with the same umber of time steps is reasoable for evaluatig the parallel code. Table 2 shows the reductio i the total executio time with a icrease umbers of processors. The simulatio was ru o from 32 processors up to 512 processors by cosecutively doublig the umber of processors. The results clearly idicate that the executio time is sigificatly reduced, as the umber of processors icreases. Table 2 also shows the time required for differet computatioal tasks usig differet umbers of processors. Whe less tha 128 processors are used, doublig the processor umber will reduce the total executio time by more tha half. From the table, we ca fid that the best parallel performace is i solvig-liear equatio systems. Data iput ad output of the program are carried out through a sigle processor, which will limit the performace of the parallel code for those parts. Figure 6 illustrates the speedup of the parallel code. The speedup is defied based o the performace of 32 processors as 32T 32 /T p, where T p deotes the total executio time usig p processors. The speedups from 32 to 64, 128, 256, ad 512 processors icrease by factors of 2.63, 2.16, 1.87 ad 1.54, respectively. Super-liear speedup appears durig the processor umber doublig from 32 to 64, ad to 128 with a speedup of 2.63 ad The overall speedup for 512 processors is 523. The super-liear speedup is mostly due to the precoditioer i solvig liear equatio system where the time requiremet is proportioal to 2, with beig the umber of gridblocks i each processor. I cotrast, the time requiremet for the startup phase (iput, partitio, distributio, ad iitializatio) i Table 2 icreases whe the processor umber is doubled from 256 to 512 (istead of decreasig). It idicates that a saturatio poit has reached. This results from the icrease of commuicatio overhead whe icreasig the umber of processors, which cacels the time savig by requirig more processors i this rage. The partitioig algorithm ca also sigificatly impact parallel code performace. The ideal case is that the gridblocks ca be evely distributed amog the processors with ot oly approximately the same umber of iteral gridblocks, but also roughly the same umber of exteral blocks per processor. For ustructured grids, this ideal situatio may be difficult to achieve i practice. However, i our problem gridblocks are almost evely divided amog processors. For example, o 128 processors, the average umber of iteral blocks is 8,402 at each processor, the maximum umber is 8,657 ad miimum umber is 8,156. It is oly about 6% differet betwee the maximum ad miimum umber. A cosiderable imbalace arises for the exteral blocks. I this problem, the average umber of exteral blocks is 2,447, while the maximum umber is as large as 3,650 ad the miimum as small as 918. This large rage idicates that the commuicatio volume ca be four times higher for oe processor tha aother. The imbalace i commuicatio volume results i a cosiderable amout of time wasted o waitig for certai processors to complete their jobs durig the solvig of equatio systems. I geeral, the memory capacity of a sigle processor may be too small to solve a problem with more tha oe millio gridblocks. The distributio of memory requiremet amog all the processors will solve the storage problem of iput data. For the Yucca Moutai oe-millio block problem, the parallel-computig performace is satisfactory for both computatio time ad memory requiremet. Coclusios Massive parallel computig techology has bee implemeted ito the TOUGH2 code for applicatio to large-scale reservoir simulatios. I the parallel code, both computig efforts ad memory requiremets are distributed amog ad shared by all processors of a multi-cpu computer. This parallel computig scheme makes it possible to solve large simulatio problems usig a parallel processor computer. The METIS graph partitioig program was adopted for the grid partitioig, ad the Aztec package was used for solvig the liear equatio systems. The parallel TOUGH2 code has bee tested o a Cray T3E system with 512 processors. Its performaces are evaluated through modelig flow i the usaturated zoe at Yucca Moutai usig differet umbers of processors with more tha a millio gridblocks. The total executio time is reduced from 10,101 secods o 32 processors to 618 secods o 512 processors for the field-scale variably saturated flow problem. A super-liear speedup of 523 for 512 processors has bee reached. Test results idicate that the overall performace of the parallel code shows sigificat improvemet i both efficiecy ad ability for large-scale reservoir simulatios. The major beefits of the code are that it (1) allows accurate represetatio of reservoirs with sufficiet resolutio i space, (2) allows adequate descriptio of reservoir heterogeeities, ad (3) ehaces the speed of simulatio. Ackowledgmet The authors would like to thak Jiachu Liu ad Da Hawkes for their review of this paper. The authors are grateful to Lehua Pa for his help i desigig the 3-D grid used for the test problem. This work was supported by the Laboratory Directed Research ad Developmet (LDRD) program of Lawrece Berkeley Natioal Laboratory. The support is provided to Berkeley Lab through the U. S. Departmet of Eergy Cotract No. DE-AC03-76SF Refereces 1. Pruess, K.: TOUGH2 A geeral-purpose umerical simulator for multiphase fluid ad heart flow, Lawrece Berkeley Laboratory Report LBL-29400, Berkeley, CA, 1991.

7 PARALLEL COMPUTING TECHNIQUES FOR LARGE-SCALE RESERVOIR SIMULATION SPE OF MULTI-COMPONENT AND MULTIPHASE FLUID FLOW 7 2. Pruess, K. Oldeburg, C., ad Moridis, G.: TOUGH2 User s Guide, V2.0, Lawrece Berkeley Natioal Laboratory, Berkeley, CA, Moridis, G. ad Pruess, K.: A ehaced package of solvers for the TOUGH2 family of reservoir simulatio codes, Geothermics (1998) 27, No.4, Dogru, A. H.: Megacell reservoir simulatio, JPT (MAY 2000), Elmroth, E., Dig, C., ad Wu, Y.: High performace computatios for large scale simulatios of subsurface multiphase fluid ad heat flow, accepted by The Joural of Supercomputig, Message Passig Formum: A message-passig iterface stadard, Iteratioal Joural of Supercomputig Applicatios ad High performace Computig, 8(3-4), Karypsis, G. ad Kumar, V.: A software package for partitioig ustructured graphs, partitioig meshes, ad computig fillreducig orderigs of sparse matrices, V4.0, Techical Report, Departmet of Computer Sciece, Uiversity of Miesota, Tumiaro, R. S., Heroux, M., Hutchiso, S. A., ad Shadid J. N.: Official Aztec user s guide, Ver 2.1, Massively Parallel Computig Research Laboratory, Sadia Natioal Laboratories, Albuquerque, NM, Wu, Y. S., Liu, J., Xu, T., Haukwa, C., Zhag, W., Liu, H. H., ad Ahlers, C. F.: UZ Flow Models ad Submodels, Report MDL- NBS-HS , Berkeley, Califoria: Lawrece Berkeley Natioal Laboratory, Las Vegas, Nevada, CRWMS M&O, Wu, Y.S., Haukwa, C., ad Bodvarsso, G. S.: A Site-Scale Model for Fluid ad Heat Flow i the Usaturated Zoe of Yucca Moutai, Nevada, Joural of Cotamiat Hydrology (1999), 38 (1-3), pp Edwards, A. L.: TRUMP: a computer program for trasiet ad steady state temperature distributios i multidimesioal systems, Natioal Techical Iformatio Service, Natioal Bureau of Stadards, Spigfield, VA Narasimha, T. N. ad Witherspoo P. A.: A itegrated fiite differece method for aalyzig fluid flow i porous media, Water Resour. Res. (1976), 12, 1, Table 1: Example of Domai Partitioig ad Local Numberig Processor 0 Processor 1 Processor 2 Update Exteral Iteral Border Gridblocks Local umberig Gridblocks Local Numberig Gridblocks Local umberig Table 2. Breakup of Executio Times (Secods) for the Yucca Moutai Problem Ruig 200 Time Steps. PE umber Iput, partitio, distributio, ad iitializatio Update thermophysical parameters, setup Jacobia matrix ad save results Solve liear equatios Total executio time

8 8 K. ZHANG, Y. S. WU, SPE, C. DING, K. PRUESS, SPE, AND E. ELMROTH SPE Processor Processor Processor (a) A 12-elemets domai partitioig o 3 processors Elemets xadj adj 2 1,3,7 2,4,10 3,5 4,6 5,11 2,8 7,9 8,10 3,9,11 6,10,12 11 (b) CSR format Figure 1. A example of domai partitioig ad CSR format for storig coectios Speedup Num ber of processors Figure 6. Speedup for the applicatio example o the Cray T3E-900

9 PARALLEL COMPUTING TECHNIQUES FOR LARGE-SCALE RESERVOIR SIMULATION SPE OF MULTI-COMPONENT AND MULTIPHASE FLUID FLOW 9 Start All PEs: Declare variables ad arrays, but ot allocate array space PE0: Read iput data, ot iclude property data for each block ad coectio PE0: Read mesh coectio data PE0: Broadcast parameters to all PEs PE1-PE: Receive parameters from PE0 PE0: Grid partitioig PE0: Set up globe DVBR format matrix PE0: Distribute DVBR matrix to all PEs PE1-PE: Receive local part DVBR format matrix from PE0 All PEs: Allocate memory spaces for all arrays for storig the properties of blocks ad coectios i each PE PE0: Read data of block ad coectio properties ad distribute the data PE1-PE: Receive the part of data which belogs to curret PE All PEs: Exchage exteral set of data All PEs: set up local equatio system at each PE All PEs: Solve the equatios use Newto method All PEs: Update thermophysical parameters Coverge? yes Next time step? All PEs: Reduce solutios to PE0 PE0: Output results o o yes Ed Figure 2. Simplified flow chart of parallel versio TOUGH2

10 10 K. ZHANG, Y. S. WU, SPE, C. DING, K. PRUESS, SPE, AND E. ELMROTH SPE Sever Wash Fault Pagay Wash Fault Solitario Cayo Fault Drillhole Wash Fault Ghost Dace Fault Due Wash Fault Imbricate Fault Figure 3 Pla view of the 3D simulatio domai, gird ad icorporated major faults

11 PARALLEL COMPUTING TECHNIQUES FOR LARGE-SCALE RESERVOIR SIMULATION SPE OF MULTI-COMPONENT AND MULTIPHASE FLUID FLOW 11 Figure 4 Simulated percolatio fluxes at repository horizo

12 12 K. ZHANG, Y. S. WU, SPE, C. DING, K. PRUESS, SPE, AND E. ELMROTH SPE Figure 5 Simulated percolatio fluxes at bottom of the domai