Parallel Multi-Zone Methods for Large-Scale Multidisciplinary Computational Physics Simulations

Parallel Multi-Zone Methods for Large-Scale Multidiscilinary Comutational Physics Simulations Ding Li, Guoing Xia, and Charles L. Merkle Mechanical Engineering, Purdue University Chaffee Hall, 500 Allison Road, West Lafayette, IN 47907-2014 Email: dli@urdue.edu Abstract. A arallel multi-zone method for the simulation of large-scale multidiscilinary alications involving field equations from multile branches of hysics is outlined. The equations of mathematical hysics are exressed in a unified form that enables a single algorithm and comutational code to describe roblems involving diverse, but closely couled, hysics. Secific sub-discilines include fluid and lasma dynamics, electrodynamics, radiative energy transfer, thermal/mechanical stress and strain distributions and conjugate heat transfer in solids. Efficient arallel imlementation of these couled hysics must take into account the different number of governing field equations in the various hysical zones and the close couling inside and between regions. This is accomlished by imlementing the unified comutational algorithm in terms of an arbitrary grid and a flexible data structure that allows load balancing by sub-clusters. Caabilities are demonstrated by a traed vortex liquid sray combustor, an MHD ower generator, combustor cooling in a rocket engine and a ulsed detonation engine-based combustion system for a gas turbine. The results show a variety of interesting hysical henomena and the efficacy of the comutational imlementation. Introduction High fidelity comutational simulations of comlex hysical behavior are common in many fields of hysics such as structures, lasma dynamics, fluid dynamics, electromagnetics, radiative energy transfer and neutron transort. Detailed three-dimensional simulations in any one of these fields can tax the caabilities of resent-day arallel rocessing, but the looming challenge for arallel comutation is to rovide detailed simulations of systems that coule several or all of these basic hysics discilines into a single alication. In the simlest multidiscilinary roblems, the hysics are loosely couled and individual codes from the several sub-discilines can be combined to rovide ractical solutions. Many alications, however, arise in which the multidiscilinary hysics are so intensely couled that the equations from the various sub-branches of hysics must likewise be closely couled and solved simultaneously. This imlies that the comutational algorithm, data structure, message assing and load balancing stes must all be addressed simultaneously in conjunction with the hysical asects of the roblem. In the resent aer we outline a method for dealing with such multi-hysics roblems with emhasis on the comutational formulation and arallel imlementation. The focus is on alications that involve closely couled hysics from several or all of the sub-domains listed above. Because conservation laws exressed as field variable solutions to artial differential equations are central in all of these sub-domains of hysics, the formulation is based uon a generalized imlementation of the artial differential equations that unifies the various hysical henomena. In the following sections we first resent a general conservation law for field variables and outline the numerical solution rocedure. Following this we describe the GEMS code in which these conservation laws are imlemented along with a multi-hysics zone method and comanion data structure that is used to establish our arallel comuting aroach. Reresentative results are then resented that demonstrate the degree of efficiency of the arallel imlementation as well as several ractical alications including a

traed vortex combustor, an MHD ower generator analysis, a conjugate heat transfer analysis for a rocket engine combustor and a constant volume combustion turbine system simulation Conservation Laws for Field Variables The fundamental henomena in nearly all fields of mathematical hysics are described in terms of a set of couled artial differential equations (de s) augmented by a series of constitutive algebraic relations that are used to close the system. The de s tyically describe basic conservation relations for quantitie4s such as mass, momentum, energy and electrical charge. In general, these de s involve three tyes of vector oerators, the curl, the divergence and the gradient that aear individually or in combination. When they aear alone, they tyically reresent wave henomena, while when they aear in combination (as the div-grad oerator) they tyically reresent the effects of diffusion. An imortant feature to in simulating multi-hysical henomena is that the structure of the conservation relations is essentially arallel for all branches of mathematical hysics. In contrast to the conservation equations, the constitutive relations are most often algebraic in nature. Constitutive relations are used to relate thermodynamic variables through aroriate thermal and caloric equations of state and ertinent roerty relations; to relate electric and/or magnetic fields to currents, and to relate stresses and strains to velocity and dislacement. The artial differential character of the conservation relations imly that these de s set the global structure of the comutational algorithm and the code, while the algebraic nature of the constitutive relations imlies that this auxiliary data can be rovided in subroutine fashion as needed, but does not imact the global structure of the code. These fundamental concets rovide much insight into arallel imlementations as well. Mathematically, the conservation relations for a general branch of mathematical hysics may be written as a generic set of artial differential equations of the form: Q + F D + F C + Φ = 0 t where Q and Φ are column vectors in the conservation system and F r D and F r C are tensors with similar column length whose rows corresond to the number of dimensions. (The subscrits, D and C, refer to Divergence and Curl tensors resectively.) Various comonents of these variables may be null. In writing these exressions, we have defined a generalized curl oerator that alies to vectors of length larger than three. For comutational uroses it is often useful to exand the divergence oerator to four variables and include the temoral derivative as an additional satial variable in the divergence term. These equations also imly the existence of a comlete set of indeendent variables (here denoted as Q to reresent the rimitive or most fundamental set of indeendent variables in the roblem) such that two vectors and tensors in the conservation relations may be exressed as comlete functions of the vector, Q (i.e., Q = Q( Q ), = Φ( Q ) Φ, F = F ( Q ) and F ( Q ) D D F = ). C An imortant issue for comutational uroses is that the length of the rimitive variables vector can vary dramatically among different fields of hysics. For examle, in solids where mechanical and thermal stresses are to be determined, Q, will generally include three dislacements, three velocities and one temerature for a total of seven equations. In simler solid alications where only heat conduction is included, only one artial differential equation need be solved and Q contains only one comonent. In simle fluid dynamics roblems, the variables will include three velocity comonents, the ressure and the temerature for a total of five comonents, but in more comlex alications there may be as many as six additional artial differential equations for turbulence modeling and an essentially unlimited number of secies continuity equations and hasic equations to describe finite rate chemical reactions and hase change. In electromagnetics, there are tyically two vectors of length three for a total of six, while when the MHD aroximation is used it is ossible to get by with three. Finally, for radiation, the number of equations can vary from one three-dimensional equation to couled six-dimensional equations. A summary of the number of equations in various sub-discilines is given in the Table on the following age. C (1)

Table 1. Number of Partial Differential Equations in Various Fields Physics Domain Sub-Domain Indeendent Variables Total No. Eqs. Maxwell Eqs. B (3) and E (3) Fields 6 MHD Arox. B(3) Field 3 Electromagentics B-E with divergence B (3) and E (3) Fields, 8 control Scalar otentials (2) Solid Mechanics Fluid Dynamics Small Dislacements Dislacement (3) 3 Large Dislacement, U dis (3), U vel (3) & T Thermal Stress, Phase 7 Change Laminar Flow Turbulent Flow U vel (3), and T 5 U vel (3),, T and NT Turblence (0<NT<7) 5-11 Combustion Reaction flow U vel (3),, T and NS Secies(0<NS<100) 5 100 Plasma Physics Fluids lus A combination of fluid >8 Electromagnetics dynamics lus electromagnetics Radiation (U to six dimensions) Intensity 1-10 Numerical Discretization and Solution Procedure Partial differential equations must be discretized before they can be solved numerically. Because the conservation laws from nearly all branches of hysics form matrices are wide-banded and, in addition, are nonlinear, iterative methods must be used to solve the resulting discretized systems. To accomlish the discretization and to define an aroriate iterative rocedure, we add a seudo-time derivative to the sace-time conservation relations. The discretization rocedure is then erformed by integrating over a series of control volumes of finite size to obtain an integral relation of the form: Q Γ τ Q Ω + dω + FDdΩ + FC dω + ΦdΩ = 0 t Ω Ω Ω By invoking the familiar theorems of Green, Stokes and Gauss, the volume integrals can be written as surface integrals, Q Γ τ Ω + QdΩ + n FDdΣ + n FC dσ + nφdσ = 0 t Ω Ω Ω Ω The surface integrals require the secification of a numerical flux across each face of the selected control volume and indicate that the rate of change of the rimitive variables in seudo time is determined by the sum of fluxes across the several faces of each control volume. An uwind scheme is emloyed to evaluate the numerical flux across the faces. The curl, divergence and gradient oerators each generate unique flux functions at the faces. A key issue in the discretization and in the alication to a multidiscilinary rocedure is that discretized exressions must be defined for each of the three vector oerators. As a art of the discretization ste, the seudo-time can also be used to define the numerical flux across each face of the control volume. In addition to defining the discretized equations, the coefficient matrix, Γ, in the seudo time term also introduces an artificial roerty rocedure that allows the eigenvalues of the convergence rocess to be roerly conditioned thereby roviding an efficient convergence algorithm to handle different time scale roblems. The introduction of the integral formulation for the discretized equation system allows the use of an arbitrary, structured/unstructured grid caability to enable alications to comlex geometry. This unstructured grid allows arallel load balancing on a cell-by-cell basis while enabling efficient data transfer. Secific data and code structures are imlemented in a fashion that mimics the conventional mathematical notations given above and the corresonding oerations for tensors, vectors and scalar functions. To allow for different numbers of conservation equations in different roblems, the number of equations is chosen at inut. In addition, individual roblems which contain multile zones in which different conservation equations must be solved are often encountered. To allow efficient comutation and load balancing over such regions, we use a multi-zone rocedure, with cells from each zone being Ω (2) (3)

transmitted to their own sub-cluster of rocessors. The comutational code that incororates these general equations, the arbitrary mesh and the multile hysical zones is referred to as the General Equation and Mesh Solver (GEMS) code. GEMS: General Equation and Mesh Solver The GEMS code uses contemorary numerical methods to solve couled systems of artial differential equations and auxiliary constitutive relations for ertinent engineering field variables (Fig. 1) on the basis of a generalized unstructured grid format. After converting the ertinent conservation equations from differential to integral form as noted above, the satial discretization is accomlished by a generalized Fig. 1. Comonents of GEMS code Riemann aroach for convective terms and a Galerkin aroach for diffusion terms. The numerical solution of these equations is then obtained by emloying a multi-level seudo-time marching algorithm that controls artificial dissiation and anti-diffusion for maximum accuracy, non-linear convergence effects at the outset of a comutation and convergence efficiency in linear regimes. The multi-time formulation has been adated to handle convection-dominated or diffusion-dominated roblems with similar effectiveness so that radically different field equations can be handled efficiently by a single algorithm. The solution algorithm comlements the conservation equations by means of generalized constitutive relations such as arbitrary thermal and caloric equations of state for fluids and solution-deendent electrical and thermal conductivity for fluids and solids. For multidiscilinary roblems, GEMS divides the comutational domain into distinct zones to rovide flexibility, romote load balancing in arallel imlementations, and to ensure efficiency. The detail of these techniques are given in next section. Multi-Physics Zone Method In ractical alications, we often face roblems involving multile media in which the ertinent henomena are governed by different conservation laws. For such alications, we define a hysics zone as a domain that is governed by a articular set (or sets) of conservation equations. For examle, in a conjugate heat transfer roblem, there are two hysics zone. The continuity, momentum and energy equations are solved in the fluid zone, while only the energy equation is solved in the solid zone. Similarly, MHD roblems can be divided into four hysical zones, the fluid, electric conductor, dielectric rings and surrounding vacuum zones (see figure 2). The continuity, momentum, energy, secies and magnetic diffusion equations are solved in the fluid zone; the energy and magnetic diffusion equations are solved in the electric conductor and dielectric solid zones, and only the magnetic diffusion equation is solved in the vacuum zone. In an arc-heater roblem, the inner lasma region in which fluids, radiation and electromagnetics coexist would be one zone; the heater walls in which electromagnetics and conjugate heat transfer are desired would be a second zone, and the external environment where only the EM equations are solved would be a third zone. To accomlish this effect, the number (and tye) of equations to be solved in each Fig. 2. Four hysical zones in MHD ower generator allocated more storage elements er cell, and etc. zone is an inut quantity. This zonal aroach rovides economies in terms of machine storage and CPU requirements while also simlifying load balancing. Regions with larger numbers of conservation equations are distributed to more rocessors and

This division into zones couled with the unstructured grid makes load-balancing on arallel comuters quite straightforward. The comlete comutational domain can be subdivided into several sub zones each of which is comuted on a searate rocessor. For examle, Zone 1 in Fig. 3 is rocessed in a 4-rocessor sub-cluster, while Zone 2 is rocessed in a 3-rocessor sub-cluster 2, and Zone 3 is rocessed in a 2- rocessor sub-cluster. The roblem sketched in Fig. 3 uses a cluster of 9 rocessors. In order to otimize the arallel comuting time, each rocessor loading has to be balanced. Because each hysics zone has different grid numbers and also different numbers of equations, the combination of the number of equations and the number of grids has to be balanced and otimized. The interface between multile hysics zones is treated as internal and external boundary conditions. Sharing information between two hysics zones satisfies the internal boundary conditions while the external boundary conditions are treated as normal boundary conditions. The interface between rocessors in each sub cluster for a given hysics zone has the same unknown Fig. 3. Sketch of multi-hysical zones with their interfaces variables and is treated as a normal inter-rocessor boundary. (Note that this hysical zone definition is different from a multile block grid of the tye obtained from structured grid generators. Normally, all grid zones in a multile block comutation have the same conservation equations. There could be multile grid blocks in each hysical zone shown in Figure 3). PC-Linux Cluster Architecture Our rimary comutational imlementation has been done on two PC clusters. The SIMBA linux cluster built in early 2001 has a head node with 50 slave nodes. All nodes have a single Intel Pentium 4 1.8 Ghz CPU with 1 Gb memory and 10/100 BaseT network adater and are linked by 60 orts on a 10/100 Fast Ethernet switch. Simba has a custom installation Beowulf cluster software including Redhat OS with a custom kernel for erformance, otimized message assing software (MPICH), libraries, custom scrits for node reboot, ower off and file coy across the cluster, C & Fortran comilers, Ganglia cluster monitoring utility and grahical user interface tools, security atches including reconfigured ort maer, IP chains and oen PBS batch scheduling system The second cluster is an Oteron cluster with 100 nodes each of which has a dual AMD Oteron 24 1.6 GHz CPU, with 4 Gb of memory which we are just bringing on line. These rocessors are connected by high erformance non-blocking infiniband network switches for low latency and high seed. A new 10 TeraGb storage system is available for data reository and data mining. The cluster has Intel, Portland and Pathscale Fortran comilers otimized for the 64-bit AMD Oteron CPU. Parallel Aroach As a arallel rogram GEMS must communicate between different rocessors. Our multi-hysics zone method uses a fully imlicit algorithm that is highly couled inside each rocessor while loosely couled between rocessors so that only small arts of data adjacent to the interface between two artitions need to be communicated between rocessors (see Fig. 4). A arallel oint-to-oint technique is used to reduce host control time and traffic between the nodes and also to handle any number of nodes in the cluster architecture. current artition interface receiving data sending data Fig. 4. Diagram of interface between artitions This technique should not introduce significant overhead to the comutation until the number of rocessors in the cluster is increased to over 100. With the emergence of massively arallel comuting architectures with otential for teraflo erformance, any code develoment activity must effectively utilize the

comuter architecture in achieving the roer load balance with minimum inter-nodal data communication. The massively arallel rocessing has been imlemented in GEMS for cross discilines such as comutational fluid dynamics, comutational structural dynamics and comutational electromagnetic simulations for both structured grid and unstructured grid arrangements. The hybrid unstructured gridbased finite-volume GEMS code was develoed and otimized for distributed memory arallel architectures. The code handles inter-rocessor communication and other functions unique to the arallel imlementation using MPI libraries which allows one to execute the code on a variety of latforms such as a PC-linux cluster, IBM SP2, Cray T3D and T3E, and SGI origin, as well as on windows workstation clusters. Very few MPI calls are used in GEMS code due to well defined data structure of shared data that is detailed later. Only the mi_sendrecv subroutine is used for sending and receiving data between rocessors to udate interface information after each iteration. GEMS loads mesh and other data into a master rocessor and then distributes this data to aroriate rocessors thereby making the code is ortable and flexible. The efficiency of the arallel methods considered in GEMS is rests uon the storage scheme used for the data shared between cluster nodes. Any single rocessor needs to send data to several other nodes while receiving data from an alternative array of nodes. The magnitude of the data sent from any rocessor is not necessarily equal Fig. 5. The exchanging rototye matrix for sending and receiving data to the amount it receives. To manage these inter-rocessor communications, we designed an exchanging rototye matrix (see figure 5.). In the figure 5 the index of the row reresents the index of the sending rocessor while the index of the column reresents the index of the receiving rocessor. A zero element in the matrix imlies there is no communication between the two subject rocessors reresented by the row and column indexes while nonzero elements of the matrix reresent the number of data ackets sent to the row index rocessor from the column index rocessor. The sum of row-wise elements is the total number of data received by the row index rocessor while the sum of columns-wise elements is the total number of data sent to the column index rocessor. The diagonal elements of the matrix are always equal to zero as there is no data sent or received inside the rocessor. Having set u this communications matrix, we can now use effective comressed row storage (CRS) format to collect and ack data into a contiguous ointer array sorted in the order of the rocessors. The ointer array hysically links to the storage in memory for the hysical locations of those data. This exchange data structure is necessary to achieve the oint-to-oint message assing oeration. In the GEMS code, two data ointer stacks, sending and receiving, are allocated to collect the exchange data between rocessors (fig. 6.). Each stack includes two arrays in CRS format: one is an index and the other is data. The index array stores the amount of data that is sent to individual rocessors that also indicate the current node has data to send to the node with non-zero value. Otherwise, there are no assing oerations token. As an examle, consider a data stack shown in Figure 6. The current node has 3 data messages to be sent to Node 1, 8 data messages to Node 4 and 5 data messages to Node 8. Similarly, receiving stack has two arrays that receive the data messages from nodes sent. Figure 6 shows that the current node has 1 data message received from Node 2, 3 data messages from Node 3, 2 data messages from Node 4 and 12 data messages from Fig. 6. Shared data structure of data communication between nodes 6. This communication, which is send directly from one node to another, brings significant communications efficiency to a distributed memory cluster, esecially as the number of nodes in the cluster increases. In addition, the exchange data structure used in the ointer array allows the data in the hysical

WTime/cells/iterations 0.0003 0.0002 0.0001 wtime/cells/iterations wall time 2500 2000 1500 1000 500 0 0 5 10 15 20 25 30 35 40 Numberof Processors Fig. 7. Wall clock time and wall clock time er cell er iteration vs. number of rocessor domain to be automatically udated when the message assing oerations are comleted. No additional stes are required to accomlish the udating. Figure 7 shows the total wall clock time and the wall clock time er cell and er iteration vs. the number of rocessors for a twodimensional hyersonic fluid flow calculation with about a half million cells. The comutational domain is artitioned to 5, 10, 20 and 40 artitions resectively and tested in our SIMBA cluster. The wall clock time is decreased when the number of rocessors increases while the average value of wall clock time of each cell and iteration (wtime) should be constant in the ideal case (in the absence of communication costs or other system oerations). When the number of rocessors is less than 10, the wtime is almost constant while as the number of rocessors are increased above 20 the wtime is increased by about (wtime_30-wtime_10)/wtime_10=1.5%. Wall Time (s) Reresentative Alications The multi-hysics zone GEMS code has been successfully alied to a variety of alications including a traed vortex combustor (TVC) with liquid fuel sray, an MHD ower generator with a lasma channel enclosed by dielectric/conductor walls of sandwich construction and the surrounding atmoshere in which the magnetic field decays to zero, the conjugate heat transfer in a rocket engine combustor and a combined ulsed detonation combustion with unsteady ejectors in oerating in cyclic fashion. Results from these various cases are outlined below. Traed Vortex Combustor The TVC concet is a revolutionary technology with otential ayoffs in almost every category of gas turbine combustor erformance including heat release rate, oerability, weight, and cost. The TVC dearts from a conventional gas turbine combustor in several ways, the most substantial of which is the mechanism that stabilizes the flame. TVC stabilizes the flame by traing a vortex in cavities located in the walls of the combustor as shown in Figure 8. Strategically laced air and fuel injection oints in the forward and aft walls of the cavity create a vortex in the cavity. The Fig. 8. Traed Vortex Combustor Simulation resulting vortex recirculates the hot combustion gases within the cavity which are then exhausted out of the cavity and transorted along the face of the combustor. In this alications, the aroximately 1 million grid oints are artitioned and distributed to 40 rocessors. Figure 8 is an overview of the stream article traces near the cavity region with temerature contours and an iso-surface of the 800k degree temerature contour. The chord wise vortex was observed inside the cavity and then trailed down the flame holder (left bottom). The right to is the iso-surface of the 100 Ka ressure surface while the right bottom charts show temerature contour slices.

MHD ower generator Magnetohydro-dynamic ower generators use the flow of an ionized gas through a magnetic field to roduce electrical ower. Although several classical geometrical configurations are available, the gas in the generator tyically asses through a flow channel comosed of individual sections of high electrical conductivity searated from each other by insulating, dielectric sections. The articular arrangement of the conductor / dielectric sections distinguishes the tyes of MHD generator and their Fig. 9. Magnetohydrodynamics Alication ensuing electrical outut characteristics. A characteristic of MHD generators is that the combined electro-magnetic/hydrodynamic fields are highly comlex and inherently threedimensional in nature. Our aroach is based uon a couled threedimensional solution of the magnetic diffusion equations and the Reynolds averaged Navier- Stokes equations that rovides the threedimensional magnetic field, electric current and fluid flow characteristics of an MHD ower generator. The formulation solves the Navier- Stokes equations in conjunction with the magnetic diffusion equation and a two-equation turbulence model on a hybrid unstructured grid. Physical difficulties with boundary conditions are removed by extending the comutational domain to encomass the lasma channel, the conducting and dielectric walls and the surrounding air. The secific roblem of interest in the resent aer is a diagonal MHD channel immersed in the field of an external magnet. Suersonic flow enters the channel from one end and exits through the other. The air is seeded with otassium to rovide electrical conductivity. The channel is comosed of a series of conducting regions searated from each other by stris of insulator stacked along the channel and oriented at an angle with resect to the axis. The multile diagonal sections in combination with the electrically conducting lasma give the advantage of increased outut voltage through an effectively series connection. Reresentative conditions of interest involve low magnetic Reynolds numbers and large Hall arameters. Figure 9 shows a combination of alications of an MHD ower generator, an electric conductor channel and the magnetic field in a wire. Constant Volume Combustion Turbine System Constant volume combustion has the otential to rovide substantial erformance imrovements in air-breathing roulsion systems although these imrovements are difficult to realize in ractical systems. In the resent examle, we look at a series of ulsed detonation tubes as a ossible means for imlementing constant volume combustion in a gas turbine system. The analysis involve both reacting and non-reacting flow solutions to the Navier-Stokes equations using the GEMS code that enables generalized fluids and generalized grids. The ressure contours of an Fig. 11. Constant volume hybrid combustion system for turbine engine alication unsteady, three-dimensional constant volume combustor are shown in Fig. 11 along with detailed diagnostics of the flow through a single PDE tube that is combined with a straight-tube ejector. The latter solution is comared with exeriment to validate the PDE simulations.

Summary We have described a unified arallel framework for dealing with multi-hysics roblems with emhasis on the comutational imlementation. The equations of motion are written in a generalized form with divergence, curl and gradient oerators that allow solids, liquids, gases, suercritical fluids and multi-hase or multi-comonent mixtures to be treated in a common manner. The artial differential equations are comlemented by arbitrary thermodynamic and caloric equations of state in fluid hases and constitutive equations for solid hases. The comutational imlementation of the general solver for the system uses an arbitrary, structured/unstructured grid caability to enable alications to comlex geometry while the general conservation formulation allows simulations of arbitrary material. Data and code structures are imlemented in a fashion that mimics conventional mathematical notation and oeration for tensors, vectors and artial differential equations. A arallel oint-to-oint technique is used to reduce host control time and traffic between the rocessors and to handle any number of rocessors in the cluster architecture that has otential to aly in the fast growing grid comuting arena. Timings on arallel rocessor indicates efficient oeration u to 40 rocessors. Current testing to cluster sizes exceeding 100 rocessors is in rogress. The broad caabilities of the unified framework are demonstrated using a series of test cases that involve a traed vortex combustor, an MHD ower generator analysis and a constant volume combustion turbine system simulation. Each of the examles showcases the effectiveness of using arallel comuting for handling comlex multidiscilinary hysics roblems. The results show a variety of interesting hysical henomena and the efficacy of the comutational imlementation Acknowledgement Portions of this work have been suorted by the Air Force Office of Scientific Research Test and Evaluation Program (AFOSR). References D. Li and Charles L. Merkle, Fundament of GEMS Code, rivate reort, 2001 D. Li, A User s Guide to GEMS, rivate reort, 2002 D. Li, S. Venkateswaran, Jules Lindau and Charles L. Merkle, A Unified Comutational Formulation for Multi-Comonent and Multi-Phase Flows, 43 rd AIAA aerosace sciences meeting and exhibit, AIAA 2005-1391, Reno, NV, January 10-13, 2005. G, Xia, D. Li, and C. L., Merkle, Modeling of Pulsed Detonation Tubes in Turbine Systems, 43 rd AIAA aerosace sciences meeting and exhibit, AIAA 2005-0225, Reno, NV, January 10-13, 2005. D. Li, D. Keefer, R. Rhodes, C. L. Merkle, K. Kolokolnikov and R. Thibodeaux, Analysis of MHD Generator Power Generation, AIAA2003-5050, July 2003. D. Li and C. L. Merkle, Analysis of Real Fluid Flows in Converging Diverging Nozzles, AIAA-2003-4132, July 2003. X. Zeng, S. Venkateswaran, C. L. Merkle and D. Li, Designing Dual-Time Algorithms for Steady-State Calculations, AIAA-2003-3707,, Orlando Fl, June23-26, 2003. S. Venkateswaran, D. Li and C. L. Merkle, Influence of Stagnation Regions on Preconditioned Solutions at Low Seeds, AIAA-2003-0435, Reno, NV, January 6-9, 2003. D. Li, F. Hakhari, S. Venkateswaran, and Charles L. Merkle, Convergence Assessment of General Fluid Equations on Unstructured Hybrid Grids, AIAA 2001-2557, Anaheim, CA, June 11-14, 2001. P.S. Pacheco, A User s Guide to MPI, Det. of Mathematics, University of San Francisco, San Francisco, CA 94117