Interactive Distributed Fluid Simulation on the GPU

Transcription

1 Inteactive Distibuted Fluid Simulation on the GPU Tamás Umenhoffe and László Szimay-Kalos Depatment of Contol Engineeing and Infomation Budapest Univesity of Technology and Economics, Budapest, Hungay Abstact - Fluid dynamics is descibed by the Navie- Stokes diffeential equations, which need to be solved by fluid simulatos. The numeical solution of these equations in D has high computational and data stoing costs. In this pape we pesent a eal-time fluid simulation and visualization method that exploits the computational powe of a GPU cluste to solve this task inteactively. We use object space decomposition of the D volume. A single CPU/GPU node solves the Navie-Stokes equations only fo its subvolume, taking into account the data fom neighboing nodes as bounday conditions. Mathematically, the numeical simulation is executed on a egula gid, i.e. an Euleian appoach is taken. Fluid data at the gid points, including the velocity field and a custom display vaiable ae stoed in specially oganized D textues. In ode to guaantee the stability of the solution we have adopted the Stable Fluid appoach in ou distibuted solution, which have been extended by voticity confinement and level of detail contol using the ealy z-culling featue of the gaphics hadwae. The GPU of a node executes both the simulation and the endeing of the subvolume. We used a textue slicing endeing method with alpha blending to display a custom scala field that is caied aound by the simulated flow in the subvolume. The images of diffeent nodes ae composited togethe by the paallel pipeline algoithm of HP's PaaComp libay. Ou implementation has good scalability and makes the simulation of lage data sets possible at acceptable fame ates. I. INTRODUCTION One inteesting, and ecently widely eseached aea of compute gaphics is the simulation of fluid motion. Many phenomena that can be seen in natue like smoke, cloud fomation, fie and explosion show fluid-like behavio. Undestandably thee is a high need fo good and fast fluid solves both in special effects and game industy, and even in scientific aeas. While engineeing tasks equie physically accuate simulation, in compute gaphics the main inteest is on pefomance and visual appeaance. As the motion of fluids is vey complex, high quality simulation in eal-time is still a challenging poblem. With the evolution of gaphics hadwae in ecent yeas new algoithms have been developed that take advantage of the high paallelization of GPUs, making it possible to see fluid dynamic simulation even in compute games. Howeve thei quality is still limited by the available computational powe and memoy of cuent GPUs. In this pape we pesent a distibuted implementation of fluid simulation. Ou algoithm is based on the Euleian solution of the Navie-Stokes equations, and uns the simulation on a GPU cluste. The distibuted implementation makes it possible to solve the equations on highe esolution data sets than in case of a single compute application while still peseving inteactive fame ates. II. PREVIOUS WORK Up to now numeous aticles and books about fluid dynamics and thei solutions have been published in vaious aeas. Ealy appoaches to visualize these phenomena used paticle systems [8], whee the motion of the paticles is diven by use defined foces like wind field o gavity. Additional detail can be added with inceasing the paticle count - which geatly deceases pefomance - o using semi-tanspaent animated textues. The complexity of motion was geatly impoved with the intoduction of andom tubulent fields. Paticle systems wee used to model smoke, fie explosions, watefall, snow falls and blizzads successfully, and they ae still widely used in special effects industy and in compute games. The geatest disadvantage of this model is that it is only patially physically based, thus achieving a ealistic esult equies huge modeling labo. Solving the equations of fluid dynamics automatically povides a natual esult. A widely used model fo fluid motion is the pai of Navie-Stokes equations. Numeous methods have been used to solve these equations but basically thee ae two main classes, the Euleian and the Lagangian appoaches. Euleian appoaches discetize a well defined pat of the space and use vecto and scala fields, which define the media at fix points of a gid. Lagangian appoaches solve the equations fo tajectoies of a set of paticles descibing the flow. Intuitively, the Euleian viewpoint takes discete samples fom a egula gid, while the sample points of the Lagangian viewpoint move with the flow. The Lattice Boltzmann Model combines the two appoaches, while maintaining a gid, it woks with paticles. In ou implementation we used the Euleian appoach like in [4] [5] [9], and simulated the flow on a thee dimensional gid. Foste and Metaxas used the full thee dimensional Navie- Stokes equations to simulate smoke motion on a coase gid []. Thei algoithm used finite diffeencing of the Navie-Stokes equations and an explicit time solve. Because of the explicit integation scheme, thei algoithm was only stable fo time steps o velocities small enough compaed to the voxel size used in the simulation. This poblem was solved by Stam [9], who intoduced a semi-lagangian advection method and implicit solves in his stable fluid simulation. Stable solves made fluid dynamics simulation applicable both in offline special effect endeing and in eal-time compute gaphics.

2 Fedkiw et. al intoduced voticity confinement as an addition to this model, which is used to estoe fine detail tubulent motion dissipated by the discetized algoithm []. Hais built his simulation algoithm on Stam's stable fluid method [5]. He edesigned the algoithm to un entiely on the pocessing units of moden gaphics cads. He tested seveal solves that can be implemented on gaphics hadwae and finally ecommended a GPU optimized Jacobi iteation. He successfully used these methods to simulate cloud dynamics at inteactive fame ates. He also gave an excellent suvey on fluid simulation on gaphics hadwae [4]. In ecent yeas seveal papes wee published that pesented algoithms based on fluid dynamics on gaphics hadwae. Some of them focused on content ceation (like in [6]), othes pesented speed up possibilities of the simulation pocess [7]. Due to the geat eseach and publications in this topic, it is possible nowadays to see effects simulated with fluid dynamics even in compute games. The exploitation of GPU clustes to attack simulation and endeing poblems is a elatively new field. We follow the line of [], but instead of the Lattice Boltzman method, an extended stable fluid algoithm is poted to a GPU cluste. III. FLUID DYNAMICS The following subsections intoduce the Navie--Stokes equations and thei solution. The solution can be divided into seveal steps accoding to the subpoblems of advection, diffusion, pojection and voticity confinement. The stuctue of these subsections eflect this division. A. Equations of Fluid Motion A fluid with constant density and tempeatue can be descibed by its velocity u and pessue p fields. These values both vay in space and in time: u = u( x,t), p = p( x,t). The motion of a fluid is descibed by the Navie-Stokes equations: u = ( u ) u p + v u + F () t ρ u = 0 () whee ρ is the density, ν is the viscosity of the fluid, and in a Catesian coodinate system = ( / x, / y, / z). Equation () descibes the consevation of momentum while equation () states the consevation of mass (i.e. that the velocity field is divegence fee). These equations should also be associated with the definition of the bounday conditions. The fist tem on the ight side of equation () expesses the advection of the velocity field itself. This tem makes the Navie-Stokes equation nonlinea. The second tem shows the acceleation caused by the pessue gadient. The thid tem descibes diffusion that is scaled by the viscosity, which a measue of how esistive the fluid is to flow. Finally F denotes the influence of extenal foces. In the Navie-Stokes equations we have fou scala equations (the consevation of momentum is a vecto equation) with fou unknowns ( u, p). The pessue and the velocity fields ae elated accoding to these equations thus we can eliminate the pessue fom them educing the numbe of equations to thee. The elimination is based on a mathematical technique, called Helmholtz-Hodge decomposition. Accoding to the equation of pesevation of mass, the velocity field must be divegence fee. The Helmholtz-Hodge theoem states that any vecto field w can be decomposed into the sum of a divegence fee vecto field u and the gadient of a scala field s that is zeo at the bounday: w = u + s. () So if we ignoe the equiement of mass pesevation duing the computation, this equiement can be met late by pojection step P accoding to equation (). Rewiting this equation in anothe fom gives us the fomulae to make ou field mass conseving: u = P w = w s ( ). Using equation () and applying the divegence opeato on both sides, we get: s = w. (4) If we know w, then scala field s is computed by solving this equation, then pojection opeato P( w ) = w s can be evaluated to obtain u. Assuming a egula gid whee vecto field w is available, equation (4) becomes a linea systems of equations in the fom of A x = b, whee A is a matix, x is the vecto of unknown samples of s and b is a vecto of known samples descibing the divegence of w. Just like in [4] we used Jacobi iteation to solve these equations. Let us apply pojection opeato P to the momentum pesevation equation: u = P( ( u ) u + v u F ) + t since P( u ) = u and P ( p ) = 0. Note that this way we could enfoce the pesevation of mass by eliminating the pessue fom the unknowns. The time function of velocity field u (t) is obtained by stepping time t by δ t. The solution of the pojected momentum pesevation equation consists of seveal steps. Taking u (t), the velocity field at t, the tems of the ight side ae evaluated and added one by one, i.e. self-advected obtaining w, modified accoding to the diffusion getting w, acceleated accoding to extenal foces and voticity confinement esulting in velocity field w, and finally pojection opeato P is applied to guaantee the consevation of the mass: diffuse poject advect acceleate u(t) a w a w a w a u(t + δt) Let us conside these steps sepaately. B. Advection Advection descibes how the fluid caies itself aound:

3 w - u = ( u ) u. (5) δt One possibility is to expess w fom this equation w = u u uδ ( ) t and eplace diffeentials by finite diffeences. Howeve, this type of fowad Eule appoach is numeically unstable. Instead a backwad method is applied as poposed by Stam [9], that guaantees stability. A moving fluid tanspots objects with itself accoding to the diection of its velocity. The advection (o convection) of custom quantity Q in the fluid can be witten in the following implicit fom: Q ( x, t + δt) = Q( x u( x, t) δt, t). This equation means that we tace a paticle backwad along the path it tavels, and use the quantities of its fome position. This semi-lagangian method is called the method of chaacteistics. Selecting velocity u ( x, t) to be custom quantity Q we obtain the following equation fo the advected velocity field: u ( x, t + δt) = u( x u( x, t) δt, t). C. Diffusion The diffusion tem descibes how the fluid motion is damped. This damping is contolled by the viscosity of the fluid. Highly viscous fluids like syup stick togethe, while low-viscosity fluids like gases flow feely. The diffusion tem u = v u t can be ewitten in the following implicit fom: ( υδt ) w ( x) = w ( x) (6) whee w is the advected velocity field and w is the velocity field that also takes into account diffusion. The solution of such an equation is simila to that of equation (4) and is descibed in subsection E. As diffusion has noticeable effect on the simulation only in case of highly viscous fluids, this tem can be omitted fo low-viscosity fluids. D. Voticity confinement Many phenomena like smoke o ai mixtue have high detail tubulent stuctues in thei motion. These fine chaacteistics will be damped out by the numeical dissipation of the simulation. One way to estoe these featues is to add a pseudo-andom petubation to the velocity field. Fedkiw et al. poposed anothe method which added these details back only whee they should appea []. We also used thei solutions. The fist step is to compute the voticity of the velocity field (i.e. the cul o ot of the vecto field, which expesses the ate of otation o the ciculation density, that is the diection of the axis of otation and the magnitude of the otation): ψ = u. Then nomalized voticity location vectos ae computed, which point fom lowe voticity to highe voticity aeas: N η =, η = ψ. η Finally the diection and scale of the voticity foce is computed: = ε N ψ F vc ( ) whee ε contols the amount of the detail to be estoed to the flow field. The computed voticity foce should simply be added to the velocity just like any othe extenal foces. This tem can be omitted if small scale tubulent featues ae not needed. D. Adding the extenal and voticity foces Due to the extenal and voticity foces the flow will acceleate, i.e. the velocity field will change popotionally to the foces: w = w + ( F + Fvc ) δt. In ou application the use can inset mateial inside the fluid, and also can add foces with custom diection to make the fluid flow. E. Pojection The last step of the simulation is the pojection by P : u t + δt = P ( ) ( ). IV. SOLUTION OF LINEAR EQUATIONS The pojection and the diffusion steps equie the solution of a linea systems of equations in the fom of A x = b. We can use Jacobi iteation to solve these equations, which has quadatic complexity, but is still the most time consuming opeation of the simulation. A single iteation step equies the endeing of a full sceen quadilateal, and letting the pixel shade gathe the values of x fom neighboing pixels accoding to spase matix A. Howeve, whee the neighboing elements ae small, we can skip the gathe opeation, taking advantage of the ealy z-culling featue of the GPU [7]. The depth value is set popotionally to the maximum element in the neighbohood and to the iteation count. This way, as the iteation poceeds the GPU pocesses less and less numbe of fagments, and can concentate on impotant egions. Accoding to ou measuements, this optimization educes the total endeing by 40%. w V. VISUALIZING THE FLOW FIELD The discussed method computes the velocity field of the flow. If needed, the pessue can also be expessed fom the Navie-Stokes equations. When we wish to visualize the flow, one option is to map these paametes to some colo and opacity function, which in tun can be endeed by volume visualization algoithms. Instead of using the

4 velocity and the pessue, we can also assume that the flow caies a scala display vaiable with itself. The display vaiable is the analogy of some paint poued into the fluid. Using the advection fomula fo display vaiable d, the scala field can also be updated in paallel with the simulation: d ( x, t + δt) = d( x u( x, t) δt, t). At a time, the colo and opacity of a point can be obtained fom the display vaiable using a use contolled tansfe function. VI. SIMULATION ON THE GPU We used a collocated gid discetization to stoe ou velocity, pessue and density data. These vaiables ae defined at the cente of the voxels of a gid. At each time step the content of these data sets should be efeshed. As we do all the simulation on the GPU, we should stoe this data on the gaphics cad's memoy in a way that it can be efficiently ead and witten by the gaphics pocesso. In GPU pogamming we usually stoe ou data in textues. The poblem with thee dimensional data is that although the gaphics cads suppot D data sets as D textues, no endeing is allowed to them. One can solve this poblem with updating each slice of the volume into a sepaate textue and copy this data to the volume afte efesh. This copy opeation can take too much time making inteactive endeing impossible fo lage data (this poblem is solved by Shade Model 4 GPUs). laye pixels Slice bounday pixels bounday laye bounday laye D textue Fig.. Volume slices and Flat D textue. 5 Flat D textue Fo efficiency easons we used flat D textues as descibed by Hais [5]. A flat D textue tiles the slices of a D volume into a D textue (see Fig..). The esult of the simulation is not only contolled by the extenal foces but also by the bounday conditions. These conditions descibe what happens with the fluid at its bounday, i.e. on the faces of the cube encapsulating the gid volume. To stoe these conditions, we should extend each slice of the volume with one pixel bode and we should also add two exta layes at the top and at the bottom of the volume. Once the textues ae eady, one simulation step of a laye of the volume can be done by the endeing of a quadilateal, while bounday pixels ae efeshed with endeing line pimitives. As indexing a flat D textue needs exta wok compaed to usual D textues, we pepae a lookup textue which helps to index the oigin of any laye in the flat D textue. The endeed quadilateals also stoe infomation about the location of the layes ight above and below in the flat D textue. Unlike in tue D textues intepolating between two volume slices cannot be done by the textue units, thus intepolation is computed by custom shades. The simulation steps equie to ead and to wite into a D data at the same time. Calculation one simulation step means endeing to a textue, but accessing a textue which is cuently bound as a ende taget is not allowed. To ovecome this limitation we should make a copy of ou flat D textues - one will be sampled and one will be bound as a ende taget - and swap them afte ende (pingponging). We used a D textue slicing endeing method to display the esulting display vaiable field, which means that we place semi-tanspaent polygons pependicula to the view plane and blend them togethe in back to font ode. The colo and the opacity of the D textue is the function of the D display vaiable field. Hee we also had good use of the lookup textue to index ou flat D textues to find the location of an abitay point in space in the tiled stuctue. Fig.4. shows one state of the flat D textues descibing the velocity (ight column) and display vaiable (left column) fields in a simulation. The top ow shows these values with voticity confinement tuned off, and the bottom ow shows the effect of voticity confinement. VII. SOLID OBSTACLES If solid objects ae pesent inside the fluid, the simulation should be modified. The density inside the voxels occupied by these objects should be zeo, and the velocity should be set to the velocity of thee object []. Futhemoe, the bounday conditions should be popely set at the solid-fluid bounday. The appopiate bounday condition is a fee-slip condition, which states that fluid velocity equals to the object velocity in the diection of the bounday nomal. This pevents the fluid fom enteing the solid object, but can feely flow along its suface. The fee slip condition can be enfoced afte pessue pojection. We should coect the esult of the pojection step in the following way: if the bounday voxel along a diection is a solid object, then we should use the coesponding component of its velocity. This equies the identification of voxels occupied by solid objects, and the detemination of thei velocity. In othe wods the object should be voxelized. To do this we ceate a sepaate gid containing the inside-outside infomation of the voxels and also the velocity values in the solid objects. We used an appoach simila to [], but we applied alpha blending instead of stencil opeations. We ende objects into each slice of the gid using an othogaphic pojection. The fa clipping plane is set at infinity and the nea clipping plane at the depth of the cuent slice. We ende the object twice, once with back facing polygons and once with font facing polygons. We set up fo additive blending and use value fo back faces and - fo font faces. If the object is a closed shell, the esult will contain a value if the voxel is inside of the object and zeo othewise. If the mesh is a igid mesh and does not defom, we can also wite its velocity infomation into the fist thee components of this textue, while the alpha component stoes the inside-outside infomation. In case of defomable objects like skinned chaactes special handling of the exact bounday is [].

5 Afte voxelization the effect of solid object can be added to the velocity and density fields similaly to extenal foces. VIII. DISTRIBUTED SYSTEM Because of the data stoing needs and computational cost descibed so fa one can easily un into a point whee a single compute implementation becomes insufficient. Using flat D textues has a disadvantage that they limit the size of the gid that can be simulated, as thee is a limited textue size. Tiling the gid slices can easily esult in a flat D textue with too high esolutions. The maximum esolution of the gid that can be computed is We should note hee that this gid esolution equies flat D textue esolution since we need two exta slices and one exta pixel bode in each slice fo the bounday conditions. image as one famelet. The composition of famelets is done with alpha blending in a paallel way using the paallel pipeline algoithm [0]. We used object space distibution of the D data (see Fig.). We subdivided the gid and each pat was teated as a sepaate data set. Each node in the distibuted system should send the adjacent gid slices to its neighbos as a bounday condition to be used. In Fig. the dashed aows show the communication between the nodes in case of a thee node system. Fo example, the topmost node sends its lowest laye to its lowe neighbo, which puts this data to its topmost bounday condition laye. This neighboing node also sends its topmost slice to its uppe neighbo, which puts this data in its lowest bounday condition slice. Thee data should be tansfeed: the adjacent slices of the velocity, divegence vaiable s, and the display vaiable fields. We used an MPI implementation fo bounday laye communication between nodes. Node Node 4 D data with slices Node Fig.. Volume endeed with ou distibuted endeing algoithm. The wie cubes show one pat of the gid which is simulated on one node of the cluste. One can step ove this limit by subdividing the gid into smalle pats and stoe them in sepaate flat D textues, but can still un out of the GPU memoy. On the othe hand, the high computational cost of simulating high esolution gids does not allow inteactive fame ates. The solution of these poblems is the application of distibuted GPGPU simulation and endeing. We implemented ou algoithm on a GPU cluste. This is a shaed memoy paallel endeing and compositing envionment that uses the PaaComp libay ( of the HP Scalable Visualization Aay. The main aspect of this libay is that each host contibutes to the pixels of ectangula image aeas, called famelets. The pocess of meging the pixels of famelets into a single image is called compositing. The PaaComp libay allows not only the famelet computation but also the composition to un paallely on the cluste. In ou implementation one node takes the ole of the maste, which means that it has additional tasks beside endeing and compositing. The maste sets the pope ode of the nodes fo composition and displays the composited image on sceen. All nodes (including the maste) do simulation steps on one potion of the data set, ende the subvolume, and contibutes this Fig.. Data communication between the nodes of the cluste. As we used a slicing endeing method to display the volume we should only ensue that the images poduced by each node ae composited in back to font ode with alpha blending. IX. RESULTS Fo ou expeiments we used a Hewlett-Packad's Scalable Visualization Aay consisting of five computing nodes. Each node has a dual-coe AMD Opteon 46 pocesso, an nvidia Quado FX450 gaphics contolle, and an InfiniBand netwok adapte. One node is only esponsible fo compositing and managing the famelet geneations and does not take pat in the endeing pocesses, so we could divide ou data set into maximum fou pats. Table I. shows the pefomance esults of the simulation of Figues 4 and 6. It can be seen that using nodes gives bette pefomance than the single compute vesion. Using moe than nodes becomes useful only in case of lage data sets, fo small data sets the communication between the nodes becomes a bottleneck. The N/A sign means that a data set cannot be simulated on a single compute since it has too high memoy needs. It is woth examining the esolutions of and The 80 esolution needs about twice as many voxels as the 64 one. It can be clealy seen that the

6 pefomance of the two node implementation is the double of the pefomance obtained on a single compute. Similaly the two node implementation nealy doubles pefomance in almost all cases. X. CONCLUSIONS This pape showed that the computation of fluid simulation on a Catesian gid can be effectively shaed between nodes of a GPU cluste. Distibuted simulation and endeing of fluid phenomena ae necessay if the gid volume has highe esolution than the esolution a single GPU can cope with. Ou implementation shows good scalability, and the simulation esults do not suffe fom visual atifacts. TABLE I PERFORMANCE RESULTS Gid Resolution Node Node 4 Node 5 FPS 7 FPS 0 FPS FPS FPS 5 FPS FPS 0 FPS FPS FPS 6 FPS 9 FPS FPS 0 FPS FPS FPS.7 FPS 5.5 FPS FPS 6 FPS 8 FPS N/A 0. FPS 0.4 FPS ACKNOWLEDGEMENTS This wok has been suppoted by the National Office fo Reseach and Technology (Hungay), Hewlett-Packad, by OTKA K-799, and by the Coatian-Hungaian Action Fund CRO-5/006. REFERENCES [] Zhe Fan, Feng Qiu, Aie Kaufman, and Suzanne Yoakum- Stove, GPU cluste fo high pefomance computing, ACM/IEEE Supecomputing Confeence, 004. [] R. Fedkiw, J. Stam, and H. W. Jensen, Visual simulation of smoke, ACM SIGGRAPH 00, pages 5, 00. [] Nick Foste and Dimitis Metaxas, Modeling the motion of a hot, tubulent gas, Compute Gaphics, (Annual Confeence Seies):8 88, 997. [4] Mak Hais, Fast fluid dynamics simulation on the GPU, SIGGRAPH 05: ACM SIGGRAPH 005 Couses, page 0, New Yok, NY, USA, 005. ACM Pess. [5] M. Hais, W. Baxte, T. Scheuemann, and A. Lasta, Simulation of cloud dynamics on gaphics hadwae, Euogaphics Gaphics Hadwae 00, 00. [6] J. Küge and R. Westemann, GPU simulation and endeing of volumetic effects fo compute games and vitual envionments, Compute Gaphics Foum, 4():685 69, 005. [7] Natalya Tatachuk Pedo V. Sande and Jason L. Mitchell, Ealy-z culling fo efficient GPU-based fluid simulation, in Wolfgang Engel, edito, ShadeX5: Advanced Rendeing Techniques, chapte 9.6, page Chales Rive Media, Cambidge, MA, 006. [8] W. T. Reeves, Paticle systems - techniques fo modeling a class of fuzzy objects, SIGGRAPH 8 Poceedings, pages 59 76, 98. [9] Jos Stam, Stable fluids, Poceedings of SIGGRAPH 99, Compute Gaphics Poceedings, Annual Confeence Seies, pages 8, August 999. [0] T.-Y. Lee, C. Raghavenda, and J. B. Nicholas, Image composition schemes fo sot-last polygon endeing on D mesh multicomputes, IEEE Tansactions on Visualization and Compute Gaphics, (996). [] Keenan Cane, Ignacio Llamas, Saah Taiq. Real-Time Simulation and Rendeing of D Fluids. GPU Gems, Edited by Hubet Nguyen, Addison-Wesley 007. pp Fig. 4. Inteactive fluid simulation. Emittes ae contolled by the cuso.

7 Fig. 5. Inteactive fluid simulation. Emittes ae contolled by the cuso. Fig. 6. Flat D velocity (uppe ow) and density (bottom ow) textues of a simulation shown by Figue 4. The ight column shows the simulation with voticity confinement.