walberla: A software framework for CFD applications on Compute Cores

Transcription

1 walberla: A software framework for CFD applications on Compute Cores J. Götz (LSS Erlangen, [email protected]), K. Iglberger, S. Donath, C. Feichtinger, U. Rüde Lehrstuhl für Informatik 10 (Systemsimulation) www10.informatik.uni-erlangen.de Multiscale Fluid Dynamics with the Lattice Boltzmann Method 15 February, 2011 Lorentz Center Leiden 1

2 walberla: A software framework for CFD applications on Compute Cores J. Götz (LSS Erlangen, [email protected]), K. Iglberger, S. Donath, C. Feichtinger, U. Rüde Lehrstuhl für Informatik 10 (Systemsimulation) www10.informatik.uni-erlangen.de Multiscale Fluid Dynamics with the Lattice Boltzmann Method 15 February, 2011 Lorentz Center Leiden 1

3 Overview Motivation Why another CFD Package? Why Parallel Programming? The Software walberla: A Software Framework for CFD Fluid-Structure Interaction with Moving Rigid Objects Rigid Body Dynamics for Granular Media Free Surface Flow Simulations GPU Computing Conclusions 2

4 Motivation 3

5 Why we need another CFD program? In the last years many PhD students at our chair wrote nice programs for different CFD applications, but: Programming and testing basic functionality takes a lot of time Parallelizing takes even more time When the PhD student leaves the chair, the program most times was not used any more, since nobody knows how to use it 4

6 Why Parallel Programming? 5

7 Why Parallel Programming? 5

8 Why Parallel Programming? (2) Latest standard processors are multicore processors The free lunch is over To exploit multicore performance, parallel algorithms are essential CPUs will have 2, 4, 8, 16,..., 128,...,??? cores Want to simulate problems which are not possible on standard computers 6

9 The Software 7

10 walberla Created for desktop PCs and supercomputers Supporting multi-core PCs and GPUs Modular software concept Supports various applications: Blood flow in aneurysms Moving particles and agglomerates Free surfaces to simulate foams, fuel cells, a.m.m. Charged colloids 8

11 walberla Created for desktop PCs and supercomputers Supporting multi-core PCs and GPUs Modular software concept Supports various applications: Blood flow in aneurysms Moving particles and agglomerates Free surfaces to simulate foams, fuel cells, a.m.m. Charged colloids 8

12 Fluid-Structure Interaction with Moving Rigid Objects 9

13 Why Simulate Fluid-Structure Interaction? Transport of solid particles is crucial for: Understanding of physical phenomena Industrial processes But: Fully resolved simulation of the obstacles is computational expensive Up to now only moderate number of obstacles can be simulated 10

14 Fluid-Structure Interaction 11

15 Rigid Body Dynamics Newton s laws of motion including rotations Contact detection in each time step Collisions modelled by cofficient of restitution: forces in normal direction friction laws: forces in tangential direction 12

16 Rigid Body Dynamics Newton s laws of motion including rotations Contact detection in each time step Collisions modelled by cofficient of restitution: forces in normal direction friction laws: forces in tangential direction 12

17 Rigid Body Dynamics Newton s laws of motion including rotations Contact detection in each time step Collisions modelled by cofficient of restitution: forces in normal direction friction laws: forces in tangential direction 12

18 Hourglass Simulation spherical particles, 256 CPUs, time steps, runtime: 48 h (including data output) 13

19 Hourglass Simulation spherical particles, 256 CPUs, time steps, runtime: 48 h (including data output) 13

20 Mapping Moving Obstacles into the LBM Fluid Grid An Example 14

21 Mapping Moving Obstacles into the LBM Fluid Grid An Example 14

22 Mapping Moving Obstacles into the LBM Fluid Grid An Example 14

23 Mapping Moving Obstacles into the LBM Fluid Grid An Example (2) Cells with state change from Particle to Fluid Cell change from particle to fluid 15

24 Mapping Moving Obstacles into the LBM Fluid Grid An Example (2) Cells with state change from Fluid to Particle Cell change from fluid to particle 15

25 Mapping Moving Obstacles into the LBM Fluid Grid An Example (2) PDF acting as Force Momentum calculation 15

26 The Algorithm 16

27 Virtual Fluidized Bed 512 processors Simulation Domain Size: 180x198x360 cells of LBM 900 capsules and 1008 spheres = 1908 objects Number of time steps: Run Time: 252,000 07h 12 min 17

28 Virtual Fluidized Bed 512 processors Simulation Domain Size: 180x198x360 cells of LBM 900 capsules and 1008 spheres = 1908 objects Number of time steps: Run Time: 252,000 07h 12 min 17

29 Simulation of a Segregation Process Segregation simulation of objects. Density values of 0.8 kg/dm3 and 1.2 kg/dm3 are used for the objects in water. 18

30 Weak Scaling 1 Efficiency Jugene Blue Gene/P Jülich Supercomputer Center 40x40x40 lattice cells per core 80x80x80 lattice cells per core Number of Cores Scaling 64 to cores Densely packed particles lattice cells rigid spherical objects 19

31 Weak Scaling Efficiency Jugene Blue Gene/P Jülich Supercomputer Center Largest simulation to date : 8 Trillion (10 12 ) variables per time step (LBM alone) 50 TByte x40x40 lattice cells per core 80x80x80 lattice cells per core Number of Cores Scaling 64 to cores Densely packed particles lattice cells rigid spherical objects 19

32 Free Surface Flow Simulation for foams, fuel cells, food processing, etc. 20

33 Free Surface Flow Simulation Example applications: Engineering: metal foam simulations Food processing Fuel cells Based on LBM: Free surfaces Surface tension and wetting modell Parallelization with MPI 21

34 Free Surface Flow Simulation Example applications: Engineering: metal foam simulations Food processing Fuel cells Based on LBM: Free surfaces Surface tension and wetting modell Parallelization with MPI 21

35 Free Surface Flow Simulation Example applications: Engineering: metal foam simulations Food processing Fuel cells Based on LBM: Free surfaces Surface tension and wetting modell Parallelization with MPI 21

36 The interface between liquid and gas Volume-of-Fluids like approach Flag field: Compute only in fluid Special free surface conditions in interface cells 22

37 23

38 LBM on Clusters with GPUs 24

39 Motivation Why should we use GPUs for LBM simulations? GPUs currently offer a very high peak performance Basic LBM performs well on GPUs Programming GPUs is getting simpler than years ago Why should we use heterogeneous simulations CPUs are available anyway on GPU nodes Do not waste these resources NVidia Fermi Intel Xeon Node GFLOPS/sp GFLOPS/dp Peak Memory bandwidth GB/s

40 Conclusions 26

41 Conclusions Desktop PCs and notebooks will get 2,4,8,16,...,128,... cores

42 Conclusions Desktop PCs and notebooks will get 2,4,8,16,...,128,... cores Parallel programming is necessary for multicore usage

43 Conclusions Desktop PCs and notebooks will get 2,4,8,16,...,128,... cores Parallel programming is necessary for multicore usage walberla supports: Multicore systems Supercomputers Accelerators (GPU, Cell) A variety of different applications

44 Future Work OK, the framework is working fine for many applications

45 Future Work OK, the framework is working fine for many applications, but:

46 Future Work OK, the framework is working fine for many applications, but: Test cases for validation of particulate flows and free surfaces Any suggestions for moving particles (especially ensembles) are welcome ;-) Grid refinement + load balancing How to deal with massive parallelization: Node crashes Postprocessing Restart mechanisms How to maintain the software?

47 Thank you for your attention!