walberla: A software framework for CFD applications

Transcription

1 walberla: A software framework for CFD applications U. Rüde, S. Donath, C. Feichtinger, K. Iglberger, F. Deserno, M. Stürmer, C. Mihoubi, T. Preclic, D. Haspel (all LSS Erlangen), N. Thürey (LSS Erlangen/ now ETH Zürich) J. Götz (LSS Erlangen, [email protected]) In collaboration with many more Lehrstuhl für Informatik 10 (Systemsimulation) Universität Erlangen-Nürnberg www10.informatik.uni-erlangen.de 12. November 2008 Workshop: Computational Bio-Mechanics, IGSSE/BGCE, München 1

2 Overview Motivation Why we need yet another software for CFD? walberla: A LBM framework for CFD Flow Simulation with Lattice Boltzmann Methods LBM introduction Computational Haemodynamics Simulations on real patient data Accelerating the simulation Non-Newtonian simulation and blood clotting Towards fluid-structure interaction with moving objects Conclusions 2

3 Part I Motivation 3

4 Why we need another CFD program? In the last years many PhD students at our chair wrote nice programs for different CFD applications, but: Programming and testing basic functionality takes a lot of time Parallelizing takes even more time When the PhD student leaves the chair, the program most times was not used any more, since nobody knows how to use it 4

5 Why parallel programming? Want to simulate problems which are not possible on standard computers And: Latest standard processors are multicore processors: The free lunch is over Without explicitly parallel algorithms, the performance potential cannot be used any more CPUs will have 2, 4, 8, 16,..., 128,...,??? cores 5

6 Part II walberla: A LBM framework for CFD 6

7 walberla Widely applicable Lattice Boltzmann from Erlangen Project started in November 2006 CFD project based on lattice Boltzmann method Team of 5 PhD students: Christian Feichtinger Klaus Iglberger Stefan Donath Jan Götz Frank Deserno 7

8 walberla Modular software concept Supports various applications: Blood flow Moving particles and agglomerates Free surfaces to simulate foams, fuel cells, a.m.m. Charged colloids Integration in efficient massivelyparallel environment 8

9 walberla Modular software concept Supports various applications: Blood flow Moving particles and agglomerates Free surfaces to simulate foams, fuel cells, a.m.m. Charged colloids Integration in efficient massivelyparallel environment 8

10 walberla Patch concept enables Parallelization Combining functionality Load balancing Memory reduction 9

11 walberla Patch concept enables Parallelization Combining functionality Load balancing Memory reduction E.g. in a brain geometry only about 3-10% of the nodes are filled with blood 9

12 Parallel performance on the Woodcrest cluster RRZE LRZ Garching 10

13 Part III Flow Simulation with Lattice Boltzmann Methods 11

14 Part III-a Lattice Boltzmann Methods LBM Introduction 12

15 The Lattice Boltzmann Method Discretization in cubes (cells) 9 (or 19) numbers per cell = number of particles traveling towards neighboring cells Repeat (many times) stream collide 13

16 The stream step Move particle (numbers) into neighboring cells 14

17 The collide step Compute new particle numbers according to the collisions 15

18 Stream/Collide: LBM in equations Equilibrium DF: 16

19 Boundary treatment No-slip boundary condition for a rigid wall 17

20 Part III-b Lattice Boltzmann Methods Computational Haemodynamics 18

21 Aneurysms Aneurysm are local dilatations of the blood vessels Localized mostly at large arteries in soft tissue (e.g. aorta, brain vessels) Can be diagnosed by modern imaging techniques (e.g. MRT,DSA) Can be treated e.g by clipping or coiling 19

22 Simulation chain 20

23 Simulation chain 20

24 Simulation chain 20

25 Simulation chain 20

26 Simulation chain 20

27 Motivation Why simulate aneurysms? Unruptured aneurysms are a major public health issue in every developed nation The flow situation could be crucial for further treatment of the patient Goals Help to understand the development of aneurysms To support therapy planning Challenges Current imaging techniques result in data sets of and more Long runtimes on standard PCs and workstations For intra-surgery planning the algorithm should perform quasi realtime 21

28 Results Velocity near the wall in an aneurysm Oscillatory shear stress near the wall in an aneurysm 22

29 Pulsating blood flow in an aneurysm Collaboration between: Neuro-Radiology (Prof. Dörfler, Dr. Richter) Computer Science Simulation Imaging CFD Datensatz 23

30 Pulsating blood flow in an aneurysm Collaboration between: Neuro-Radiology (Prof. Dörfler, Dr. Richter) Computer Science Simulation Imaging CFD Datensatz 23

31 Pulsating blood flow in an aneurysm Collaboration between: Neuro-Radiology (Prof. Dörfler, Dr. Richter) Computer Science Simulation Imaging CFD Datensatz 23

32 Accellerating the simulation with the STI Cell processor Goal: demanding (flow) simulations at moderate cost but very fast Available cell systems: Roadrunner Blades Playstation 3 24

33 Performance results 50,0 43,8 37,5 25,0 21,1 12,5 9,1 11,7 0 Xeon 5160* Playstation 3 1 core 1 CPU *performance optimized code by LB-DC 25

34 Simulation of clotting processes using non-newtonian blood models Motivation Knowledge about the clotting of blood is important for many medical applications: Diagnostics Surgery 26

35 Non-Newtonian blood model: The Casson model Constant viscosity Shear strain rate Shear stress Yield stress 27

36 Reaction chain of clotting is well known Problems of simulating the whole clotting reaction chain: Lots of different species to be simulated Many parameters Many parameters are not known 28

37 So what to do? Usage of a well established analogous for blood clotting investigations: milk clotting Lot simpler reaction chain Similar clot forming properties 29

38 Part III - c Flow Simulation Fluid Structure Interaction: Moving Objects 30

39 Fluid structure interaction: Moving objects The rigid body physics engine pe Framework for accurate and games rigid body physics Calculation of contact forces and torque Accurate friction calculation during collision Coupling to different simulation frameworks 31

40 Rigid body dynamics with friction and objects of more general shape 32

41 Rigid body dynamics with friction and objects of more general shape 32

42 Rigid body dynamics with friction and objects of more general shape 32

43 Coupling the rigid body physics engine to the Lattice Boltzmann solver

44 Coupling the rigid body physics engine to the Lattice Boltzmann solver

45 Coupling the rigid body physics engine to the Lattice Boltzmann solver

46 Coupling the rigid body physics engine to the Lattice Boltzmann solver

47 Coupling the rigid body physics engine to the Lattice Boltzmann solver

48 Coupling the rigid body physics engine to the Lattice Boltzmann solver

49 Massively parallel simulations with many particles (LSS cluster) xsize: 540 ysize: 240 zsize: ,84 x 10 6 lattice cells 31 processes/cores on LSS cluster 750 objects timesteps 9:45h 34

50 Massively parallel simulations with many particles (LSS cluster) xsize: 540 ysize: 240 zsize: ,84 x 10 6 lattice cells 31 processes/cores on LSS cluster 750 objects timesteps 9:45h 34

51 Massively parallel simulations with many particles (Woody cluster) xsize: 540 ysize: 500 zsize: x 10 6 lattice cells 129 processes/ cores (on woody cluster) 2,500 objects 27,000 time steps 12:54h 35

52 Massively parallel simulations with many particles (Woody cluster) xsize: 540 ysize: 500 zsize: x 10 6 lattice cells 129 processes/ cores (on woody cluster) 2,500 objects 27,000 time steps 12:54h 35

53 Parallelization of flow with many particles 36

54 Part IV Conclusions 37

55 Conclusions walberla s modular software concept supports: Various applications: Blood flow Moving particles and agglomerates Free surfaces Charged colloids Integration in efficient massively-parallel environment Tested on different supercomputers Parallel support for compiling, restarting and visualization Extensions Work with master students/external PhD students Extensible framework for further LBM HPC software development 38

56 39

57 Talk is Over Slides, reports, thesis, animations available for download at: www10.informatik.uni-erlangen.de 40

58 Talk is Over Please wake up! Slides, reports, thesis, animations available for download at: www10.informatik.uni-erlangen.de 40

59 Talk is Over Please wake up! Slides, reports, thesis, animations available for download at: www10.informatik.uni-erlangen.de 40