walberla: A software framework for CFD applications U. Rüde, S. Donath, C. Feichtinger, K. Iglberger, F. Deserno, M. Stürmer, C. Mihoubi, T. Preclic, D. Haspel (all LSS Erlangen), N. Thürey (LSS Erlangen/ now ETH Zürich) J. Götz (LSS Erlangen, jan.goetz@cs.fau.de) In collaboration with many more Lehrstuhl für Informatik 10 (Systemsimulation) Universität Erlangen-Nürnberg www10.informatik.uni-erlangen.de 12. November 2008 Workshop: Computational Bio-Mechanics, IGSSE/BGCE, München 1
Overview Motivation Why we need yet another software for CFD? walberla: A LBM framework for CFD Flow Simulation with Lattice Boltzmann Methods LBM introduction Computational Haemodynamics Simulations on real patient data Accelerating the simulation Non-Newtonian simulation and blood clotting Towards fluid-structure interaction with moving objects Conclusions 2
Part I Motivation 3
Why we need another CFD program? In the last years many PhD students at our chair wrote nice programs for different CFD applications, but: Programming and testing basic functionality takes a lot of time Parallelizing takes even more time When the PhD student leaves the chair, the program most times was not used any more, since nobody knows how to use it 4
Why parallel programming? Want to simulate problems which are not possible on standard computers And: Latest standard processors are multicore processors: The free lunch is over Without explicitly parallel algorithms, the performance potential cannot be used any more CPUs will have 2, 4, 8, 16,..., 128,...,??? cores 5
Part II walberla: A LBM framework for CFD 6
walberla Widely applicable Lattice Boltzmann from Erlangen Project started in November 2006 CFD project based on lattice Boltzmann method Team of 5 PhD students: Christian Feichtinger Klaus Iglberger Stefan Donath Jan Götz Frank Deserno 7
walberla Modular software concept Supports various applications: Blood flow Moving particles and agglomerates Free surfaces to simulate foams, fuel cells, a.m.m. Charged colloids Integration in efficient massivelyparallel environment 8
walberla Modular software concept Supports various applications: Blood flow Moving particles and agglomerates Free surfaces to simulate foams, fuel cells, a.m.m. Charged colloids Integration in efficient massivelyparallel environment 8
walberla Patch concept enables Parallelization Combining functionality Load balancing Memory reduction 9
walberla Patch concept enables Parallelization Combining functionality Load balancing Memory reduction E.g. in a brain geometry only about 3-10% of the nodes are filled with blood 9
Parallel performance on the Woodcrest cluster Woody-Cluster @ RRZE HLRB-II @ LRZ Garching 10
Part III Flow Simulation with Lattice Boltzmann Methods 11
Part III-a Lattice Boltzmann Methods LBM Introduction 12
The Lattice Boltzmann Method Discretization in cubes (cells) 9 (or 19) numbers per cell = number of particles traveling towards neighboring cells Repeat (many times) stream collide 13
The stream step Move particle (numbers) into neighboring cells 14
The collide step Compute new particle numbers according to the collisions 15
Stream/Collide: LBM in equations Equilibrium DF: 16
Boundary treatment No-slip boundary condition for a rigid wall 17
Part III-b Lattice Boltzmann Methods Computational Haemodynamics 18
Aneurysms Aneurysm are local dilatations of the blood vessels Localized mostly at large arteries in soft tissue (e.g. aorta, brain vessels) Can be diagnosed by modern imaging techniques (e.g. MRT,DSA) Can be treated e.g by clipping or coiling 19
Simulation chain 20
Simulation chain 20
Simulation chain 20
Simulation chain 20
Simulation chain 20
Motivation Why simulate aneurysms? Unruptured aneurysms are a major public health issue in every developed nation The flow situation could be crucial for further treatment of the patient Goals Help to understand the development of aneurysms To support therapy planning Challenges Current imaging techniques result in data sets of 512 3 and more Long runtimes on standard PCs and workstations For intra-surgery planning the algorithm should perform quasi realtime 21
Results Velocity near the wall in an aneurysm Oscillatory shear stress near the wall in an aneurysm 22
Pulsating blood flow in an aneurysm Collaboration between: Neuro-Radiology (Prof. Dörfler, Dr. Richter) Computer Science Simulation Imaging CFD Datensatz 23
Pulsating blood flow in an aneurysm Collaboration between: Neuro-Radiology (Prof. Dörfler, Dr. Richter) Computer Science Simulation Imaging CFD Datensatz 23
Pulsating blood flow in an aneurysm Collaboration between: Neuro-Radiology (Prof. Dörfler, Dr. Richter) Computer Science Simulation Imaging CFD Datensatz 23
Accellerating the simulation with the STI Cell processor Goal: demanding (flow) simulations at moderate cost but very fast Available cell systems: Roadrunner Blades Playstation 3 24
Performance results 50,0 43,8 37,5 25,0 21,1 12,5 9,1 11,7 0 Xeon 5160* Playstation 3 1 core 1 CPU *performance optimized code by LB-DC 25
Simulation of clotting processes using non-newtonian blood models Motivation Knowledge about the clotting of blood is important for many medical applications: Diagnostics Surgery 26
Non-Newtonian blood model: The Casson model Constant viscosity Shear strain rate Shear stress Yield stress 27
Reaction chain of clotting is well known Problems of simulating the whole clotting reaction chain: Lots of different species to be simulated Many parameters Many parameters are not known 28
So what to do? Usage of a well established analogous for blood clotting investigations: milk clotting Lot simpler reaction chain Similar clot forming properties 29
Part III - c Flow Simulation Fluid Structure Interaction: Moving Objects 30
Fluid structure interaction: Moving objects The rigid body physics engine pe Framework for accurate and games rigid body physics Calculation of contact forces and torque Accurate friction calculation during collision Coupling to different simulation frameworks 31
Rigid body dynamics with friction and objects of more general shape 32
Rigid body dynamics with friction and objects of more general shape 32
Rigid body dynamics with friction and objects of more general shape 32
Coupling the rigid body physics engine to the Lattice Boltzmann solver
Coupling the rigid body physics engine to the Lattice Boltzmann solver
Coupling the rigid body physics engine to the Lattice Boltzmann solver
Coupling the rigid body physics engine to the Lattice Boltzmann solver
Coupling the rigid body physics engine to the Lattice Boltzmann solver
Coupling the rigid body physics engine to the Lattice Boltzmann solver
Massively parallel simulations with many particles (LSS cluster) xsize: 540 ysize: 240 zsize: 400 51,84 x 10 6 lattice cells 31 processes/cores on LSS cluster 750 objects 10000 timesteps 9:45h 34
Massively parallel simulations with many particles (LSS cluster) xsize: 540 ysize: 240 zsize: 400 51,84 x 10 6 lattice cells 31 processes/cores on LSS cluster 750 objects 10000 timesteps 9:45h 34
Massively parallel simulations with many particles (Woody cluster) xsize: 540 ysize: 500 zsize: 500 135 x 10 6 lattice cells 129 processes/ cores (on woody cluster) 2,500 objects 27,000 time steps 12:54h 35
Massively parallel simulations with many particles (Woody cluster) xsize: 540 ysize: 500 zsize: 500 135 x 10 6 lattice cells 129 processes/ cores (on woody cluster) 2,500 objects 27,000 time steps 12:54h 35
Parallelization of flow with many particles 36
Part IV Conclusions 37
Conclusions walberla s modular software concept supports: Various applications: Blood flow Moving particles and agglomerates Free surfaces Charged colloids Integration in efficient massively-parallel environment Tested on different supercomputers Parallel support for compiling, restarting and visualization Extensions Work with master students/external PhD students Extensible framework for further LBM HPC software development 38
39
Talk is Over Slides, reports, thesis, animations available for download at: www10.informatik.uni-erlangen.de 40
Talk is Over Please wake up! Slides, reports, thesis, animations available for download at: www10.informatik.uni-erlangen.de 40
Talk is Over Please wake up! Slides, reports, thesis, animations available for download at: www10.informatik.uni-erlangen.de 40