OpenFOAM Tomáš Brzobohatý Supercomputing for industry SC4Industry Research Programme 3 - Libraries for Parallel Computing, VŠB-TU Ostrava
OpenFOAM - Introduction - Features - OpenFOAM Case Outline OpenFOAM on ANSELM - Running applications - Running applications in parallel - Compiling applications and libraries Benchmarks
What OpenFOAM is? OpenFOAM - Introduction The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox based on C++. OpenFOAM is free and open source software, released under the GNU General Public License. OpenFOAM includes over 80 solver applications that simulate specific problems in engineering mechanics and over 170 utility applications that perform pre- and post-processing tasks, e.g. meshing, data visualization, etc. Friendly syntax for partial differential equations Unstructured polyhedral grid capabilities Automatic parallelization of applications written using OpenFOAM high-level syntax Commercial support and training provided by the developers
OpenFOAM - Introduction OpenFOAM syntax for tensor operations and partial differential equations ρu t For example the equation is represented by the code: + φu- μ U = p Solve ( ); fvm::ddt(rho,u) + fvm::div(phi,u) - fvm::laplacian(mu,u) == - fvc::grad(p)
History OpenFOAM - Introduction The original development of OpenFOAM started in the late 1980s at Imperial College, London. The FOAM, was sold by UK company Nabla Ltd. before being released as open source in 2004. On 15 August 2011, OpenCFD announced its acquisition by Silicon Graphics International (SGI). On September 12, 2012, the ESI Group announced the acquisition of OpenFOAM Ltd from SGI. Where we can find info? Homepage - http://www.openfoam.com/ Wiki - http://openfoamwiki.net/ CFD Forum - http://www.cfd-online.com/forums/openfoam/
OpenFOAM Features
Incompressible flows OpenFOAM - Features adjointshapeoptimizationfoam icofoam MRFSimpleFoam nonnewtonianicofoam pimpledymfoam pimplefoam pisofoam poroussimplefoam simplefoam SRFSimpleFoam
Incompressible flows OpenFOAM - Features adjointshapeoptimizationfoam icofoam MRFSimpleFoam nonnewtonianicofoam pimpledymfoam pimplefoam pisofoam poroussimplefoam simplefoam SRFSimpleFoam Multiphase flows bubblefoam cavitatingfoam compressibleinterfoam interfoam interdymfoam intermixingfoam interphasechangefoam LTSInterFoam MRFInterFoam MRFMultiphaseInterFoam multiphaseinterfoam porousinterfoam settlingfoam twoliquidmixingfoam twophaseeulerfoam simplefoam SRFSimpleFoam
Compressible flows OpenFOAM - Features rhocentralfoam rhocentraldymfoam rhopimplefoam rhoporousmrfltspimplefoam rhoporoussimplefoam rhoporousmrfpimplefoam rhosimplecfoam rhosimplefoam sonicdymfoam sonicliquidfoam
Compressible flows OpenFOAM - Features rhocentralfoam rhocentraldymfoam rhopimplefoam rhoporousmrfltspimplefoam rhoporousmrfsimplefoam rhoporousmrfpimplefoam rhosimplecfoam rhosimplefoam sonicdymfoam sonicliquidfoam chemfoam coldenginefoam dieselenginefoam dieselfoam enginefoam firefoam PDRFoam reactingfoam Combustion rhoreactingfoam XiFoam
Compressible flows OpenFOAM - Features rhocentralfoam rhocentraldymfoam rhopimplefoam rhoporousmrfltspimplefoam rhoporousmrfsimplefoam rhoporousmrfpimplefoam rhosimplecfoam rhosimplefoam sonicdymfoam sonicliquidfoam chemfoam coldenginefoam dieselenginefoam dieselfoam enginefoam firefoam PDRFoam reactingfoam Combustion rhoreactingfoam XiFoam Conjugate Heat Transfer buoyantbafflesimplefoam buoyantboussinesqpimplefoam buoyantboussinesqsimplefoam buoyantpimplefoam buoyantsimplefoam chtmultiregionfoam
RAS Turbulence Models OpenFOAM - Features laminar kepsilon komega komegasst RNGkEpsilon NonlinearKEShih LienCubicKE qzeta LaunderSharmaKE LamBremhorstKE LienCubicKELowRe LienLeschzinerLowRe LaunderGibsonRSTM realizableke SpalartAllmaras
RAS Turbulence Models OpenFOAM - Features laminar kepsilon komega komegasst RNGkEpsilon NonlinearKEShih LienCubicKE qzeta LaunderSharmaKE LamBremhorstKE LienCubicKELowRe LienLeschzinerLowRe LaunderGibsonRSTM realizableke SpalartAllmaras LES Turbulence Models laminar Smagorinsky Smagorinsky2 dynsmagorinsky dynlagrangian scalesimilarity mixedsmagorinsky dynmixedsmagorinsky komegasstsas oneeqeddy dynoneeqeddy locdynoneeqeddy specteddyvisc LRDDiffStress DeardorffDiffStress SpalartAllmaras SpalartAllmarasDDES SpalartAllmarasIDDES
Transport models OpenFOAM - Features Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerlaw interfaceproperties
Transport models OpenFOAM - Features Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerlaw interfaceproperties ThermoProperties icopolynomial perfectgas hconstthermo janafthermo speciethermo consttransport polynomialtransport sutherlandtransport hpolynomialthermo
Transport models OpenFOAM - Features Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerlaw interfaceproperties ThermoProperties icopolynomial perfectgas hconstthermo janafthermo speciethermo consttransport polynomialtransport sutherlandtransport hpolynomialthermo Radiation Models P1 fvdom viewfactor
Transport models OpenFOAM - Features Newtonian CrossPowerLaw BirdCarreau HerschelBulkley powerlaw interfaceproperties ThermoProperties icopolynomial perfectgas hconstthermo janafthermo speciethermo consttransport polynomialtransport sutherlandtransport hpolynomialthermo Radiation Models Reaction Models P1 fvdom viewfactor hpsimixturethermo hspsimixturethermo hrhomixturethermo hsrhomixturethermo homogeneousmixture inhomogeneousmixture veryinhomogeneousmixture dieselmixture basicmulticomponentmixture multicomponentmixture reactingmixture egrmixture
Mesh Generation OpenFOAM - Features blockmesh extrudemesh extrude2dmesh extrudetoregionmesh snappyhexmesh
Mesh Generation OpenFOAM - Features blockmesh extrudemesh extrude2dmesh extrudetoregionmesh snappyhexmesh Mesh Manipulation checkmesh mergemeshes movemesh polydualmesh refinemesh renumbermesh splitmesh transformpoints autorefinemesh modifymesh refinehexmesh refinementlevel refinewalllayer
Mesh Generation OpenFOAM - Features blockmesh extrudemesh extrude2dmesh extrudetoregionmesh snappyhexmesh Mesh Conversion Mesh Manipulation checkmesh mergemeshes movemesh polydualmesh refinemesh renumbermesh splitmesh transformpoints autorefinemesh modifymesh refinehexmesh refinementlevel refinewalllayer ansystofoam cfx4tofoam fluentmeshtofoam foammeshtofluent foamtostarmesh gambittofoam gmshtofoam ideasunvtofoam mshtofoam netgenneutraltofoam plot3dtofoam star4tofoam tetgentofoam writemeshobj
Pre-Processing OpenFOAM - Features applyboundarylayer applywallboundaryconditions changedictionary mapfields setfields viewfactorsgen
Pre-Processing OpenFOAM - Features applyboundarylayer applywallboundaryconditions changedictionary mapfields setfields viewfactorsgen Post-Processing foamdatatofluent foamtoensight foamtoensightparts foamtogmv foamtotecplot360 foamtovtk Co Lambda2 Mach
OpenFOAM case OpenFOAM case Constant - contains a full description of the case mesh in a subdirectory polymesh and files specifying physical properties for the application concerned, e.g.transportproperties. System - for setting parameters associated with the solution procedure itself. Time directories - containing individual files of data for particular fields Processors - directories are named processorn where N=0,1 represents a processor number and contains a time directory, containing the decomposed field descriptions, and a constant/polymesh directory containing the decomposed mesh description. properties polymesh constant Points Faces Cells boundary processors* case system controldict fvschemes fvsolution decomposepardict 0 time directories > 0
OpenFOAM on ANSELM
Available engineering modules [brz021@login2.anselm ~]$ module avail ------------------- /opt/modules/modulefiles/engineering -------------------- ansys/14.5.x matlab/r2013a-edu comsol/43b-com openfoam/2.2.1-gcc481-openmpi1.6.5-dp comsol/43b-edu openfoam/2.2.1-gcc481-openmpi1.6.5-sp lsdyna/7.x.x openfoam/2.2.1-icc-openmpi1.6.5-dp matlab/r2013a-com To create OpenFOAM environment on ANSELM give the commands: [brz021@login2.anselm ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-dp [brz021@login2.anselm ~]$ source $FOAM_BASHRC
Getting Started Create a project directory within the $HOME/OpenFOAM directory named <USER>-2.2.1 and create a directory named run within it, e.g. by typing: [brz021@login2.anselm ~]$ mkdir -p $FOAM_RUN Project directory is now available by typing [brz021@login2.anselm ~]$ cd /home/brz021/openfoam/brz021-2.2.1/run or [brz021@login2.anselm ~]$ cd $FOAM_RUN Copy the tutorial examples directory in the OpenFOAM distribution to the run directory [brz021@login2.anselm ~]$ cp -r $FOAM_TUTORIALS $FOAM_RUN Run the first example case of incompressible laminar flow in a cavity: [brz021@login2.anselm ~]$ cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity [brz021@login2.anselm ~]$ blockmesh [brz021@login2.anselm ~]$ icofoam
Running Serial Applications Create a Bash script: pokus.sh #!/bin/bash module load openfoam/2.2.1-icc-openmpi1.6.5-dp source $FOAM_BASHRC # source to run functions. $WM_PROJECT_DIR/bin/tools/RunFunctions cd $FOAM_RUN/tutorials/incompressible/icoFoam/cavity runapplication blockmesh runapplication icofoam Job Submission $ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh
OpenFOAM on ANSELM Running applications in parallel
Running applications in parallel #!/bin/bash #PBS N motorbike #PBS l select=2:ncpus=16 #PBS l walltime=01:00:00 #PBS q qprod #PBS A IT4I-1-4 pokusparallel.pbs module load openfoam/2.2.1-icc-openmpi1.6.5-dp source $FOAM_BASHRC cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike nproc = 32 mpirun hostfile ${PBS_NODEFILE} np $nproc snappyhexmesh overwrite parallel tee snappyhexmesh.log mpirun hostfile ${PBS_NODEFILE} np $nproc potentialfoam -nofunctionobject-writep parallel tee potentialfoam.log mpirun hostfile ${PBS_NODEFILE} np $nproc simplefoam parallel tee simplefoam.log
Running applications in parallel decomposepardict The mesh and fields are decomposed using the decomposepar utility. constant case system properties polymesh Points Faces Cells boundary 0 controldict fvschemes fvsolution decomposepardict time directories > 0 processors*
Running applications in parallel numberofsubdomains 32; method simple; decomposepardict simplecoeffs { n ( 4 4 2 ); delta 0.001; } hierarchicalcoeffs { n ( 1 1 1 ); delta 0.001; order xyz; } manualcoeffs { datafile ""; }
Running applications in parallel simple - simple geometric decomposition in which the domain is split into pieces by direction hierarchical - hierarchical geometric decomposition which is the same as simple except the user specifies the order in which the directional split is done scotch - scotch decomposition which requires no geometric input from the user and attempts to minimise the number of processor boundaries. manual - manual decomposition, where the user directly specifies the allocation of each cell to a particular processor.
Running applications in parallel Create a Bash script: pokus.sh #!/bin/bash module load openfoam/2.2.1-icc-openmpi1.6.5-dp source $FOAM_BASHRC # source to run functions. $WM_PROJECT_DIR/bin/tools/RunFunctions cd $FOAM_RUN/tutorials/incompressible/simpleFoam/motorBike runapplication blockmesh runapplication decomposepar Job Submission $ qsub -A OPEN-0-0 -q qprod -l select=1:ncpus=1,walltime=03:00:00 pokus.sh $ qsub pokusparallel.pbs
OpenFOAM on ANSELM Compile an own solvers
Compile an own solver Initialize OpenFOAM environment before compiling your solver [brz021@login2.anselm ~]$ module load openfoam/2.2.1-icc-openmpi1.6.5-dp [brz021@login2.anselm ~]$ source $FOAM_BASHRC $ cd $FOAM_RUN/ Create directory applications/solvers in user directory $ mkdir p applications/solvers $ cd applications/solvers Copy icofoam solver s source files $ cp r $FOAM_SOLVERS/incompressible/icoFoam/ My_icoFoam $ cd My_icoFoam Rename icofoam.c to My_icoFOAM.C $ mv icofoam.c My_icoFoam.C
Compile an own solver Edit files file in Make directory: icofoam.c EXE = $(FOAM_APPBIN)/icoFoam My_icoFoam.C EXE = $(FOAM_USER_APPBIN)/My_icoFoam In directory My_icoFoam give the compilation command: $ wmake
BENCHMARKS
Boeing 787 Dreamliner Benchmarks
Boeing 787 Dreamliner Benchmarks 106.2 M Cells 637.2 M Unknowns Mesh generated by snappyhexmesh simplefoam k-omegasst 2000 time step iter. solver parameters p U k omega - GAMG - smoothsolver - smoothsolver - smoothsolver Tolerance 1e-7 relativetolerance 0.1
Boeing 787 Dreamliner Benchmarks
Lamborghini Aventador Benchmarks
Lamborghini Aventador 225.3 M Cells 1351.8 M Unknowns Mesh generated by snappyhexmesh simplefoam k-omegasst 2000 time step iter. Benchmarks solver parameters p U k omega - GAMG - smoothsolver - smoothsolver - smoothsolver Tolerance 1e-8 relativetolerance 0.001
Lamborghini Aventador Benchmarks
Hydraulic Valve Benchmarks 52.3 M Cells Mesh generated by ANSYS Workbench simplefoam k-omegasst 1000 time step iter.
Hydraulic Valve Benchmarks
Time [s] Hydraulic Valve Benchmarks 2000 1500 1000 500 32 64 256 0 6 14 52 N Cells [M]
Thermal Sensor Benchmarks 5.3 M Cells Mesh generated by ANSYS Workbench chtmultiregionsimplefoam k-epsilon 1000 time step iter. 11 regions
Thermal Sensor Benchmarks
Lamp Benchmarks 17.3 M Cells Mesh generated by ANSYS Workbench chtmultiregionsimplefoam fvdom Radiation Model laminar 20000 time step iter. 6 Regions
Lamp Benchmarks
Lamborghini Aventador Benchmarks Transient solver - pisofoam 23 M Cells ncorrectors 2 k-omegasst solver parameters p - PCG U - PBiCG k - PBiCG omega - PBiCG
Lamborghini Aventador Benchmarks 20 000 time steps solution time - 40 hr 256 cores
VSB Campus Benchmarks
VSB Campus Benchmarks 18 M Cells Mesh generated by snappyhexmesh simplefoam k-epsilon 2000 time step iter.
VSB Campus Benchmarks
snappyhexmesh
Stonavka - Olše Benchmarks
Stonavka - Olše Benchmarks
snappyhexmesh Benchmarks VSB Campus 1,51 1,01 0,51 0,01 0 20 40 60 80 100 120 140
Boeing 787 Dreamliner Benchmarks snappyhexmesh scalability N cores 256 512 1024 Time [h] 1.5 2 1.9
Lamborghini Aventador Benchmarks snappyhexmesh scalability N cores 128 256 512 1024 Time [h] 3.7 2.5 3.1 3.7
Supermarine Spitfire Benchmarks
VW Beetle Benchmarks
Other Open Source Benchmarks Elmer Code Aster Code Saturne Palabos SU 2
benchmarket.it4i.cz Benchmarks