Large-scale ocean modelling with adapting unstructured mesh methods

Transcription

1 Large-scale ocean modelling with adapting unstructured mesh methods Matthew Piggott (Imperial College London) Annual Science Meeting Liverpool 1-3 June 2009 Page 1

2 Acknowledgments A large number of people have contributed to this work including groups at: Imperial College London Applied Modelling and Computation Group Grantham Institute for Climate Change Department of Earth Science and Engineering Department of Aeronautics Department of Computing High Performance Computing Service University of Oxford Department of Physics - AOPP National Oceanography Centre Ocean Modelling and Forecasting Ocean Observing and Climate Proudman Oceanographic Laboratory Daresbury Laboratory (HECToR) Fujitsu Laboratories of Europe and others Page 2

3 Some aims of this work To work towards a computational modelling framework with attributes: Open source community model development Flexible application to multi-scale ocean problems from laboratory to global scale Non-hydrostatic, Boussinesq/non-Boussinesq dynamics Choices between incompressible/compressible and multi-material/phase Optimised representation of coastlines and bathymetry Cartesian space no pole singularities Finite element/control volume discretisation options Element pairs including continuous/discontinuous and higher order Respect large scale geophysical balances h, r and p type mesh adaptivity algorithms Solvers/preconditioners which perform for poorly conditioned matrices Make use of massively parallel computing systems Cross-fertilisation of methods from other science/engineering areas Page 3

4 1 minute The future Resolution is critical, both to define adequately the domain of interest and also to resolve the relevant dynamics within it 6 minute Eddy resolving Increased realism, but the expense of massive computational costs Models used today use uniform structured meshes. 20 minute Eddy permitting 60 minute Climate Page 4 An unstructured (and adaptive) mesh is able to give an optimal representation of coasts, Imperial College bathymetry London and the dynamics, without the need for high resolution everywhere

5 Hence the interest in the flexibility for variable resolution that unstructured meshes provide However this flexibility does comes at a cost of higher CPU times The aim is to mitigate (and more!) this through the use of static or dynamic mesh adaptivity to focus computational resource where it is most required This can be due to numerical errors, or simply what is of relevance to the simulation being conducted, or the region you are interested in Mesh optimised for coastlines and bathymetry, using the adaptive methods presented here Page 5

6 GFD example: The lock exchange problem (Hiester 2009) Compare speed of head at free and no slip boundaries against DNS results (Hartel et al, 2000): & Page 6

7 Validation against Laboratory experiments (Maddison; Power) The differentially heated rotating annulus at two rotation rates. The bounds of the normalised temperature have been limited to aid visualisation at the end of the movie Page 7 R. Hide and P.J. Mason 1975: Sloping convection in a rotating fluid. Adv. Phys. 24,

8 Two snapshots in time at the faster rotation rate Here the maximum computational cost corresponds to the time of maximum kinetic energy Page 8

9 Internal wave breaking at the continental shelf break important for mixing in the vertical, biogeochemistry on the shelf (Martin) Open ocean deep convection an important process linking the surface and deep parts of the thermohaline circulation (Bricheno) Page 9

10 Here the adaptive mesh is focusing in and preserving the integrity of the thermocline in a lab scale simulation of internal wave generation, breaking and mixing over bathymetry (Martin) Page 10 (Martin 2009)

11 Westward intensified boundary currents: Effect of varying the maximum allowed aspect ratio in the adapted mesh Approximately a two orders of magnitude improvement in the error/cost relationship between uniform and anisotropic adaptive refinement Page 11 Uniform mesh

12 Advecting a tracer field through the boundary layer (cf. Hecht et al, Hanert et al) Fixed Adaptive 160k 40k 10k 10k Adaptive 40k 160k Page 12

13 Wind driven gyre, western boundary current and eddies resolved with an adapting anisotropic mesh Page 13 Anisotropy reflecting that although the flow is intense North-South, it is in the East-West direction that the important variation (shear) occurs Wind stress

14 High aspect ratio largescale baroclinic circulation Need for stable discretisations respecting geophysical balances, fast linear solvers for ill-conditioned pressure equation, minimally dissipative advection methods Also shown here the use of 2+1D adaptivity to preserve some structural difference between horizontal and vertical Page 14

15 Evolution of the upper mixed layer with an adapting mesh over a column Here atmospheric data from Station Papa in the Pacific is used to force the model. A k-epsilon turbulence model is being validated against measurements of the mixed layer. The adaptive mesh is reflecting the seasonal nature of the stratification. Seasonal node count Summer Summer Winter Page 15

16 Biology module with four species Aim to use adaptivity to simulate important small scale physics and seasonal biological activity. Currently developing methods in single column domains. Page 16

17 Mesh movement Solve a system of equations for node locations of the form where w is a monitor function which is large in regions requiring finer resolution, e.g. Ceniceros & Hou Also working in 3D Approach being developed to track isopycnals and eddies for example Have shown improved diapycnal mixing properties for simple problems Page 17

18 Modelling geohazards: landslide generated tsunami waves (Wilson) Lituya Bay, Alaska, 1958: Largest recorded landslide induced tsunami runup. Experimental models (Fritz et al., 2001) are now being used to validate our 2D wave generation models. Page 18 Stromboli, 2002: Landslide induced tsunami destroyed coastal infrastructure. Current 3D modelling of wave generation and propagation is building on previous laboratory scale models (DiRisio et al., 2009) of the event.

19 Indian Ocean Tsunami, Boxing Day 2004 Validation against arrival times from tide gauge data and another tsunami model. Also possible to look at free surface height profiles from satellite tracks (Jason-1). Location Arrival times (minutes) Tide Gauge Delfin model, 2007 ICOM model Sibolga, Indonesia Tuticorin, India Viakhap-atam, India Colombo, India Male, Maldives Diego-Garcia, UK Hanimadhoo, Maldives Gan, Maldives An idealised tsunami on aqua-planet no problem with polar singularities with Page 19 this approach

20 Dynamic balancing of the computational load between processors as the mesh adapts Cost of adaptivity approx 10% of simulation, cost of data migration minimal Page 20 Recent scaling analysis shows good results on up to 1024 processors of a Cray XT4 (HECToR): 0.8 efficiency going from with a 5M node adapting annulus (Gorman et al. 2009)

21 Conservative Interpolation: crucial for many applications, with adaptivity, coupled models donor mesh target mesh super mesh and mapping from donor mesh Following construction of the supermesh we are able to perform projection operations to achieve conservative, bounded (etc ) interpolation schemes. Farrell et al. Comp Meth Appl Mech Eng, in press. Page 21 Also now in 3D (Farrell+Maddison)

22 Buildbot: automated code verification system (Farrell et al.) All models require rigorous validation/verification. A continuous automated approach is required as the codebase changes. A central copy of the source is kept in a subversion repository. When developers commit changes this must result in a pass of the code test suite. Buildbot checks out and builds on various platforms with several different compilers. Any errors are relayed back to developers. If a failure is detected in a test problem, the developers are notified details via . Statistical information about code quality is automatically collected from the newly validated code. This allows for the monitoring of performance. Results available via a web interface. This modern approach to software engineering has yielded dramatic improvements in code quality and programmer efficiency. Page 22

23 A nice thing about having a general Navier-Stokes code at the core is that a larger number of validation & verification cases is available E.g. Comparisons between drag calculation in flow past a sphere at a range of Reynolds numbers Computed drag coefficient compared against correlation (from Brown and Lawler, 2003) with lab data Page 23

24 Diamond: new automatic pre-processor tool to improve usability (Ham et al. Geoscientific Model Development, 2009, Vol: 2, Pages: 33-42) An xml schema file describes the rules that govern ICOM options Diamond uses this to automatically generate a GUI based on the schema Options are entered and output as another xml file containing the options values This is written to an options library accessible from anywhere in ICOM Includes many features, including the ability to define python functions executed at run time by ICOM Page 24

25 Current status: Parallelisation of latest discretisation methods which respect balance and are needed for largescale high aspect ratio applications complexity (higher order and discontinuous solution fields) means a complete revamp of our entire parallel framework needed which is nearing completion Non-hydrostatic model means that we have costly elliptic equations to solve, high aspect ratios leads to poor matrix conditioning. New algebraic multi-grid methods have been developed which perform better than other solvers available. These have now been parallelised For tracer advection the twin requirements of resolving weak stratification as well as sharp fronts/interfaces makes this a very challenging problem for standard advection methods. Control volume limited and discontinuous based methods are being investigated Above are being combined on the large scale baroclinic double gyre problem as a stepping stone to basin/global scale applications Further development of subgrid-scale parameterisations Verification and validation on idealised problems and laboratory data; range of process studies Interfacing with NetCDF and bulk formulae for realistic forcing; open boundary conditions Development of a biology module Coupling with ice (sea ice and ice shelves), atmosphere, structures, Work towards a full adjoint and also adjoint produced from reduced order models (POD) Pre and post processing tools; documentation etc Important cross-fertilisation and re-application of technology and code with other areas: coastal engineering, atmospheric pollution, geohazards, geodynamics, oil reservoir engineering, Page 25 underground waste repositories,

26 Page 26