Abstract. Introduction

Transcription

1 Fast solution of te Sallow Water Equations using GPU tecnology A Crossley, R Lamb, S Waller JBA Consulting, Sout Barn, Brougton Hall, Skipton, Nort Yorksire, BD23 3AE. amanda.crossley@baconsulting.co.uk Abstract Te increasing reliance on 2D flood modelling provides a strong incentive to reduce model run times, especially for studies involving uncertainty analysis or simulation of multiple scenarios. Tis paper reports a 2D Sallow Water Equation (SWE) flood model tat as been developed to run on te grapics card of a desktop PC, exploiting te fast parallel floating point performance of te grapics processing unit (GPU). We give an overview of te Riemann based finite volume sceme and its implementation in serial and parallelised algoritms. We sow test results to demonstrate tat te sceme produces dept and velocity simulations tat compare well wit oter models widely used in te industry for a realistic urban flood modelling application. We also sow tat te parallel GPU implementation of our sceme runs significantly faster tan te corresponding serial code on a conventional processor. In addition to numerical results from te new SWE model, we report on practical aspects in te use of GPU tecnology for flood modelling, building on earlier experience wit a 2D diffusion wave model. We discuss recent improvements in ardware and software, and te cost-efficiency of te tecnology for flood risk modelling and researc applications. Introduction In recent years te use of two dimensional (2D) flood inundation models as become well establised in flood risk management. A range of models as emerged based on different approaces to modelling te flow of water over a variable topograpy. One of te most popular approaces is to apply te Sallow Water Equations (SWE), wic are simplifications of te Navier-Stokes fluid flow equations for dept-averaged, incompressible flows. In practical flood modelling applications tere is a need for robust, stable numerical scemes tat can cope wit features observed in real flood situations, suc as a flood wave advancing over dry ground and weir-like flows over breaks in slope. In addition, calculation is becoming particularly important because of te use of scenario analysis and tecniques suc as Monte Carlo modelling tat require many different simulations to be performed. Te calculations tat are performed to solve a numerical sceme for te sallow water equations involve tousands or millions of iterations were te same aritmetic operations are repeated. Often, parts of te solution can be written in suc a way tat aritmetic operations are carried out largely independently of eac oter. Tis makes it possible to do te numerical work in parallel, using multi-core processors were aritmetic and logic instructions can be executed concurrently on eac core. Current tecnology trends point towards greater use of parallel processing rater tan unlimited growt in te speed of eac individual core. One of te emerging tecnologies for parallel computation is te Grapics Processor Unit (GPU). Tis as been based on te development of fast, igly parallel processors to support

2 te video games and computer grapics markets. Altoug GPUs began as specialised grapics accelerators, te tecnology as now advanced suc tat grapics ardware supports scientific calculation. GPU ardware is relatively ceap and can easily be deployed in typical office workstation PCs. We ave previously applied GPU programming tecniques to accelerate a 2D diffusion wave flood model, JFLOW (Lamb et al., (2009)). Te JFLOW-GPU code as since been used to produce national flood mapping data sets for rivers (in te UK, France Czec Republic, Slovakia and Hungary) and also areas susceptible to surface water flooding, as well as numerous smaller-scale studies. However, a SWE-based model can provide better results in situations were momentum is important and also more realistic prediction of velocities. Building on te advances made in te first generation of GPU flood modelling, we ave terefore developed a SWE model for te GPU platform. Te sceme reported in tis paper is a new formulation tat will be made available in future releases of te JFLOW software, ence we refer to it as JFLOW SWE. Model formulation In two dimensions te sallow water equations can be written as U t F x G y R were U is te vector of conserved variable, F and G represent te flux terms and R is te source term vector. Te vectors can be expressed as U u, v u F u g, 2 uv v G uv and v g 2 0 R g( S0 S x f g( S0 S y f x y ) ) were is te water dept and u and v represent te velocity components in te ortogonal grid directions. Te source terms are comprised of a bed slope term and a friction component (wic can be expressed in terms of Manning s n) wereby and S x S f x z x 0, n u u S0 y v z y 2 n v u v , S 4/3 f y 4/ 3 In te absence of source terms, te model equations written in tis form describe a series of conservation laws were water dept and momentum are conserved. Te sceme we use is a finite volume implementation of Roe s sceme (Roe and Pike, 1984, Toro, 2001, Le Veque, 2002), wic can be written for a 2D system of conservation laws in te form

3 U t x * * t * * n F F G G t n1 n i, Ui, i1/ 2, i1/ 2, i, 1/ 2 i, 1/ 2 Ri, y were te numerical fluxes are F 1 F U 2 n n ~ i1, F U ~ i, k ~ k ek i1/ 2, * i 1/ 2, G 1 G U 2 n n ~ ~ ~ i, 1 G Ui, k k ek i, 1/. * i, 1/ 2 2 We make use of a regular, structured rectangular grid tat allows immediate use of raster Digital Elevation Model (DEM) data sets for flood mapping. Te i and subscripts refer to te cell indexes witin te Cartesian grid. Te use of a tilde (~) denotes a Roe averaged quantity across te specified interface, were te averaging takes account of te local wave speed. For te x direction, te average values are defined as c~ u~ ~ u 1 2 c ~ c~ u~ u, ~ 2 2 c ~ ~ u ~ c~ ~, u ~ c~, ~ u ~ 1 1 ~ e u ~ c~ 1, v~ 2 1 ~ e u ~ c~ 2, v~, ~ v v~ 3 ~ e were u ~ L L R R, u L R u v L L R R v ~, L R v ~ 2 1 L R, c~ ~ g were te operator implies te difference between te rigt and left states. Te sceme falls under te classification of a Godunov type approac and is based on te approximate solution of Riemann problems yielding an upwinded sock capturing metod. We ave cosen to implement a first order version of te sceme. It is possible to extend te metod to second order troug te introduction of flux/slope limiters. We ave also adopted a unsplit strategy for solving te 2D equations. Tis means tat te update is performed in a single sweep troug te domain. As Roe s sceme is explicit, it is subect to a stability constraint of te form t a 1 (1) x

4 were a is a caracteristic velocity and te left and side of Equation 1 is known as te Courant-Fredric-Levy (CFL) number (Courant et al. (1967)). For te 1D sallow water equations, te appropriate velocity term is a = u +c. Te maximum permissible time step for eac cell is calculated and ten te minimum of tese values over te wole grid is used for te next update. In 2D, a number of alternative formulations are possible (Brandford and Sanders, 2002, Brufau et al., 2004, Delis et al., 2008, Toro, 1992 and Zou et al., 2001). Here, a formulation based on te 1D strategy as been adopted wereby Min x y t, i 0.5. ui, ci, vi, ci, Te ground slope and friction slope enter te sceme as te source terms R. Recent studies (Bermudez, A. and Vázquez (1994), Vázquez-Cendón (1999), Brufau et al. (2002)) ave igligted te benefits of employing upwind source term treatments wereby te left and rigt components are effectively weigted based upon local wave speeds. We utilise an upwind treatment for te bed slope source term, wic, coupled wit te Roe flux, results in a well balanced metod, meaning tat a pysically realistic equilibrium solution can be obtained for a lake at rest wereby te water level is constant and te velocity is zero. Te friction source terms are calculated in a pointwise fasion. Te application of a 2D SWE code to real world situations requires some additional considerations because te domain may contain bot wet and dry regions, and cells may become flooded or may dry out during a simulation. Te situations were a special treatment is required are 1. Water surface advancing over an adverse dry bed slope. 2. Water cascading from cells of ig elevation to neigbouring cells wose water level is below te ground elevation. 3. Dry cells and cells drying out. Cases 1 and 2 are closely related and are treated essentially by applying similar calculations as for a transmissive boundary condition locally witin te model grid. A local wall condition is implemented to prevent flow advancing unrealistically over adverse bed slopes. For dry or drying cells, a dept tresold is used to determine weter to perform te flux calculations for a given cell interface. Flux calculations are only performed if te dept in one of te cells exceeds te specified value, wic may be treated as an additional model parameter. Te tresold is also used to determine wen to apply te friction formulation and te velocity components are set to zero if te dept falls below te tresold. Implementation Te sceme summarised above as been implemented in code targeted at two types of computer processor. One implementation is written in C++ for a single x86 processor core, wic is te standard type of CPU in most desktop PCs running Windows. Tis version of te algoritm is a serial code, in oter words all loops and calculations are executed one after anoter on te same processor core.

5 Te second implementation runs on NVIDIA grapics cards using te Grapics Processing Unit (GPU) as a fast parallel numerical co-processor. Tis makes use of CUDA, a language based on C and developed by NVIDIA to enable programmers to exploit te multiple aritmetic cores and fast memory available in modern grapics cards. Altoug bot codes implement an identical algoritm, tere are inevitably some subtle differences in te outputs because te two codes target radically different processor arcitectures. However, for practical purposes te results are numerically nearly identical, typically to at least five or six significant figures. Tis is an advance over te earlier GPU implementation of te JFLOW diffusion wave model, were te x86 and GPU codes produced results tat were similar, but could be visibly distinguised on plots of results suc as water level outputs. Test cases Te software implementations of te above 2D SWE solution sceme produce comparable results to tose publised by oter autors for standard idealised test cases suc as instantaneous lateral and radial dam breaks, flows over a plane surface and steady state equilibrium solutions. Suc tests are of interest in evaluating te beaviour of te numerical sceme wit respect to analytical or reference results, but are not real world applications. Here we present tests results for a realistic model application to a detailed urban flood bencmark case. Urban flood bencmark case Tis test is based on modelling flooding from a ypotetical culvert blockage scenario in an area near to Glasgow, making use of data used in previous 2D model comparison studies (Hunter et al., (2008), Lamb et al., (2009)) and, currently, in a bencmark study commissioned by te Environment Agency of England and Wales. Te case study uses a 2 m x 2 m DEM grid (Figure 1) and a spatially varying Manning s n rougness grid. A ypotetical, plausible flow ydrograp is applied as an input to te model at te point marked in te upper rigt of te grid and te simulation allowed to proceed for two ours (model time). Figure 2 sows maximum flood depts simulated using te SWE sceme presented in tis paper. Wilst tere are no observations available for tis ypotetical case study, tere are results from oter well known 2D ydraulic models including TUFLOW, wic is widely used in flood modelling studies in te UK. For comparison te results from TUFLOW are also sown (using an implicit finite difference ADI solver for te sallow water equations).

6 Figure 1. Glasgow urban test case DEM sowing numbered monitor point locations and inflow location (red dot in upper rigt of te image). Figure 2. TUFLOW. Maximum dept grids. Upper panel: JFLOW SWE sceme. Lower panel:

7 Altoug te different numerical scemes and software implementations produce some sligt differences in modelled depts, it can be seen tat overall te two models generate very similar results. Similarly, te patterns of maximum velocity (Figure 3) produced by te two models are similar. Tere is some evidence tat te TUFLOW comparison results are sligtly smooter, as migt be expected for a model tat uses an implicit solution sceme. Figure 3. TUFLOW. Maximum velocity grids. Upper panel: JFLOW SWE sceme. Lower panel: Time series plots are sown for dept (Figure 4) and velocity (Figure 5) at te monitor points labelled 1 4 in Figure 1. Again, te results of our sceme are compared wit a TUFLOW simulation ere. Te data can be compared wit several oter 2D ydraulic models by reference to figures sown in Hunter et al. (2008).

8 Figure 4. Dept time series at monitor points. Figure 5. Velocity time series at monitor points.

9 Run time Te Glasgow bencmark tests were performed using an NVIDIA GeForce GTX285 grapics card. Run time for te JFLOW SWE GPU test simulation was 46 seconds. Tis compares wit previously publised results of 9.5 minutes for te previous JFLOW-GPU diffusion wave code. Te ost PC as an AMD Penom II 2.8 Gz CPU wit 3GB RAM. Relative to te serial C++ (x86 CPU) code implementation of te JFLOW SWE sceme, te corresponding GPU code was 116 times faster. Practical considerations Tere is now considerable experience in te use of te GPU computing platform for flood modelling. Applications include flood mapping, as noted earlier, along wit researc proects were multiple Monte Carlo simulations ave been carried out. An example is te area around Mexboroug were work in FRMRC2 involved running 1000 simulations of te JFLOW diffusion wave model to explore uncertainty associated wit specification of Manning s n for te floodplain. Tis type of analysis is possible on a desktop PC wit GPU ardware tat costs around 300. For large scale (e.g. national) flood mapping, our experience as been tat it is practical and cost effective to build a grid of GPU-based PCs. Te development of appropriate software tools enables suc a resource to be used eiter for large batc processing obs, suc as Monte Carlo analysis, or for te sceduling of individual model runs. Wit te advent of fast ydraulic calculations, it is often te case tat computational bottlenecks appear in te databases serving information, particularly DEM data, rater tan te ydraulic model itself. Hardware reliability is very occasionally an issue wen running a cluster of GPUs. Previous generations of GPU cards ave lacked some of te features found in conventional (and far more expensive) supercomputers. However, as part of te trend towards increasing use of GPUs for tecnical computing, te latest generation of processors include safety features suc as error-correcting memory protection. In our practical experience running a grid of more tan 50 GPUs, we ave encountered very few issues wit te reliability of te GPU ardware itself. Were reliability is of particular importance, grapics ardware vendors supply professional versions of teir ardware, albeit at prices several times more expensive tan te equivalent mass market consumer products. References Bermudez, A. and Vázquez, M.E Upwind metods for yperbolic conservation laws wit source terms. Computers Fluids, 23, Bradford, S.F. and Sanders, B.F Finite-volume model for sallow-water flooding of arbitrary topograpy. J. Hydraul. Eng. 128, Brufau, P., Vázquez-Cendón, M.E. and García-Navarro, P A numerical model for te flooding and drying of irregular domains. Int. J. Numer. Met. Fluids., 39,

10 Brufau, P., García-Navarro, P. and Vázquez-Cendón, M.E Zero mass error using unstrady wetting-drying conditions in sallow flows over dry irregular topograpy. Int. J. Numer. Met. Fluids., 45, Courant, R., Friedrics, K.O. and Lewy, H On te partial difference equations of matematical pysics. IBM Journal, Delis, A.I., Kazolea, M. and Kampanis, N.A A robust ig-resolution finite volume sceme for te simulation of long waves over complex domains. Int. J. Numer. Met. Fluids., 56, Hunter, N.M. Bates, P.D. Neelz, S. Pender, G. Villanueva, I., Wrigt, N.G., Liang, D., Falconer, R.A., Lin, B., Waller, S., Crossley, A.J. and Mason, D.C Bencmarking 2D ydraulic models for urban flood simulations. Proceedings of te Institution of Civil Engineers: Water Management, 161, Lamb, R., Crossley M.and Waller, S A fast two-dimensional floodplain inundation model. Proceedings of te Institution of Civil Engineers: Water Management, 162, Le Veque, R. J Finite Volume Metods for Hyperbolic Problems. Cambridge University Press. Roe, P.L. and Pike, J Efficient construction and utilistaion of approximate riemann solutions. In: Glowinski, R. and Lions, J.L. (ed). Proc. of te Sixt Int. Symposium on Computer Metods in Applied Sciences and Engineering, Toro, E.F Riemann problems and te WAF metod for solving te two-dimensional sallow water equations. Pil. Trans. R. Soc. Lond. A, 338, Toro, E.F Sock Capturing Metods for Free-Surface Sallow Flows. Wiley. Vázquez-Cendón, M.E Improved treatment of source terms in upwind scemes for te sallow water equations in cannels wit irregular geometry. J. Comput. Pys., 148, Zou, J.G., Causon, D.M., Mingam, C.G. and Ingram D.M Te surface gradient metod for te treatment of source terms in te sallow-water equations. J. Comput. Pys., 168, 1-25.