Summary Flood Modelling for Cities using Cloud Computing FINAL REPORT Vassilis Glenis, Vedrana Kutija, Stephen McGough, Simon Woodman, Chris Kilsby Assessment of pluvial flood risk is particularly difficult because it is sensitive to the spatialtemporal characteristics of rainfall, topography of the terrain and surface flow processes influenced by buildings and other man-made features. CityCat is an urban flood modelling, analysis and visualisation tool which uses very accurate and computationally efficient solutions for free surface flow equations. In this pilot project a Cloud Computing compatible version of CityCat was developed and for the first time applied for estimating spatial and temporal flood risk at a city-scale. The use of Cloud Computing enabled modelling of flooding of larger domains (up to 1100km 2 ) with much higher resolution (up to 16.000.000 computational cells) than it has been done previously. This project has demonstrated that the use of Cloud Computing can enable efficient and detailed modelling of flooding at a city or even regional scale, with high resolution, using standard terrain and rainfall data and powerful Cloud enabled software which does not impose limitations on the computational domain size. Background The risk from pluvial flooding, where intense direct rainfall overwhelms urban drainage systems, in the UK cities is considerable and as a result of 2005 and 2007 floods, local authorities are now required to develop Surface Water Management Plans (SWMPs) However, assessment of pluvial flood risk is particularly difficult because it is sensitive to the spatial-temporal characteristics of rainfall, topography of the terrain, local runoff and surface flow processes influenced by buildings and other man-made features and the performance of urban drainage systems. Conventional assessments of urban flood risk are generally carried out at relatively small scales using commercial software resulting in very restricted coverage in space and number of design storms (Hunter et al., 2008). Alternatively, simplified codes may be used for city scales (Neal et al., 2009).The use of fully detailed numerical codes for larger areas is in its infancy and the requirement for HPC or cluster facilities means it is limited to institutional platforms such as Condor grids which are restrictive in terms of power as well as, crucially, access for non-institutional users in industry. On the other hand, Cloud Computing (CC) offers (apparently) huge amount of processing power to anyone. Also, Cloud Computing has been successfully used within the business community allowing on-demand access to this power at a (relatively) cheap price. The motivation for this project was exploration of suitability of Cloud Computing for the assessment of city-scale flood risk
CityCat is an urban flood modelling, analysis and visualisation tool which uses state of the art numerical solutions for flow equations in a user-friendly visual environment. Its numerical solutions of the 2D free surface flow equations are very accurate and computationally efficient but it has been recognised that users would benefit from further enhancement of the computational power brought in by Cloud Computing as that will enable modelling of larger domains, extended simulation periods and/or different future climate scenarios. Objectives A meaningful assessment of city-scale flood risk requires a very large number of simulations on large domains that cannot be achieved by existing institutional HPC or cluster facilities. Therefore in this pilot study we developed for the first time a truly city-scale application of the hydrodynamic model CityCat for estimating spatial and temporal flood risk using Cloud Computing. This has been achieved through the following objectives: 1. Porting the existing state-of-the-art hydrodynamic model CityCat from desktop to Cloud 2. Selecting and using only readily available data sets for CityCat simulations so that this kind of flood risk assessment can be easily applied nationwide 3. Generating a large number of design storms with different return periods and durations as rainfall input for simulations 4. Applying CityCat to different large domains (4km 2 to 1100 km 2 ) for a variety of extreme storm events. Methodology and results 1. Porting CityCat from desktop to Cloud Prior to this project, CityCat was written in Delphi and compiled only under Windows. Also, CityCat s Graphical User Interface (GUI) was used for preparation and visualisation of results. Therefore in order to run CityCat on the Cloud the following changes were made: a. The numerical engine was separated from the GUI and input files were used for setting up the model. Also, the variables: water depth, velocity in the x direction and velocity in the y direction were saved at predetermined time step intervals. b. A new version of CityCat was developed and compiled under Linux because if the Windows version was used then the Windows OS would need to be installed at each PC. That would have increased the cost and due to our limited resources it would in fact have reduced the number of runs we could do.
In this project we adopted a high throughput model of computation on the Cloud in which a Condor (http://research.cs.wisc.edu/condor/) cluster of nodes were deployed as a set of virtual machines instances on the Amazon Cloud. Each instance was a standard Ubuntu Linux image with the addition of the Condor deployment configured to use the large scratch space provided with these images. A set of parameter sweep jobs were deployed by modifying the original source code such that each job could be instantiated by passing a single integer number as part of the command line arguments to the program. This caused the correct configuration files to be selected. A simple script was used to wrap each job and would first decompress the files needed for each run before executing the main program and then compressing the results back up before returning the results to a central Condor computer on the Cloud. These files were then staged back to computers within Newcastle University. 2. Standard Datasets needed for CityCat simulations CityCat uses standard datasets for simulations. For the topography the digital terrain model (DTM) is used (see Fig. 1) and the numerical grid is generated automatically using the cell sizes of the DTM. In addition to the terrain data, CityCAT uses the buildings layer from the OS- MasterMap (see Figure2) in order to exclude the buildings footprint from the computational grid. The cells which are removed from the computational grid are characterised as building and depending on different options/properties, the water captured on the roof of each building is either directly distributed to the neighbouring cells or slowly released and distributed to the neighbouring cells if a green roof is introduced. Exclusion of the buildings from the computational domain (Fig.3) improves the ability of the model to Figure 1 - Terrain given by a Digital Terrain Model (DTM) capture realistically the flow patterns in urban areas while the different options for the roof drainage enable assessment of the different adaptation techniques for flood risk reduction. In most of the simulations within this project the buildings were cut-out of the computational domain and the rainfall falling on the roofs was directly distributed to the neighbouring cells. Hydrodynamic simulations are driven by boundary conditions which are sources of water in the model. Usually, these are time dependent functions which describe rainfall over the domain or water entering the domain from an external source. External sources of water can either introduce some volume of water in the domain (for example from a burst pipe) or they can force the water level at the boundary of the Figure 2 -An example of Master Map coverage. Solid representation
Rainfall (mm) domain to follow a certain condition (e.g. river water level or tide). In this project most of the simulations were driven by rainfall as most of urban flooding is of that origin. However, two examples of forced water level boundary condition were also tested to explore the model s ability to simulate such complex flows. 3. Rainfall events A set of storm events, for different return periods and different storm durations, have been created following the standard procedure from the Flood Estimation Handbook (FEH) and a summer profile rainstorm located at Newcastle was used. Modelling different return periods is necessary if we want to assess different level of risk as a rainfall event with a return period of n years has probability of 1/n to happen in any given year. The storms with higher return period are less likely to happen but they are larger in magnitude so should they happen the flooding would be more likely. Different rainfall durations are used because, it is not clear which storm duration would be critical for a given situation. The shorter duration rainfall events usually have larger rainfall intensity but the overall rainfall volume is not very large, whereas rainfall events with larger duration have larger volumes of rainwater. Susceptibility to flooding of any particular area is determined by its topography and other features and depends on the combination of these two factors (rain intensity and total volume of rain) in a complex way. Therefore, being able to model an extensive range of 10 9 8 7 6 5 4 3 2 1 0 Figure 3 -A detail of an example CityCat computational domain without buildings 0 5 10 15 20 25 30 35 40 45 50 55 60 Time (mins) Figure 3 -Rainfall event no. 21 (duration 60min return period 50yrs) rainfall events with different durations, is one of the clear advantages of the use of Cloud Computing in comparison with the standard engineering practice where one storm of a particular duration has to be chosen as a critical one. Altogether, 36 rainfall events were created and used in simulations. The events covered 6 different return periods (2, 10, 20, 50, 100 and 200 years) and 6 different rainfall durations (15 minutes, 30 minutes, 1 hour, 2 hours, 3 hours and 6 hours). The duration
and the return period of each storm event are shown in Table 1 while an example of one rainfall event is given in Figure 4. Table 1. Rainfall events Rainfall event number Return period (years) Duration (min) Rainfall event number Return period (years) Duration (min) Rainfall event (years) Return period number Duration (min) 1 2 15 13 20 15 25 100 15 2 2 30 14 20 30 26 100 30 3 2 60 15 20 60 27 100 60 4 2 120 16 20 120 28 100 120 5 2 180 17 20 180 29 100 180 6 2 360 18 20 360 30 100 360 7 10 15 19 50 15 31 200 15 8 10 30 20 50 30 32 200 30 9 10 60 21 50 60 33 200 60 10 10 120 22 50 120 34 200 120 11 10 180 23 50 180 35 200 180 12 10 360 24 50 360 36 200 360 4. Application of CityCat to different domains Within this project CityCat has been applied to large areas and for extensive duration of simulations to test its limits and to be able to ascertain the benefits of the use of Cloud Computing. Three different domains, ranging in size from 4km 2 to 1100km 2 were used, with even the smallest of them being much larger than the domains used in current engineering practice. Additionally, for one of the domains, 4 different grid sizes were used which resulted in very different model sizes. Most of the models were then run in parallel for multiple rainfall events. All these simulations required very different memory, CPU effort and total run time and benefitted in different ways from Cloud Computing. In Table 2, a summary of all the runs performed on the Cloud is given and more detailed analysis for each simulation is presented below. Table2. Summary of all the simulations Domain Area Cell size Number of cells Boundary conditions 1 Newcastle 4km 2 2m 1.000.000 Rainfall events city centre 1-36 2 Newcastle 4km 2 1m 4.000.000 Rainfall events city centre 1-36 3 Newcastle 4km 2 0.5m 16.000.000 Rainfall events city centre 1-36 4 Newcastle 4km 2 2m 1.000.000 Hypothetical city centre flood wave 5 Whole Newcastle 120km 2 4m 7.500.000 Rainfall events City Council area 1-36 6 Thames estuary ~1100km 2 15m ~5.000.000 Tidal surge water level Event duration Number of runs see rainfall 36 events, Table 1 see rainfall 36 events, Table 1 see rainfall 36 events, Table 1 2 hrs 1 see rainfall events, Table 1 33 hrs and 21 hrs 36 2
All these simulations, and some of them with multiple runs, produced a huge amount of results which need to be converted into maps and videos in order to become meaningful to end users. The development of flood risk maps requires the results of different rainfall events to be compared in order to identify the most critical ones. Within this pilot project, there was not enough time to carry out all this analysis due to the huge amount of results that was generated. Therefore, within this project, just some of the results were turned into maps and analysed, however, the rest of the results will be used in further studies. This opened up the question if Cloud Computing could be used for the analysis and presentation of results in flood risk studies and this will be explored in the future. Domain 1 Newcastle city centre simulations 1 to 4 in Table2 A domain of 4km2 including the city centre of Newcastle was chosen as the main domain area for this project. When this domain is compared with the other domains used in this project it might seem small but it is in reality much bigger that the areas usually modelled in current engineering practice, especially when the grid size is taken into account. A large consultancy project would typically model a domain of 1 km 2 with a grid of 5m which gives 40.000 cells while our simulations 1, 2 and 3 have 1.000.000, 4.000.000 and 16.000.000 cells respectively. These simulations could not have been done with most Figure 4 - Domain1 - Newcastle City Centrebuildings layer from the MasterMap software used for flood simulation as there are usually software limitations to the number of cells in the domain. However, CityCat does not have any limits on the size of the domain but we still could not have run it on our university facilities due to the memory limitations of our PCs. Use of PCs with large memory via Cloud Computing (see table 3) enabled us to run these simulations. Additionally, using Cloud Computing we were able to run all 36 rainfall events (see table1) in parallel either on one or more PC instances. Table 3 Instances of PCs used in simulations 1,2 and 3 Simula tion Number of cells Cell size Required memory PC Instances used 1 1.000.000 2m 3GB Standard instances 7.5GB 2 2 4.000.000 1m 11GB High-Memory Instances 68.4 GB 5 3 16.000.000 0.5m 40GB High-Memory Instances 68.4 GB 1 Number of jobs submitted per instance The motivation to use different cell sizes for the same domain was not only to test the limits of the software and the use of Cloud but also to investigate how different cell sizes influence the results. See Figure 6.
Figure 5 - Influence of the grid size on results - up left- cell size of 1m and up right- cell size of 2m There are numerical reasons why one could expect different results when using different cell sizes but in the case of CityCat the main difference comes from the way the buildings are cut-out of the domain. Larger cell sizes could produce unrealistic situations in the models where buildings that are not connected in reality would appear connected in the model and that could completely distort the flow pattern. From the previous investigations, it has been concluded that the grid sizes of 2m or smaller, are not prone to large errors of that type. In this study even smaller cell sizes were used to check if they bring any further improvements. The initial conclusion is that the smaller cell sizes capture better the topography and improve the model results but unless some small passageways are completely cut-off when the buildings are cut-out, the overall result a) time =15 min c) time= 1 hour 20 min a) time = 15 min d) time = 2 hours Figure 7. Flow in Newcastle City Centre induced by a fictitious constant water level at the north boundary of the domain. does not seem to be significantly improved with the use of smaller cell size grid. However, if one is interested in changes of flood risk due to man-made interventions like rising of
pavements or building of walls, then models based on very small grid sizes (i.e. 0.5m) are beneficial as they give a more precise picture of the change in flow conditions. In simulation 4 the same domain as in simulation 1 was used with different boundary condition in order to test the capability of the model to handle extreme flood conditions from other sources. This was achieved by introducing a hypothetical flood wave entering the north boundary of the domain. (See Figure 7.) The results of this model are completely fictitious but they show that the model is able to capture the flow patterns and calculate the water depth and the velocities in extreme flow conditions due to external sources (e.g. fluvial or tidal flooding). Domain 2 Whole Newcastle City Council area simulation 5 in Table2 Figure8 - Flood map for the whole area of Newcastle City Council for the rain event 27 at 60min The second domain is the whole area of Newcastle City Council which covers approximately 120km 2. We have chosen this area to demonstrate that by using Cloud Computing and CityCat local authorities would be able to model such big areas for Surface Water Management Plans. In a way it is a move towards producing flood models at a scale larger than a city and moving towards a region. Newcastle City council area is a good example of this as it has urban, suburban and rural areas. See figure 8. Due to the fact that the area of this domain is large, a 4m cell size was used which resulted in 7.5million cells. Any further reduction in the cell size was considered unfeasible at that stage of the project. On the enlarged detail in figure 9, it can be seen that cutting-out of buildings, formed continuous barriers within the domain which have a Figure 9 Detail from -suburban areas significant influence on the flow patterns. However, these are mainly dettached and semi detached dwellings and although a lot of them have been extended and almost form rows of terraced houses, there are still a lot of gaps between them which are not captured well
by the model. This is a consequence of the 4m grid and this points out to our previous conclusion that the 2m grid provides much more realistic flow patterns when the buildings are cut-out. Domain 3 Thames estuary simulation 6 in Table2 The domain of the final simulation was the Thames estuary with the area of ~1100km 2. Due to the domain being so large the grid size was chosen to be 15m and buildings were not cutout. CityCat does not impose limitations on the size of the domain, however, larger or higher resolution domains were not used due to the limited duration of the project. The propagation of the tidal surge upstream along the Thames was modelled to see if CityCat could be used in flood risk studies for the Thames estuary including London. The boundary condition for this simulation was the tidal water level given as a function of time and placed at the east boundary of the domain. Due to the fact that we did not have the bathymetry of the river we decided not to include any discharge coming down the river. Two different runs were performed using two different tidal boundary conditions. Time = 27 hours Time = 33 hours Figure 10 Water depth in the Thames estuary due to the extreme storm surge (1000 years return period) The obtained results look encouraging and furthermore it was demonstrated that CityCat running on the Cloud can handle such huge domains for events of prolonged duration and could be used in the future for flood risk studies of large areas such as the Thames estuary.
Conclusions Although CityCat does not impose any intrinsic limitations on the size of the modelled domain, prior to this project, hardware limitations prevented modelling of very large domains. The use of Cloud Computing enabled modelling of flooding in cities on a much larger scale; with much higher resolution and longer duration of events than it is current available in engineering practice. Domains of up to 1100km 2 and 16.000.000 computational cells were modelled, and it has been demonstrated that even much larger ones can be used. Another major advantage of using Cloud computing was the ability to run models of the same domain, with different rainfall inputs, all at the same time. For the majority of the simulations 36 different rainfall events were run in parallel. This number can be much larger if a flood risk analysis under climate change is undertaken. This would have been possible, however, within this pilot project there was no scope to develop an efficient way to analyse and visualize the already vast amount of the results which have been generated with 36 events. Based on this experience, the automatic visualisation and analysis of the results using Cloud Computing has been identified as a potential area for further exploration. In this project it was demonstrated that the use of CityCat on the Cloud can perform high resolution large scale modelling of flooding that can be used for the assessment of city-scale flood risk under climate change using only readily available data and producing results much faster than all other currently available models and computing methods in general engineering practice.