386 Multidimensional Marine Environmental Data Conversion and Visualization Using Python and GIS Manuel RAINER Abstract The Tropical Marine Science Institute (TMSI) at the National University of Singapore (NUS) generates huge amounts of predicted tidal currents data every day. The data are used for a hydrodynamics forecast system of the Singapore Strait. This paper deals with the access, extraction, and visualization of those spatial data, stored in netcdf (network Common Data Format) file format. The expected software solution had to fulfil several requirements. Hydrodynamics time series must to be selected and converted into the Shapefile format. Geoprocessing operations on the Shapefiles have to perform the currents visualization and thematic map layer creation. The exported map images should be animated along time. The final result of the three aforementioned procedures is an animated hydrodynamics map of the Singapore Strait which shows the predicted tidal current changes within a certain time interval. For these given requirements, the developed software framework consists of two main tools. For the conversion process a stand-alone and multi-platform Python tool has been implemented. The tool has integrated and enhanced existing open-source modules and packages. The Shapefile post-processing and visualization task is done by the created HydroViz ModelBuilder geoprocessing tool for ArcGIS. The hydrodynamics visualization framework as well as all intermediate output data and the final map can be used and adapted by TMSI researchers for interactive visualization and analysis of marine environment. 1 Introduction The observation and modeling of coastal and oceanographic phenomena is an important topic since the emergence of Geographic Information Systems (GIS) and the increasing computer power in recent years (HAIDVOGEL et al. 2000). Especially the physical processes occurring in the oceans pose a significant research area for the use in different areas, e.g. water pollution (TMSI), coastal management, marine renewable energy (MUELLER & WALLACE 2008) or particle transportation (KULAWIAK et al. 2010 and TMSI). Among those phenomena is the field of water hydrodynamics. Hydrodynamic models have been used for decades. The first models, which were created in lavish procedures, have been continuously substituted by more detailed and complex automatically generated models, based on routinely collected sensors data (HAIDVOGEL et al. 2000). Nowadays, hydrodynamic models are the basis for sophisticated tidal current now-/forecast systems (TKALICH et al. 2004). The process from a sensor measurement, computation and file compression to single record access and data visualization is complex and comprehensive. This work deals with files in a self-describing data format (REW 2006), which contain the computed parameter values for predicted currents in a certain observation area, defined by coordinate Jekel, T., Car, A., Strobl, J. & Griesebner, G. (Eds.) (2012): GI_Forum 2012: Geovizualisation, Society and Learning. Herbert Wichmann Verlag, VDE VERLAG GMBH, Berlin/Offenbach. ISBN 978-3-87907-521-8.
Multidimensional Marine Environmental Data Conversion 387 vectors and geographic coordinates. The most relevant part of the research presented in this work is the extraction, conversion and visualization of the multidimensional oceanographic data using customized tools and proprietary GIS. 1.1 Problem Definition At the Tropical Marine Science Institute (TMSI) huge amounts of data are generated, including the measures for the hydrodynamic models of the Singapore Strait. These daily predicted hydrodynamics data are compressed into one large (~700 Megabytes) netcdf (network Common Data Format) file. The file stores 3D point coordinates for different time slices, which contain depth averaged water current speed and the distance from still water level to sea bed. The data are defined as 4D because of their time and space components. A netcdf file (REW 2006) can only be accessed through special software programs to extract and visualize the datasets. The first task was thus to read the selected data from the netcdf files. The extracted content has to be saved in a geospatial file format. These geospatial files can subsequently be used for many kinds of geoprocessing, e.g. editing, manipulating and interactive spatial analysis together with other environmental thematic layers. The final product should be an animated map, which shows tidal current changes within a certain time period. An additional requirement is to find methods for optimizing the processing performance as well as to handle processing of multiple files. In order to accomplish this, methods to handle different tasks involved in the data extraction, conversion, and visualization process need to be identified. Throughout the entire process, the following questions need to be answered: How to handle the self-describing netcdf data structure? How to define the selection parameters and their values? What kind of system architecture is suitable? How to define the user interaction that plausible for all researchers? Another objective is to highlight application areas of hydrodynamic models of the Singapore Strait. 1.2 Groundwork TKALICH et al. (2002 & 2004) from the Physical Oceanography Research Laboratory (PORL) at TMSI have developed the Tropical Marine Hydrodynamic Model (TMH) and have subsequently generated the first forecast model for the Singapore Strait. This coastal hydrodynamics forecast system, a local and internet-based now-/forecast system, does a similar job compared to the conversion and visualization procedures proposed in this paper. The core of the system is the Model INTegrator (MINT), which manipulates 4D data and allows access to hydrodynamics and water quality modules. A Java-based interface is responsible for post- and pre-processing of model data and parameters. The system is used for online-visualization and has model-independent capabilities through open-source standards. Information is stored and shared using the netcdf data format. The specific research proposed in this paper investigates the methods and techniques required to visualize the time series data stored in netcdf format interactively. The netcdf files are an intermediate output of the existing system mentioned above and at the same time basic data for further research purposes. The current forecast system does not have any functionality which converts the netcdf data into Shapefiles. However, this is necessary, because the entire system does not allow an easy access for researchers, who have to work on hydrodynamic models, especially for environmental analysis. The developed tools/ models/scripts can be used and adapted by PORL researchers for interactive visualization and analysis of marine environment.
388 M. Rainer 1.3 Methodology The two main technologies used in this research are the programming language Python and the GIS software ArcGIS. To read and extract data from netcdf files, a Python tool was designed and implemented. It converts the selected hydrodynamics data into a geospatial data format. Because of the compatibility with ArcGIS and other open-source GIS, Shapefile (ESRI 1998) is considered as output file format. The conversion tool is a standalone program which consists of Python code and additional open-source Python modules and packages for netcdf data handling, array operations, Shapefile creation and GUI (Graphical User Interface) building. The programming task of extracting time slices with correct values and coordinates from the self-describing data structure is solved by using Python dictionaries which can be accessed by each other, i.e. the values from one dictionary are used to access records from another dictionary. The created Shapefiles run through an ArcGIS ModelBuilder (ESRI 2004) script for geoprocessing operations on feature layers to get the necessary visualization results. Some geoprocessing scripts, which are added to the model workflow, are developed with ArcPy (ArcGIS Python module). The animated hydrodynamics map of the Singapore Strait will be an animated GIF (Graphics Interchange Format) showing a sequence of map pictures containing tidal currents for a certain time. This animation can be done by using free or proprietary graphic editors. 1.4 Expected Results The developed software framework will consist of two independent tools: 1. The NetCDF to Shapefile Conversion Tool 2. The HydroViz ModelBuilder Geoprocessing Tool in ArcGIS The framework is used to generate hydrodynamics maps from netcdf files, which show the tidal currents predictions of the Singapore Strait. The tool for netcdf to Shapefile conversion is a stand-alone and cross-platform Python tool including open-source modules and packages. Its GUI allows user-defined input, parameter selection and output as well as file batch processing. The HydroViz ModelBuilder tool works on the output data of the previous tool and consists of integrated geoprocessing scripts and creates hydrodynamics maps with thematic layers. The feature symbology indicates current magnitude and direction. HydroViz provides user-defined projection, clipping, point grid scaling, and magnitude raster generation. The animated hydrodynamics map is realized by creating an animated GIF from the map images using the Adobe Fireworks graphics editor. 2 About Modelling Hydrodynamics and the Singapore Strait A description of the developed hydrodynamics model, information about the domain of interest, and facts about current projects and application areas are discussed next. 2.1 Tropical Marine Hydrodynamics Model The modeling of hydrodynamics of the Singapore Strait is developed by the Tropical Marine Hydrodynamics Model (TMH) of the TMSI s Physical Oceanographic Research Lab (PORL). TMH is a free 3D surface sigma-coordinate primitive equation ocean model
Multidimensional Marine Environmental Data Conversion 389 developed by TKALICH et al. (2002) based on the open-source Princeton Ocean Model (POM). PORL handles regional projects related to hydrodynamics now-/forecast as well as water quality modeling in the Singapore Strait. The predicted current tidal data for the area of the Singapore Strait is stored into netcdf data files. These files, containing predicted hydrodynamics for one day a week in advance, are routinely created using a tidal prediction software program. An entire file includes 49 time slices, which represent the time period of 24 hours starting from 00:00 am in steps of half an hour intervals. Each slice is a record of observation values for each half an hour interval. The file size of a compressed TMH netcdf file is around 700 Megabytes. An uncompressed TMH file amounts to 1.1 Gigabytes. 2.2 Singapore Strait The Singapore Strait is among the busiest shipping routes in the world. Thousands of ships and cargo are passing through this strategic important strait between the Indian Ocean and South China Sea every day. During the last 50 years an extensive network of port and harbor facilities has been built along the coasts of the Singapore Strait, followed by intensive land reclamation (TKALICH et al. 2002). Coastal management and marine industries need predictions of changes in tidal current patterns, indicated by the TMH, and the resulting effects on salinity and sediment transport along the coast (TMSI). 2.3 Research Projects and Application Areas The study of predictions based on hydrodynamic models is an essential part of the TMSI project Marine Environmental Monitoring, Impact Assessment and Enhancement for Singapore Waters. The models calculate a 3D currents prediction to assist in coastal management, environmental monitoring and emergency response (TMSI). Other current TSMI research and development projects on the hydrodynamics of the Singapore Strait include: Development of operational hydrodynamic models Daily tidal elevation and stream modeling and prediction On-demand particle tracking simulation Research and development on basic model mechanics and numerical schemes 3 Implementation and Results This section provides both an overview as well as detailed descriptions of the developed hydrodynamics visualization framework. Results are listed after the discussion of each tool. 3.1 Hydrodynamics Visualization Framework Figure 1 shows the workflow of the entire framework. The main components are explained in the next sections. The result is an animated hydrodynamics map of the Singapore Strait.
390 M. Rainer Fig. 1: Workflow of the developed Hydrodynamics Visualization Framework. It includes the procedure from the netcdf file input, Shapefile conversion (1), ArcGIS geoprocessing (2) and map image animation (2) to the final animated hydro map. 3.2 NetCDF to Shapefile Conversion As has been already mentioned, the tidal current observation data is packed into netcdf (network Common Data Format) files, a self-describing scientific data format (REW 2006). The file content can only be extracted and visualized with special software tools. Most of the researchers at PORL are familiar with ArcGIS, therefore the netcdf files needed to be converted into the Shapefile format (ESRI 1998). A fast and simple stand-alone conversion tool was thus proposed, which provides a user-definable file and time slice selection. The developed conversion tool consists of pure Python code together with the NumPy/SciPy packages and Tkinter/Ttk as well as the NetCDF4-python and GDAL/OGR modules. All these packages and modules are open-source. Wingware and PyWin were used as IDE (Integrated Development Environment). The simple user interface proceeds with the conversion after setting a few input and output parameters. Multiple netcdf data files can be loaded for conversion from any folder. Then, the list boxes for date and time selection are automatically filled with values from the input files. The user can select dates and several times slices to start the conversion, with the result being put into a specified output folder. The following tasks are accomplished during the conversion process:
Multidimensional Marine Environmental Data Conversion 391 1) NetCDF data access and extraction Depending on the selected date and time, the corresponding netcdf file records, including all dimensions, variables and global attributes, are accessed and extracted into new datasets. For a fast array operation the values must be directly (without any loops or if/else clauses) transferred from the dataset. The arrays get flattened, transposed, stacked and merged so that they have the right dimension and length. In the end, three dictionaries are generated from the data arrays. The first dictionary contains the time slice index (as key) as well as the corresponding date and time. The second dictionary also contains the time slice index (as key), the water surface elevation from still water level, x/y coordinates and the depth average water current speed values along the x- and y-axes (u/v). The third dictionary contains the x/y coordinates in the form of vectors (as key), the geographic coordinates (long/lat), and the distance from still water level to seabed. 2) Water current magnitude and direction calculation Water current magnitude and direction values are added to each point as an additional column in the attribute table. The magnitude and direction can be calculated with the depth average water current speed values along the x- and y-axes in meters per second (u/v vectors). The equations used for this process are from NOAA. The current magnitude and direction are essential for the symbology assignment in the geoprocessing task. 3) Shapefile creation OGR, as part of the Python GDAL (Geospatial Data Abstraction Library) module, can create Shapefiles without using ArcGIS. That makes the conversion tool absolutely independent from proprietary software products. The data source (Shapefile), data layer, and attribute table are declared first. Then the features within the layer are defined by iterating over the Python dictionaries and selecting the necessary values for each point feature. The special trick is to use the x and y coordinates from a record in the data content dictionary as key for the corresponding longitude/latitude coordinates and water depth in the coordinate s dictionary (as can be seen in figure 2). The same methodology is used to access the right date and time. The key from the data content dictionary is also used as key for the date time dictionary. That is the reason why netcdf is called a self-describing data format. All information is packed into one file and must be extracted and joined together in the right way. After all values are read and the feature points are created, the Shapefile is saved into the previously defined output directory. As already mentioned, this tool allows multiple netcdf file access, time slice selection and export into one or more Shapefiles. Moreover, it can be used to view and compare the data structure of different netcdf files (see figure 3). After the conversion, Shapefiles which contain at least one time slice record layer are created. The layer consists of point features with current and bathymetrical attributes for the area of the Singapore Strait. The Shapefiles can be used and edited with any GIS software. The conversion time for one time slice, which usually has more than 1 million observations, is approximately 3 minutes.
392 M. Rainer Fig. 2: Dictionary interaction and value access for setting the Shapefile feature attributes Fig. 3: Screenshot of the NetCDF to Shapefile Conversion Tool 3.3 HydroViz Tool for Geoprocessing & Visualization The post-processing and visualization task includes all geoprocessing operations on the created Shapefiles using ArcGIS. The HydroViz tool, developed with ModelBuilder (ESRI 2004), creates hydrodynamic maps. Most of the geoprocessing tools are already available in ArcGIS others have been developed and modified with ArcPy (ArcGIS Python module). The entire HydroViz model workflow is represented in figure 4.
Multidimensional Marine Environmental Data Conversion 393 Fig. 4: Model diagram of the HydroViz ModelBuilder tool Two output layers are generated. The first output layer is a raster layer which visualizes the current magnitude values. The second is a hydrodynamics point layer. HydroViz consists of integrated geoprocessing models which perform the following operations: Check and define the coordinate system Create a magnitude raster, based on the current magnitude values Point resampling for representation in different scales Clipping and point symbology assignment The model workflow can process one input file or batch a list of files. After running the entire model, all output layers are displayed in ArcMap. Layout elements, such as legend, north arrow or labels must be inserted and modified manually. A hydrodynamics map image shows the current magnitude and direction using arrow markers which have different size and angle, depending on the current attribute values. The background raster layer highlights the sea surface in respect to the tidal current magnitude. Each layer showing the predicted hydrodynamics for a certain time can be exported as map image (see figure 5). 3.4 Animated Hydrodynamics Map Creation After exporting each hydrodynamic map as image, the files are merged into one animation file, saved as GIF (Graphics Interchange Format). This can be done by most of the proprietary software packages (e.g. Adobe Fireworks, Photoshop) and also by some free graphics editors (e.g. GIMP). The final result, an animated hydrodynamics map of the Singapore Strait shows the tidal current changes in a specified time interval (see figure 5).
394 M. Rainer Fig. 5: Maps showing hydrodynamic changes in steps of four hours from 10 am to 6 pm 4 Conclusion and Outlook The results presented in this work meet the requirements and objectives as put forward in the problem definitions and research questions (see section 1). The stand-alone tool can be used as part of the visualization framework or can be used independently. The HydroViz Modelbuilder tool automatically creates hydrodynamic maps from Shapefiles and can easily be modified. For map image animation a graphics editor can be used. All in all, the creation of an animated hydrodynamics map from the initial netcdf (network Common Data Format) file is accomplished with a few mouse clicks and in less than ten minutes. To guarantee domain specific output, all supported values and parameters are user-definable. The models, scripts and tools developed in this work provide the basis for further implementation, especially for coastal marine information platforms, forecast systems, and WebGIS. Moreover, a server-sided integration of the conversion tool will be realized. Acknowledgements: Many thanks to Dr. Durairaju Kumaran Raju and his team at the Tropical Marine Science Institute, National University of Singapore; and to Dr. Victor Garcia-Barrios, Dr. Michael Leitner and Dr. Gernot Paulus from the Department of Geoinformation & Environmental Technologies, Carinthia University of Applied Sciences.
Multidimensional Marine Environmental Data Conversion 395 References ESRI ESRI Shapefile Technical Description (1998), ESRI White Paper. Environmental Systems Research Institute, Inc., 1-34. ESRI ArcGIS 9 Geoprocessing in ArcGIS (2004), ESRI White Paper. Environmental Systems Research Institute, Inc. HAIDVOGEL, D. B., BLANTON, J., KINDLE, J. C. & LYNCH, D. R. (2000), Coastal Ocean Modeling: Processes and Real-Time Systems. Oceanography, 13 (Special Issue), 35-46. KULAWIAK, M., PROSPATHOPOULOS, A., PERIVOLIOTIS, L., ŁUBA, M., KIOROGLOU, S. & STEPNOWSKI, A. (2010), Interactive visualization of marine pollution monitoring and forecasting data via a Web-based GIS. Computers & Geosciences, 36, 1069-1080. MUELLER, M. & WALLACE, R. (2008), Enabling science and technology for marine renewable energy. Energy Policy, 36, 4376-4382. NOAA, Transport Winds, National Oceanographic and Atmospheric Association. http://www.wrh.noaa.gov/sew/fire/olm/transport.htm (last accessed: 8 May 2011). REW, R., HARTNETT, E. & CARON, J. (2006), NETCDF-4: Software Implementing an Enhanced Data Model for the Geoscience. Proceedings of the 22nd AMS Conference on Interactive Information and Processing Systems for Meteorology, American Meteorological Society, Section 6.6. TKALICH, P., PANG, W. C., & SUNDARAMBAL, P. (2002), Hydrodynamics and Eutrophication Modelling for Singapore Straits. Proceeding to the Seventh Workshop on Ocean Models for the APEC Region (WOM-7), 5-1 5-9. TKALICH, P., ZEMSKY, P., WEI CHONG, P. & ENG SOON, C. (2004), Coastal Hydrodynamics Forecast System. 6th International Conference on Hydroinformatics, World Scientific Publishing Company, 1-8. TMSI, Singapore Straits HydroDynamics & Water Quality Model, Tropical Marine Science Institute, PORL, National University of Singapore. http://www.porl.nus.edu.sg/main/research/singapore-str (last accessed: 10 April 2011). TMSI, Hydrodynamics of Singapore Strait, Tropical Marine Science Institute, Physical Oceanography Research Laboratory, National University of Singapore. http://www.porl.nus.edu.sg/main/research/hydro-model (last accessed: 10 April 2011). Software and Package Links ARCGIS, http://www.arcgis.com (last accessed: 20 March 2011) GDAL, http://www.gdal.org (last accessed: 1 April 2011) NETCDF4-PYTHON, http://code.google.com/p/netcdf4-python/ (last accessed: 1 April 2011) NUMPY, http://numpy.scipy.org (last accessed: 27 March 2011) PYTHON, http://www.python.org (last accessed: 26 March 2011) PYWIN, http://sourceforge.net/projects/pywin32/ (last accessed: 7 May 2011) TKINTER, http://www.pythonware.com/library/tkinter/ (last accessed: 1 April 2011) WINGWARE, http://www.wingware.com (last accessed: 7 May 2011)