Ricardo-AEA. AirDART. Air quality Data Analysis & Retrieval Tool. DMUG, 9 th December Andrea Fraser.

Transcription

1 Ricardo-AEA AirDART Air quality Data Analysis & Retrieval Tool DMUG, 9 th December 2013 Andrea Fraser

2 2 Air quality Data Analysis Retrieval Tool - AirDART no this isn t a new weapon (well it is but not in that sense!)... AirDART (Air quality Data Analysis Retrieval Tool) is an internet tool, built by us for the Environment Agency. It allows users to access terabytes of CMAQ air quality modelling data in a simple way, allowing powerful visualisation of huge data sets. AirDART has generated genuine excitement among the CMAQ modellers that have seen it so far its beauty lies in its relative simplicity.

3 3 AirDART Air quality Data Analysis & Retrieval Tool Background The aim of the Environment Agency project was to develop a data retrieval tool with simple visualisation to allow air quality professionals access to CMAQ model data created in previous contracts. To allow data to be incorporated into other models and analytical tools used by the Agency. Provide modelling evidence to inform and support strategic decisions. Team: Ricardo-AEA: Andrea Fraser, Daniel Brookes, Trevor Davies, Neil Morris, Alistair Dorman-Smith University of Hertfordshire: Charles Chemel, Xavier Francis, Ranjeet Sokhi, Xin Kong Environment Agency: Bernard Fisher, Roger Timmis

4 Why develop a new tool? Existing tools are aimed at: experts working with the models model evaluation and focus on results at monitoring sites Who is it for? Aimed at Environment Agency staff and other air quality researchers including the modelling community CMAQ modellers Where possible give consideration to developing a tool that can be used by other modelling groups within the wider modelling community What are the problems? the amount of data data complexity speed of access 4 It will not include all CMAQ model data and model experts will continue to provide data and interpretation for more complex questions.

5 5 Original Approach Similar to other tools available extract data for selected locations, store in a database and use as required. BUT At the beginning of the project it became apparent that to be of the greatest value the tool required flexibility and that it should: allow selection of data for any latitude and longitude coordinates for time series or grids only allow choice of species and species combinations which are comprehensible ensure all data selected are accompanied by metadata and have an option to download the data as a file suitable for import into Excel and other environmental models provide simple graphs and maps for basic data quality checks, but not attempt development of a data visualization or model evaluation tool where possible use R for development The research community may prefer access to more detail e.g. all species and 3D data. This was considered to be a more specialised requirement and outside the scope of this project. Where possible give consideration to utilisation of the tool within the wider modelling community

6 6 DATA for AirDART CMAQ Model data from Environment Agency projects: CREMO PM 2.5 CMAQ model data: Gas conc. 3D hourly 74 species PM conc. 3D hourly 50 species Deposition 2D hourly 136 species Emissions 3D hourly 30 species Meteorology 2/3D hourly 80 species Terabytes of data What are the problems? the amount of data data complexity speed of access Date Available in AirDART AirDART raw data: Gas conc. 2D hourly 23 species PM conc. 2D hourly 16 species Deposition 2D hourly 30 species Emissions 2D hourly 17 species Meteorology 2D hourly 11 species Gigabytes of data AirDART summary data: Gas conc. PM conc. Deposition Emissions Meteorology 2D all species Daily/Monthly/Yearly Max/Average 2D all species Daily/Monthly/Yearly Sum selected parameters Megabytes of data

7 7 DATA for AirDART CMAQ Model data from Environment Agency projects: CREMO PM 2.5 CMAQ model data: Gas conc. 3D hourly 74 species PM conc. 3D hourly 50 species Deposition 2D hourly 136 species Emissions 3D hourly 30 species Meteorology 2/3D hourly 80 species Terabytes of data Metadata: Model scenario metadata Details of the geographic grid Species descriptions Data Preparation Stage 1: Create new 2D files of hourly data for selected species and combinations of species Reduces the number of species to those of interest: Removing the minor species related to intermediate in the chemical mechanism. Combining the wide range of PM and VOC species into meaningful groups. Converting units use standard file names ready for AirDART Data Preparation Stage 2: Create summarised data for daily, monthly and annual files of pre calculated metrics - maximum, average and total sum. Results in: Smaller files for faster access Reduces data to common summaries. Date Available in AirDART AirDART raw data: Gas conc. 2D hourly 23 species PM conc. 2D hourly 16 species Deposition 2D hourly 30 species Emissions 2D hourly 17 species Meteorology 2D hourly 11 species Gigabytes of data AirDART summary data: Gas conc. PM conc. Deposition Emissions Meteorology 2D all species Daily/Monthly/Yearly Max/Average 2D all species Daily/Monthly/Yearly Sum selected parameters Megabytes of data

8 8 CMAQ Data TRACEABLE and TRANSPARENT AirDART output

9 9 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

10 10 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

11 11 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

12 12 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

13 13 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

14 14 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

15 15 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

16 16 Selection options are on the left hand side and a series of tabs in the body of the page for different output options: Summary - A summary of the selected data, including map markers to show the area selected. Scenario Description - A description of the model used for the scenario Data Output a table of the selected data which can be downloaded as a file. For a time series data is for the selected location and includes wind speed and direction. For a grid it is a single time step. Map Output two simple maps showing the spatial distribution, a map of the whole UK and a map of the selected area. Plot Output the graphs vary a little depending on the selection, for grids a frequency for time series a bar chart with a stacked bar chart for PM components.

17 17

18 18

19 19 How we did it EA-AirDART Includes the Environment Agency CMAQ data EA-AirDART Web access to AirDART including EA CMAQ data. AirDART AirDART R-Shiny Web access to summarised CMAQ data R-Shiny - Shared Server Environment Interactive data selection reflecting the model data Type of variable (gas, PM, deposition, emissions or meteorology) Time selection Variable Location (latitude, longitude) R- function Data extract function EXTRACT - R function Data Selection Extract data from the netcdf files using parameters passed from the web interface. Creates data tables, graphs and maps Can operate on model (netcdf) files independent of AirDART Underling Linux computing environment which includes: netcdf file reading programs ioapi utility programs Access to R in a shared server environment (shiny) Selected R packages working with netcdf and ioapi files GDAL-Geospatial Data Abstraction Library

20 20 How we did it? EA-AirDART Includes the Environment Agency CMAQ data AirDART R-Shiny Web access to summarised CMAQ data R- function Data extract function What are the problems? the amount of data data complexity speed of access Example: Less than 1 minutes to select and download a year of data for Harwell into Excel It took less than 10 min to create the examples used in this presentation. R works faster the more you access the same file Using R-Shines reduces the computing overheads everything below the web page header is written in R. Including the selection panel and output tabs.

21 21 What are the benefit? Data mining with AirDART allows even experienced modeller to discover relationships in the data that would be more difficult to notice with more traditional tools. Exploit the one atmosphere features of CMAQ by making data available to a wider group of experts. Makes it possible to use the data in other models and analytical tools openair, excel, R, ArcGIS. Data can be selected based on latitude and longitude, analysis is not restricted to selected locations e.g. monitoring sites. A modular tool that has a range of benefits for the wider modelling community. R function - Can be used with other CMAQ or netcdf gridded model data. AirDART - Can be used with other sets of pre prepared CMAQ data. EA-AirDART Environment Agency can access there own data.

22 22 Using AirDART to look at deposition of oxidised sulphur.

23 23 WET DEPOSITION of OXIDISED SULPHUR Annual deposition of Oxidised Sulphur

24 24 DRY and WET DEPOSITION of OXIDISED SULPHUR Annual deposition

25 25 DRY and WET DEPOSITION of OXIDISED SULPHUR Annual deposition

26 26 DRY and WET DEPOSITION of OXIDISED SULPHUR Annual average The exercise can quickly be repeated for monthly deposition, gas and PM species. You can explore the data it identify the relationships you wish to investigate further.

27 27 AirDART - Air quality Data Analysis & Retrieval Tool AirDART is an internet based tool that allows data created during air quality modelling projects to be retrieved from an archive without the need for a complex set of computer programs. It provides graphs and maps to allow data quality checking and exploration. Why do we need AirDART? To increase the value of the investment that has been made in generating CMAQ air quality model data by making it easier to access. Maximising the potential value of the data. Enable data exploration the air quality model CMAQ adopts a one atmosphere perspective. Only a fraction of the data has been explored. AirDART is NOT A model evaluation tool but the data can be used for evaluation A full data analysis tool but it could be enhanced

28 28 Benefits Data mining with AirDART allows even experienced modeller to discover relationships in the data that would be more difficult to notice with more traditional tools. R and R-shiney have been used to demonstrate an efficient method of using the internet to share model data with colleagues that don t have access to the specialised model software. A modular tool that has a range of benefits for the wider modelling community. R function - Can be used with other CMAQ or netcdf gridded model data. AirDART - Can be used with other sets of pre prepared CMAQ data. EA-AirDART Environment Agency can access there own data. AirDART has been built of CMAQ but this type of application can be built over many different types of model data.

29 Thank you Any Questions? Andrea Fraser Ricardo-AEA Ltd The Gemini Building Fermi Avenue Harwell, Didcot, OX11 0QR T: E: W: +44(0)

30 30 AirDART is a prototype What would we have liked to do, what could have been done better? The prototype was developed with an early version of R-Shiny (v0.4.0) and features are continuously improving (now up to v0.7.0) The EA requested basic visualisation, the strength of R and R-Shiny is data visualisation A lot can be learned with 3 simple plots Data can easily be interfaced with R for additional analysis

31 31 AirDART Air quality Data Analysis & Retrieval Tool PM 2.5 at Harwell Demonstration of using the AirDART data in other R visualisation packages Wind speed and direction for Harwell Openair Wind Rose Wind Contours function (centred on North)

32 32 AirDART Air quality Data Analysis & Retrieval Tool PM 2.5 at Harwell OPENAIR polar plot All emissions Excluding the industrial sources of emissions

33 33 Using AirDART to look at the monthly average PM 2.5 components Monthly average VOC emissions in the Brecon Beacons Monthly average PM 2.5 components