Evaluation of GEMS Regional Air Quality Forecasts

Transcription

1 Evaluation of GEMS Regional Air Quality Forecasts UK Met Office P. Agnew, M.P. Mittermaier INERIS C. Honore CNRS I. Coll, R. Vautard Meteo-France V.-H. Peuch University of Cologne H. Elbern February 2007

2 Abstract The GEMS Regional Air Quality (RAQ) sub-project will produce limited area forecasts of chemical species concentration and air quality, based on initial and boundary conditions derived from the GEMS aerosol and global reactive gas sub-projects. In order to assess the skill of the RAQ models a verification procedure is required to compare forecast values with observations. This report describes a proposed procedure for evaluating RAQ forecasts. Basic statistical metrics for assessing (forecast-observation) bias, error and correlation are described and the use of Taylor diagrams to produce a convenient visualisation of the models performance is recommended. The ability of RAQ models to forecast exceedance events is a key performance metric and a skill score based on the odds ratio is proposed for this case. This measure is not dependent on the event base rate, is not sensitive to the magnitude of the thresholds chosen and has a number of other properties which make it suitable for the evaluation of RAQ forecasts. Another crucial aspect of forecast verification is the choice of measurements sites. Some key issues related to this topic are discussed and recommendations made regarding the selection of sites to be used in GEMS verification. In order to assess the effects of chemical pollutants on humans and crops, a number of air quality indices are in common use throughout the world. We review these indices and make recommendations regarding those to be adopted for GEMS forecasts.

3 Contents 1. INTRODUCTION REVIEW OF MODEL INTERCOMPARISONS REVIEW OF MODELS AND EXISTING VERIFICATION METHODOLOGIES AMONGST GEMS PARTNERS REVIEW OF ISSUES RELATED TO OBSERVATION SITES INTRODUCTION EVIDENCE FOR THE LIMITED SPATIAL REPRESENTATIVENESS OF OBSERVATION SITES AN EXAMPLE OF OBSERVATION SITE CLASSIFICATION: THE EUROAIRNET AIR QUALITY MONITORING NETWORK MODEL EVALUATION WITHIN THE CONTEXT OF OPERATIONAL AIR QUALITY FORECASTING SYSTEM: THE EXAMPLE OF PREV'AIR VERIFICATION METHODOLOGIES BASIC STATISTICAL EVALUATION OF CHEMICAL SPECIES FORECASTS QUANTITIES TO BE VERIFIED TAYLOR DIAGRAMS THRESHOLD EXCEEDANCE SKILL SCORES MODEL DIAGNOSTICS: SCALE DECOMPOSITION TECHNIQUES ISSUES RELATING TO CITY-LEVEL FORECASTS GENERAL REMARKS CONTINUOUS SCORES DICHOTOMOUS SCORES AIR QUALITY INDICES FOR HUMAN HEALTH QUANTIFYING ENVIRONMENTAL RISK WITH INDICES AIR QUALITY INDICES AIR QUALITY INDICES FOR GEMS REVIEW OF ISSUES RELATED TO CROP DAMAGE INDICES OZONE CRITICAL LEVELS FOR THE PROTECTION OF VEGETATION IN EUROPE STOMATAL FLUX MEASURES OF CROP DAMAGE COMMENTS AND RECOMMENDATIONS SUMMARY OF PROPOSED VERIFICATION METHODOLOGIES...32 ACKNOWLEDGEMENTS...34 REFERENCES...34 APPENDIX 1 TESTING A SCALE-INTENSITY TECHNIQUE FOR ASSESSING OPERATIONAL NWP FORECASTS...40 A.1.1. SCALE-INTENSITY METHOD...40 A.1.2. EXAMPLE SCALE DECOMPOSITION...42 A.1.4. CONCLUDING REMARKS...43 APPENDIX 2 AIR QUALITY INDICES AROUND THE WORLD...44 A.2.1. AQ INDICES IN US AND CANADA...44 A.2.2. AQ INDICES IN ASIA...45 A.2.3. AQ INDICES IN EUROPE...48 A.2.4. AQ INDICES IN AUSTRALIA...51 APPENDIX 3 CROP DAMAGE INDICES: FURTHER DISCUSSION...53

4 A.3.1 RELATIVE CONSISTENCY OF AOT AND STOMATAL FLUX APPROACHES...53 A.3.2 AN EXAMPLE OF PARAMETERIZATION OF STOMATAL FLUXES : THE EMEP MODEL...54 A.3.3 UNECE RECOMMENDATIONS...54

5 1. Introduction GEMS (Global and regional Earth-system (Atmosphere) Monitoring using Satellite and in-situ data) is an ambitious EU-funded project aiming to exploit advanced data assimilation of satellite and in-situ data in order to characterise the chemical composition of the atmosphere. The project comprises five subprojects, the first three being: Green House Gases (GHG), Global Reactive Gases (GRG) and Aerosols (AER). In each case the aim is to develop a system to monitor concentrations of key species in the atmosphere and their associated surface sources and sinks. The systems primarily utilise satellite remote sensing measurements to gather global data, augmented with limited use of other in-situ data. These global data will be assimilated in a specially adapted version of the IFS forecast model at the European Centre for Medium-Range Weather Forecasting (ECMWF). The GEMS IFS will be coupled to other chemical transport models and used to produce global forecasts of the chemical composition of the atmosphere. The work by the central GEMS team at ECMWF to develop a global chemical forecast is designated the PRO sub-project. The remaining sub-project, Regional Air Quality (RAQ), has as its focus the production of higher resolution regional forecasts of chemical species and air quality indices. GEMS RAQ partners will take forecast initial and boundary conditions from the ECMWF global model and use these to drive limited area models running on the domain shown in figure 1.1. Figure 1.1 GEMS RAQ domain A key aspect of the RAQ sub-project is the on-going evaluation of the partner forecasts. Only through continued quantitative verification of forecasts can an assessment of model skill be made and improvements to models evaluated. 1

6 The main purpose of this report is to review suitable verification methods and produce a set of recommendations for implementation as part of a central verification suite to be run centrally at ECMWF. Model intercomparisons have been conducted in the past and the present report begins with a brief overview of some of these in section 2. In section 3 some of the RAQ models currently operated by GEMS partners and the verification procedures employed in evaluating them are reviewed. Chemical forecast verification involves comparing model predictions, averaged over a finite area (usually a grid box) with observation values at a single point. An awareness of issues relating to the representativeness of the sites is important for interpreting the verification measures produced and in section 4 we discuss these issues. Evidence to demonstrate the degree of variability one can encounter is presented and the EuroAirnet site classification scheme for sites is described. This section concludes with a review of how these issues have been addressed in the Prev Air system. Section 5 then moves on to review a basic formalism for the evaluation of chemical composition forecasts. This section forms the basis for the proposed GEMS central verification methodologies which will be carried out at ECMWF. In addition, it also includes some remarks concerning advanced model diagnostic techniques which, although not intended to form part of the routine verification, may be valuable for groups interested in diagnosing the variation of model skill with spatial scale. An appendix is included which describes the application of these methods to precipitation verification. Some issues of producing and verifying forecasts at the urban scale are discussed in section 6. Although no GEMS partners are planning city-scale forecasts in the initial stages of the project, this may be addressed in the later stages. We have therefore reviewed some of the issues pertinent to this problem. Forecasts of chemical composition giving detailed information regarding the concentration of pollutants can be difficult to interpret and it is common in many forecast systems to combine information relating to individual species into an overall air quality index which is used to give guidance regarding effects on human health. Many different indices have been developed and are in use throughout the world. In section 7 we present an overview of the issues relating to the choice of an appropriate index to be used within GEMS RAQ. In Appendix 2 a review of the main indices used in different countries is presented. The influence of air quality (and in particular ozone) on crops is a key area of concern to plant biologists. The effects tend to be of a longer-term nature and cumulative over a long period. It is not intended to forecast parameters relating to crop damage within GEMS RAQ. However scientists studying this area may find the atmospheric chemical analyses produced by GEMS to be a valuable tool for correlating air quality with crop growth. In view of this we have included in section 8 a review of the issues related to the definition of 2

7 suitable metrics to quantify the dose of damaging ozone to which crops are subjected. This discussion is extended in Appendix 3 to give a more detailed comparison of the different ways of evaluating crop exposure to ozone. Finally, in section 9, a summary of the proposed verification procedure to be developed and implemented at ECMWF is presented. This takes into account the issues discussed in previous sections and addresses the needs of the different users of GEMS forecasts. 2. Review of model intercomparisons There has been extended work on individual regional air quality model evaluation during the last two decades for ozone and particulate matter (for reviews see: Seinfeld, 1988; Peters et al., 1995; Russell et al., 2000 and references therein). In most studies models are evaluated against observations from air quality monitoring data or data taken from field campaigns over short periods of time. It was found by Russell et al. (2000) that model advancement would benefit from coordinated thorough intercomparisons. Putting models together and comparing them over the same set of data and over the same period of time requires a high coordination effort. Since model simulations not only depend on the models formulations but also on numerous input data, the intercomparisons are made even more difficult. Forcing several models to use the same input data (meteorology, emissions, boundary conditions) is an ideal concept, but is very difficult to set up in practice, since models use different meteorological variables, some of them recalculate meteorological variables, etc. Thus in most intercomparison studies the entire regional air quality simulation or group of forecasting systems are evaluated and not just the models. In air quality analysis mode, intercomparisons of air quality modelling systems have been successfully achieved in several major air quality projects (see e.g. the NARSTO air quality model intercomparison, Sonoma, 1998). In Europe the first intercomparison study was made over a very short summer time period (Hass et al., 1997). With the development of air quality monitoring networks and the increase in computer power and memory, modelling systems intercomparisons have been recently been carried out over increasingly larger periods of time and a greater number of observations. Within the EUROTRAC/TOR project, 6 regional air quality models were used over two summer seasons to evaluate the effects of decadal emission changes on ozone (Roemer et al., 2003), and were intercompared. More recently the question of air quality and its changes due to emission control policies in European cities and their neighbourhood was addressed using several modelling systems over 4 cities in Europe (Cuvelier et al., 2006; Vautard et al., 2006), both for ozone and particulate matter. At the European scale, a recent intercomparison study involving seven regional air quality modelling systems has been carried out (Van Loon et al., 2004; Van Loon et al., 2006). Other intercomparison studies have been carried out, 3

8 along with major air quality field campaigns such as ESCOMPTE (Peuch et al., 2006). Several regional air quality models have also been used and intercompared in forecast mode over several days (Delle Monache and Stull, 2003), or larger periods of time (Tilmes et al., 2002; McKeen et al., 2005). They were demonstrated to have equivalent skill for ozone. When used as an ensemble, by averaging concentrations, the forecasts showed improved skill for ozone. These studies show that models generally display reasonable skill for ozone daily maxima (correlations of the order of ) while for particulate matter the skill is poorer (r= ), with large underestimations in Europe (30-50%). It appears that ozone models have difficulties in capturing concentrations near sources or during night time, whilst daytime concentrations are quite well reproduced, especially at large scale. The spread of ozone simulations by regional air quality models has recently been shown to fairly well describe the uncertainty of ozone simulations 3. Review of models and existing verification methodologies amongst GEMS partners A number of RAQ partners currently run operational or semi-operational air quality models, of varying degrees of sophistication. The most advanced models also have access to near real-time data and have implemented verification procedures. In order to collate basic information about partner models a review has been conducted of the basic model details and verification methods employed in assessing forecast skill. The information collected is summarised below. FMI FMI operate SILAM, which is a Lagrangian dispersion model driven by either HIRLAM or ECMWF meteorology. No routine air quality forecasts are currently conducted (as of November 05) and hence no routine verification is conducted. However a set of software tools, Model and Measurement Assessment Software (MMAS) has been developed to compare forecasts with observations and produce a wide range of basic error statistics, such as bias, RMSE, correlation etc. Meteo-France/INERIS The MOCAGE model, operated by Meteo-France, is a 3D chemistry transport model which can be driven by meteorology from either ARPEGE/ALADIN or ECMWF. The model is run operationally and provides both global and regional forecasts to the Prev Air platform, operated by INERIS. For the purpose of verification data from a combination of in-situ surface sites and sondes are used to evaluate ozone, NO2 and SO2 forecasts. The basic statistical error indicators of bias, RMSE, correlation etc. are used to assess the 4

9 forecast quality, with options to compute these measures on either raw hourly data, daily peak values or 8 hour running means. In order to investigate the spatial variation in model skill a number of additional methods are employed, including probability distribution functions and histograms of modelled versus observed values. University of Athens The Comprehensive Air quality Model with extensions (CAMx) is an Eulerian photochemical dispersion model that allows for an integrated oneatmosphere assessment of gaseous and particulate air pollution (ozone, PM2.5, PM10, air toxics, mercury) over scales ranging from sub-urban to continental. The model is driven by MM5 regional meteorology, which in turn is driven by NCEP global analyses. Evaluation of the forecasts is not carried out on a routine basis, but for case studies verification data is provided by surface observations from the standard ground-level monitoring network of the Greek Ministry of Environment. University of Cologne The EURAD model, operated by the University of Cologne, is an Eulerian model driven by MM5 regional meteorology, which in turn is driven by NCEP global analyses from the GFS. The model is run operationally, with 3D variational data assimilation of satellite data being employed. Routine verification of the forecasts is carried out against surface observations. Scatter plots of forecast versus observed values are used to give a visual demonstration of the forecast skill and the standard statistical error indicators (bias, RMSE, correlation etc.) are computed for these plots. In addition hit rates at the 20% and 50% levels are evaluated. Institute of Atmospheric Sciences and Climate ISAC operate an Eulerian model driven by meteorology from ECMWF. The model is currently run in an experimental mode and no routine forecast verification is carried out. UK Met Office The Met Office currently produces operational air quality forecasts using the Lagrangian dispersion model NAME. The model is driven by meteorology from the Met Office s Unified Model. No routine quantitative verification of the forecasts is conducted, although for particular episodes some comparison with observations is conducted. CNRS/INERIS The CHIMERE model, developed by CNRS and more recently with INERIS, is an Eulerian chemical transport model driven by MM5 regional meteorology, which in turn is driven by NCEP global analyses from the GFS. Routine verification of forecasts is performed using near real-time air quality surface data, provided by regional air quality networks. These are collected at the French national level by the Agency for Environment and Energy 5

10 Management (ADEME). The following basic statistical indicators are computed: Bias (absolute and relative to observation values) RMSE (absolute and relative to observation values) Variability ratio (i.e. standard deviation of forecast values versus standard deviation of observation values) Correlation coefficient between observations and forecast values Contingency tables defined with respect to thresholds Histogram of absolute and relative errors, from which error percentiles are derived. These indicators are computed for ozone peak concentrations, and nitrogen dioxide and particle (PM2.5 and PM10) mean concentrations. DMI The Danish Meteorological Institute operates both a Lagrangian and an Eulerian model, with the latter intended for use within GEMS. The Lagrangian model is operational and produces forecasts four times each day for public display. The model can be driven by either ECMWF or DMI-HIRLAM meteorology. Routine verification of forecasts over Denmark is carried out, based on surface observations. The indicators employed include bias, mean square error, correlation, contingency tables and forecast skill for the daily maximum ozone concentration. 4. Review of issues related to observation sites 4.1 Introduction When evaluating a model against measurement data, besides the definition of common statistical skill indicators, the question arises of selecting the "right" observation sites on which statistical indicators are to be evaluated. Indeed, on one hand, three dimensional air quality models compute the evolution of pollutant concentrations on grids; the concentrations can be thought of as averaged concentrations over the volume of each grid cell. On the other hand, observations are available from fixed measurement sites; they are local data and might be (and usually are) influenced by local processes. These two values are different in nature and it is not straightforward to relate grid cell average concentrations to point concentrations. Given an observation site, one has to answer the following question (McNair et. al., 1996): "How well do the measurements reflect the air quality surrounding the monitoring station?". It is the general question of spatial representativeness 1 of observations, dependent on the characteristics of the site (topography, proximity to emission sources) and on the chemical species under evaluation (lifetime). 1 This review deals with the question of spatial representativeness. Time representativeness problems also arise. This is generally treated by averaging concentrations at the model time resolution. 6

11 The selection of the "right" observation sites depends also on the model itself (typically, the horizontal and vertical resolution) and on the purpose of the evaluation (Schmidt et. al., 2001): do we need a model to produce an operational forecast or do we need a comprehensive model dedicated for example - to the prediction of pollutant concentrations in emission reduction scenarios? In the following, section 4.2 first presents different approaches to assess and evaluate quantitatively the spatial representativeness of a given observation site. Section 4.3 then deals with some criteria developed by the European Air Quality monitoring network (EuroAirnet) to provide a classification of European observation sites. Finally, the example of model evaluation within the context of operational air quality forecasting (PREV'AIR) is presented. 4.2 Evidence for the limited spatial representativeness of observation sites The simplest approach to evaluate the spatial representativeness of a given observation site is a direct comparison of the measurement data with data collected in the surroundings of the site. For example, Figure 4.1 displays the ozone concentration measured at two different locations in the Los Angeles area. The two measurement points are located only 5km from each other, whereas the ozone peak concentrations range from 50 to 100 ppb. This very simple picture shows that at a 5km scale, ozone concentrations are highly inhomogeneous. Figure 4.1 Ozone concentration measured at two locations in the Los Angeles area (McNair et al., 1996). McNair et al. (1996) use a second approach to assess the degree of spatial inhomogeneity of the available data set (the SCAQS database, consisting of a set of photochemical measurements collected in the Los Angeles area in 1987). For each observation site, all other sites within 25 km are used to predict the average pollutant concentration at the original site via 1/r 2 interpolation. The interpolated concentrations are then compared with the observations at the corresponding site. A large difference between the observed and interpolated values indicates strong concentration gradients in the vicinity of observation site. The authors conclude that "the spatial variability in observed CO concentrations is higher than for other pollutants such as ozone, with normalized gross error statistics for interpolated CO and 7

12 ozone at 45% vs 27% respectively. Observed CO concentrations are more susceptible than ozone to local conditions such as proximity of the air monitoring site to direct emissions sources". Park (2005) applied the "data-withholding" method to PM2.5 measurements over the United States, with a radius of 120km (180km when no station was available in a 120km radius). The spatial variability of PM2.5 and PM2.5 species was calculated using the mean fractional error (MFE) computed between observed and interpolated concentrations. The MFE ranges from 28% to 84% in the United States for different species (Table 4.1). Table 4.1 Spatial variability of PM2.5 mass and species in the United States (Park, 2005). This study shows that the differences in spatial variability depends on the chemical species: it is higher for primary pollutants, e.g. elemental carbon and soil dust; lower for secondary pollutants, e.g. sulphate, nitrate and ammonium. These differences are explained by particular emissions and formation characteristics. Tilmes (2001) considers ozone time series from about 360 sites all over Germany, and forecasts from a 3D regional CTM. A large number of stations are close to emission sources (from industry or traffic) and thus are referred to as "polluted sites". In order to estimate the variances in the observation errors 2 and forecast errors, Tilmes plots the correlations of the observation increments (differences between observations and forecast data) as a function of the horizontal distance between the sites for two different times of the day: 14h-15h UTC (maximum diurnal value) and 04-05h UTC (minimum concentrations at polluted sites). The correlations are considered between non polluted sites, polluted sites and between one polluted site and one non polluted site. From these graphs the variances in the observation and background error are estimated (see Table 4.2). The largest observation errors are in the complete set of data (all sites, all times). This points out the limited representativeness of the data in case no distinction is made between the sites and the time of the day. The smallest observation error is found for the afternoon hours from non-polluted sites. 2 In this study, observation errors consist of the sum of measurement uncertainties and representativeness errors. 8

13 Table 4.2 Forecast and observation errors [ppb2/ppb2], for different choices of sites (P: polluted, N: non-polluted) and times of day (UTC) (Tilmes, 2001). The comparison between model outputs and individual measurement stations also provides useful information about the representativeness of the observation data (Tilmes et. al., 2002). For example, the comparison of five Eulerian air pollution forecasting systems with summer 1999 German ozone data showed that all models behave similarly with respect to the bias indicator in large parts of Saxony. This fact points to a limited representativeness of some measurement locations in this area, under strong urban influence. Figure 4.2 Comparison of DWD and NERI models in terms of the spatial distribution of the bias, using the daily maximum ozone values for the period July August (Tilmes et al., 2002). In addition to emission sources, the topography surrounding the site represents another source of spatial heterogeneity for observations. Sites in remote mountainous locations are difficult to deal with in the context of model evaluation, usually due to the model resolution being too coarse to resolve the orography. In an ongoing study, Chevalier et al. (2006) collected ozone time series over 21 measurement sites in France (from the MERA - French acronym for MEsure des Retombées Atmosphériques and PAES - French acronym for Pollution Atmosphérique à l'echelle Synoptique networks) and Switzerland between 2001 and The altitude of the MERA sites ranges from 115m to 1750m above sea level. The altitude of the PAES observation sites - representative of the free troposphere - ranges between 755m and 2877m (Pic du Midi). The altitude of the Swiss stations ranges between 770m and 3500m (Jungfraujoch). Chevalier et al. (2006) plotted the correlations between the observed ozone daily mean concentrations as a function of a) the difference of altitude between the sites; b) the horizontal distance between the sites (Figure 4.3). They show that i) the correlation between ozone concentrations decreases as the difference of 9

14 altitude between the sites increases; ii) for a given distance between two observation sites, the correlation between ozone concentrations is higher when the site altitude is lower. a) b) Figure 4.3 Impact on the correlations between the observed ozone daily mean concentrations at two different sites of a) the difference of altitude and b) the horizontal distance between the sites. The size of the points is proportional to a) the distance; b) the altitude between the sites (Chevalier et al., 2006). Gheusi at al. (2006) reports the first results of the PIC 2005 campaign, carried out in Central Pyrenees in summer 2005, in the framework of the PAES monitoring network. One objective of the PIC2005 campaign was to study the short range variability of ozone concentrations in the lower part of the troposphere. The experimental set-up is shown in Figure 4.4 (left panel). Figure 4.4 On the left: PIC 2005 experimental set-up. PDM site: 2875m a.s.l.; AYA site: 1050m a.s.l.; ozone measurements. CHI site: meteorological data. The CHI and AYA sites are located about 4km away from the PDM site. CRA site: 650m a.s.l., located 28km away from PDM. On the right hand side, mean diurnal evolution of ozone concentrations (in ppb) for the PDM site, AYA (=CHI) and CRA. (Gheusi et al., 2006). Figure 4.4 (right panel) displays the mean diurnal ozone cycle for the Pic Du Midi (PDM), Chiroulet (AYA) and Lannemezan (CRA) sites. The AYA and CRA sites show a classical ozone cycle (photochemical production at daytime; destruction at night), with maximum ozone concentrations observed during 10

15 the afternoon. On the other hand, the PDM site shows a lower-amplitude but reversed ozone diurnal cycle, with minimum concentrations at daytime. These different evolutions of the ozone concentrations at different sites located within a few kilometres one from another are related to a thermal breeze regime. 4.3 An example of observation site classification: The EuroAirnet air quality monitoring network We have just mentioned some criteria to provide a quantitative estimate of the representativeness of observation sites. This topic is well known by people involved in air quality monitoring networks: for their observation data to be usable by people that do not actually know the observation sites, they have to provide a simple classification of these sites. In the following, we provide an example of such a classification for EuroAirnet, the European Air Quality monitoring network. EuroAirnet - developed since is a selection of air quality monitoring stations in Europe, out of more than 6000 existing sites in about 30 countries. The selection is based on several criteria (EEA Technical Report No. 12, 1999), among which are i) the classification of monitoring stations ; ii) the area of representativeness of monitoring stations. These criteria were specified in order to provide a consistent set of monitoring stations across Europe. The classification scheme of monitoring stations in EuroAirnet is based on three sub-criteria: type, zone, characterization of zone (see Table 4.3). Table 4.3 EuroAirnet site classes (EEA, 1999). Sub-classes are defined to distinguish between different types of background stations, based on precise requirements in term of minimum distances to emission sources. Remote stations are located far away from emission sources and are used to monitor natural background levels and long-range transport of air pollutants. They are exposed to air masses representative of background levels. Some sites are avoided: valleys and locations subject to formation of stagnant air under inversion conditions, mountain tops and cols (topography constraint); coastal sites subject to pronounced diurnal wind variations (land-sea breeze); sites under vegetation shelter, where concentrations might be lowered. Regional ("Rural background") stations are used to monitor regional/rural background levels resulting from long-range transport of air pollutants or regional emissions. The emission distance 11

16 requirements for important emission sources are less strict than for remote stations. Stations can be located in agricultural areas. Near city background stations are used to monitor regional background levels resulting from long-range transport of air pollutants or regional emissions. The emission distance requirements for important emission sources are less strict than for regional stations. They are located outside cities, possibly in areas with many cities close to each other. Urban/suburban background stations are used to monitor "average" air pollution levels in urban areas resulting from transport of air masses from outside the urban area and from emissions in the city itself. They are not directly under the influence of dominating emission sources (traffic, industry). For each monitoring station, an evaluation of its area of representativeness in terms of a radius is provided. This is defined as the area in which the concentration does not differ from the concentration measured at the station by more than a specified amount. The area of representativeness varies with the station type. It depends on the concentration difference allowed in the definition and on the environment of the station, its morphology and sources. Determining the area of representativeness of a station requires either monitoring around the station or dispersion model calculations for the area in question and its surroundings. Such determinations are rarely performed. Thus the determination of station class is accompanied by an evaluation of the station s area of representativeness, taking into account the emission variations in the surroundings and any localised influence of sources further away, topographical features influencing the dispersion and transport of the emissions. Table 4.4 lists typical ranges of the area of representativeness (radius of area) for the various station types. Table 4.4 Radius of area for various station classes. *) For traffic stations, the area of representativeness can rather be defined in terms of length of road: traffic stations suitable for comparison with others should represent a road/street length of some 100m or more in central city areas and some 1,000m or more in suburban/other areas (EEA, 1999). 4.4 Model evaluation within the context of operational air quality forecasting system: the example of PREV'AIR Since 2003, the PREV'AIR system has been delivering information about air quality. Forecasts and observation maps of air pollutants - ozone, nitrogen dioxide and particles - are published on a daily basis on the Internet. The 12

17 forecasts are delivered up to three days in advance, at various spatial scales (global, Europe or France) depending on the pollutant. Additionally, numerical forecast data are available enabling air quality monitoring qualified associations to run their own air quality forecast or diagnostic tools at a finer resolution. Near-real time observation data 3 are used in the framework of PREV AIR to evaluate in near-real time (NRT) the system capability to forecast air pollutant concentrations, and to produce NRT surface ozone analyses by combining model and observations. The question of choosing right observation sites i.e. suited to the purpose of evaluating the PREV AIR system has arisen since the beginning of the project. An important step in the observation site selection procedure consisted in launching a survey by the French qualified air quality monitoring associations about the representativeness of observation sites. For ozone, this survey led to the following conclusions: i) all background sites but one should be considered for the model performance evaluation; ii) additional industrial sites should be included for the model performance evaluation, since they behave as background sites, with respect to ozone concentrations. Thus for the model performance evaluation within PREV AIR, observations are compared with the outputs in the first model layer. Sites from different zones (i.e. urban, suburban and rural) are treated separately since it is expected that air quality models behave differently with respect to observations depending on the spatial representativeness of the observation sites. In view of this experience it is proposed that initially all observation sites providing data are used in the forecast verification. A large number of observations sites should become available (some several thousand ultimately) and the influence of a small number of unrepresentative sites on the overall performance statistics will be small. Quality control of those observations, so that only reliable measurements are used, is an important topic but lies outside the scope of the present report. As experience with the system increases it may be possible to identify sites which systematically produce suspect or misleading results and remove them from the evaluation. 5. Verification methodologies The GEMS RAQ sub-project will ultimately deliver around 10 separate regional air quality forecasts which must be verified against observations. Although all partners will be forecasting for the same domain, the resolution will be dependent on local model configurations and computing resources. It is well known that no single statistical indicator is capable of capturing all aspects of model behaviour thus a suitable verification strategy should produce a range of indicators which demonstrate the relative skill of the models to capture various features of the measured air quality. In section 5.1 an evaluation of some basic field statistics is described. These are useful in 3 These data are collected locally by the French qualified air quality monitoring associations, and gathered in the BASTER near real-time database, transferred to the PREV AIR system. 13

18 giving an overview of the performance of the model over the entire domain in terms of a simple bias, correlation and error statistic. However a problem arises in making a meaningful comparison of the GEMS RAQ models on this basis since they will all run at different spatial resolutions. It is well known that the skill of a given model (as represented by these basic statistics) run at low resolution may appear to be greater than that of a higher resolution simulation, due to the smoothing associated with the larger grid size. The higher resolution model is more susceptible to the double penalty problem, whereby a local maximum is correctly forecast but in a slightly the wrong place, giving rise to two areas of error. However the lower resolution model would tend to show less skill in forecasting extremes and therefore it is important that any model assessment considers both performance aspects. In section 5.4 we therefore discuss the use of metrics to assess the skill of models in exceeding thresholds. 5.1 Basic statistical evaluation of chemical species forecasts Air quality models produce fields of pollutant concentration over a given domain and the first aspect to be considered in assessing the quality of the forecast is to determine how closely the field of forecast values matches observed values over the entire domain. An evaluation of some basic statistical measures to compare fields, similar to those currently employed by partners running operational models, is sought. The forecast values are produced on a regular grid and therefore give a value which is essentially averaged over an area equal to that of the grid box. Observed values on the other hand are available at isolated and irregularly spaced points. As described in Section 4, the spatial variations in chemical species may frequently be more rapid than the grid size and hence the model is unable to represent these sub-grid scale variations. One option for making the comparison would be to use the point observations to produce a regular field of observed values on a grid corresponding to that of the forecast values, using a statistical interpolation methodology such as kriging. A practical drawback of this approach is that, since the GEMS RAQ forecasts will be produced on different grids a separate re-gridding of the observations would be required for each model evaluation. A more fundamental objection is that this would tend to smooth the observations and does not give a true picture of the observed variability. An attempt to compare area average values of observations and forecasts has a similar problem. Combining observations from nearest neighbour sites to produce area averages would inevitably produce a smoother variation than exists in the point observations. It is preferable to retain the true variability in the observed values and recognise the inability of a model to represent this variability as an inherent limitation related to the grid size of the model. An alternative approach therefore is to take the forecast values and interpolate these to the observation points. 14

19 Given forecast and observed values f i, o i for a given species at site i, it is possible to compute a variety of error statistics. Traditional metrics include the mean bias 1 B = ( f i o i ) (5.1) N and the root mean square error i ( i i) (5.2) i 1 E = f o N A further desirable step is to normalize these metrics so that they provide a relative error. This is essential when comparing the bias of different chemical species which may be present in the atmosphere at very different concentration levels. The usual choice is to use the observed values for normalisation, giving the normalised mean bias and normalised rmse B n 1 ( fi oi) = (5.3) N o i i E n fi o i = N i o (5.4) i In these equations some authors prefer to normalise with the sum of the observed values and thus sum separately over the numerator and denominator. Whilst this is acceptable for relatively small domains where the observations do not vary widely, with a large domain it is preferable to maintain normalisation by values at individual sites. One problem with the use of B n is that there is an asymmetry between the case of under and overprediction. For over-prediction B n can grow to very high values much greater than unity, whilst for under prediction it is limited to -1. In comparing different GEMS RAQ models some may exhibit a trend to over-prediction and some to under-prediction. It is desirable to use a metric which treats both of these model deficiencies in a symmetric manner. A solution is to employ a normalisation comprised of the arithmetic mean of the observed and forecast value, giving a modified mean bias: B 2 f o = N i fi + oi (5.5) i i ' n This gives a measure of forecast bias bounded by the values -2 to +2 and which performs symmetrically with respect to under and over-prediction. One weakness of this procedure is that the normalisation employed varies 15

20 slightly between different forecast models due to the dependence on forecast values. However given that partner models will evolve from the same initial conditions, in most cases the variation is not expected to significantly distort the calculated bias. This approach is adopted by Seigneur et.al. (2000) and Cox and Tikvart (1990). The traditional rationale for employing the (normalised) rmse as an indicator of overall forecast error is two fold: (i) by squaring the errors before combining, this measure removes any cancellation of under and overprediction; (ii) in cases where the spread of errors approximates to a wellknown distribution (e.g. normal, binomial, Poisson etc.) the rmse can be attributed with a physical significance. In the present case the errors are not expected to conform to any well-known distribution. In addition, the rmse suffers from similar deficiencies as the mean bias, displaying an asymmetry with respect to under and over forecasting. A further issue is that the rmse gives added weight to those errors having greater magnitude, as a consequence of the squaring operation. In view of these issues we propose the use of the fractional gross error, E f, as the indicator of overall forecast error E f = 2 fi oi (5.6) N f + o i i i This is essentially a relative version of the commonly used mean absolute error. The modified mean bias indicates the extent to which the model under or over-predicts the set of observations, whilst the fractional gross error gives a measure of the overall forecast error. An additional metric proposed for comparing forecast and observation fields over the whole GEMS domain is the correlation coefficient. This is needed to indicate the extent to which patterns in the forecast match those in the observations. The correlation coefficient R between the forecast and observed values is defined as R = 1 N i ( f f )( o o) i σ σ f o i (5.7) where f and o are the mean values of the forecast and observed values and σ f and σ o are the corresponding standard deviations. The correlation coefficient has a maximum value of unity when, for each observation site, ( f f ) = c( o o (5.8) i i ) where c is a positive constant. In this case the two datasets have the same pattern of variation but are not identical unless c=1 for all sites. 16

21 5.2 Quantities to be verified Verification statistics are of interest to both the scientists developing and analysing model performance, forecast end-users interested in the skill with which models can forecast key threshold exceedances. Our verification system must evaluate results which are informative to both groups. For the first group an analysis based on the hourly data produced by partner models will be computed for the species of primary interest from an air quality viewpoint: NO 2, SO 2, PM 10, CO, ozone and overall air quality index (to be discussed in section 7). For the second group quantities closely related to the average values commonly stated in air quality objectives are more relevant. In this case the following quantities are proposed for verification Species Quantity to be verified NO 2 Daily maximum of hourly values SO 2 24 hour mean of hourly values PM hour mean of hourly values CO Daily maximum of 8-hourly running mean of hourly values ozone Daily maximum of 8-hourly running mean of hourly values AQ Index Daily maximum of hourly values Table 5.1. Air quality parameters to be verified. 5.3 Taylor diagrams The statistical measures discussed above, when taken together provide a valuable indication of model performance over the entire spatial domain. The results are expressed numerically and a time series of each quantity (bias, gross error, correlation) can be plotted to show trends over time. However Taylor (2001) has shown that a single diagram can be used to summarise the basic statistical measures of pattern rmse and correlation and thus display the relative performance of a number of forecast models in a visually accessible manner. The overall mse, defined in equation (5.2), can be decomposed into contributions from the square of the mean bias and the centred pattern mse E p, defined by p = ( i ) ( i ) N o (5.9) i E f f o The other symbols have been defined in equations (5.1) and (5.7). The following relationship can be shown to hold between the forecast/observation variances, the pattern mse and the correlation coefficient E = σ + σ 2σ σ R (5.10) p f o f o 17

22 Taylor exploited the similarity between this relationship and the cosine rule to plot a polar diagram having the radial coordinate represent the field standard deviation and the angular coordinate represent the correlation between the two fields. A reference field, such as an analysis field or in our case the set of observations, is normally plotted along the x-axis. Normalisation of the forecast and observation fields via the observation standard deviation places the reference point at a value of unity along the x-axis. The radial distance from this point to that of the forecast field gives the pattern rmse (normalised by σ o 2 ). An example is given below. Here a set of relative humidity forecast fields, taken at 12 hourly lead times from T+12 out to T+120, are plotted, with reference to the analysis field along the x-axis. The forecast fields all have similar normalised standard deviation values of around unity. However as the lead time reduces from T+120 to T+12 the correlation improves from ~0.78 to ~0.97 whilst the normalised pattern rmse falls from ~0.66 to ~0.22. Since their introduction in 2001 Taylor plots have found frequent application in comparing model performance, e.g. Brunner et.al. (2003), Cuvelier et al. (2006), Vautard et al. (2006a). They provide a visual summary of model performance which is easily interpreted when comparing a number of different models and are thus ideally suited to the evaluation of GEMS RAQ forecasts. We propose to display Taylor diagrams for the daily mean values of NO 2, CO, SO 2, PM 10, ozone and the overall air quality index to be described in section 7. 18

23 5.4 Threshold exceedance skill scores The verification measures described above provide information about the forecast errors under all conditions, regardless of the magnitude of pollutant concentration. However it is desirable to have metrics which provide information regarding forecast skill specifically at those times when pollutant levels are elevated and pose a greater risk to human health. For a given chemical species threshold levels have been identified for which it is recommended that the public be either informed or given warnings regarding possible adverse health effects. For example for ozone an information threshold of 180µgm -3 and a warning threshold of 240µgm -3 have been specified (Directive of the European Parliament, 2005). It is of interest to assess the skill that models possess in predicting exceedance of given thresholds. The method traditionally employed in this case is to construct a 2x2 contingency table in the following manner: Event Observed Yes No Event Yes a b Forecast No c d Table 5.2. A 2x2 Contingency Table In this table a denotes the number of occasions on which the event (in this case an exceedance of the relevant threshold) was forecast to occur and did indeed occur (a hit ), b denotes the number of false alarms, c the number of misses and d the number of correct rejections. The sum of all of these is the total number of events, n. Once a contingency table has been compiled it is straightforward to compute a range of skill scores. A variety of scores have been used in meteorological studies, such as Proportion Correct, Heidke Skill Score, Gilbert Skill Score etc. (see Wilks, 2006). Each of these scores has particular characteristics which may be suitable for a given application. For the evaluation of GEMS RAQ forecasts we require a skill score which has the following properties: Simple to calculate and interpret Not sensitive to the thresholds chosen Not sensitive to the base rate Robust not easily hedged Can be tested for significance if required A metric which meets these requirements is the odds ratio skill score (Stephenson, 2000). The odds is defined as the ratio of probability that an event (such as the exceedance of a threshold) occurs, to probability that it does not occur. It is a non-negative number with a value greater than unity when a success (good forecast) is more likely than a failure (bad forecast). The odds ratio θ is calculated on the assumption that forecast skill can be judged by comparing the odds of a success (a hit) to the odds of failure (false alarm): 19

24 H F θ = 1 H 1 F 1 (5.11) In this equation the hit rate and false alarm rate are given by a H = (5.12) a+ c b F = (5.13) b+ d The odds ratio is easily calculated from contingency tables via the equation ad θ = (5.14) bc Stephenson (2000) has emphasised that since the odds ratio depends solely on the conditional joint probabilities and not on the marginal totals it is independent of any bias between the forecast and the observations. When b or c are equal to zero the odds ratio is indeterminate. However in this case it can be shown (see next section) that the odds ratio is not statistically significant and should not be applied. In practice, after a reasonable length of forecast trial this will not present a problem. The odds ratio is a symmetric statistic in the sense that it behaves equally for greater than a threshold as for less than the threshold. For some purposes asymmetric scores are required especially for rare events for which the consequence of exceeding the threshold is very severe. However for the purpose of air quality model verification the use of a symmetric score gives a more balanced assessment. A skill score can be constructed from the odds ratio via a simple transformation θ 1 ORSS = (5.15) θ + 1 This mapping produces a score ranging from -1 to +1. Forecasts having a strong negative (positive) association with observations have ORSS values tending to -1 (+1), whilst random forecasts have ORSS tending to zero. Significance testing The odds ratio has further properties which enable significance testing of any results to be carried out. Whilst the odds ratio itself is skewed, its logarithm is asymptotically normally distributed (Agresti, 1996), with a standard deviation given by 20