Report on operational implementation with analysis of opportunity for ISO 9001 service certification

Transcription

1 MACC-II Deliverable D108.2 Report on operational implementation with analysis of opportunity for ISO 9001 service certification Date: 07/2014 Lead Beneficiary: MF-CNRM (#23) Nature: R Dissemination level: PU Grant agreement n

2 Work-package 108 (ENS, Transition to operations) Deliverable D_108.2 Title Report on operational implementation with analysis of opportunity for ISO 9001 service certification Nature R Dissemination PU Lead Beneficiary MF-CNRM (#23) Date 07/2014 Status Final version Authors Approved by Contact Michael Gauss (MET.NO#22), Alvaro Valdebenito (MET.NO#22), Anna Benedictow (MET.NO#22), Julius Vira (FMI#15), Mikhail Sofiev (FMI#15),Hendrik Elbern (RIUUK#27), Elmar Friese (RIUUK#27), Patricia Schmid (Rhenish Institute for Environmental Research at the University of Cologne), Frederik Meleux (INERIS#17), Anthony Ung (INERIS#17), Laurence Rouïl (INERIS#17), Solen Queguiner (MF-CNRM#23), Marion Pithon (MF- CNRM#23), Sylvie Guidotti (MF-CNRM#23), Virginie Marécal (MF- CNRM#23), Matthieu Plu (MF-CNRM#23), Lennart Robertson (SMHI#28), Henk Eskes (KNMI#21), Robert van Versendaal (KNMI#21), Ujjwal Kumar (KNMI#21), Arjo Segers (TNO#30) Virginie Marécal info@gmes-atmosphere.eu [In case the deliverable is not a report: provide a description of it inside this box.] 2 / This document has been produced in the context of the MACC-II project (Monitoring Atmospheric Composition and Climate - Interim Implementation). The research leading to these results has received

3 Executive Summary / Abstract The MACC II (Modelling Atmospheric Composition and Climate, project is establishing the core global and regional atmospheric environmental service delivered as a component of Europe's GMES (Global Monitoring for Environment and Security) initiative. The regional forecasting service provides daily 4-days forecasts and analyses for the past 24-hours of the main air quality species (ozone, NO 2, SO 2, CO, PM2.5, PM10 and birch pollen during spring, plus 4 additional species for fine scale model boundary conditions (NO, NH 3, total NMVOCs, PAN+PANprecursors) over Europe from 7 state-of-theart atmospheric chemistry models and from the median ensemble calculated from the 7 models. The MACC-II service is in the continuity of MACC but includes the organization of the transition from prototype operational services to an operational phase, post MACC-II (WP108). It is important to note that the funds given in MACC-II do not provide the possibility of a 7 days/7 days and 24h/24h monitoring. Consequently, the current system has to be regarded as the demonstrator of the fully operational system that is planned for the Copernicus atmosphere service phase. The NRT (Nearly-Real Time) regional air-quality services are distributed over the seven production centers providing robustness and richness to the system, thus requiring to organise the operations in a distributed context. Météo-France plays a central role in the European air quality production since it centralizes the 7 model forecasts and analyses and the observations used for the assimilation, it produces the ensemble forecasts and analyses, the plots for the website, it develops/maintains the web site, it produces the files for the numerical data server and it develops/maintains the numerical data server. This report presents the details of the operationalisation at the end of MACC-II project of the seven individual production lines, of the ensemble production and associated central services. Although the production chains have already a good reliability (over 90% for most of them), work has been done within MACC-II to consolidate these chains where necessary in order to increase their reliability based on operational procedures, softwares and databases similar to those used for Numerical Weather Prediction. In this report, the opportunity of a service certification under ISO 9001 standard is also detailed for the individual production chains at the 7 centers in charge of these chains and for the central production (Météo-France). 3 /

4 Table of Contents 1. General organisation of the regional daily production CHIMERE production Forecast production Analysis production Risks of failure, resiliency and monitoring of the system Feasibility of a certification ISO EMEP production Forecast production Analysis production Risks of failure, resiliency and monitoring of the system Feasibility of a certification ISO EURAD-IM production Forecast production Analysis production Risks of failure, resiliency and monitoring of the system Feasibility of a certification ISO LOTOS-EUROS production Forecast production Analysis production (Semi-operational): Risks of failure, resiliency and monitoring of the system Feasibility of ISO9001 certification MATCH production Forecast production Analysis production Risks of failure, resiliency and monitoring of the system Feasibility of a certification ISO MOCAGE production /

5 7.1 Forecast production Analysis production Risks of failure, resiliency and monitoring of the system Feasibility of a certification ISO SILAM production Common suite for data preparation Forecast production Analysis production Risks of failure, resiliency and monitoring of the system Feasibility of a certification ISO Central production services (observations for assimilation, processing of ensemble products and verification, website and numerical data server) Introduction Observations Handling of the 7 individual model outputs Ensemble calculations Verification and statistics Website and numerical data server The Website The download service of numerical data Status Risks of failure, resiliency and monitoring of the system Feasibility of a certification ISO /

6 1. General organisation of the regional daily production The regional air quality service delivers daily forecast products, analysis products in numerical format (GRIB-edition2 and NETCDF) and graphical format on the MACC-II regional air quality website. Besides, some verification plots and statistics are computed and displayed on the website. Forecasts and analyses are produced by 7 individual models that are managed locally by 7 partners. This is what is called local production in the rest of the document. Each local model production uses the same meteorological forcings (from ECMWF IFS), the same chemical boundary conditions (from ECMWF IFS-MOZART and soon from C-IFS), and the same aerosol boundary conditions (from IFS) and the same observations for analyses (collected by Meteo-France). Meteo-France is in charge of collecting the 7 individual productions, of producing an ensemble to have the best estimate of pollutant concentrations, of distributing the data to the users, and of displaying the graphical outputs (maps of the models and of the ensemble, verification and statistics). This is called central production in the rest of the document. The regional ensemble forecast production is based on the following steps (Fig. 1.1): 1. Each partners is responsible for the local forecast production based on its model, which consists in the concentration of different pollutants, from 0h to 96h term by 1- hour step, in GRIB2 format, 2. Each partner pushes this outputs (GRIB2 format) onto a ftp server at Météo-France 3. Météo-France inserts these model outputs (after regridding if necessary) in operational databases and afterwards produces the ensemble and all the plots and data to be present in the website or on the numerical data server. All these steps are running on the operational system. The first 48-hours forecast terms of the ensemble are produced before 7h UTC in the morning and the following terms are produced as early as possible after. Figure 1.1 : Schematic of the organization of the multi-model Ensemble forecast system 6 /

7 The regional ensemble analysis production is based on the following steps (Fig. 1.2): 1. Météo-France transfers the NRT observation data from the EEA database, preprocess them, produces CSV files and makes them available for the partners on the Météo-France ftp server. 2. Each partner gets the surface observations from Météo-France ftp server and produces an analysis in its own infrastructure. 3. Each partner pushes its production on the Météo-France ftp server. 4. Météo-France inserts these model outputs (after regridding if necessary) in operational databases and afterwards produces the ensemble and all the plots and data to be present in the website or on the numerical data server. Since the observations are delivered by Météo-France around 7h20 UTC for the day before, the individual analyses may be produced no later than 10h30 UTC. The ensemble analysis is then produced before 11h UTC everyday. However, such timing constrains do not enable to start the individual forecasts from the analyses. Figure 1.2 : Schematic of the organization of the multi-model Ensemble analysis system 7 /

8 2. CHIMERE production 2.1 Forecast production A synthetic summary of the model configuration is given in the table 2.1. To reach the requested targets in terms of resolution and output time delivery, we make the choice to consider the input data provision from the ECMWF 00 UT production (IFS meteorological fields and IFS-MOZART/C-IFS for the chemical boundary conditions). Emission data are processed upstream of the modelling chain from the recovery of emission inventory made by TNO. The reference year currently used is 2009 which was rescaled with the annual total emission for The timetable of the operations in charge of gathering all the input data needed to run CHIMERE are described in the figure 2.1. The first transfer is the download of IFS data which starts early in the morning as we currently use only the meteorological forecast issued at 00 UT. The dissemination ECMWF application provides the daily IFS data in near real time on our ecgate domain. The download script is composed of a main script which get the IFS files from ecgate and a second script checking the number and size of the necessary files and in case of lack or Domain -25 E 45 E ; 30 N 70 N Horizontal resolution 0.1 Vertical resolution Variable, 8 levels from the surface up to 500 hpa Gas phase chemistry MELCHIOR2, comprising 44 species and 120 reactions Heterogeneous chemistry yes Aerosol size distribution 8 bins from 10 nm to 40 μm Inorganic aerosols Primary particle material, nitrate, sulphate, ammonium Secondary organic aerosols Biogenic, anthropogenic Aqueous phase chemistry yes Dry deposition/sedimentation Classical resistance approach Mineral dust Dusts are considered Sea Salt Inert sea salt Boundary values Values provided by MACC-GRG Initial values 24h forecast from the day before Anthropogenic emissions TNO inventory Biogenic emissions Pre-processing using MEGAN Forecast System Meteorological driver 00:00 UTC operational IFS forecast from the day before Assimilation System Assimilation method Optimal Interpolation Observations Selection of background surface stations for O3 and PM10 Frequency of assimilation Every hour over the day before Meteorological driver 00:00 UTC operational IFS forecast for the day before Tableau 2.1: Description of the CHIMERE outline for the purposes of MACCII unusual size a new download is launched. 8 /

9 Two similar processes occur later in the day starting around 6 pm. One to get the MOZART files for the CHIMERE gaseous boundary conditions. This download concerns the Mozart outputs pre-processed by ECMWF and stored on ecgate. The second one to extract from mars the aerosol boundary conditions (dust, sea salt...) that are used by Chimere. These files are converted in netcdf using mars requests before being downloaded. Figure 2.1: Schedule of the operations involved in the generation of a CHIMERE forecast Once all the mandatory files are downloaded on our system, the CHIMERE execution can start. The model run is split in two phases. The first one is dedicated to the calculation of the forcing (meteorological fields, emissions and boundary conditions). These three independent pre-processing are starting separately when their input data are available. Once all the required files issuing from the pre-processing phase are present, the second phase which is the CHIMERE execution starts in parallel on 256 cpus. The run usually lasts 7 hours and is available early in the morning depending most of the time on the time availability of the Mozart files. The treatment to convert CHIMERE output in grib2 format for the requested fields starts at 1 am and generates the files in near real time during the forecast computing. The grib2 transfer to Meteo-France ftp is automatically triggered once the grib2 file is produced. 9 /

10 Data volume and file production During the several stages described in the previous section, lot of files are produced as input, intermediate and output files. All of them represent a large volume of data per day evaluated in the table 2.2. Data volume / day size save format kind of data IFS download 7.3 Go just J-1 grib2 input OBS download 10 Mo yes csv input MOZART download 1.2 Go no netcdf input Emission files /month 85 Go 1 month netcdf CHIMERE full output file 187 Go no netcdf output CHIMERE grib2 files 3 Go 7 days grib2 input preprocessed MACCII output CHIMERE light output file 32 Go yes netcdf output Analyses 120 Mo yes netcdf output Analyses grib2 files 53 Mo 7 days grib2 MACCII output Emission pre-processing 30 Go no netcdf intermediate Boundary pre-processing 8.7 Go no netcdf intermediate Meteo pre-processing 14 Go no netcdf intermediate Tableau 2.2: Characteristics of the data used to run CHIMERE and produced by the CHIMERE modelling chain The total volume of data per day is around 375 Go which can be described approximately as 95 Go for the input data, 55 Go for the intermediate files and 225 Go for the output files. Among these data, we decided to back-up only a part of them highlighted in green in table 2. Finally, we store around 35 Go per day. Cluster infrastructure Chimere currently runs on a cluster of 320 CPU with an allocation of 256 cpu. Times of different parts of the mono-processor and parallel run are shown in Table /

11 Processing time Nb proc Emission pre-processing 80 1 Boundary pre-processing Meteo pre-processing CHIMERE run Table 2.3 Processing characteristics The CHIMERE production for MACCII is currently inserted in the timetable of the French platform PREV AIR. Especially, MACCII forecast is used to provide the boundary conditions of high resolution forecasts over France. The other characteristics of the cluster are 4 To of disk with high speed access and 70 To of storage disk. In addition a system is archiving the data on bands. 2.2 Analysis production The data download (fig. 2.1) about in-situ observations occurs around 2am for the previous day observations and 11 am for the updates of the J-2,-3,-4 observations. For all these data, requests are sent to EEA defining pollutant list and date required. Using this dataset, analyses for O 3 and PM10 are built using the observation data available for the eve based on an optimal interpolation methodology. The analyse runs on a single processor and the task lasts 45 each. 53 Mo are daily produced and saves on the system for seven days. The analyses which are produced in netcdf format are converted in grib2 files and transfer to the Meteo-France FTP server at 11 am. 2.3 Risks of failure, resiliency and monitoring of the system Lot of efforts has been made in the recent past to improve the reliability of the system especially in terms of input data provisions. The failures are linked to disruption of the input data transfer from ECMWF to INERIS, and to the production of the CHIMERE daily forecast on a cluster dedicated to the French platform with a tight timetable that can lead to disruption in production and delays in the grib2 files delivery. This HPC system is monitored on 24/7 with a high level of availability (more than 99.5 %). In addition 10 persons weekly permute with a daily constraint to monitor the system and its production. Even if the MACC production is not assessed by this people this monitoring action has benefits on the CHIMERE production for MACC. 2.4 Feasibility of a certification ISO9001 CHIMERE is running on the system at INERIS dedicated to operational air quality monitoring and forecasting. INERIS is certified ISO9001 and so all their underneath activities among them the MACC-II forecast production are certified ISO9001. Therefore they follow the recommendation in terms of continuous improvement, tests and conception. 11 /

12 3. EMEP production The EMEP production line, delivering daily Chemical Weather Forecasts, is developed and maintained at the Norwegian Meteorological Institute (MET Norway). It has been in semioperational mode since 2008 with a very low failure rate, but is still being improved with the aim of 100% reliability (no lack of model data), increased accuracy (through continuous model development data assimilation), and an optimum level of automation. Analyses, with assimilation of NO 2 columns have been operational since the end of 2012, and assimilation of additional chemical species is being introduced in Being the national meteorological service in Norway, MET Norway has reliable and longterm access to the country s leading HPC facilities, such as stallo ( University of Tromsø) and vilje ( Norwegian University of Science and Technology). MET Norway also follows the introduction of new HPC facilities and ports its production lines accordingly whenever necessary. In the following two sections, the forecast and analysis production lines are explained in some more detail. Sections 3.3 and 3.4 identify risks of failure and discuss the feasibility of an ISO 9001 certification. 3.1 Forecast production Figure 3.1 shows the flow chart of the semi-operational forecast chain (along with the analysis chain to be described in the next section). In the following description, the day at which the EMEP production chain delivers data to Météo France is referred to as day D. The day before is day D-1, the day after is day D+1, etc. The forecast meteorology for the 108-hour period starting on day D-1 at 12:00 UTC and running until day D+4 at 00:00 UTC, is downloaded from ECMWF/IFS (MARS) automatically on day D-1 by 18:15 UTC. The data format is Grib2. These data are then pre-processed at MET Norway for use in the EMEP air quality model, including the conversion into netcdf format and vertical interpolation. Boundary conditions for gaseous species, based on IFS- MOZART output, are retrieved from ECMWF operationally on day D-1 at 19:40 UTC. The files (one for every 3-hr record) are combined into one file before the EMEP model run, but the horizontal and vertical interpolation is done on-the-fly, while the EMEP model is running. Aerosols are not yet used as boundary conditions, but this will soon be included (the reason why they have not been included yet is that they do not come together with the gaseous species). Using these meteorological and boundary condition data, the EMEP model is run for the MACC domain and the 96-hour period from day D, 00:00 UTC to day D+4, 00:00 UTC. This model run is started at 00:30 local time on the HPC facility vilje and takes about 20 minutes on 64 CPUs to complete. For initialization, the run uses a restart file from the forecast of the previous day, valid at day D at 00:00 UTC. 12 /

13 Subsequently, the model results are converted from netcdf to Grib2 format (which takes less than 5 minutes) and then transferred to Météo France for analysis and visualization on the MACC webpages. As the transfer takes between 20 and 40 minutes, the EMEP model results are usually available to Météo France by around 02:00 local time on day D. The file size of the uploaded EMEP forecast results is: 97 files x 14 Mb each. Since the EMEP model is so fast, and since we deliver at night time when the ftp lines are not too busy, we post-process and deliver all 96 hours in one go. The results are not regridded at MET Norway, but only trimmed (to the domain required by Météo France). As the data files are pushed to Météo France s ftp site, they are not stored at our local ftp site anymore. But they are still published locally at emep.int. Figure 3.1: Data flow and timing of the EMEP forecast chain. Day D means today, i.e. the day when the production chain delivers data to Météo France. D-1 means yesterday, D+1 tomorrow, etc. Timings are approximate. The EMEP forecast is operational in the sense that it is run daily in an automated way. However, availability still falls below 95% sometimes (as specified in the model dossiers of ENS), as the nodes used at the HPC facility are research nodes, and in case of failure, our staff can handle these only during office hours. However, the transition to MET Norway s operational service lines (with superior availability statistics) is underway, as explained in Sections 3.3 and /

14 3.2 Analysis production The analysis chain is set up to produce model data for the 24-hour period from day D-1, 00:00 UTC to day D, 00:00 UTC, i.e. all yesterday. It assimilates observations for day D-1, i.e. yesterday, at a 6-hourly frequency. The flow of the analysis production is shown in Figure 3.2. Figure 3.2: Data flow and timing of the EMEP analysis chain. Day D means today, i.e. the day when the chain delivers data to Météo France. D-1 means yesterday, D+1 tomorrow, etc. Timings are approximate. The run crucially depends on the availability of the surface observations for day D-1, which become available to MET Norway on day D only at 07:40 UTC. Satellite observations for day D-1 become available on day D at 03:00 UTC and are retrieved at 06:30 UTC. The model simulation takes about 6 hours to complete because, until now, it can run only on one single processor (CPU). It is started already at 06:30 UTC on day D, i.e. more than one hour before the surface observations arrive, in order to save time. As surface observations for the early night hours are not available during the first stages of the model run, this head start is to be seen as a trade-off between available observations and delivery time. By the time the model simulation reaches the early morning hours, the surface observations are available. The model simulation is driven by the 00 UTC forecast meteorology that was issued on day D-1. It is initialized from the restart file created at the end of the previous analysis run, i.e. valid for day D-1, 00:00 UTC. 14 /

15 After completion of the run, the model results are post-processed and transferred to Météo France. This process takes between 5 and 10 minutes, so that the analysis is available to Météo France at about 14:00 UTC on day D. The file size of the uploaded EMEP analysis results is: 25 files x 14 Mb each. For the processing of boundary conditions and the transfer to Météo France the same applies as for the forecast chain (see Section 3.1). The 24 hours of data are processed and transferred to Météo France in one go, in the early afternoon of day D. The rather high computational demand of data assimilation poses a problem for timely operational deliveries of the analyses. Currently, efforts are ongoing to parallelize the 3D VAR code, meaning that the analysis can be run on a multitude of processors thus allowing faster delivery in the future. 3.3 Risks of failure, resiliency and monitoring of the system MET Norway is the national meteorological service in Norway and can as such be seen as a stable institution and reliable data provider. Nevertheless, the daily provision of very advanced calculations of this kind is always challenging. Although the failure rate (missing file notifications) during the last years of pre-operation has been very low, every effort has to be undertaken to go towards 100% reliability. During the last 12 months the most usual failures were related to: - interruptions of the ftp transfer or ssh connection failures - exceedances of disk space - corrupted boundary condition files - long queues at the HPC facility However, it has to be noted, that these failures are rare, amounting to less than 10% of the deliveries. In all cases of failure, the data could be delivered with short delays. Currently, the automatic forecast runs are transferred from the research nodes to the operational forecast nodes of MET Norway (the so-called testprod lines of the institute). This will ensure a high priority regarding the use of computer resources, i.e. the CWF runs will not need to compete with other research applications. Also staff can take care of failures 24 hours a day, 7 days a week, provided the failure is not severe. In severe cases the developing team has to be contacted. It is still to be decided if contacting the development team can occur outside the usual office hours. In any case, the service will be coordinated with other operational services provided by met.no, such as volcanic ash forecasts for Norway and nuclear accident simulations. Thus, most of the current vulnerabilities are going to be eliminated. Also, as has been the case for two years already, a backup system is being maintained which can provide the forecasts in the unlikely case of a failure of the first system (e.g. power failures, node failures, disk failures, etc.) 15 /

16 3.4 Feasibility of a certification ISO9001 The EMEP production chain is not yet part of the operational service lines of MET Norway. Rather, the EMEP production chain has been operated by scientists, which are part of the EMEP development team, on so-called research nodes at the HPC facilities (e.g. vilje and stallo). However, we aim at finalizing the transition to MET Norway s operational service lines by the end of MACC-II, and certainly during MACC-III. There are three operational service levels at MET Norway, differing mainly in the availability and quality requirements, and the way possible failures are handled. At the highest service level, the developers can be contacted at all times of the day in case of failure, while at lower priority levels, notifications are sent. In any case, the production is run on so-called operational nodes at the HPC facility, without having to compete with other jobs as is the case for productions running on research nodes. As of today, among the various operational service lines of MET Norway, only the (meteorological) observational network and the aviation services are ISO9001 certified. However, MET Norway is currently conducting a so-called internal control project where the most relevant processes in the various production chains of MET Norway are scrutinized with respect to risks and vulnerabilities. The project is to develop and implement comprehensive internal control mechanisms at MET Norway. Its mission is to establish systems that ensure stable and robust delivery of products and services. Work is done systematically to identify risks and vulnerabilities in the company's core processes which support operational weather and ocean forecasts but also administrative processes. In cases where vulnerabilities are identified that may prevent the achievement of goals, measures and controls are implemented. The work is intended to improve the quality of our production chain (ensuring high availability), but quality assurance of results in the sense of ISO9001 is not envisaged within this internal control project. The process of ISO9001 certification is comprehensive and requires external evaluation and is therefore also a question about the available budget. As internal control routines at MET Norway are well established in general, the formal process of ISO9001 certification has not been given a high priority yet, although it can be seen as a long-term goal. However, in case this is required, the EMEP production chain could possibly be certified, as it involves only a small part of MET Norway s infrastructure (ftp transfer, disk storage, high performance computers, job scripts and monitoring procedures, and the EMEP air quality model). 16 /

17 4. EURAD-IM production The EURopean Air pollution Dispersion Inverse Model (EURAD-IM) is a comprehensive chemistry transport model, which was designed at the Rhenish Institute of Environmental Research (RIU) since the mid 90-ties for 4-dimensional variational data assimilation with special focus on advanced assimilation of satellite data. In the framework of MACC EURAD- IM provides (pre-)operational air quality forecasts and analyses, which contribute to the MACC regional ensemble AQ service. Operational stability is mainly achieved by the following measures: 1. The service is operated twice and simultaneously, on two independent computational platforms, that is a. ECMWF on the one hand, and at b. Research Centre Jülich as a secondary installation, with a similar configuration. Both installations are operated independently, except a final mutual two-way test of delivery of final results. 2. Operational duties of the Research centre Jülich based compute platform are handed to contracted maintenance staff, to guarantee technical stability. 3. The supervisor monitor scheduler (SMS) developed at ECMWF is used to control the time schedule of the daily EURAD-IM AQ forecast and AQ analysis. SMS provides reasonable tolerance for hardware and software failures, combined with good restart capabilities Both, the EURAD-IM MACC AQ service running at ECMWF (in the following called primary system) and the AQ service running at Research Centre Jülich (in the following called backup system) depend on the global production at ECMWF. Required components of the global production are the IFS operational meteorological forecast and analysis, the MOZART global AQ forecast (MACC-II subproject GRG), the global IFS forecast including aerosols (MACC-II subproject AER), and the Global Wildfire Assimilation System (GFAS) forecast of wildfire emission data (MACC-II subproject FIR). Both the primary and backup AQ forecast depend on the EURAD-IM AQ analysis running at ECMWF. This AQ analysis is primarily designed to provide initial values for the AQ forecast. Because it is started shortly after midnight to ensure delivery of the AQ forecast on schedule, a reduced set of observations is assimilated. For this reason the AQ analysis from the backup system (which uses a more complete set of NRT observations) is delivered to the MACC-II ensemble analysis system. However, the primary and backup production of AQ forecasts and analyses may run independent from each other. A further common feature of the primary and backup systems is that both systems deliver their results to the ECMWF for further processing by Meteo-France where the MACC-II regional ensemble AQ service is hosted. 17 /

18 Figure 4.1. Data flow of the EURAD-IM AQ service running at the ECMWF (primary system). See text for further explanation. Figure 4.2. Data flow of the EURAD-IM AQ service running at Research Centre Jülich (backup system). See text for further explanation. 18 /

19 4.1 Forecast production Input data Meteorological initial and boundary values Primary system The 12:00 UTC operational IFS meteorological forecast for the previous day is used for the provision of initial and boundary values for the EURAD-IM meteorological driver WRF. Forecast data is extracted from the MARS archive at ECMWF with 3h temporal resolution. If the extraction of the IFS forecast fails, the required meteorological fields are retrieved from the GFS operational forecast. Backup system The GRIB2 files containing the interpolated meteorological fields extracted from the MARS archive by the primary system are transferred via Internet to the Research Centre Jülich. If the Internet connection cannot be established, GFS data is used instead of IFS data. Chemical boundary values Primary system The global MOZART AQ forecast (MACC-II subproject GRG) for the previous day is used for the provision of gas-phase boundary values for the EURAD-IM CTM. MOZART output files in NetCDF format are read from the band storage system (ecfs) at the ECMWF. Aerosol-phase boundary values are obtained from the global IFS forecast including aerosols (MACC-II subproject AER). The IFS-Aerosol data is extracted from the MARS archive at ECMWF. MOZART and IFS-Aerosol data are horizontally interpolated to the Lambert conformal grid used by the EURAD-IM system. If the MOZART air quality forecast or the IFS- Aerosol forecast is not available, climatological boundary values are used. Backup system The boundary values prepared by the primary system are transferred via Internet to the Research Centre Jülich. If the Internet connection cannot be established, climatological boundary values are used. Chemical initial values Primary system The EURAD-IM AQ forecast is initialised with the final state of the EURAD-IM AQ analysis for the previous day. If the analysis is not available, the forecast for the previous day is used for initialisation. The latter method of initialisation can be applied to the past 4 days in case of a forecast horizon of 96 hours. Backup system A file containing the final state of the primary EURAD-IM AQ analysis for the previous day is transferred via Internet to the Research Centre Jülich. If the data transfer cannot be established, the backup forecast is initialised with the backup forecast for the previous day. Emission data Emissions from natural fires are derived from the Global Fire Assimilation System (GFAS). The GFAS forecast of wildfire emissions of several gaseous and particulate species for the previous day is extracted from the MARS archive at ECMWF and interpolated on the Lambert conformal grid used by the EURAD-IM AQ forecast. If the GFAS forecast for the 19 /

20 previous day is not available, the forecast for the day before yesterday will be used. If the GFAS forecast cannot be obtained, wildfire emissions are omitted Model chain First step of the EURAD-IM AQ forecast is the processing of data from the global production chain at the ECMWF, i.e. generation of initial and boundary values for the WRF model from the operational IFS meteorological forecast (see Section for details) and generation of chemical boundary values from global AQ forecasts. Execution of the WRF pre-processing system WPS is triggered by the provision of meteorological initial and boundary values. Once the WPS is completed, the WRF model is executed for a 96h meteorological forecast. The WRF model uses an adaptive transport time-step to minimize the risk of a failure due to violation of the Courant-Friedrichs-Levy (CFL) Stability Criterion. The CTM is executed as soon as the initial values are available. Initial values are provided by the EURAD-IM AQ analysis running at ECMWF (See Section 4.2). If the CTM fails, it is executed again with a reduced transport time-step. Final step of the EURAD-IM air quality forecast is the postprocessing of CTM NetCDF output data. Post processing includes the production of GRIB2 files GRIB2 output Concentrations of some chemical species in the surface layer and at 50 m, 250 m, 500 m, 1000 m, 2000m, 3000 m, and 5000 m are horizontally interpolated on a geographic grid with 0.1 x 0.1 resolution and written in GRIB2 format. For each of the 96 hours forecast horizon a single file is generated. The GRIB2 files are delivered around 05:30 UTC to the Ensemble prediction system operated by MeteoFrance. Primary system GRIB2 output files are archived on ECMWF s band storage system. Backup system The GRIB2 files generated at the Research Centre Jülich are transferred via Internet to the ECMWF. If the Internet connection cannot be established, the transfer is initiated again after a time delay. This procedure may be repeated 10 times with increasing time delay. 4.2 Analysis production Input data Meteorological initial and boundary values Primary system The 12:00 UTC operational IFS meteorological forecast for the previous day is used for the provision of initial and boundary values for the EURAD-IM meteorological driver WRF. See Section for further information. Backup system The operational IFS meteorological analysis for 00:00, 06:00, 12:00, and 18:00 UTC of the previous day is used for the provision of initial and boundary values for the EURAD-IM meteorological driver WRF. 20 /

21 Chemical boundary values Preparation of boundary values for the AQ analysis is done in the same way as for the AQ forecast. See Section for details. Chemical initial values Primary system The EURAD-IM AQ analysis is initialised with the final state of the EURAD-IM analysis for the previous day. If the analysis is not available, the EURAD-IM AQ forecast for the previous day is used for initialisation. The latter method of initialisation can be applied to the past 4 days in case of a forecast horizon of 96 hours. Backup system A file containing the final state of the primary EURAD-IM analysis for the previous day is transferred via Internet to the Research Centre Jülich. If the data transfer cannot be established, the backup EURAD-IM AQ analysis is initialised with the backup analysis for the previous day Observations Both, the primary system and the backup system currently assimilate MACC NRT surface in situ data, surface in situ data from the German Umweltbundesamt, NO 2 column retrievals from OMI and GOME-2, and MOPITT CO profile retrievals. The observations are transferred via Internet from the data providers to the ECMWF and to the Research Centre Jülich. If a datasets cannot be obtained, it is excluded from assimilation Model chain First steps of the EURAD-IM AQ analysis are the processing of data from the global production chain running at ECMWF and the acquisition of NRT observation data. All available NRT data is read by the EURAD-IM observation data pre-processor and written in a common format suitable for the subsequent data assimilation procedure. The required data from the global production chain comprises initial and boundary values for the WRF model from the operational IFS production chain (see Section for details) and chemical boundary values from global AQ forecasts (see Section for details). Once the WRF preprocessing system WPS is complete, the WRF model is executed. Initial values for the CTM are provided by the EURAD-IM AQ analysis for the previous day. Available NRT data are assimilated every full hour using intermittent 3d-var with an assimilation interval of 30 minutes. If the CTM fails, it is executed again with a reduced transport time-step. Final step of the EURAD-IM AQ analysis is the production of GRIB2 files, which are delivered around 10:00 UTC to the MACC ensemble analysis system (see section for details). 4.3 Risks of failure, resiliency and monitoring of the system Risks of failure are minimized due to the independent twin back-up implementation of the EURAD-IM MACC AQ service. The largest remaining risk is the data transfer from the ECMWF to Meteo-France. This risk of failure will be reduced in future by the use of dissemination channels available at the ECMWF. 21 /

22 Two staff members of the Rhenish Institute for Environmental Research are responsible for the monitoring of the AQ service. Results from both implementations of the EURAD-IM AQ service are furthermore transferred to RIU for routinely validation and monitoring of service quality. 4.4 Feasibility of a certification ISO9001 ISO9001 certification is aspired for the forthcoming project. As yet, there are no unified criteria for the acquisition. In preparation of this, the model validation is underway according to the JRC delta tool, which can provide a candidate examination for certification. 22 /

23 5. LOTOS-EUROS production The LOTOS EUROS contribution to the MACC II ensemble forecasts and analyses is a joint effort from TNO and KNMI. TNO is mainly coordinating the development of the chemistry transport model with contributions from RIVM and KNMI. The KNMI is installing and maintaining the operational processing environment for air quality forecasts and produces the daily analyses and forecasts. Since November 2011 (and for the full duration of MACC II) the MACC LOTOS EUROS forecasts are part of the KNMI operational processing environment for air quality forecasting. This implies a fully operational status and results in a high availability (e.g. 100% during MACC II phase 1). The operational processing is performed and monitored by the IT department of KNMI. Forecasts are available before 4:00 UTC. Since September 2013, all the operational Lotos-Euros runs have been transferred to the new KNMI supercomputer BULL in operational mode. This has increased the speed of forecast/analyses run considerably by a factor of about two. 5.1 Forecast production Operational infrastructure and software For air quality forecasting, the KNMI has installed dedicated computing hardware and operational software to guarantee the availability to an agreed level. Currently this is 97%, meaning that less than 3% of the forecasts are allowed to be delayed. This operational system serves two clients: first, the RIVM and the Dutch citizen for daily air quality forecasts for the Netherlands; secondly, the MACC II ENS ensemble forecasts. A nested configuration is set up, consisting of a European domain (serving MACC II) and a higher resolution zoom domain including the Netherlands. The hardware consists of a development computer (now development is also done on the BULL Supercomputer), and computers for the operational processing. The whole operational system consists of two identical copies: in case of hardware failure the second set of computers will take over automatically. The operational hardware consists of a separate computer for the scheduling/monitoring software and a computer (now the BULL Supercomputer) for the Lotos Euros analyses/forecasts. Transfer of Input data (meteorological, boundary conditions) : The operational system uses the following inputs : Meteorological fields from the ECMWF: surface fields and 3D fields, grib format. File transfer occurs twice a day (for 0:00 and 12:00 forecasts) through an operational channel between ECMWF and KNMI (KNMI APL). 23 /

24 MACC global reactive gas concentrations (from IFS MOZART) used as boundary conditions, in netcdf format. These files are read directly by the model. MACC global dust aerosol concentrations used as boundary conditions, in netcdf format. These files are read directly by the model. MACC daily emissions from fires (GFAS v1.1 daily product). The MACC GFAS 2D fire emission fields are retrieved from MARS at ECMWF, and are converted into (small) netcdf files with only the non zero field elements and sent to KNMI. Start fields from the forecast of the previous day. Static input: anthropogenic emissions (MACC TNO), land use, boundary conditions. The availability of ECMWF operational forecasts is critical: a forecast cannot be produced when this data is missing. The other data is not critical: If the MOZART IFS output is missing the LOTOS EUROS model will fall back to climatological boundary conditions. If the MACC fire emissions are missing, then the model will run without fire emissions. If the MACC aerosol data is missing the dust boundary conditions are set to zero. All results and inputs for the past week are available on disk. The full archive is available on the MOS storage system of KNMI Schematic of Data-Flow and Organisation of the production chain Figure 5.1 shows the ingest, distribute and processing data flows in the operational system. Note that this system serves both MACC and the national air quality forecasts issued by RIVM. The "EU" labels refer to the MACC European forecasts, and the "NL" labels refer to the zoom domain over The Netherlands. The following output of LOTOS EUROS is stored on the KNMI mass storage: Three dimensional hourly model state for a large number of species (netcdf). Hourly surface concentrations for a large number of species (netcdf). Three dimensional hourly meteorological fields on the model grid (netcdf). Model state at 0:00h of each forecast day, for restart. Assimilation statistics. The size of this output is about 7Gb per forecast run. In a post processing step the fields required for ENS are extracted from the model output. These fields are stored in hourly grib2 files, for example: KNM_ grib2, KNM_ grib2 24 /

25 Figure 5.1. Flow diagram of the KNMI processing chain for air quality forecasts. The processing time and time of availability: The forecasts are produced in about 2 1/2 hours, and are available before 4:00 utc. The grib-2 files are made available on the KNMI ftp site for download to the MACC-ENS central processing site at Meteo France. Documentation and testing of the operational system The operational aspects of the forecast system are documented in detail. The current documentation includes: Production Rules document (PRD); Product Specification document, (PSD) specification of all data products; Interface Control Document (ICD), list of external interfaces; Second Interface Control Document (ICD), listing interfaces with Lotos Euros; System Specification Document (SDD), describing the Lotos Euros processing chain; Software Verification and Validation Plan (SVVP), for the acceptance tests; Software Test Report (STR), with results from the Site Acceptance Test (SAT); Failure Procedures, describing actions to be taken in case of failure; Test scripts and test data for performing the acceptance tests; 25 /

26 Yearly updates of the operational system Updates of the operational system occur once every year (the target is 1 April). Such upgrades follow a strict procedure: A new and documented model version of Lotos Euros is released by TNO and is installed on the development machine at KNMI. A test run is performed for the years (no assimilation, no forecasts). User RIVM analyses/validates the output of these runs by comparing with the Dutch air quality surface measurements (LML network of RIVM). Based on these runs the improvement of the model is documented with respect to the previous model version. Biases of the model are estimated, focussing on ozone and PM10. Depending on these results RIVM will advice to proceed. A combined set of assimilation/hindcast runs is performed for the summer of 2003 (April September), with the configuration of Lotos Euros which will be employed operationally. The 2003 forecast and analyses results are evaluated by RIVM. The new Lotos Euros configuration is then incorporated into the NADC software and tested. Updates in input and output are implemented and changes to the operational system are documented in detail. This new operational system is subsequently led through a set of end to end tests, checking, e.g., how the system responds to missing datasets. Operational documentation (see above) is updated. Then the system is migrated to the operational hardware. A test production (e suite) will be performed for at least one week before the new version becomes operational. When all components are performing according to expectations, the operational system is replaced by the new one. We would like to remark that, because the system is fully operational, the upgrade process requires a substantial amount of work. Ideally such upgrades are performed only once a year. For LOTOS EUROS the preferred date for the upgrade is 1 April (the start of the new Summer smog season). Intermediate update in 2012 In MACC II a major upgrade has occurred. This upgrade consisted of: Extension of the domain to include Iceland, Turkey; Extension of the forecast to 4 days; (For LOTOS EUROS) Use is made of the near real time fire emissions from the FIR subproject (GFAS product); use is made of the global dust analyses to improve the boundary conditions. The impact of this upgrade on the surface concentrations was tested, and after a testing period the upgrade became operational on 29 October The processing time has increased with about 30 minutes as a result of the changes listed above Update in 2013 (Migration to KNMI Supercomputer BULL system) Since 1-Sep-2013, a major upgrade was launched. Apart from upgrading LOTOS-EUROS to new version 1.9 and inclusion of birch pollen module, all the operational runs of LOTOS- 26 /

27 EUROS were migrated to the KNMI supercomputer BULL. This decreased the run time of the models considerably almost by a factor of 2 or higher. 5.2 Analysis production (Semi-operational): During MACC II, the daily analyses (for R ENS) have been produced in a semi operational way on the KNMI development hardware, under the responsibility of the researchers involved in MACC. The availability of the forecasts and analyses was checked manually by the project scientists at KNMI. In case of hick ups, due to, e.g., missing input data, the processing was re started manually. This semi operational mode gave more flexibility for, e.g., more frequent updates of the Lotos Euros code and settings. The MACC analysis is decoupled from the forecasts. The reason is the late delivery of the surface observations. In practice, a forecast run is produced in the early night time (results available by 3:40 utc). The analysis starts when the surface observations are available on the Météo France MACC server. On 29-Oct-2013, the MACC analysis has been transferred to the KNMI supercomputer BULL. Till April-2014, the observations were availably only halfway the afternoon. However, since april-2014, with the new EEA observations available by early morning (about 7:00 to 7:30 UTC), accordingly the analyses run has been shifted earlier in the morning set to start at 8:30 UTC. Now, the analyses run output are available latest by 11:30 UTC. At this moment, the forecasts and analyses are not coupled. The results from the analyses have no impact on the forecast and the forecast uses its own start field from the run of yesterday. With the nearly half a day delay of the analysis it is not useful to establish the coupling. As soon as the European surface observations become available in near real time, the two approaches will be coupled and one combined analysis forecast will be produced. Transfer of input data, such as meteorological, boundary conditions etc remains same as that is used for the forecasts as explained in section 5.1. On top of this, the analysis system uses: European surface observations distributed by ENS (Météo France). As pre processing step these files are converted to the netcdf format which can be read by LOTOS EUROS. Start fields for each of the ensemble members of the Ensemble Kalman Filter. Following schematic shows the data-flow in case of analyses production. 27 /