Concepts of operational validation of the Copernicus Atmosphere Monitoring Service

Transcription

1 MACC-II Report n D153.5 Concepts of operational validation of the Copernicus Atmosphere Monitoring Service Date: August 2014 Lead Beneficiary: IASB-BIRA (#6) Nature: R Dissemination level: PU Grant agreement n

2 Work-package 153 (MAN, Service specification and validation) Deliverable D_153.5 Title Concepts of operational validation of the Copernicus Atmosphere Monitoring Service (CAMS) Nature R Dissemination PU Lead Beneficiary IASB-BIRA (#6) Date December 2014 Status Final Authors A. De Rudder and J.-C. Lambert (IASB-BIRA) Editors A. De Rudder (IASB-BIRA) Reviewers V.-H. Peuch (ECMWF) Contact info@gmes-atmosphere.eu This document has been produced in the context of the MACC-II project (Monitoring Atmospheric Composition and Climate - Interim Implementation). The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7 THEME [SPA ]) under grant agreement n All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and liability. For the avoidance of all doubts, the European Commission has no liability in respect of this document, which is merely representing the authors view. 2 / 18

3 Executive Summary / Abstract Monitoring Atmospheric Composition and Climate Interim Implementation (MACC-II) took over from the EU FP7 project MACC to establish the core global and regional atmospheric environmental service of the Earth observation European programme GMES/Copernicus. The present document overviews concepts applicable to operational aspects of validation related tasks for the upcoming Copernicus Atmosphere Monitoring Service (CAMS). Based on the ensemble of documentation published by the MACC-II project partners, it reviews the operational components already in place, investigates gaps, emits suggestions for future development directions and considers possible automation. 3 / 18

4 Table of Contents 1 Introduction Concept of operational (validation) system From R&D validation to operational validation Operational validation of the Copernicus Atmosphere Monitoring Service Collection of user requirements Data validation Validation method Traceability Validation data Assessment of compliance with specifications Assessment of compliance with user requirements Validation reporting Metadata Service validation Automation Conclusion and summary of recommendations Names and acronyms References / 18

5 1 Introduction Together with the European Global Satellite-Based Navigation System GALILEO [1] and the INSPIRE Directive [2], the Programme for the Establishment of a European Capacity for Earth Observation COPERNICUS constitutes the European contribution to the Global Earth Observation System of Systems (GEOSS) (Figure 1). Through linking together past, current and future Earth observing systems around the world, the GEOSS, under the umbrella of the Group on Earth Observations (GEO), aims at providing decision makers and the society worldwide with a wide range of information and decision-supporting tools in nine societal benefit areas (SBA), namely disasters, energy, agriculture, biodiversity, ecosystems, health, climate, water and weather. Figure 1. With INSPIRE and GALILEO, COPERNICUS forms the European contribution to the GEOSS and addresses its nine societal benefit areas. In this prospective, data quality and, consequently, data validation are of crucial importance. Driven by the wish to enable all users to assess the extent to which some given scientific information is fit for their purpose, the GEO secretariat mandated the Committee on Earth Observation Satellites (CEOS) to establish, with support from IEEE and GSICS, the general data quality strategy for the GEOSS. This action resulted in the creation of the Quality Assurance Framework for Earth Observation (QA4EO), which strives for harmonisation and dissemination of best practices in use in the Earth Observation (EO) communities and provides guidance in this area. The overarching principles of QA4EO are the traceability of the data and data derivation, and the provision of fully traceable, useroriented quality indicators (QI) associated with the data [3]. The objective of COPERNICUS being to provide its users, in particular policy makers, with quality assessed information in order to support the formulation of informed decisions, the operational validation of the COPERNICUS Atmosphere Monitoring Service (CAMS) should provide appropriate and documented quality indicators enabling COPERNICUS users to determine the fitness-for-purpose of the delivered information and make informed decisions accordingly. 5 / 18

6 2 Concept of operational (validation) system Reaching the operational stage which, for each COPERNICUS service, should occur close to the first launch of the associated Sentinel satellite (in 2016 for the Atmosphere Monitoring Service, with the launch of the S-5p precursor TropOMI mission) is referred to as COPERNICUS most crucial milestone in all the documentation about the programme. However, what is meant or entailed by operational remains implicit. Military sense put aside, the Collins dictionary offers two meanings of the word: relating to operations and in working order, ready for use [4]. Webster s online dictionary adds being in effect or operation [5]. In the context of the Copernicus services supported by the space and ground segments of the programme (Figure 2), it is the second meaning which is obviously applicable: all components of the programme will become operational when they perform as planned. Figure 2. The three components of COPERNICUS are the space segment laid out around ESA/EUMETSAT s seven Sentinel satellite missions, the ground segment treating and dispatching the remotely sensed data, and the service segment encompassing currently six areas of application: emergency, security, climate, marine, land and atmosphere. The projects MACC and MACC-II laid the foundations of the latter. With COPERNICUS becoming operational, its Atmosphere Monitoring Service (CAMS) will systematically exploit data from the Sentinel missions dedicated to the observation and monitoring of the Earth atmosphere, namely Sentinel-5p ( ), Sentinel-4 and Sentinel-5. The latter two are scheduled to start respectively in 2019 and 2020 with launches of the geostationary Meteosat Third Generation (MTG) and polar-orbiting MetOp Second Generation satellites on which they will be embarked. In practice, the notion of an operational system often embeds or implies a number of overlapping properties, without being restricted to any of them : Usefulness (fitness for purpose) Working as expected, running smoothly, without unwanted discontinuities Unfolding as routine operation Regularity Sustainability Possibility of mechanisation / defined in an algorithmic way Possibility of partial or total automation Delivery of the required information in ready-to-use form(at) 6 / 18

7 It should be stressed that all the properties listed above, but the first one, emphasise the mechanical aspects of the system, that is, its operation stricto sensu. Paradoxically, there is therefore some pitfall, in focusing on these with a view to maintain the system efficiency, of losing sight of its actual purpose, which would of course be missing the point. This danger is analogous to the peril which lies in favouring formalism at the expense of meaning for the sake of interoperability. One may reasonably consider that being in working order and ready for use implies that the use has been formerly defined and that the considered tool fulfils the corresponding requirements, that is, that it is fit for purpose (first property in the list above). When the system in question is based on scientific knowledge, in constant evolution by nature, being fit for purpose will imply integrating the most recent findings and methods. In order to prevent it from falling into the above-mentioned trap, a science-based operational system must hence remain closely and continuously connected to and fed by specific but also more general research and development (R&D) activities. As soon as this bond, acting like an umbilical cord, is broken, the system will grow obsolete and run idle or collapse by itself, even if it offers, at some stage, the guarantee that its wheels are perfectly oiled. The same is true for those wheels: in a world in permanent evolution, information/outreach systems and their underlying technical/it hardware must adapt continuously or become quickly obsolete, unattractive, unused. This is also true for a validation system, generally one component of a more complex system, as it is the case for the CAMS. As the CAMS validation system grows from R&D to operations, it should remain flexible enough to continue to integrate new features such as, for example, new validation data sets as they become available, and the evolution of validation concepts and methods developed not only within EU projects (including e.g. FP7 NORS and QA4ECV and H2020 GAIA-CLIM) but also with space agencies (in particular ESA CCI and S5p and EUMETSAT SAFs) and more generally in the GEOSS development actors (e.g., QA4EO board, GSICS and CEOS WGCV, ACC and WG-Climate), failing which it would soon lose its state-of-the-art status. As stated in the MACC-II Service Validation Protocol (SVP) [6], the top-level functions of the CAMS QA/Val system are: to assess the accuracy (classical uncertainties, stability, information content ) of the scientific data delivered by the service in a systematic and harmonised manner; to check that the production chain of the data is documented and quality assessed from end to end (traceability of data production); to ensure that any required information pertaining to the data (e.g. observations and ancillary data used in the data product generation) and their use (metadata) is available to the user in an intelligible form; to ensure that documentation on the end-to-end validation chain is provided to the service providers and users (traceability of validation, including error propagation); to assess data and service properties and performance against service specifications; 7 / 18

8 to assess data and service properties and performance against user requirements. In order to be operational, the QA/Val system must fulfil each of these functions on a continuous or periodic basis, and apply them to the variety of products and subservices forming the CAMS, which can only be achieved through central QA/Val governance organising and supporting distributed operation of validation tasks according to the required expertise and competence. 3 From R&D validation to operational validation The MACC-II service status level matrix takes on seven criteria to locate a data-providing service on a scale of four operational grades, from in development to operational. More elaborated and generic metrics have been proposed, one of the most relevant being the maturity matrix developed in the context of the FP7 project CORE-CLIMAX) and reproduced hereafter in Table 1, to measure how far a data-providing service stands on its way from R&D to operations. The CORE-CLIMAX maturity model is an adaptation of Bates et al. (2012) [7], which was revised to be more generic so that it can be applied to all atmospheric and climate data records (in situ, combined satellite and in situ, and reanalyses). The adapted approach has been discussed and agreed with many leading initiatives in Europe such as the EUMETSAT network of Satellite Application Facilities (SAF) and the ESA Climate Change Initiative. Internationally, the approach is supported by the WMO, CEOS-CGMS WG Climate, NOAA and USGS. It is currently applied by the CEOS-CGMS WG Climate to assess the status of the global collection of climate data records contained in the CEOS, CGMS and WMO ECV inventory ( 8 / 18

9 File: MACCII_MAN_DEL_D153.5.doc/.pdf Table 1: Top level CORE-CLIMAX Maturity Matrix showing the key areas to be assessed on a scale between / 18

10 On the same model, but using specifically adapted criteria, one could define some maturity matrix applicable to a QA/Val system itself, to assess how close to operational it stands. It is not the object of this document to come up with a formal definition of such a matrix, but some evaluation criteria could include the existence of a central QA/Val governance supported by metrology experts coordinating and harmonising QA practice according to some established validation protocol; compliance with applicable metrology standards (SI, GUM, QA4EO ); consistent application of harmonised validation procedures to different data products or services; fitness-for-purpose of validation methods and provided quality indicators (QI); practice of uncertainty propagation analysis; use of quality-checked independent validation data for comparison; sustainability and regularity of validation data procurement; practice of information content analysis; provision of complete traceability information; provision of harmonised metadata compatible with applicable standards; assessment against product and service specifications; assessment of compliance with user requirements; evaluation of product and service fitness-for-purpose; validation reporting (appropriate combination of man-made and automated); availability of online validation tools (static, interactive); existence of validation archive; provision of numerical QI in the data files; existence of a metadata searchable catalogue. In the same way as it takes time, through progressive evolution, to a R&D data-providing service to reach the operational stage, it will take time to any QA/Val system to reach fully mature operations, as some of the above criteria are gradually met. Priorities may have to be set up after some cost-benefit assessment of the implementation of each specific upgrade, and some of them might end up to be abandoned. Finally, the progress of the QA/Val system from R&D to operational stage may be monitored through periodic audit by a temporary or permanent board of external experts (such cyclic auditing worked efficiently in the past with ESA s GMES Service Element (GSE) Atmosphere PROMOTE project (annual audit) and is envisioned in current FP7 projects like QA4ECV). 10 / 18

11 4 Operational validation of the Copernicus Atmosphere Monitoring Service 4.1 Collection of user requirements CAMS user requirements have been collected in several rounds during MACC and MACC-II - and precursor projects - and were compiled in the User Requirements Document (URD), the first issue of which was followed by four updates. A brief overview of the last edition of the User Requirements Document [8] can be found in the MACC-II final Service Validation Report (SVR) [9] (Section 4). As suggested in this document, operational functioning should include the continued collection and update of user requirements, with the following additional inclusions with respect to what was done in MACC-II. Focused questions on the type of validation information needed for the molecule or product used (e.g. is the user more interested in the overall uncertainty, in the latitude dependence, in the decadal stability, or in the 5-day variability of some quantity?). Reason for the requirement (to provide the product supplier an indication on the most appropriate validation method). Relevant quality indicators and target values for these quality indicators 1 (to feed the validation of compliance with user requirements). 4.2 Data validation Validation method The choice of the validation method applied to data on some given molecule should be user- and application-driven, which is one of the reasons (not the only one) for prior collection of user requirements that can be translated into validation requirements. In all cases, application of the validation method should output numerical values of appropriate quality indicators. For example, if a trend of 1% / decade is required to be detectable, a specific validation task must assess the decadal stability of CAMS data against a validation data source of documented stability and using a robust drift detection method outputting drift estimates as well as uncertainty estimates on the drift itself; if a total uncertainty of x % is required at all latitudes and in all seasons, a specific validation task must assess the bias and spread of CAMS data against reference measurements of documented uncertainty (systematic and random uncertainties) distributed from pole to pole and in all seasons. 1 Such target values were provided by the Implementation Group at the beginning of the MACC-II project and should be updated as appropriate. 11 / 18

12 4.2.2 Traceability As stated in the MACC-II SVP [6], the overarching principle of traceability requires that the whole validation process is documented as an unbroken chain of individual processes, including data collection, file format conversion, data manipulation (change in representation system, e.g., from geopotential altitude to atmospheric pressure levels; from number density to volume mixing ratio; spatial or temporal interpolation), data selection (use of quality flags, co-location criteria, ), data comparison, reporting Each individual process and the complete validation chain must refer to and be compliant with the applicable international standards Validation data Inventory of validation data sources New or unused candidate validation data sources with well documented quality must be periodically identified, screened and, possibly, fetched and secured. For example, which NDACC stations provide currently unused data that would be useful to complete CAMS product validation? An initial such inventory was included in the MACC-II SVP [6] and is presented in the SVR [9] in the form of a table (Table 2, p. 6). The inventory of data sources actually used as reference validation data in the periodic MACC-II NRT global atmospheric composition and UV radiation validation report issues [10] also appears in the SVR (Table 7, p.16). As an example, Table 2 below shows which of the candidate validation data listed in the SVP (1 st part of the table) and which validation data not listed in the SVP (2 nd part of the table) were actually used as validation data in the NRT validation reports [10] (see detail in last three columns). A similar exercise should be systematically conducted for all validation activities and all uses of input data since assimilated data can of course not be used for validating the assimilation results. Table 2. Data sources proposed as candidate validation data in the MACC-II SVP (top section) and additional validation datasets actually considered in the MACC-II Global NRT reports (bottom section). Where applicable, the last three columns provide detail on the use of the dataset as validation data in the MACC-II global NRT reports (possible use in other validation reports has not been included). Validation data sources listed in the MACC-II Service Validation Protocol (SVP) Programme / Provider GAW Parameters Surface CO 2, CH 4, CFC, N 2O, O 3, CO, NOx, SO 2, VOC AOD properties, column & profile, backscattering & extinction coefficients WDCGG WDCA Archive Validated product(s) Surface O 3 and CO Use in MACC-II NRT validation reports NRT validation report(s) Remark Global NRT reports stations delivering NRT ozone. 12 / 18

13 ALE CFC, HCFC, CH 4, N 2O GATECH GAGE AGAGE CCGG CASN AERONET GAW GO3OS NDACC MOZAIC IAGOS MPLNET EOS MetOp CO 2 + stable isotopes, CH 4 + stable isotopes, CO, H 2, N 2O, SF 6, VOC Aerosol optical properties O 3 vertical column (Brewer, Dobson, DOAS/SAOZ) and profile (ozonesondes and Umhker) Atmospheric trace gas profiles, total & partial columns, particles, UV radiation & physical parameters of the atmosphere In-situ measurements of O 3, CO, CO 2, NOx, NOy, H 2O, aerosols & cloud particles NOAA / ESRL / GMD NASA / GSFC Aerosol Global NRT reports 1-10 WOUDC NDACC DHF SHADOZ DC Aerosol dust OD Global NRT reports 7-10 See below (NDACC data used for stratospheric ozone validation) NDACC DHF Stratospheric O 3 Global NRT reports 7-10 Microwave data in reports 7-10 FTIR data in reports 9-10 Lidar data in report 10 (only a few stations, all delivering NRT data to the NORS system) CNES-CNRS / INSU / ETHER CARIBIC Tropospheric O 3 Tropospheric CO Global NRT reports 1-10 Aerosol & cloud vertical NASA / GSFC structure CO total column & NASA / ASDC Tropospheric CO Global NRT reports 2-10 profile O 3, NO 2, SO 2, BrO, NASA / GSFC Aerosol dust OD Global NRT reports 7-11 OClO, aerosol, UV GOME-2A/B O 3, total & Eumetsat O3M- Tropospheric NO 2 Global NRT reports 3-10 tropospheric NO 2 SAF Tropospheric columns, HCHO HCHO column, OClO column Stratospheric NO 2 IASI CO column Eumetsat O3M- Tropospheric CO Global NRT reports 6-10 SAF Envisat O 3, NO 2, BrO, SO 2, HCHO, OClO, H 20, CH 4, CO, CO 2, clouds, aerosol, UV GOSAT CO 2 & CH 4 columns ESA ACE BSRN Volume mixing ratio profiles of a large number of atmospheric trace gases Short- & long-wave radiation surface fluxes UV radiation measurements / Tropospheric NO 2 Tropospheric HCHO Stratospheric NO 2 Global NRT reports 1-10 / Stratospheric O 3 Global NRT reports 2-6 WRMC EUVDB NSF DB WOUDC Additional validation data sources quoted in MACC-II global NRT validation reports Use in MACC-II NRT validation reports Programme / Provider Parameters Archive Validated product(s) NRT validation report(s) Remark EMEP CCC Surface concentrations of a large number of tropospheric chemicals NILU Tropospheric O 3 Global NRT report 1 Availability at the time of Global NRT report 1: Sep - Dec 2009 (used for validation of the reanalysis). Many of the 171 stations reporting ozone observations now provide data for 2010 & 2011, and some for 2012 & Data from 1 station (Finokalia) is used for NRT validation. 13 / 18

14 NOAA / ESRL / GMD Atmospheric constituents. For ozone: surface concentration, vertical column, profile. NOAA / ESRL / GMD Tropospheric O 3 MODIS Aerosol dust Global NRT reports 7-10 OD MSR Aerosol dust Global NRT reports 7-10 OD BASCOE IASB-BIRA Stratospheric Global NRT reports 1-10 O 3 Global NRT reports 3-10 Among 49 stations observing ozone, 12 deliver NRT data and are used for NRT validation. Stratospheric Global NRT reports 1-4 & Model NOx & HNO 3 comparison NOx 6 dropped from Report 7 onwards Stratospheric Global NRT report 6 HNO 3 SACADA DLR Stratospheric Global NRT reports 3-10 FRIUUK O 3 TM3DAM KNMI Stratospheric Global NRT reports 3-10 COST UV DB FMI UV index Global NRT reports 8-10 O Data flow Some input observational data flows (satellite, surface, aircraft) have been secured in MACC-II by the OBS sub-project, which also pointed to some obstacles to fetching other input data. This effort is continued under MACC-III in order to take CAMS closer to operational stage Assessment of compliance with specifications Verifying the continued compliance with data specifications is some new task that needs to be implemented for the CAMS. As already pointed out in the SVR [9], the Product and Service Specifications Document [11] should be completed with quantitative information on data quality (QI target values) and service performance Assessment of compliance with user requirements Assessment of the fitness-for-purpose property of the data by its confrontation to user requirements is key to the CAMS operations. Prerequisites to this task include that quality indicators (QI) are an output of the data validation (see Section 4.2.1); that validation itself has been designed fit-for-purpose (see Section 4.2.1); that users are interviewed about their requirements in terms of quantitative QI (see Section 4.1) Validation reporting Validation reporting should be supported by some validation checklist. A common report template ensuring the use of common terminology and set of practices as recommended in the SVR [9], will ensure harmonisation and guide product providers in their validation task, 14 / 18

15 and support users with harmonised, cross-domain (by opposition to domain specific jargon), user friendly quality information. 4.3 Metadata Metadata are essential to enable potential users to locate, identify and access the data they need, to assess their fitness for purpose, understand their meaning and limitations and actually use the data in particular applications. Moreover, in order to allow interoperability, it is desirable that the metadata fulfil applicable existing metadata standards and refer to agreed terminologies. Data exchange standards, together with some associated metadata rules, were agreed within the processing chains of the individual sub-services of MACC-II but a generic data exchange protocol for all services was not planned, and formats were selected in agreement with core users. Note that, to be compliant with INSPIRE rules on Data Specifications, if standard guidelines are not to be applied for meteorology and atmosphere data and services due to the size and dynamics of envisaged data flows, at least one of the three following formats should then be used: GRIB, BUFR and/or CF-NetCDF, which all include some standardisation of metadata. Update, maintenance and possible upgrade of the MACC-II online metadata catalogue is key to CAMS operation. 4.4 Service validation Service validation should examine delivery aspects (access, timeliness, regularity); sustainability (as documented on the MACC-II project relevant webpage); the provision of online automated graphs and tables (with full identification of the data like e.g. data version, type and version of validation data used etc.); the long-term archival and availability of documents, including validation reports. It should encompass the assessment of continued compliance with service specifications; the assessment of service performance against corresponding collected user requirements; service validation reporting. 5 Automation Some of the tasks listed above lend themselves to partial or full automation. When considering automation, the determining factor will of course be the cost/benefit ratio, 15 / 18

16 which needs to be assessed. In particular, some significant cost will be associated to preliminary standardisation or formatting of the pieces of information to be handled. As pointed out in Section 2, it should also be kept in mind that some intelligent doublechecking system should be set up to protect automated operations from unforeseen or undetected drifts. As underlined in the SVR [9] (Section 3.2), the FP7 NORS validation server was already a demonstration of automation capabilities for the validation of CAMS data [12]. Tasks typically likely to be partly or fully automated include input data flow, including data flow monitoring and data QA/QC; output data flow or data delivery, including QA/QC monitoring; comparison to reference data, including automated co-location, representativeness and units conversion, and similar preliminary tasks; comparison of coded tabulated information such as specific product or service characteristics versus their equivalent specifications or user requirements (e.g., QI values); validation reporting; generation of warnings in case of unexpected atmospheric events, unwanted levels of discrepancy between CAMS and validation data... 6 Conclusion and summary of recommendations This section summarises the recommended implementations to near operational functioning of the CAMS QA/Val system, that have been emitted in previous sections. Some of these (in blue) already appear in the SVR [9]. A central governance of QA/Val would ensure a consistent harmonised approach of QA and would offer support and guidance to the data providers. A common validation methodology should be promoted and applied uniformly to all products: validation methods and applicable validation data should be identified from user requirements; emphasis should be put on rigorous traceability and the systematic provision of numerical quality indicators. QA/Val audits would help monitor the progress in QA maturity, provide confidence to the data providers and offer guarantees to the users. More comprehensive use of existing validation data should be optimised. Candidate datasets should be systematically envisaged and data flows secured. Data quality and service performance should be documented in the Product and Service Specifications Document. Data quality and service performance should be systematically evaluated with regard to a representative selection of the most relevant user requirements in order to allow users to assess fitness-for-purpose of the products. 16 / 18

17 7 Names and acronyms ACC CEOS Atmospheric Composition Constellation CAMS COPERNICUS Atmosphere Monitoring Service CEOS Committee on Earth Observation Satellites COPERNICUS Programme for the Establishment of a European Capacity for Earth Observation CORE-CLIMAX Coordinating Earth Observation Data Validation for Re-analysis for Climate Services (EU FP7 project) EC European Commission EO Earth Observation EU European Union FP7 Seventh Framework Programme for Research GAIA-CLIM Gap Analysis for Integrated Atmospheric ECV CLImate Monitoring (EU H2020 project, March 2015 February 2018) GALILEO European Global Satellite-Based Navigation System GEO Group on Earth Observations GEOSS Global Earth Observing System of Systems GMES Global Monitoring for Environment and Security (now Copernicus) GSICS Gobal Space-based Inter-Calibration System GUM Guide to the expression of uncertainty in measurement [13] IEEE Institute of Electrical and Electronics Engineers INSPIRE Infrastructure for Spatial Information in the European Community MACC Monitoring Atmospheric Composition and Climate (EU FP7 project, June 2009 October 2011) MACC-II Monitoring Atmospheric Composition and Climate Interim Implementation (EU FP7 project, November 2011 July 2014) MTG Meteosat Third Generation NORS Demonstration Network of Ground-based Remote Sensing Observations in Support of the GMES Atmospheric Service (EU FP7 project, November 2011 July 2014) QA Quality Assurance QA4ECV Quality Assurance for Essential Climate Variables (EU FP7 project, February 2014 January 2018) QA4EO GEO Quality Assurance Framework for Earth Observation QI Quality Indicator(s) R&D Research and Development SBA Societal Benefit Area(s) SI Système International (International Unit System) SVP Service Validation Protocol SVR Service Validation Report URD User Requirements Document Val Validation WG-Climate CEOS Working Group on Climate WGCV CEOS Working Group on Calibration and Validation 17 / 18

18 8 References 1 Galileo navigation homepage on ESA s website: 2 EC, Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 establishing an Infrastructure for Spatial Information in the European Community (INSPIRE), Official Journal of the European Union, L 108, pp. 1 14, 25 April QA4EO task team, A Quality Assurance Framework for Earth Observation: Principles, v4.0, 19 pp., 14 January Collins English Dictionary online: 5 Webster s Online Dictionary: 6 Lambert, J.-C., Atmospheric Service Validation Protocol, MACC-II Deliverable D_153.1, May Bates, J. J., and J. L. Privette, A maturity model for assessing the completeness of climate data records, Eos, Transactions American Geophysical Union, 93, p. 441, October Holzer-Popp, T., L. Klüser and F. Schnell, User Requirements Document v5.0, MACC-II Deliverable D_141.10, July De Rudder, A., Service Validation Report, MACC-II Deliverable D_153.3, July Eskes, H., V. Huijnen, A. Wagner, M. Schulz and E. Botek, ed., Validation report of the MACC near-real time global atmospheric composition service, System evolution and performance statistics, Status up to 1 March 2014, MACC-II Deliverable D_82.12, June 2014 (last issue out of ten). 11 Engelen, R., X. Yang and M. Razinger, Product and Service Specifications Document V2, MACC-II Deliverable D_153.4, July Breebaart, L. and S. Niemeijer, Validation server design document, NORS Deliverable D8.2, July JCGM, Evaluation of measurement data Guide to the expression of uncertainty in measurement (GUM), v2, JCGM 100:2008, / 18