Specification of the EUMETNET operational Weather Radar Data Center

Transcription

1 Specification of the EUMETNET operational Weather Radar Data Center OPERA, Working document, WD_2008_02 5 June 2009 J.-L. Chèze, S. Hafner, I. Holleman, S. Matthews, and D. Michelson

2 Executive Summary In line with the Program decision for the third phase of OPERA a specification document for the EUMETNET operational Weather Radar Data Center has been composed. For this a project team consisting of OPERA delegates from France, Germany, Netherlands, Sweden, and United Kingdom has been formed and during the last year the team has had several discussion and working meetings. In addition contacts with important users (for NMSs) of continental-scale weather radar data have been established and user requirements have been discussed. At the April 2008 plenary meeting of OPERA this specification document has been presented to the whole group, discussed in detail, and approved. The specification document starts with an evaluation of the pilot Data Hub which has been developed during the second phase of OPERA and is now running for several years. This experience has been extremely helpful for the project team and important lessons are learnt. Furthermore a review of other operational radar data centers has been performed and a summary is given in this document. Four priority user groups (Forecasting and nowcasting, Numerical Weather Prediction, Aviation, and Hydrology) of radar data from NMSs have been identified and their user requirements for continental-scale weather radar data are presented. There is a strong requirement for a European radar composite, quality information, and operational exchange of radar volume data (to improve composite quality and for assimilation in NWP). From the user requirements three functional options for the operational radar Data Center are proposed. The functional specifications for these options and some general specifications are given in this document. The data center options are defined according to increasing data flows, complexity of processing, and content of products: 2D European composites from 2D single-site products (Option 1), 2D composites from polar volume data (Option 2), and 3D composites from polar volume data (Option 3). The last option may appear futuristic but it already exists on a continental-scale in the USA operated by the National Weather Service. Within EUMETNET the possible centralization of data centers is being discussed and it is now proposed to only centralize monitoring (both automated and interactive) and (internet) visualisation. For the production it is proposed to standardize and integrate the operational data centers in so-called External Data Collection and Production Centers (DCPC) in the framework of the new WMO Information System. The radar products from the operational Data Center will be included in the ECOMET catalogue of the NMS hosting the Center and thus all existing ECOMET regulations will apply. EUMETNET/OPERA WD_2008_02

3 Table of Contents Executive Summary Introduction Evaluation of Pilot Data Hub Other radar data centers CWINDE data hub for wind profiles NORDRAD and BALTRAD Data Centers US National Mosaic and Quantitative Precipitation Initiative User requirements for European-scale radar data Approach Core services forecasting and nowcasting Numerical weather prediction Assimilation Numerical weather prediction Verification Civil and military aviation Hydrology and water management Summary Functional specifications and options Options for radar Data Center Input to Data Center Monitoring and display Data Processing Operational resilience and portability Change and Release management Products of Data Center Development and running costs Benefits for European users and institutions Core services forecasting and nowcasting Numerical weather prediction Assimilation Numerical weather prediction Verification Civil and military aviation Hydrology and water management Summary External factors EUMETNET Centralized Observation Monitoring and Production GEOSS and GMES WMO Information System ECOMET Data policy Next steps and timeline...32 Appendix A...33 EUMETNET/OPERA WD_2008_02

4 1. Introduction During OPERA-2 a pilot Data Hub (PDH) has been established, collecting and storing single-site and national composite data, and producing the first prototype European-wide composite for quality control purposes and official duties of the NMSs. Around 100 single-site products have been contributed to this European composite thus far. This was a major achievement of the second phase of the OPERA program. During this phase of OPERA an operational Data Collection and Production Center for the weather radar network should be specified, developed, and operated. This operational Data Center is crucial for reaching the main objective of OPERA-3, i.e., establishing the weather radar networking as a solid element of the European infrastructure. The operational Data Center will enhance and monitor availability of radar data, facilitate quality control of single-site radar data, stimulate exchange of volume radar data and quality information, and produce an homogeneous European radar composite. Furthermore the Data Center will deliver radar data to users inside the NMSs for official duties and also to other users under ECOMET regulations. The specifications of the operational Data Center have been drafted by a project group of OPERA with delegates from France, Germany, Sweden, and the United Kingdom and chaired by the OPERA Program manager. This project group had two dedicated discussion and working meetings in between the general OPERA meetings, and in addition project group members paid visits to potential users and operators of other data centers. The kick-off meeting was held in June 2007 and was devoted to user requirements and external factors (e.g. data policy). The second meeting in February 2008 focused on functional specifications. Experiences with the Pilot Data Hub and other operational radar data centers have been considered carefully. This document starts in Chapter 2 with an evaluation of the pilot Data Hub of OPERA which has been running for a few years now. Subsequently a summary of a review of other operational radar data centers is presented. In Chapter 4 the user requirements for an operational Data Center of OPERA as collected by the subgroup are given. The functional specifications of the operational Data Center and options are described in Chapter 5. The key benefits of the different data center options for European users and institutions are highlighted in Chapter 6. External factors impacting on the operational Data Center, like EUMETNET Centralized Observation Monitoring and Production, WMO Information System, and ECOMET data policy, are discussed in Chapter 7. The last chapter is devoted to the next steps and timeline. EUMETNET/OPERA WD_2008_02

5 Figure 1: Single-site radar data (left image) and national composites (right image) that are being delivered to the pilot data hub. 2. Evaluation of Pilot Data Hub (Based on OPERA working documents WD_2006_11 and WD_2008_01) In June 2004 PB-OBS identified a requirement for the creation of a central weather radar hub and suggested that the establishment of such a facility should become the subject of a funded work package under the OPERA-2. A proposal submitted by the UK Met Office was selected by the group. The overall aim of this pilot weather radar data hub (hereafter referred to as the Pilot Data Hub or PDH) was to facilitate the production of Europe-wide high quality rainfall products for use in research and development activities, particularly Numerical Weather Prediction (NWP). The Pilot Data Hub was scoped to handle data from up to 100 radar sites. Its specific functions would be to 1. receive radar data from participating organisations 2. perform agreed quality control and correction procedures 3. maintain and operate a data storage and retrieval system 4. disseminate data and products to participating members for use in research and development activities 5. inform the specification of an operational data hub. The work package proposal defined two types of composite products. The first one to be based on National or Regional composites, whose data have undergone prior processing. The second based on single-site data, supplied in an unprocessed form, which would be subject to a set of quality control and correction algorithms applied within the Pilot Data Hub system, prior to compositing. EUMETNET/OPERA WD_2008_02

6 The Pilot Data Hub shares hardware with the UK Met Office s centralized radar data processing system (Radarnet). This is based on a UNIX cluster, which was expanded from two to four processing nodes in May 2005 to create capacity required for the Pilot Data Hub. The cluster is split across the two IT halls at the Exeter site to maximize resilience. The cluster management software will automatically switch packages from one processing node to another in the event of hardware/system failures on any one node. In the event of, say, loss of facilities (e.g. power failure) in a single IT hall, all operational packages can be run on the two processing nodes within a single IT hall. During OPERA-2 the Pilot Data Hub has been established, collecting and storing single-site and national composite data, and producing the first prototype European-wide composite (see first page of this document for example) for quality control purposes and official duties of the NMSs. Around 100 single-site products and many national composites have been contributed to this European composite thus far (see Figure 1). The original OPERA-2 budget for the development of the PDH was 50k, but total costs of the Met Office internal project were nearer to 300k. The Met Office project included work already planned, including the procurement of new hardware to provide resilience across two IT Halls. The actual costs of processing the extra sites, creating the OPERA composites and developing the external web site was approximately double the original OPERA-2 budget. The development of the PDH web site accounted for approximately 25% of the development costs. Of the radar processing costs, approximately 75% of these were dedicated to the configuration of various input projections and products The project team working on the specification of the operational data hub has evaluated the pilot data hub and has summarized the lessons learnt : Used Functionality FTP server for the delivery of products, used by most OPERA members FTP access also given to ECMWF, EUMETSAT and EUMETNET Extensive use of European composite with maximum coverage (i.e. combination of single site and composite products) Some qualitative use of the processed single site data Unused Functionality Website is hardly used Images on website of single-site products not at all Statistics are not used much Lessons (The good things) EUMETNET/OPERA WD_2008_02

7 A PDH was developed quickly by using existing system High availability due to running on existing operational hardware Higher profile of OPERA and radar activities in Europe There is much good will and enthusiasm for a European Radar data hub Quickly incorporated 100 sites into PDH and continues to grow Lessons (The not so good things) The acceptance of all data enabled wide geographic coverage but at the expense of a high quality product and significantly increasing development cost Apart from forecasters, who wanted a general overview of the European situation, little use is made within European NMSs. There was no User Requirement The functionality of and requirements for a web site were not fully understood The reliability of the GTS and local GTS nodes resulted in blinking composites (absence and presence of parts of the composite) The lessons learnt from the Pilot Data Hub will be used in the specification and development of the operational radar data center. EUMETNET/OPERA WD_2008_02

8 3. Other radar data centers (Based on OPERA working document WD_2007_16) In this chapter a short review of relevant other operational radar data centers, most notably the CWINDE data hub, the NORDRAD/BALTRAD datacenters, and the US National Mosaic and Quantitative Precipitation Initiative (NMQ), is presented. This review will be of use for setting the functional requirements in the next chapters and also led to important liaison between OPERA and the other radar data center operators. 3.1 CWINDE data hub for wind profiles The CWINDE data hub is operated as part of the EUMETNET program WINPROF-2 focused on wind profilers and weather radar wind profiles. In this program Austria, Finland, France, Germany, Hungary, Ireland, The Netherlands, Switzerland and the United Kingdom are participating. CWINDE can be accessed via internet using: data hub was developed by the UK Met Office as part of the COST-76 Action of wind profilers and it is now running for more than 10 years. Currently the CWINDE data hub collects wind profiles from 27 wind profilers and 91 weather radars. The profiles are displayed on the internet (see Figure 2) and monthly statistics are determined. In short CWINDE s characteristics are: Figure 2: Left image shows the weather radar sites that currently are supplying real-time wind profile data to CWINDE. The right plot shows an example of the wind profiles from the weather radar in Den Helder (NL). EUMETNET/OPERA WD_2008_02

9 Data collection from the different wind profiler types and weather radar (wind data only); A total of about 14,000 messages per day; Real time web display for quality control purposes; Data monitoring (data availability, comparisons against UK NWP system, NWP assimilation statistics, monthly reports); Data distribution to the meteorological data bank of the Met Office, ECMWF and other GTS centres, Web interface, FTP link Data archiving No user requirements exists, evolutionary development No data processing and product generation at data hub The CWINDE data hub is running on a single Linux server and 0.3 FTE of a technical officer is available for operational maintenance. 3.2 NORDRAD and BALTRAD Data Centers Figure 3: Example of BALTRAD radar composite EUMETNET/OPERA WD_2008_02

10 NORDRAD is the operational weather radar network in the Nordic countries, with member countries Finland, Sweden, and Norway. Estonia and Latvia are associated members, and Danish data is included by special agreement. There are currently 32 radars in NORDRAD. BALTRAD is a virtual network used for R&D purposes by and for the Baltic Sea Experiment (BALTEX), which is the European GEWEX Hydroclimate project. In addition to the countries contributing to NORDRAD, BALTRAD contains radar data from Poland (8) and Germany (5). Wind profiles from the Netherlands (2) were once part of BALTRAD, but with the increased availability of WRWPs on the GTS, BALTRAD is no longer a driving force for such data. BALTRAD datasets are produced by the BALTEX Radar Data Centre (BRDC) at SMHI. The NORDRAD network is managed through an activity of the same name, within the NORDMET framework of meteorological institutes. An operations subgroup deals with the technical details. Radar R&D in NORDMET is organized in another Activity called NORDMET Radar Applications (NORA), which holds annual workshops together with other interested participants, e.g. IMO, DWD, KNMI, MeteoCat, LEGMA, Rosshydromet, and others. Common R&D projects, dealing with e.g. beam propagation and VPR issues, are conducted in NORA. Output radar composites commonly have a 2x2 km horizontal resolution, 15 minute temporal resolution (NORDRAD exchange is regulated at 15 minutes), and 8-bit depth reflectivity. Higher-order products based on these composites, e.g. gauge-adjusted QPE, are not produced in every country and are not regulated by the NORDRAD cooperation agreement. An example composite is shown in Figure US National Mosaic and Quantitative Precipitation Initiative The information in this section is taken from the website ( several publications (Langston et al 2007, JAOT; Zhang et al 2005, JAOT; Zhang et al 2004, ERAD), and an interview and s with Dr. Jin Zhang (Cooperative Institute for Mesoscale Meteorological Studies, University of Oklahoma, and National Severe Storms Laboratory, Norman, OK). The National Mosaic and Quantitative Precipitation Initiative (NMQ) project is a joint initiative between the National Severe Storms Laboratory, Federal Aviation Administration, National Weather Service/Office of Hydrologic Development, the Office of Climate, Water and Weather Services and the University of Oklahoma Cooperative Institute in Mesoscale Meteorological Studies. EUMETNET/OPERA WD_2008_02

11 Figure 4: Snapshot of National Radar Mosaic website The NMQ serves as an international testbed for research, development, evaluation and science to operations infusion of high resolution 3D Mosaic of multiple radars and radar networks for model assimilation and aviation applications, Quantitative Precipitation Information (QPI) including Multiple Sensor Quantitative Precipitation Estimation (MSQPE) and Very Short Term Quantitative Precipitation Forecasts (VSTQPF) for the monitoring and warnings of floods and flash floods and in support of comprehensive hydrology and ecosystem modeling. The continental US domain is divided into eight so-called "tiles" covering lower 48 states and southern Canada. The tile domain size is a function of number of radars and climate. On average each tile contains about 35 weather radars. This is designed to balance computational load. The composite calculated for each tile has a spatial resolution of 0.01 degrees (about 1 km) and it has 31 constant height levels. The National Mosaic has a 5-minute update cycle. A snapshot of the website of the National Radar Mosaic is shown in Figure 4. EUMETNET/OPERA WD_2008_02

12 4. User requirements for European-scale radar data 4.1 Approach The project group for the specification of the operational data hub has put a significant effort in collecting the user requirements for European-scale weather radar data. For this the group has identified the four most important user groups of weather radar data for the National Meteorological Services and thus for the OPERA program. These priority user groups for the OPERA radar data center are: Core services forecasting and nowcasting, e.g. National Meteorological Services, EUMETSAT Numerical Weather Prediction (Assimilation and Verification), e.g. EUMETMET-SRNWP and underlying consortia, ECMWF Civil and military aviation, e.g. Eurocontrol Hydrology and water management, e.g. JRC Ispra For each of these groups the user requirements have been collected. Note that for Numerical Weather Prediction (NWP) the user requirements for assimilation and verification have been treated separately. At the first meeting of the project group (Norrköping, June 2007) a first guess of the user requirements for the priority groups has been made. Subsequently, several user groups inside and outside the NMSs have been contacted and feedback on the first guess was obtained. Amongst others the following contacts with users were established and/or dedicated visits were paid: Inside NMSs of project group members (DWD, KNMI, Météo France, SMHI, UKMO). The user requirements were presented or distributed within the institutes and feedback was collected. EUCOS: Reports from project group meetings were sent to EUCOS program team and feedback was obtained. In addition OPERA program manager participated in meetings organized by EUCOS on Centralized data hub. SRNWP: The user requirements for NWP assimilation and verification were sent to the SRNWP program manager. He distributed the user requirements and collected feedback from several modeling consortia and NMSs (Denmark, France, Germany, Italy, Poland, Slovakia, Switzerland, HIRLAM). In addition a summarized feedback on assimilation and verification requirements was provided. ECMWF: A meeting with data assimilation and verification experts was organized (24 September 2007). COST 731 (Propagation of uncertainty in hydrometeorological models): The user requirements for NWP and hydrology have been presented at a plenary meeting (October 2007) and some feedback was received. EUMETNET/OPERA WD_2008_02

13 JRC Ispra (Flood forecasting): Contacts with JRC were established and draft user requirements for hydrological applications were sent. JRC showed interest in the European-composite radar data (test dataset will be provided) and supported the draft user requirements. Eurocontrol: A visit to Maastricht Upper-Air Control was paid by an OPERA delegation (KNMI, DWD) on 16 August At this meeting also a delegate from Eurocontrol Head Quarters was present. The use of radar data by MUAC and the draft user requirements for aviation from OPERA were presented and discussed in detail. Eurocontrol has indicated which high-level requirements are important for aviation and this information has been used to improve the user requirements for aviation. OPERA-2: During the second phase of OPERA, the use of radar data by operational user communities has been evaluated (WP1.3, Deliverable OPERA_2005_17). Different radar data users (Hydrology, Aviation, Meteorology and Climatology, Road and Railway, Agriculture, and Hydropower) in Finland, France, Germany, Norway, Spain, and Sweden were interviewed and the final reports from COST 717 (Use of Radar Observations in Hydrological and NWP Models) and CARPE DIEM (FP5, Improvement of rainfall accumulations) were reviewed. Input for user requirements has been extracted from this deliverable. During the second meeting of the project group (Dublin, October 2007) the first guess of the user requirements for all groups was updated and at the third meeting (Offenbach, February 2008) the user requirements given below were finalized. 4.2 Core services forecasting and nowcasting The following user requirements on the European weather radar data are obtained for the core service forecasting and nowcasting applications: 2D reflectivity composite and radial velocity imagery now, and 3D composite of Z and V in (near) future Timeliness, at least before next product High update frequency, 5-15 min Horizontal resolution km 8-bit depth of data Quality information: e.g. coverage, blockage, height of observation Non-precipitating echoes removed Homogeneous behavior, e.g. no boundaries High availability and wide coverage 4.3 Numerical weather prediction Assimilation EUMETNET/OPERA WD_2008_02

14 The following user requirements on the European weather radar data are obtained for data assimilation in Numerical Weather Prediction models: Single-site volume data of reflectivity and radial velocity 2D Reflectivity single-site data when height is known Weather radar wind profiles Hydrometeor classification from dual-polarization radars Timeliness, ~15 minutes Update frequency, 5-15 minutes Horizontal resolution 1-5 km 8-bit depth data Detailed quality information for all data Stringent removal or identification of non-precipitating echoes No data better than bad data 4.4 Numerical weather prediction Verification For verification of NWP models, the following user requirements on the European weather radar data are obtained: Surface rainfall accumulation (1 hour) composite Hydrometeor classification from dual-polarization radars 3D Cartesian composite of wind and reflectivity Timeliness not relevant Update frequency, 5-15 min Horizontal resolution 1-4 km 8-bit depth of data Quality information: e.g. coverage, blockage, height index Non-precipitating echoes removed Archive 4.5 Civil and military aviation The following user requirements on the European weather radar data are obtained for civil and military aviation: 2D reflectivity composite and echotop imagery Also wind profiles, horizontal wind analyses, hydrometeor classification 3D composite of Z, V, and W in future Timeliness: instantaneous for some applications! High update frequency, e.g. <=5 min Horizontal resolution km 8-bit depth of data Non-precipitating echoes removed Certification, e.g. ISO 9001, SES EUMETNET/OPERA WD_2008_02

15 4.6 Hydrology and water management For hydrological applications and water management, the following user requirements on the European weather radar data are obtained: Surface precipitation intensity and accumulation composite Hydrometeor classification, especially rain/snow Timeliness, at least before next product High update frequency, e.g min Horizontal resolution km Stringent removal of non-precipitating echoes Floating point data or 16-bit for accumulations Quality information All processing and corrections applied, including real-time gauge adjustment if possible 4.7 Summary The User Requirements from the identified user groups, i.e., forecasting and nowcasting, NWP assimilation and verification, civil and military aviation, and hydrology and water management, have been summarized in Table 1. Common requirements have been highlighted in the table as well. There is a clear requirement for operational exchange of single-site volume data (3D), either directly or indirectly via the requirement for a 3D radar composite. Obviously a European radar composite, 2D for now but 3D in near future, is strongly required by most user groups. Apart from radar reflectivity, also surface rainfall and radial velocity are required by several user groups. The amount of processing on the data required by the users varies from stringent removal of non-precipitating echoes only to application of all kinds of correction algorithms and real-time gauge adjustment. Data quality and homogeneity are relevant for all user groups, but several users also require detailed (spatial) information on the data quality of the radar products, e.g. beam height and coverage. The required update frequency varies from once up to three times per 15 minute and the timeliness is very important (maximum 15 minutes delay is acceptable, but shorter delays are really appreciated). With respect to the horizontal resolution of the products the user requirements range from 5 km down to 0.5 km. EUMETNET/OPERA WD_2008_02

16 Table 1: Summary of user requirements for European weather radar data. Forecasting and Nowcasting NWPassimilation NWPverification Civil and military aviation Hydrology and water management All Single-site data Radial V imagery Volume data of Z and V, 2D Reflectivity 2D Surface rainfall intensity Volume data of Z and V, 2D Data Products and 2D composites 3D Composite 2D Reflectivity composite Z and V in (near) future Wind profiles, Hydrometeorclassification Surface rainfall accumulation (1h), Hydrometeor classification Z and V 2D Reflectivity composite, Echotop imagery, Wind profiles, Hydrometeor classification, Mostnotably Z, but also V and maybe W Surface rainfall accumulation (1h), Hydrometeor class (rain/snow) 2D Reflectivity composite, Surface rainfall accumulation, Hydrometeorclassification, 3D Composite of Z and V Z and V Horizontal km 1-5 km 1-4 km km km km resolution Data resolution 8 bit 8 bit 8 bit 8 bit 16 bit or floating point 8 bit Update frequency 5-15 Minutes 5-15 Minutes 5-15 Minutes <5 Minutes 5-15 Minutes 5-15 Minutes Timeliness Before next product 15 Minutes Not relevant 0-5 Minutes Before next product 0-15 Minutes Availability High High High High High Non precipitating echoes Homogeneous composite Processing Quality information Removed Stringent removal or identification Removed Stringent removal Yes Yes Yes Yes Yes Coverage, Blockages, Height of observation. Detailed for all data VPR, Attenuation Coverage, Blockages, Height of VPR, Attenuation Yes Stringent removal VPR, Attenuation Coverage, Blockages, Height of observation observation. Archiving Yes Yes Certification ISO/SES ISO/SES EUMETNET/OPERA WD_2008_02

17 5. Functional specifications and options After gathering and analyzing the user requirements, the project group produced the following functional specification. The functional requirements were considered for three possible data center options. It is noted that EUMETNET is discussing a centralized observation monitoring and production (see Section 7.1), and when running the operational data center could be positioned as a Data Collection and Production Center (DCPC) according to WMO Information System regulations. 5.1 Options for radar Data Center The following options have been considered by the OPERA project group for the operational radar data center: Option 1: 2D Compositing of 2D single-site products Option 2: 2D Compositing of polar volume data Option 3: 3D Compositing of polar volume data Note that the level of complexity, data flows, and processing are increasing when moving from option 1 to 3. Option 1 is basically an operational version of the pilot Data Hub where only single-site products are accepted. Option 3 may seem undoable and risky but it actually exists on a continental scale in the USA (see Chapter 3.3). 5.2 Input to Data Center General For all three options, only singles-site data will be accepted. National and Regional composites will not be accepted (as was the case for the pilot Data Hub). Availability and quality acceptance targets should be set for all incoming data. These should include: o Annual availability targets (Suggest greater than 90%) o Change control procedures at supplying NMS to be agreed. o Procedures for informing data hub support team of changes to input data need to be set up o Quality standards for acceptance o Agreed acceptance testing period Archiving of input data is the responsibility for the supplying NMS and is a low-priority requirement for all data hub options. An option for the redistribution of input data, possibly onto V-GISC (see Chapter 7.2) or just using a FTP account, should be considered e.g. for NWP data assimilation. EUMETNET/OPERA WD_2008_02

18 Data types The operational Data Center will receive data as either 2D singe-site reflectivity (Option 1 only), or Polar Volume (Options 2 and 3) 2D single-site reflectivity (Input Option 1) Only pseudocappi (lowest height) or lowest useable data (height information for each pixel will be required for this product) Reflectivity values only (Z) 8 bit depth of data Quality information should be supplied following the OPERA quality framework. See OPERA work packages for definitions Horizontal resolution of 2 km or better The projection should be as close to original site-specific Cartesian projection or the projection of the data center composite. The fastest possible communication should be used (FTP, RMDCN) The data should arrive earlier than 5 minutes before composite creation time. Ideally DataTime+5 minutes (DT+5) but mandatory by DT+10. Mandatory for data to be provided in the official OPERA formats (BUFR and HDF5) only and according to a single information model Polar Volume (Input Options 2 and 3) Polar volume data Reflectivity (Z) and Wind (V) 8 bit depth of data with sufficient level of detail Quality information should be supplied following the OPERA quality framework. The bin resolution should be 1 degree or better and of 2 km or better length (desirably less than 1 km) The fastest possible communication should be used (FTP, RMDCN) For the 15 minute composite options an arrival time earlier than 5 minutes before composite creation time is required. A DT+5 is desirable and mandatory by DT+10. For the 5 minute composites a DT+5 would be mandatory. Note that the dissemination strategy for volume data complicates the concept of timeliness. Some NMS may wish to send the whole volume in a single file. Other may wish to send data scan by scan. To meet user requirements the strict time limits are necessary Mandatory for data to be provided in the official OPERA formats (BUFR and HDF5) only and according to a single information model 5.3 Monitoring and display EUMETNET/OPERA WD_2008_02

19 Operational monitoring of the arrival of the data from the contributing weather radars is essential. It is noted that EUMETNET is discussing a centralized observation monitoring and production (see Section 7.1), and when running this facility could (partly) provide the functionality required below. Monitoring arrival The monitoring of data arrival will be a mandatory requirement for the data center. Display of arrival logs is optional. Arrival warnings Mandatory for automatic warnings of non-delivery of data to be produced. A mechanism for the dissemination of warning messages to agreed distribution list is also mandatory. Arrival statistics Mandatory to produce arrival time statistics. A feed-back mechanism should be developed. Removal of radar from network procedures Procedures for the manual removal of radars from the network/composite should be developed. These procedures should take into account user requirements. Display of radar data The operational data center will output the European composite in some kind of graphical format (see below). Monitoring of quality Options for the monitoring of quality should be included. Some possible options include: Radar versus Radar Radar versus Gauge (available on GTS for 6 and 18 UTC) Radar versus Sun Radar radial velocity versus NWP (in collaboration with NWP center) 5.4 Data Processing To provide a product of homogeneous quality, processing of input data should form part of the data center. The following processing options should be considered but are not mandatory. Other options will be considered. Blockage Residual clutter removal (e.g. by using Meteosat data) VPR (Options 2 and 3) Gauge adjustment Attenuation correction (Options 2 and 3) EUMETNET/OPERA WD_2008_02

20 Data may arrive having already passed through some or all of these procedures. Therefore, it is mandatory for each processing stage to be optional for each site. Description of the methods being applied should be documented and made available. 5.5 Operational resilience and portability The data center hardware should be designed and operated as a truly operational system including a fully identical hot and cold server, and possibly also server for development and testing. The operational redundancy could also be implemented in a virtual data center where the backup systems run at different locations. The data center is expected to have Service Level Agreements (SLAs) with customers and these SLAs will be used to produce a minimum Operating Level agreement (OLA) for the Data Center as a whole. The data center may have more than one standard Service level depending on the requirements of the customer. The OLA will contain targets for availability, timeliness, problem resolution expectations and possibly quality. Based on the user requirements specified earlier in this document, Appendix A illustrates the likely Operating Level of the data center. Software and hardware of the operational data center will be assets of the OPERA program and must be transferable to the next responsible member at the end of the program. This implies that the software developed for the operational data center should be portable and not be tied to the ICT infrastructure of the host NMS. 5.6 Change and Release management The operational data center should provide the capability and capacity to allow for the testing of: new software; new algorithms; new sites; etc; before deployment into the operational environment. The data center should therefore have development, testing, next release, parallel running functionality. All of these functions should have no impact on the operational service. An option for the testing of new algorithms using case studies should be considered. Acceptance procedures before operational deployment of (new) algorithms should be developed. It is suggested to establish an OPERA subgroup for EUMETNET/OPERA WD_2008_02

21 coordination and preparations of decisions. Major changes must be agreed in the plenary meetings of OPERA. 5.7 Products of Data Center General Mandatory for quality information to be included with all composite products Output products would ideally be produced in a single format. An option for a single graphical format (e.g. png) should be considered Composite products should be created on a single domain. Mandatory for products to be available on an FTP sever, connected to RMDCN Options for products to be available on V-GISC, GTS and on a web site should be considered The data center should ensure that products are available to all NMS at agreed consistent time (e.g. range DT+15 to DT+20) All composite products generated by the data center should be archived. A method or procedure for extracting data from the archive should be demonstrated 2D Composite (All Options) The following features of the 2D composite will be required European domain Reflectivity, Surface rain rate and 1 hour accumulation Horizontal resolution: 2 km Updated every 15 minutes (Option 3 with optional 5 minute update) Available at DT+15 (Option 3 DT+10 or better if possible) Depth 8 bit Strict time window for input to be included Availability > 99% Projection to be specified to meet the user requirements 3D Composite (Reflectivity), Option 3 only In addition the 3D composite in Option 3 should: 15 minutes or 5 minutes Vertical resolution of 500 m Height max 15 km Vertical resolution 2 km 3D Wind vector composite, Option 3 only 15 minutes or 5 minute Vertical resolution of 500 m EUMETNET/OPERA WD_2008_02

22 Table 2: Estimated development effort and running costs. Option 1 Option 2 Option 3 Development effort x x x Hardware costs x x x Maintenance effort x x x Operating costs x x x Total costs 1) x x x 1) Total costs estimates include development and two years of operation assuming 6 k /month, i.e. not according to integral costs. Thus when taking into account the integral costs the total costs may double. Height max 12 km Horizontal 2 km u, v and w 5.8 Development and running costs The project group has made an attempt to estimate the development and running costs of the operational radar Data Center. The estimations have been made for the three proposed options. For this the following cost items have been considered: development effort, hardware costs (computers and mass storage), maintenance effort (production and monitoring), and operating costs (communication, licenses, maintenance). Table 2 lists the estimated costs for the three options. It is stressed that these figures are indicative only and the costs can only be fixed when tenders for development and operation of the data center have been received. It is noted that a conversion of the OPERA Pilot Data Hub into an operational radar data center is not a sensible option. First of all, this system is not satisfying the requirements imposed by European user groups for e.g. homogeneity of products and by the EUMETNET Council for e.g. portability and open selection of responsible member. Additionally, the development and running costs of the pilot data hub as an operational radar data center will only be marginally cheaper than Option 1 (Limited development still needed and other costs are similar). The UK Met Office has developed and run the Pilot Data Hub at a (small) fraction of the total integral costs. EUMETNET/OPERA WD_2008_02

23 6. Benefits for European users and institutions In this chapter the key benefits of the different data center options for European users and institutions are discussed. Basically a matching between the User Requirements from Chapter 4 and the Functional Specifications from Chapter 5 has been performed. The key benefits of the different options for the priority users groups are highlighted below. 6.1 Core services forecasting and nowcasting From Option 1 for the operational weather radar data center the following key benefits are expected for core services forecasting and nowcasting: Operational availability of continental-scale radar reflectivity composites for monitoring of (severe) weather and issuing of warnings Improved timeliness of radar products due to FTP data transfer instead of GTS distribution Access to some quality information on products and possibility for limited use in quantitative applications Potential cost saving due to reduced need for generation of large-scale composite at National Meteorological Services Radar data center can serve as back-up for national infrastructure From Option 2 for the operational weather radar data center the following additional benefits are expected: Increased cross-border continuity and homogeneity of the continentalscale products due centralized processing and product generation Improved quality and quality information for the continental-scale products Operational availability of continental-scale radar velocity products From Option 3 for the operational weather radar data center the following benefits are expected in addition to those listed above: Operational availability of three-dimensional continental-scale reflectivity and velocity products enabling e.g. interactive cross-sections and quantitative applications Increased update frequency and spatial resolution of the products 6.2 Numerical weather prediction Assimilation From Option 1 for the operational weather radar data center the following key benefits are expected for core services forecasting and nowcasting: Operational continental-scale radar reflectivity and surface-rainfall products may be used for data assimilation, e.g. latent heat nudging EUMETNET/OPERA WD_2008_02

24 From Option 2 for the operational weather radar data center the following additional benefits are expected: Operational access to the polar volume data (reflectivity and velocity) from the European weather radar network boosting the data assimilation in (non-hydrostatic) NWP models Increased cross-border continuity and homogeneity of the continentalscale products due centralized processing and product generation Improved quality and quality information for the continental-scale products From Option 3 for the operational weather radar data center the following benefits are expected in addition to those listed above: Operational availability of three-dimensional continental-scale reflectivity products enabling e.g. more-sophisticated latent heat nudging Increased update frequency of polar volume data and the continentalscale products 6.3 Numerical weather prediction Verification From Option 1 for the operational weather radar data center the following key benefits are expected for core services forecasting and nowcasting: Operational availability of continental-scale surface-rainfall and accumulation products for verification of NWP forecasts Access to some quality information on products Archive of continental-scale weather radar products for verification studies From Option 2 for the operational weather radar data center the following additional benefits are expected: Increased cross-border continuity and homogeneity of the continentalscale products due centralized processing and product generation Improved quality and quality information for the continental-scale products Operational availability of continental-scale radar velocity products From Option 3 for the operational weather radar data center the following benefits are expected in addition to those listed above: Operational availability of three-dimensional continental-scale reflectivity and velocity products enabling more detailed verification of the models Increased update frequency and spatial resolution of the products 6.4 Civil and military aviation From Option 1 for the operational weather radar data center the following key benefits are expected for core services forecasting and nowcasting: EUMETNET/OPERA WD_2008_02

25 Operational availability of continental-scale radar reflectivity composites for monitoring of (severe) weather and optimizing air traffic Improved timeliness of radar products due to FTP data transfer instead of GTS distribution Access to some quality information on products Improved data integrity by use of radar products from single source Potential cost saving due to reduced need for generation of regional composites From Option 2 for the operational weather radar data center the following additional benefits are expected: Increased cross-border continuity and homogeneity of the continentalscale products due centralized processing and product generation Improved quality and quality information for the continental-scale radar products Operational availability of continental-scale radar velocity products From Option 3 for the operational weather radar data center the following benefits are expected in addition to those listed above: Operational availability of three-dimensional continental-scale reflectivity and velocity products enabling e.g. interactive cross-sections and matching of radar data to three-dimensional air space blocks Increase update frequency and spatial resolution of the products 6.5 Hydrology and water management From Option 1 for the operational weather radar data center the following key benefits are expected for core services forecasting and nowcasting: Operational availability of continental-scale radar surface-rainfall and accumulation products for monitoring of (intense) precipitation and issuing of warnings Improved timeliness of radar products due to FTP data transfer instead of GTS distribution Access to some quality information on products and possibility for limited use in quantitative applications Archive of continental-scale weather radar products for verification studies From Option 2 for the operational weather radar data center the following additional benefits are expected: Increased cross-border continuity and homogeneity of the continentalscale products due centralized processing and product generation Improved quality and quality information for the continental-scale products EUMETNET/OPERA WD_2008_02

26 Table 3: User requirements versus functional specifications for operational data hub options. Non compliance is indicated by (double) minus, neutrality by =, and compliance by (double) plusses. Option 1 Option 2 Option 3 From Option 3 for the operational weather radar data center the following benefits are expected in addition to those listed above: Increase update frequency and spatial resolution of the products 6.6 Summary Single-site data = Products and 2D composites = D Composites Horizontal resolution Data resolution = Update frequency Timeliness Availability Non precipitating echoes = + + Homogeneous composite Processing Quality information = + ++ Archiving Certification Table 3 lists the user requirements for the operational Data Hub of OPERA versus the functional specifications associated with the three proposed options. For all user requirements identified in Table 1 and all proposed options, it has been indicated to what extent the specifications meet the requirements. It is obvious that Options 2 and 3 meet most of the user requirements for an operational weather radar Data Center. The requirement for starting an operational exchange of polar volume data is strong both for obtaining more EUMETNET/OPERA WD_2008_02

27 homogeneous composite products and for assimilation of radar data in NWP. Clearly Option 1 fails to meet this important requirement. EUMETNET/OPERA WD_2008_02

28 7. External factors 7.1 EUMETNET Centralized Observation Monitoring and Production At its 31 st meeting the EUMETNET Council agreed on work to scope the requirements of a new E-Programme with a centralised data hub and quality monitoring function. The Council authorised the development of a proposal for a new E-Programme on a centralised data hub by a drafting group. As members for this drafting group were nominated: EUCOS Team, WINPROF, OPERA and E-GVAP Programme Managers, E-AMDAR Programme Manager; Chairman: EUCOS Programme Manager. The group should deliver results for the PB-OBS meeting in June The evolution of the current data hubs within EUMETNET is dependent on the individual expansion of the programmes concerned. In most cases the hub configuration depends on available expertise and resources in the participating NMSs and developed by stepwise modifications. User requirements do not exist for most data hubs, it is OPERA which now compiles complete user requirements for its operational hub. The main tasks of the existing data hubs can be divided into three categories: Quality monitoring o Fully automated monitoring of data availability, timeliness and accuracy if possible; o Interactive monitoring to exclude erroneous data sets from GTS distribution or from product generation; fault correction procedures with NMSs or other operators; resumption of data distribution after fault correction; Data visualisation for quality monitoring purposes; Product generation. The advantages and disadvantages of centralization are different for these categories. It is evident that the user would benefit from a centralized quality monitoring of operational observations and a single portal for access to the data. A centralized monitoring (both automated and interactive) and (internet) visualisation may also result in some cost savings. A further investigation of the centralization of these tasks will be proposed to PB-OBS. Because of the different layouts of the product generation within the existing data hubs and the amount of expertise involved in the product generation, it is felt that this task should not be centralized in the near future. However, by conversion of the operational data hubs in so-called External Data Collection and Production Centres (DCPC) in the framework of the new WMO Information System (WIS, see Section 7.3) a further standardisation and integration is feasible. EUMETNET/OPERA WD_2008_02

29 7.2 GEOSS and GMES Metadata standards and software tools from the Open Geospatial Consortium (OGC) have become a de facto standard for harmonized management and availability of spatially distributed environmental data. Metadata are organized according to the Geography Markup Language (GML), which is based on ISO standards but free. Software tools are in the form of Web Map Services and Web Feature Services which are used to exchange and merge multi-source information from many different scientific disciplines. In February 2005, 61 countries around the World agreed on a 10 year plan to build open systems for sharing geospatial data and services across different platforms worldwide. This system is known as the Global Earth Observation System of Systems (GEOSS). The objective of GEOSS focuses on easy access to environmental data and interoperability across different systems allowing participating countries to measure the pulse of the planet in an effort to advance society. OGC solutions are at the heart of GEOSS. The main European contribution to GEOSS is in the form of the Global Monitoring for Environment and Security initiative (GMES). Numerous national agencies of various kinds (e.g. meteorology, hydrology, forestry, environment, rescue services, seismology, space etc.) are involved in GMES. The main challenge lies in harmonized access to and management of geospatial information. 7.3 WMO Information System The WMO is planning to introduce the successor to the GTS: the WMO Information System (WIS). To clarify the concept of WIS, three functional components are defined: National Centres (NC), Data Collection and Production Centres (DCPC) and Global Information System Centres (GISC). One physical centre could perform the functions of one or more of these components. Likewise, several physical centres could cooperate to perform the functions of a single functional centre. In Europe, the main candidate constituting the technological basis for the WIS is in a gridded networking framework currently approaching the end of its development phase. This technology is currently known as SIMDAT and is an EU IST project targeted to end in September The SIMDAT meteorological activity is led by the ECMWF with the UKMO, Météo-France, and DWD as participants. NORDRAD countries have expressed an interest in introducing exchange mechanisms for radar data using SIMDAT, and this has been encouraged. EUMETNET/OPERA WD_2008_02

30 Figure 5: Schematic overview of the Virtual-GISC as proposed for West-Europe. Already now, before the end of the SIMDAT development project, Météo-France, UKMO, and DWD have volunteered to jointly work out the concept of a virtual GISC (V-GISC) shared by their services and to include the ECMWF and EUMETSAT as DCPCs into the concept (see Figure 5). The OPERA Operational Data Hub would not only fit well into this framework, as an External DCPC, but the whole WIS concept including the Operational Data Hub would constitute a qualified contribution to GMES and GEOSS if implemented appropriately. 7.4 ECOMET Data policy (Based on OPERA working document WD_2007_17) The current data policy for the prototype European-wide composite from the pilot Data Hub has evolved in three steps: 1. At first the pilot Data Hub was not allowed the generate a European radar composite (23 rd Council) 2. The pilot Data Hub got permission to generate a European radar composite for the purpose of quality control in the Hub (26 th Council) 3. At the 28 th Council the use of composite images for official duties within the NMSs, excluding commercial reuse, was approved At the same time, new commercial services have developed and a Northwest- European radar composite is now available on the internet in real time, see e.g. Commercial service providers actually take all efforts to extend this radar composite beyond the current originating countries (France, EUMETNET/OPERA WD_2008_02

31 UK, Ireland, Germany, Belgium, and Netherlands) through bilateral license agreements. The operational Data Hub of OPERA needs an agreed data policy before operation can start and data can be delivered to users. Several options to establish the data policy for the Operational Data Hub were discussed at the 16 th meeting of PB-OBS and it was clear that only one option is valid. The radar products from the Operational Data Hub will be included in the ECOMET catalogue of the NMS hosting the Hub and thus all existing ECOMET regulations will apply. This data policy has the following advantages: Conflicts with existing data policy and responsible bodies (ECOMET) are avoided Availability of the Data Hub products for official duties of NMSs and lowthreshold access for non-commercial research outside of NMSs (WMO Resolution 40) are secured Commercial reuse of the Data Hub products (inside and outside NMSs) is possible, and a mechanism for product pricing and paying shares to participating originators is in place Disadvantages of this data policy may be: It will be difficult for a non-ecomet member to host the Operational Data Hub (Out of the 24 EUMETNET members: Latvia is a prospect member, and Cyprus, Estonia, and Slovenia are not members of ECOMET) For non-ecomet members providing data to the Operational Data Hub bilateral contracts will be required EUMETNET/OPERA WD_2008_02

32 8. Next steps and timeline In agreement with the OPERA-3 Program Proposal, the following steps towards the development of the European Operational Radar Data Center of EUMETNET/OPERA are foreseen: 1. Distribution of this draft specification document to the OPERA group (7 April 2008) 2. Presentation of the draft specifications and discussion at the 19 th meeting of OPERA (22-24 April 2008) 3. Update of draft specification document sent to PB-OBS (1 June 2008) 4. Presentation of the draft specifications, discussion, and approval at the 17 th PB-OBS meeting (24-27 June 2008) 5. Release of final specification document (1 Aug 2008) 6. Approval of (part of) budget for development by Council (Sep 2008) and/or EU/GMES Proposal (Time scale?) 7. Request for proposals by PM OPERA for development of Operational Data Center (End 2008) 8. Deadline for submission of proposals (1 March 2009) 9. Selection of the proposal for development of Operational Data Center at 21 th OPERA meeting (April 2009) 10. Approval by Council? 11. Start of development of Operational Data Center (1 June 2009) 12. Start of operation of the Radar Data Center and end of Pilot Data Hub (1 January 2010) It is noted that timeline is really tight and thus at the start of operation the radar data center may not have all functionality yet. For NMSs to be able to rely on the operational radar data center a certain continuity of the system should be guaranteed. What will happen after the two years of operation during this phase of OPERA? EUMETNET/OPERA WD_2008_02

33 Appendix A The table below illustrates the likely operating levels that will be expected of the radar data center. Quality Expected performance Performance Measure Fault resolution Products generated by the data centre from raw site data are created using common algorithms with common product service standards The target for availability of composite products will be an average of 99.8% Composite products delivered within 10 minutes of data time on at least 95% of occasions Delivery timeliness will be recorded as the time leaving the data centre Percentage availability will be measured over a rolling 12 month period For system or hardware faults that affect availability, the target will be to respond and fix the fault within 2 hours of notification on 98% of occasions. For faults that degrade the quality of a product, the target will be to respond and fix the problem within 1 working day of notification on 95% of occasions. Contingency The data centre service is to be maintained despite an IT infrastructure failure. Contingency could be provided by - alternative method on site (i.e. replicate production on an alternative platform); production from another site; alternative production from another site Support Cover IT support to be provided 24 hours a every day. (24/7/365) Service Failures A tolerable level of service failure would be: one break of up to 15 minutes in any 7 day period one break of up to 60 minutes in any quarter one break over 60 minutes in any one year, with service being restored within 4 hours A break denotes a reduction in service delivery, however the service will be deemed to be met if the agreed alternative output is being supplied. An alternative output is deemed acceptable for up to 12 hours EUMETNET/OPERA WD_2008_02