Using standards for ocean data

Transcription

1 Pan-European infrastructure for marine and ocean data management Using standards for ocean data By Dick M.A. Schaap SeaDataNet Technical Coordinator Vienna, April 09 Supported by the European Commission FP6 - Research Infrastructures

2 SeaDataNet Infrastructure objectives SeaDataNet aims to set up and operate an efficient Pan-European distributed infrastructure for managing marine and ocean data by connecting: 40 National Oceanographic Data Centres (NODC s), national oceanographic focal points, and ocean satellite data centres, in Europe these Data Centres are mostly divisions of major national marine research institutes and based in 35 countries, surrounding the European seas SeaDataNet aims to ensure a comparable quality of data sets and to make data sets easily accessible on-line through a unique portal, while the data sets are stored and managed at the Data Centres.

3 SeaDataNet Participating Countries 49 Partners = 40 Data Centres + ICES + IOC-IODE + JRC + specialists in statistics, products and informatics

4 SeaDataNet Infrastructure versions and planning Version 0 = Continuation and maintenance of existing Sea-Search metadata systems and population of the Common Data Index (CDI) V0 system Version 1 = Harmonised and upgraded metadatabases adopting ISO standard; Transparent data access involving all Data Centres via implementation of Common Data Index (CDI) V1 system and downloading services. Version 2 = Adding OGC viewing services and further virtualisation of data access.

5 SeaDataNet System Approach for V1 and V2 An approach has been adopted, which is in line with INSPIRE. The SeaDataNet infrastructure should consist of the following services: Discovery services = Metadata directories Security services = Authentication, Authorization & Accounting (AAA) Delivery services = Data access & downloading of data sets Viewing services = Visualisation of metadata, data and data products Product services = Generic and standard products Monitoring services = Statistics on usage and performance of the system Maintenance services = Updating of metadata by Data Centres A network of interconnected Data Centres and a central Portal, that will give users access to the various SeaDataNet services, and information on data management standards, tools and protocols.

6 SeaDataNet Technical Task Team Oversees and coordinates the technical developments Includes 11 core technical partners 3 4 Meetings per year Methodology: Agenda with relevant topics, leading speakers, open discussion and brainstorming, possible subgroups for further explorations and analyses, resulting in technical specifications of system architecture and its modules, followed by implementation, evaluation and operation and transfer to other partners All presentations and documents are included in the SeaDataNet Extranet for distribution among partners Documentation, once definite, is distributed publicly via the SeaDataNet Website

7 SeaDataNet V1 Portal website

8 SeaDataNet User s portal architecture V1 General request Metadata request Data request Status of request User registration Data download Project info metadata data Registr. Standards Cross search My transact. Software Vocabularies EDIOS CSR EDMED EDMERP CDI EDMO Shopping basket Requests status manager AAA Organisation + data source id User Register CSR EDIOS CDI Ifremer Database EDMED EDMO BODC Database at BSH at BODC Metadata & Data catalogues + EDMERP at MARIS BSH Database Entry point for access hits services for metadata Data downloading services Visualization services (WMS) for V2... Download managers in Data centers

9 SeaDataNet Interoperability Interoperability is the key to distributed data management system success. This is achieved in SeaDataNet via: Adopting the ISO metadata standard for all metadata directories Providing XML Validation Services to quality control the metadata maintenance Providing standard metadata entry tools Using common and controlled vocabularies, including international content governance Using common quality control protocols and flag scale Using harmonised Data Transport Formats for data sets delivery Adopting of OGC standards for mapping and viewing services Using SOAP Web Services in the SeaDataNet architecture Organising training and instruction workshops for transfer of expertise and hands-on training

10 SeaDataNet infrastructure V1 Discovery services EDMED - Data Sets EDMERP Research Projects CSR Cruise Summary Reports EDIOS Monitoring systems EDMO Marine Organisations CDI Common Data Index

11 SeaDataNet Discovery services Activities undertaken for: Reviewing and streamlining the logical formats of each of the Directories Setting up and populating Common vocabularies, making these available as Web services Defining XML schema s and formats, using the ISO metadata standard as basis Defining and developing maintenance modalities for each of the Directories

12 SeaDataNet Discovery services Defining and developing new User Interfaces for each of the Directories Defining and implementing XML Validation Web services, that can be used to validate XML output from data centres, before import into the public Directories Developing Web services for the Directories

13 SeaDataNet : Formats review Review of the format and use of common vocabularies for each of the Directories to achieve harmonisation and integration, and paving the path to data access via the CDI EDMED Vocabs CSR EDIOS CDI EDMERP Data Access EDMO

14 SeaDataNet XML Schema s The ISO content model is the basis for the XML formats and exchange schema s (XSD). Guidance document has been prepared on how to use XML for SeaDataNet, including how to declare references to Common Vocabularies, EDMO and EDMERP For each Directory has been prepared: Description of the format and XML tags XML Schema XML example file Extended XML Schema s have been prepared, using Schematron and OCL to support the checking of mandatory fields, use of codes from the Common Vocabularies and use of organization codes from EDMO. These schema s are used in the XML Validation Web services

15 SeaDataNet Common Vocabularies Use of common vocabularies in all metadatabases and data formats is an important prerequisite towards consistency and interoperability. The SeaDataNet Vocabulary service is based upon the NERC DataGrid (NDG) vocabulary Web service. The Web service is compliant to WS Basis Profile 1.1, which is adopted as standard for all Web services in SeaDataNet. For end-users there is a vocabulary Client Interface for searching and browsing and to export selected entries in csv format. Content governance of the vocabularies is very important and is done by a combined SeaDataNet and MarineXML Vocabulary Content Governance Group (SeaVoX), moderated by BODC, and including experts from SeaDataNet, MMI, MOTIIVE, JCOMMOPS and more international groups. SeaVox operates by mailing list server.

16 SeaDataNet Common Vocabularies Vocabularies Client User Interface

17 SeaDataNet Maintenance modalities Maintenance: Depending on the Directory, the following maintenance modalities are provided: Online maintenance via online Content Management System XML export from local system Local XML export can be produced by partners via: own software using the MIKADO Java tool, that has been developed for entering and editing partner s shares of the V1 Directories. MIKADO interacts with the Web services of the vocabularies, EDMO and EDMERP and produces valid XML files, that can be imported into the central V1 Directories. MIKADO also includes functionality for coupling to partner s local database(s) for generating CDI XML files in bulk

18 SeaDataNet Content Management Systems EDMERP EDMO EDMED CSR

19 SeaDataNet MIKADO Java tool Available under multiple environments : Microsoft : Windows 2000, XP, VISTA APPLE MIKADO Java code Unix - Solaris Linux Manual Automati c XML files for SeaDataNet catalogues CSR EDMED EDMERP CDI [EDIOS] DATABASE JDBC Java DataBase Connectivity Native Drivers MYSQL ORACLE POSTGRES SQLServer Bridge Drivers using Microsoft ODBC (ACCESS, EXCEL, SQL SERVER) Other Drivers Downloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory) EXCEL File

20 SeaDataNet V1 User Interfaces EDMERP CSR EDIOS EDMO

21 SeaDataNet Access to Data - CDI The Common Data Index (CDI) is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record). CDI paves the way for direct online data access through file downloads or online data requests. CDI XML Schema is based upon the ISO standard CDI V1 CDI V1 gives users not only a homogeneous metadatabase of available data, but also a unique interface for getting access to the data sets themselves, that are managed in distibuted SeaDataNet Data Centres

22 SeaDataNet CDI V1 shopping principle

23 SeaDataNet Delivery Services CDI V1 Central and homogeneous metadatabase plus transparent access to the distributed data sets via a unique user interface at the SeaDataNet portal and downloading services

24 SeaDataNet CDI V1 process Search find - bind

25 SeaDataNet CDI V1 Data Access example Check Status In RSM Search Request Confirmed Include in Basket Results Ready at DC x Shopping list Submit + Authentication Download Data SDN format

26 SeaDataNet CDI V1 Quick Search interface Using an intuitive approach which is easy for users

27 SeaDataNet CDI V1 Data Coverage CDI V1 coverage at mid April 2009: ca CDI entries so far from 11 Data Centres; connecting other SeaDataNet partners is underway

28 SeaDataNet V2 challenges INSPIRE is delivering its implementing rules. SeaDataNet must make sure that it is compliant. SeaDataNet is developing Product services, that will provide generic regional products from available datasets. These require delivery and viewing services. Products and analyses are supported by SeaDataNet analysis & presentation tools (ODV, DIVA). There is a challenge is to make these applications available as online applications.

29 INSPIRE Implementing Rules - excerpt Catalogue Service for the Web CSW ISO AP version (OGC) to enable Discovery Services (= INSPIRE name) ISO profile for discovery metadata / for services (ISO as technical format). SOAP for the bindings (communication)

30 SeaDataNet V2 pilot with CSW Making the exchange of metadata between data centres and European directories more efficient by applying the harvesting functionalities of CSW Search GO CSW (harvesting) Metadata repository ISO19115 metadata (V1) Download Web Service Data center1 Data center2 Data center3 Data center4

31 SeaDataNet multiple resources SDN V1 Discovery Services CDI, EDMED, CSR, EDIOS, EDMERP SDN Generic Product services Realize Cross-search INSPIRE compliant = CSW ISO Combine with other relevant ocean and marine resources THREDDS OPeNDAP NetCDF model data sets OGC WMS / WCS servers and maps

32 Use GI-cat broker / mediator service Developed as OS by University of Florence (Stefano Nativi) Harmonises metadata of resources to ISO core profile and produces CSW-ISO output

33 INSPIRE Community SDN Portal GUI Set up of full CDI chain CSW-ISO SDN GI-cat mediator Core metadata CDI WS CDI Metadata CDI RSM service DC Search Find Bind Data Delivery Services

34 SeaDataNet V1 presentation and analysis tools Ocean Data View (ODV) General data analysis and visualization software >10,000 registered users Upgrading ODV software into ODV4 for: Extending ODV s graphical display capabilities and interactive controls for automatic and visual quality control and data quality flagging Seamless connection to SeaDataNet output: SDN ODV4 data formats, quality flag scale Integration of DIVA gridding software Important tool for the SeaDataNet regional Data Products DIVA software (Data-Interpolating Variational Analysis) allows to spatially interpolate observations on a regular grid in an optimal way. The analysis is performed on a finite element grid allowing for a spatial variable resolution and a good representation of the coastline and isobaths.

35 SeaDataNet V1 presentation and analysis tools

36 SeaDataNet V2 Challenge

37