SRS BIO OPTICAL WORKFLOW Version 2.0 22 nd March 2013 Data Workflows emii, the data management facility for IMOS, has developed workflows for each IMOS sub facility to describe the flow of IMOS data from planning through data collection to data delivery and public data access. The primary goals of this workflow are to: Improve data flow and data handoff, making tracking of data status easy and preventing data loss Identify and delimit precisely the responsibilities of each person involved Improve communication at the interface between IMOS facilities (i.e. between emii and other IMOS facilities) Improve transparency for end users by providing more details to populate metadata records (i.e. limitations and processing methods applied to datasets) Assist in reporting planned deployments against actual deployments and data delivery The workflow is available on the next page of this document. Additional information (i.e. timeline, input, output, step description) for each operation step is available in the Supporting Information section. The role and contact details of people involved in the workflow are summarised in a table and suggested potential improvements are listed at the end of the document. An Appendix describes access permissions for IMOS data directories and lists useful links.
Data collection and data processing Data delivery on the IMOS portal SRS Bio optical database Gather data collected on various cruises Create a file per cruise using the Excel template Perform QA/QC on the data Create a metadata record for each cruise List of acronyms: BODBAW: Bio Optical Database of Australian waters CSIRO: Commonwealth Scientific and Industrial research Organisation emii: emarine Information Infrastructure MEST: IMOS Metadata Catalogue QA/QC: Quality Assurance/Quality Control SRS: Satellite Remote Sensing WMS: Web Map Service Send files to the emii by email Prepare Excel template Harvest metadata records into IMOS MEST emii Convert data into NetCDF format Publish data on the Data Fabric Create/update database tables Create/update a WMS layer in Configure IMOS portal menu Data and metadata publicly available through the IMOS portal
Supporting information Phase Operation step Timeline Input Output Step description Step operator Prepare Excel template Gather data collected on various cruises Data collected during previous cruises Excel template created Data from previous cruises gathered Creation of an Excel template to store the data. This template contains both metadata and data. Create a file per cruise using the Excel template Created Excel template Data from previous cruises gathered One Excel data file populated per cruise Copy the data in the Excel template. Data collection and processing Create a metadata record for each cruise Perform QA/QC on the data Send files to the by email Convert data into NetCDF format Harvest metadata records into IMOS MEST Publish data on the Data Fabric Two to three weeks Data from previous cruises gathered One Excel data file populated per cruise QA/QCed Excel data files Excel files at emii One metadata record created per cruise NetCDF files Excel files.csv files One metadata record created per cruise Metadata records once created are available on the MarLIN CSIRO website (MarLIN format) and then duplicated in Geonetwork in order to be harvested by the IMOS instance of Geonetwork. QA/QCed Excel data files QA/QCed Excel data files at emii NetCDF files Excel files.csv files Metadata records populated NetCDF, Excel and.csv files publicly available in the Opendap and Public folder of the Data Fabric Run a Matlab script to convert automatically the Excel data files into IMOS compliant NetCDF files. Harvest the CSIRO MEST catalogue and publish the records on the IMOS MEST. Move manually NetCDF files into the Opendap folder of the Data Fabric. Move manually Excel and.csv files into the Public folder of the Data Fabric. Create/update database tables Excel files at emii Metadata records populated Database tables populated Run a Matlab script to update the IMOS portal database using information extracted from the Excel files (e.g. filename, platform code, start/end time and location, bounding box, data and metadata links).
Data delivery on the IMOS portal Create/update a WMS layer in Configure IMOS portal menu Creation: One week Update: One day Creation: One week Update: One day Populated database tables WMS layer created in WMS layer created in Data and metadata accessible on the IMOS portal Use of a database table as a data source for the creation of a WMS layer in. Configure the pop up window displaying information from the database table (e.g. start/end time and location, link to data and metadata). Create filters. Create a link to the metadata records. Access to the admin interface of the IMOS portal. Select and include newly created layers to the portal menu. Save configuration. Update internet browser and check that the portal menu has been updated. and Software engineer
Contact details SRS Bio optical database emii Role Name Institution Email address Phone Comments Facility leader Dr. Edward King CSIRO edward.king@csiro.au (02) 6246 5894 Sub facility leader Dr. Vittorio Brando CSIRO vittorio.brando@csiro.au (02) 6246 5716 Dr. Lesley Clementson CSIRO lesley.clementson@csiro.au (03) 6232 5337 Mr. Laurent Besnard UTAS Laurent.Besnard@utas.edu.au (03) 6226 8570 Data services team leader Mr. Sebastien Mancini UTAS Sebastien.Mancini@utas.edu.au (03) 6226 8571 Mr. Philip Bohm UTAS Philip.Bohm@utas.edu.au (03) 6226 1975
Suggested improvements Make available monthly reports created by emii to the Facility (emii suggestion).
Appendix Access permissions for each directory of the Data Fabric IMOS datasets are stored on the ARCS Data Fabric. The Data Fabric is a virtual file system that allows data to be distributed across sites, but appear under a uniform structure. The base directory for IMOS is located at http://df.arcs.org.au/arcs/projects/imos/public/. Under this path the directories are as follows: Staging the place for facilities to upload their data (processed and unprocessed); accessed by facility and emii; no public access Archive for raw unprocessed files and other materials; accessed by facility and emii; no public access Public for processed QA/QC data for general access which is not suitable for OPeNDAP (like AUV images, excel spreadsheets, PDF document ); accessed by facility, emii and the public. Opendap location for files to be accessible via THREDDS/OPeNDAP; accessed by facility, emii and the public Directory Facilities emii End users Staging read/write read/write not visible Public read read/write read Opendap read read/write read Archive read read/write not visible Supporting links IMOS portal: http://imos.aodn.org.au/webportal/ IMOS THREDDS server (access to NetCDF files): http://opendap vpac.arcs.org.au/thredds/catalog/imos/srs/catalog.html Data Fabric: http://df.arcs.org.au/arcs/projects/imos/public/srs/ IMOS MEST (Metadata catalogue): http://imosmest.aodn.org.au/geonetwork/srv/en/main.home IMOS website: http://www.imos.org.au/ SRS facility: http://imos.org.au/srs.html SRS Bio optical database: http://imos.org.au/bwg.html