Insights from 40 years of data services at NCAR s Research Data Archive. Douglas Schuster and Steven Worley

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Insights from 40 years of data services at NCAR s Research Data Archive. Douglas Schuster and Steven Worley"

Transcription

1 Insights from 40 years of data services at NCAR s Research Archive Douglas Schuster and Steven Worley

2

3 Highlights What is the? Evolution of Services User Registration Policies Future Directions

4 What is the? Purposes Support climate & weather research at NCAR and UCAR Universities Collections Ocean & atmosphere observations, analyses, reanalyses, operational NWP outputs Basic Metrics Established in 1960s 600+ datasets, 8M files, 1.8 PB +70 datasets growing daily monthly 4

5 What is the? Science educated staff Expert consultants and data engineers Free and open access

6 Evolution of services General Goals Securely maintain community reference information for a long period Facilitate easy discovery of holdings Enable simple access to holdings Provide the community with the capability to build upon or reproduce the research of others 6

7 Evolution of services 1960s to early 1990s Specialist Driven / Storage Discovery Delivery 7

8 Evolution of services Mid 1990s to Early 2000s Specialist Driven / Storage Discovery Delivery 8

9 Evolution of services Mid 2000s to Present Driven / Storage Discovery Delivery 9

10 Current Services External Users ~10K/year Geoportal! Control and DB Internal Users ~100s/year Download GCMD Journal Article Faceted Search Tools OPeNDAP, WCS, WMS OAI-PMH! Web Interface and Services THREDDS Web and Server Processing Central File System 500 TB of set NCAR/CISL High Performance Computing Command line apps Software HPSS Tape Library 1.8 PB of set Legend /User Interactions

11 Discovery Pathways External Users ~10K/year Geoportal! Control and DB Internal Users ~100s/year GCMD OAI-PMH! Distributed search Web and Server NCAR/CISL High Performance Computing Download Central File System 350 TB of set HPSS Tape Library 1.2 PB of set Legend /User Interactions

12 Discovery Pathways External Users ~10K/year Download Journal/DOI based discovery Journal Article Web Interface and Services Web and Server Control and DB Central File System 350 TB of set NCAR/CISL High Performance Computing HPSS Tape Library 1.2 PB of set Legend /User Interactions Internal Users ~100s/year

13 Discovery Pathways -Server Generated Citation

14 Discovery Pathways External Users ~10K/year Download In-house developed web interface search Faceted Search Web Interface and Services Web and Server Control and DB Central File System 350 TB of set NCAR/CISL High Performance Computing HPSS Tape Library 1.2 PB of set Legend /User Interactions Internal Users ~100s/year

15 Discovery Pathways THREDDS catalog search

16 Discovery Pathways -Advise User of Best set Collection(s) to Support Needs

17 Pathways External Users ~10K/year Download Web interface/service data request submission and data download Web Interface and Services Tools Web and Server Processing Control and DB Central File System 500 TB of set NCAR/CISL High Performance Computing HPSS Tape Library 1.8 PB of set Legend /User Interactions Internal Users ~100s/year

18 Pathways External Users ~10K/year Control and DB Internal Users ~100s/year Remote data analysis software access through TDS service endpoints Download OPeNDAP, WCS, WMS THREDDS Web and Server Processing Central File System 500 TB of set NCAR/CISL High Performance Computing HPSS Tape Library 1.2 PB of set Legend /User Interactions

19 Pathways -Unidata TDS set Collection Virtual Aggregations Issue: Clients lack authentication support User Made Products Client Tools TDS 500 TB on Disk Plots Statistical Analyses Unidata IDV NASA Panoply Gridded set Collection Virtual Timeseries Aggregation File Gridded set Collection GRIB1 File GRIB1 File GRIB1 File netcdf File GRIB2 File GRIB2 File GRIB2 File s netcdf File netcdf File NetCDF Output files Spreadsheets NCAR Command Language Web Browser Observation set Collection Virtual Timeseries Aggregation File In-Situ Observation set Collection BUFR File BUFR File BUFR File Operational Under Development

20 Pathways External Users ~10K/year Download OPeNDAP, WCS, WMS THREDDS Web and Server Control and DB Direct data analysis on HPC systems Central File System 500 TB of set NCAR/CISL High Performance Computing Software HPSS Tape Library 1.8 PB of set Legend /User Interactions Internal Users ~100s/year

21 Future Pathways External Users ~10K/year Download OPeNDAP, WCS, WMS Control Direct data analysis Private and DB Cloud for select external users? THREDDS Web and User ServerConfigured VM Central File System 500 TB of set NCAR/CISL Private Cloud HPSS Tape Library 1.8 PB of set Legend /User Interactions Internal Users ~100s/year

22 Format management and usage Archive files stored in native data formats GRIB-1, GRIB-2, netcdf, HDF, BUFR, various binary and ASCII formats Archive may be restructured, depending on end-user research needs Time-series for single variable, or synoptic-time arrangement Climate research versus weather research preference! Prefer on-demand format conversion over maintaining datasets in multiple formats Offer format transformation as an output option Leverage protocols (e.g. OPeNDAP) that make data-format transparent to the end user

23 Current Services Overview Usage metrics In PB (virtual) processing 4000 users received data subsetting and format conversion services 45K requests were processed and 380 TB of data delivered 7300 users downloaded 750TB of data from archive files Metrics are now being collected to determine user data discovery pathways and TDS use

24 Policies (User Registration, /Use, Licensing, Limits, T&Cs) User Profile provided during registration ( only) address First and Last Name Organization Country Agree to UCAR Corporate Terms of Use Special Authorizations (access to restricted datasets) Implement requirements per third party policy agreements Enable to hold and serve unique research products Analyze metrics as a basis for scientific impact assessment and economic justification Free and open access to all archive holdings We reserve the right to request payment Why?

25 Policies (User Registration, /Use, Licensing, Limits, T&Cs) Future Might instantiate access control to services in tiered levels dependent on resource strain. Highest priority for NCAR and NSF supported education and research Broadly support all US users Use unused capacity for world-wide service

26 Future Directions Improve data access General GLOBUS connect access for all holdings Expand OPeNDAP and OGC based access ESGF integration of archives with climate model data brokering e.g. distributed satellite data matchup service with marine surface observations (ICOADS) Improve data discovery More data journal articles and DOIs Additional metadata standards available through harvestable endpoints ISO Use web standards (e.g. defined by Schema.org) to support discoverability by commercial search engines Support single-sign-on identity management Tutorial videos to enhance consulting with user forum (,YouTube, edx.org) More hosted data processing Improved multi-node/mpi data preparation Sub-setting, format conversion, customized file structures Private tiered cloud structure for non-hpc users analysis (analytics, regridding, user code execution environments) Commercial partnerships public cloud Focus on new communities general science interested public, more visualization,

27 Questions? Doug Schuster

The Arctic Observing Network and its Data Management Challenges Florence Fetterer (NSIDC/CIRES/CU), James A. Moore (NCAR/EOL), and the CADIS team

The Arctic Observing Network and its Data Management Challenges Florence Fetterer (NSIDC/CIRES/CU), James A. Moore (NCAR/EOL), and the CADIS team The Arctic Observing Network and its Data Management Challenges Florence Fetterer (NSIDC/CIRES/CU), James A. Moore (NCAR/EOL), and the CADIS team Photo courtesy Andrew Mahoney NSF Vision What is AON? a

More information

NOAA Environmental Data Management Update for Unidata SAC 2014-10-08

NOAA Environmental Data Management Update for Unidata SAC 2014-10-08 NOAA Environmental Management Update for Unidata SAC 2014-10-08 Jeff de La Beaujardière, PhD NOAA Management Architect +1 301-713-7175 NOAA data are unique, valuable, and irreplaceable Many observing systems:

More information

The THREDDS Data Repository: for Long Term Data Storage and Access

The THREDDS Data Repository: for Long Term Data Storage and Access 8B.7 The THREDDS Data Repository: for Long Term Data Storage and Access Anne Wilson, Thomas Baltzer, John Caron Unidata Program Center, UCAR, Boulder, CO 1 INTRODUCTION In order to better manage ever increasing

More information

Outcomes of the CDS Technical Infrastructure Workshop

Outcomes of the CDS Technical Infrastructure Workshop Outcomes of the CDS Technical Infrastructure Workshop Baudouin Raoult Baudouin.raoult@ecmwf.int Funded by the European Union Implemented by Evaluation & QC function C3S architecture from European commission

More information

Rosetta. A white paper on the challenges of sharing observational datasets. The Challenges of Sharing Observational Data

Rosetta. A white paper on the challenges of sharing observational datasets. The Challenges of Sharing Observational Data Rosetta A white paper on the challenges of sharing observational datasets The Challenges of Sharing Observational Data Observational studies are a critical component of geosciences education and research.

More information

PART 1. Representations of atmospheric phenomena

PART 1. Representations of atmospheric phenomena PART 1 Representations of atmospheric phenomena Atmospheric data meet all of the criteria for big data : they are large (high volume), generated or captured frequently (high velocity), and represent a

More information

Data-Intensive Science and Scientific Data Infrastructure

Data-Intensive Science and Scientific Data Infrastructure Data-Intensive Science and Scientific Data Infrastructure Russ Rew, UCAR Unidata ICTP Advanced School on High Performance and Grid Computing 13 April 2011 Overview Data-intensive science Publishing scientific

More information

WCS 2.0 Extension for netcdf-cf; CSW and THREDDS Interoperability

WCS 2.0 Extension for netcdf-cf; CSW and THREDDS Interoperability GEOSS Workshop XLIII: Sharing Climate Information & Knowledge NCAR Center Green, Boulder Colorado, 23 Sept 2011, 8:00am 5:30pm WCS 2.0 Extension for netcdf-cf; CSW and THREDDS Interoperability S.Nativi

More information

1. Overview and Status Update (Satoko) : 10min. 2. Demonstration (Yoshi) : 20min. 3. New Architecture (Yoshi): 15min. 4. Q&A, Discussion (All) : 15min

1. Overview and Status Update (Satoko) : 10min. 2. Demonstration (Yoshi) : 20min. 3. New Architecture (Yoshi): 15min. 4. Q&A, Discussion (All) : 15min 1. Overview and Status Update (Satoko) : 10min 2. Demonstration (Yoshi) : 20min 3. New Architecture (Yoshi): 15min 4. Q&A, Discussion (All) : 15min 1 CEOS Water Portal Overview and Status Update WGISS-37

More information

Introduction to the NOAA Earth Information System (NEIS)

Introduction to the NOAA Earth Information System (NEIS) Introduction to the NOAA Earth Information System (NEIS) Climate Diagnostics and Prediction Workshop October 23 rd, 2012 Jebb Q Stewart CIRA in collaboration with NOAA/OAR/ESRL/GSD Concept Take advantage

More information

Integrated Data Viewer (IDV) a visualization framework Yuan Ho Unidata Program Center Boulder, CO Presentation Outline Integrated Data Viewer (IDV) overview The IDV features IDV examples and customized

More information

Advancement of the NOMADS for Observational Data and Model Intercomparisons and the Establishment of a NCDC NOMADS Team and HelpDesk

Advancement of the NOMADS for Observational Data and Model Intercomparisons and the Establishment of a NCDC NOMADS Team and HelpDesk Advancement of the NOMADS for Observational Data and Model Intercomparisons and the Establishment of a NCDC NOMADS Team and HelpDesk A NOMADS Collaboration White Paper 5/5/2003 This white paper is intended

More information

IPSL climate data management backed by EUDAT services Agenda

IPSL climate data management backed by EUDAT services  Agenda IPSL climate data management backed by EUDAT services http://www.ipsl.fr eudat@ipsl.jussieu.fr Agenda IPSL presentation Data & calculation Eudat cooperation Page 1 What is IPSL Ins>tut Pierre Simon Laplace?

More information

SURFsara Data Services

SURFsara Data Services SURFsara Data Services SUPPORTING DATA-INTENSIVE SCIENCES Mark van de Sanden The world of the many Many different users (well organised (international) user communities, research groups, universities,

More information

13.2 THE INTEGRATED DATA VIEWER A WEB-ENABLED APPLICATION FOR SCIENTIFIC ANALYSIS AND VISUALIZATION

13.2 THE INTEGRATED DATA VIEWER A WEB-ENABLED APPLICATION FOR SCIENTIFIC ANALYSIS AND VISUALIZATION 13.2 THE INTEGRATED DATA VIEWER A WEB-ENABLED APPLICATION FOR SCIENTIFIC ANALYSIS AND VISUALIZATION Don Murray*, Jeff McWhirter, Stuart Wier, Steve Emmerson Unidata Program Center, Boulder, Colorado 1.

More information

The distribution of marine OpenData via distributed data networks and Web APIs. The example of ERDDAP, the message broker and data mediator from NOAA

The distribution of marine OpenData via distributed data networks and Web APIs. The example of ERDDAP, the message broker and data mediator from NOAA The distribution of marine OpenData via distributed data networks and Web APIs. The example of ERDDAP, the message broker and data mediator from NOAA Dr. Conor Delaney 9 April 2014 GeoMaritime, London

More information

Using Message Brokering and Data Mediation to use Distributed Data Networks of Earth Science Data to Enhance Global Maritime Situational Awareness.

Using Message Brokering and Data Mediation to use Distributed Data Networks of Earth Science Data to Enhance Global Maritime Situational Awareness. Using Message Brokering and Data Mediation to use Distributed Data Networks of Earth Science Data to Enhance Global Maritime Situational Awareness. Conor Delaney, Alfredo Alessandrini & Harm Greidanus

More information

Structure? Integrated Climate Data Center How to use the ICDC? Tools? Data Formats? User

Structure? Integrated Climate Data Center How to use the ICDC? Tools? Data Formats? User Integrated Climate Data Center? Data Formats?? Tools???? visits Structure???? User Contents Which Data Formats do we offer? What is the Structure of our data center? Which Tools do we provide? Our Aims

More information

CEOS Water Portal Status Update

CEOS Water Portal Status Update CEOS Water Portal Status Update Satoko Horiyama MIURA / Atsushi KAWAI JAXA/Mission Operations System Office WGISS-31 Introduction CEOS Water Portal is ; A web based portal system evolved from Distributed

More information

UERRA Visualization KNMI UERRA GA 2016, TOULOUSE, FR ADAGUC WEB-BASED VISUALIZATION ELSE VAN DEN BESSELAAR / MAARTEN PLIEGER KNMI R&D

UERRA Visualization KNMI UERRA GA 2016, TOULOUSE, FR ADAGUC WEB-BASED VISUALIZATION ELSE VAN DEN BESSELAAR / MAARTEN PLIEGER KNMI R&D UERRA Visualization WEBSERVICES @ KNMI UERRA GA 2016, TOULOUSE, FR ADAGUC WEB-BASED VISUALIZATION ELSE VAN DEN BESSELAAR / MAARTEN PLIEGER KNMI R&D Visualization - Web Map Services - ESGF Meteorological

More information

Nevada NSF EPSCoR Track 1 Data Management Plan

Nevada NSF EPSCoR Track 1 Data Management Plan Nevada NSF EPSCoR Track 1 Data Management Plan August 1, 2011 INTRODUCTION Our data management plan is driven by the overall project goals and aims to ensure that the following are achieved: Assure that

More information

NASA's Strategy and Activities in Server Side Analytics

NASA's Strategy and Activities in Server Side Analytics NASA's Strategy and Activities in Server Side Analytics Tsengdar Lee, Ph.D. High-end Computing Program Manager NASA Headquarters Presented at the ESGF/UVCDAT Conference Lawrence Livermore National Laboratory

More information

Globus Research Data Management: Introduction and Service Overview

Globus Research Data Management: Introduction and Service Overview Globus Research Data Management: Introduction and Service Overview Kyle Chard chard@uchicago.edu Ben Blaiszik blaiszik@uchicago.edu Thank you to our sponsors! U. S. D E P A R T M E N T OF ENERGY 2 Agenda

More information

Environmental Data Management:

Environmental Data Management: Environmental Data Management: Challenges & Opportunities Mohan Ramamurthy, Unidata University Corporation for Atmospheric Research Boulder CO 25 May 2010 Environmental Data Management Workshop Silver

More information

The ORIENTGATE data platform

The ORIENTGATE data platform Research Papers Issue RP0195 December 2013 The ORIENTGATE data platform SCO Scientific Computing and Operations Division By Alessandra Nuzzo University of Salento and Scientific Computing and Operations

More information

Using standards for ocean data

Using standards for ocean data Pan-European infrastructure for marine and ocean data management Using standards for ocean data By Dick M.A. Schaap SeaDataNet Technical Coordinator Vienna, April 09 Supported by the European Commission

More information

Climate SuperComputing at NASA Goddard Space Flight Center

Climate SuperComputing at NASA Goddard Space Flight Center Climate SuperComputing at NASA Goddard Space Flight Center Fifteenth ECMWF Workshop on the Use of High Performance Computing in Meteorology Dr. W. Phillip Webster 3 October 2012 NASA Center for Climate

More information

EUMETSAT EO Portal. End User Image Access using OGC WMS/WCS services. EUM/OPS/VWG/10/0095 Issue <1> <14/01/2010> Slide: 1

EUMETSAT EO Portal. End User Image Access using OGC WMS/WCS services. EUM/OPS/VWG/10/0095 Issue <1> <14/01/2010> Slide: 1 EUMETSAT EO Portal End User Image Access using OGC WMS/WCS services Slide: 1 Overview Introduction: status of data access and visualization EUMETSAT datasets Architecture Web Map Service implementation

More information

Introduction to IODE Data Management. Greg Reed Past Co-Chair IODE

Introduction to IODE Data Management. Greg Reed Past Co-Chair IODE Introduction to IODE Data Management Greg Reed Past Co-Chair IODE Outline Background - Introduction to IOC and IODE - IODE activities Oceanographic data management - End to end data management - Data stewardship

More information

Real-time and Archived Antarctic. Interactive Processing Tools

Real-time and Archived Antarctic. Interactive Processing Tools Real-time and Archived Antarctic Meteorological Data via a Synergy of Interactive Processing Tools Mark W. Seefeldt Department of Atmospheric and Oceanic Sciences University it of Colorado at Boulder,

More information

Data Management Perspective for an International Arctic Observing Network: Preserving the Legacy Promoting Exchange

Data Management Perspective for an International Arctic Observing Network: Preserving the Legacy Promoting Exchange Data Management Perspective for an International Arctic Observing Network: Preserving the Legacy Promoting Exchange James A. Moore 1 Molly McCammon 2 Rob Bochenek 3 1 National Center for Atmospheric Research,

More information

Web-based spatio-temporal visualization and analysis of the Siberian Earth System Science Cluster (SIB-ESS-C)

Web-based spatio-temporal visualization and analysis of the Siberian Earth System Science Cluster (SIB-ESS-C) Web-based spatio-temporal visualization and analysis of the Siberian Earth System Science Cluster (SIB-ESS-C) Roman Gerlach Supervisor: Prof. C. Schmullius (Dept. of Geography, Friedrich-Schiller-University

More information

André Karpištšenko, Co-Founder & Chief Scientist, Marinexplore Strata, 2014.02.11

André Karpištšenko, Co-Founder & Chief Scientist, Marinexplore Strata, 2014.02.11 marineos André Karpištšenko, Co-Founder & Chief Scientist, Marinexplore Strata, 2014.02.11 The Ocean's Big Data Platform marineos: a platform for organizing, analyzing and distributing machine data marineos

More information

THREDDS. THematic Real-time Environmental Distributed Data Services. Connecting people, documents and data

THREDDS. THematic Real-time Environmental Distributed Data Services. Connecting people, documents and data THREDDS THematic Real-time Environmental Distributed Data Services Connecting people, documents and data Ben Domenico, John Caron, Ethan Davis, Robb Kambic, Stefano Nativi Unidata Program Center and University

More information

REACCH PNA Data Management Plan

REACCH PNA Data Management Plan REACCH PNA Data Management Plan Regional Approaches to Climate Change (REACCH) For Pacific Northwest Agriculture 875 Perimeter Drive MS 2339 Moscow, ID 83844-2339 http://www.reacchpna.org reacch@uidaho.edu

More information

HYCOM Data Management -- opportunities & planning --

HYCOM Data Management -- opportunities & planning -- HYCOM Data Management -- opportunities & planning -- Steve Hankin (NOAA/PMEL) Ashwanth Srinivasan, (RSMAS) Peter Cornillon, OPeNDAP PI (URI) Mike Clancy, US GODAE Server (FNMOC) Jon Callahan, Kevin O Brien

More information

NASA s Big Data Challenges in Climate Science

NASA s Big Data Challenges in Climate Science NASA s Big Data Challenges in Climate Science Tsengdar Lee, Ph.D. High-end Computing Program Manager NASA Headquarters Presented at IEEE Big Data 2014 Workshop October 29, 2014 1 2 7-km GEOS-5 Nature Run

More information

New Globus Features for Campuses

New Globus Features for Campuses New Globus Features for Campuses Steve Tuecke The University of Chicago Instrument 1 Researcher initiates transfer request; or requested automatically by script, science gateway Globus SaaS: Research data

More information

The ORIENTGATE data platform

The ORIENTGATE data platform Seminar on Proposed and Revised set of indicators June 4-5, 2014 - Belgrade (Serbia) The ORIENTGATE data platform WP2, Action 2.4 Alessandra Nuzzo, Sandro Fiore, Giovanni Aloisio Scientific Computing and

More information

The CEOP Model Data Archive at the World Data Center for Climate as part of the CEOP Data Network

The CEOP Model Data Archive at the World Data Center for Climate as part of the CEOP Data Network The CEOP Model Data Archive at the World Data Center for Climate as part of the CEOP Data Network CEOP / IGWCO Joint Meeting Feb. 28th Mar. 4th, 2005, Tokyo Michael Lautenschlager Frank Toussaint, Hans

More information

MyOcean Copernicus Marine Service Architecture and data access Experience

MyOcean Copernicus Marine Service Architecture and data access Experience MyOcean Copernicus Marine Service Architecture and data access Experience Sophie Besnard CLS, Toulouse, France February 2015 MyOcean Story MyOcean Challenge & Success MyOcean Service MyOcean System MyOcean

More information

Norwegian Satellite Earth Observation Database for Marine and Polar Research http://normap.nersc.no USE CASES

Norwegian Satellite Earth Observation Database for Marine and Polar Research http://normap.nersc.no USE CASES Norwegian Satellite Earth Observation Database for Marine and Polar Research http://normap.nersc.no USE CASES The NORMAP Project team has prepared this document to present functionality of the NORMAP portal.

More information

SRS BIO OPTICAL WORKFLOW

SRS BIO OPTICAL WORKFLOW SRS BIO OPTICAL WORKFLOW Version 2.0 22 nd March 2013 Data Workflows emii, the data management facility for IMOS, has developed workflows for each IMOS sub facility to describe the flow of IMOS data from

More information

Satellite Products and Dissemination: Visualization and Data Access

Satellite Products and Dissemination: Visualization and Data Access Satellite Products and Dissemination: Visualization and Data Access Gregory Leptoukh GES DISC, NASA GSFC Dana Ostrenga GES DISC, NASA GSFC Introduction The Goddard Earth Sciences Data and Information Services

More information

Arctic Climate and Weather Forecast Model Visualization Scenario Engineering Report GEOSS Architecture Implementation Pilot Version 1.

Arctic Climate and Weather Forecast Model Visualization Scenario Engineering Report GEOSS Architecture Implementation Pilot Version 1. Arctic Climate and Weather Forecast Model Visualization Scenario GEOSS Architecture Implementation Pilot Version 1.0 Content developed by the GEO Architecture Implementation Pilot Licensed under a Creative

More information

George Mason University (GMU)

George Mason University (GMU) George Mason University (GMU) Center for Spatial Information Science and Systems Organization (CSISS) 4400 University Drive, MSN 6E1 George Mason University Fairfax, VA 22030, USA Telephone: +1 703 993

More information

A Project to Create Bias-Corrected Marine Climate Observations from ICOADS

A Project to Create Bias-Corrected Marine Climate Observations from ICOADS A Project to Create Bias-Corrected Marine Climate Observations from ICOADS Shawn R. Smith 1, Mark A. Bourassa 1, Scott Woodruff 2, Steve Worley 3, Elizabeth Kent 4, Simon Josey 4, Nick Rayner 5, and Richard

More information

Big Data Services at DKRZ

Big Data Services at DKRZ Big Data Services at DKRZ Michael Lautenschlager and Colleagues from DKRZ and Scientific Computing Research Group MPI-M Seminar Hamburg, March 31st, 2015 Big Data in Climate Research Big data is an all-encompassing

More information

Metadata for Data Discovery: The NERC Data Catalogue Service. Steve Donegan

Metadata for Data Discovery: The NERC Data Catalogue Service. Steve Donegan Metadata for Data Discovery: The NERC Data Catalogue Service Steve Donegan Introduction NERC, Science and Data Centres NERC Discovery Metadata The Data Catalogue Service NERC Data Services Case study:

More information

HYCOM Meeting. Tallahassee, FL

HYCOM Meeting. Tallahassee, FL HYCOM Data Service An overview of the current status and new developments in Data management, software and hardware Ashwanth Srinivasan & Jon Callahan COAPS FSU & PMEL HYCOM Meeting Nov 7-9, 7 2006 Tallahassee,

More information

Scientific Data Management and Dissemination

Scientific Data Management and Dissemination Federal GIS Conference February 9 10, 2015 Washington, DC Scientific Data Management and Dissemination John Fry Solution Engineer, Esri jfry@esri.com Agenda Background of Scientific Data Management through

More information

Big Data in OpenTopography

Big Data in OpenTopography Big Data in OpenTopography Vishu Nandigam San Diego Supercomputer Center NSF Big Data in Educa

More information

StormSurgeViz: A Visualization and Analysis Application for Distributed ADCIRC-based Coastal Storm Surge, Inundation, and Wave Modeling

StormSurgeViz: A Visualization and Analysis Application for Distributed ADCIRC-based Coastal Storm Surge, Inundation, and Wave Modeling : A Visualization and Analysis Application for Distributed ADCIRC-based Coastal Storm Surge, Inundation, and Wave Modeling Brian Blanton Renaissance Computing Institute University of North Carolina at

More information

Open Source Visualisation with ADAGUC Web Map Services

Open Source Visualisation with ADAGUC Web Map Services Open Source Visualisation with ADAGUC Web Map Services Maarten Plieger Ernst de Vreede John van de Vegte, Wim Som de Cerff, Raymond Sluiter, Ian van der Neut, Jan Willem Noteboom 1 ADAGUC project Cooperative

More information

GIS Initiative: Developing an atmospheric data model for GIS. Olga Wilhelmi (ESIG), Jennifer Boehnert (RAP/ESIG) and Terri Betancourt (RAP)

GIS Initiative: Developing an atmospheric data model for GIS. Olga Wilhelmi (ESIG), Jennifer Boehnert (RAP/ESIG) and Terri Betancourt (RAP) GIS Initiative: Developing an atmospheric data model for GIS Olga Wilhelmi (ESIG), Jennifer Boehnert (RAP/ESIG) and Terri Betancourt (RAP) Unidata seminar August 30, 2004 Presentation Outline Overview

More information

CEOS Water Portal Status Update

CEOS Water Portal Status Update CEOS Water Portal Status Update WGISS-33 Tokyo, Japan April 23-27, 2012 JAXA/Mission Operations System Office Satoko Miura/ Atsushi Kawai Update since WGISS-32 New features Compare service for MOLTS data

More information

National Snow and Ice Data Center A brief overview and data management projects

National Snow and Ice Data Center A brief overview and data management projects Cooperative Institute for Research in Environmental Sciences University of Colorado National Snow and Ice Data Center A brief overview and data management projects Walt Meier, Mark Parsons, and Mark Serreze

More information

JASMIN Cloud ESGF and UV- CDAT Conference 09-11 December 2014 STFC / Stephen Kill

JASMIN Cloud ESGF and UV- CDAT Conference 09-11 December 2014 STFC / Stephen Kill JASMIN Cloud ESGF and UV- CDAT Conference 09-11 December 2014 STFC / Stephen Kill Philip Kershaw (1, 2), Jonathan Churchill (5), Bryan Lawrence (1, 3, 4), Stephen Pascoe (1, 4) and MaE Pritchard (1) Centre

More information

GeoNetwork, The Open Source Solution for the interoperable management of geospatial metadata

GeoNetwork, The Open Source Solution for the interoperable management of geospatial metadata GeoNetwork, The Open Source Solution for the interoperable management of geospatial metadata Ing. Simone Giannecchini, GeoSolutions Ing. Emanuele Tajariol, GeoSolutions Outline GeoNetwork Introduction

More information

A standards-based open source processing chain for ocean modeling in the GEOSS Architecture Implementation Pilot Phase 8 (AIP-8)

A standards-based open source processing chain for ocean modeling in the GEOSS Architecture Implementation Pilot Phase 8 (AIP-8) NATO Science & Technology Organization Centre for Maritime Research and Experimentation (STO-CMRE) Viale San Bartolomeo, 400 19126 La Spezia, Italy A standards-based open source processing chain for ocean

More information

11-12 June 2015, Bari-Italy. Stefano Nativi CNR-IIA

11-12 June 2015, Bari-Italy. Stefano Nativi CNR-IIA 11-12 June 2015, Bari-Italy Stefano Nativi CNR-IIA Coordinating an Observation Network of Networks EnCompassing satellite and IN-situ to fill the Gaps in European Observations GEOSS Information System

More information

Data Lab Operations Concepts

Data Lab Operations Concepts Data Lab Operations Concepts 1 Introduction This talk will provide an overview of Data Lab components to be implemented Core infrastructure User applications Science Capabilities User Interfaces The scope

More information

GeoNetwork, The Open Source Solution for the interoperable management of geospatial metadata

GeoNetwork, The Open Source Solution for the interoperable management of geospatial metadata GeoNetwork, The Open Source Solution for the interoperable management of geospatial metadata Ing. Emanuele Tajariol, GeoSolutions Ing. Simone Giannecchini, GeoSolutions GeoSolutions GeoSolutions GeoNetwork

More information

NATIONAL CLIMATE CHANGE & WILDLIFE SCIENCE CENTER & CLIMATE SCIENCE CENTERS DATA MANAGEMENT PLAN GUIDANCE

NATIONAL CLIMATE CHANGE & WILDLIFE SCIENCE CENTER & CLIMATE SCIENCE CENTERS DATA MANAGEMENT PLAN GUIDANCE NATIONAL CLIMATE CHANGE & WILDLIFE SCIENCE CENTER & CLIMATE SCIENCE CENTERS DATA MANAGEMENT PLAN GUIDANCE Prepared by: NCCWSC/CSC Data Management Working Group US Geological Survey February 26, 2013 Version

More information

and Deployment Roadmap for Satellite Ground Systems

and Deployment Roadmap for Satellite Ground Systems A Cloud-Based Reference Model and Deployment Roadmap for Satellite Ground Systems 2012 Ground System Architectures Workshop February 29, 2012 Dr. Craig A. Lee The Aerospace Corporation The Aerospace Corporation

More information

University Bremen (UniHB) PANGAEA

University Bremen (UniHB) PANGAEA SeaDataNet 2 New Participants University Bremen (UniHB) PANGAEA Stéphane PESANT (spesant@marum.de) OUTLINE 1. HISTORY 2. PEOPLE 3. INFRASTRUCTURE 4. DATA COLLECTIONS 5. METADATA & DATA SUBMISSION 6. METADATA

More information

Guidance and Requirements for. NCCWSC/CSC Data Management Plans

Guidance and Requirements for. NCCWSC/CSC Data Management Plans Guidance and Requirements for NCCWSC/CSC Plans (Required for NCCWSC and CSC Proposals and Funded Projects) Prepared by the CSC/NCCWSC Working Group Emily Fort, Data and IT Manager for the National Climate

More information

Scalable Services for Digital Preservation

Scalable Services for Digital Preservation Scalable Services for Digital Preservation A Perspective on Cloud Computing Rainer Schmidt, Christian Sadilek, and Ross King Digital Preservation (DP) Providing long-term access to growing collections

More information

CDI/THREDDS Interoperability: the SeaDataNet developments. P. Mazzetti 1,2, S. Nativi 1,2, 1. CNR-IMAA; 2. PIN-UNIFI

CDI/THREDDS Interoperability: the SeaDataNet developments. P. Mazzetti 1,2, S. Nativi 1,2, 1. CNR-IMAA; 2. PIN-UNIFI CDI/THREDDS Interoperability: the SeaDataNet developments P. Mazzetti 1,2, S. Nativi 1,2, 1. CNR-IMAA; 2. PIN-UNIFI Outline Interoperability Issues in SeaDataNet A broker solution for CDI/THREDDS interoperability

More information

DATA MANAGEMENT (DM) - (e.g. codes, monitoring) software offered to WMO Members for exchange

DATA MANAGEMENT (DM) - (e.g. codes, monitoring) software offered to WMO Members for exchange DATA MANAGEMENT (DM) - (e.g. codes, monitoring) software offered to s for exchange Notified to WMO 10/11/2009 United States A library of functions and subroutines which can be used to encode and decode

More information

Using Databases to Manage State Information for. Globally Distributed Data

Using Databases to Manage State Information for. Globally Distributed Data Storage Resource Broker Using Databases to Manage State Information for Globally Distributed Data Reagan W. Moore San Diego Supercomputer Center moore@sdsc.edu http://www.sdsc sdsc.edu/srb Abstract The

More information

Driving Earth Systems Collaboration across the Pacific

Driving Earth Systems Collaboration across the Pacific Driving Earth Systems Collaboration across the Pacific Tim F. Pugh Centre for Australian Weather and Climate Research Australian Bureau of Meteorology What s our business? Earth System Science Study of

More information

IDL. Get the answers you need from your data. IDL

IDL. Get the answers you need from your data. IDL Get the answers you need from your data. IDL is the preferred computing environment for understanding complex data through interactive visualization and analysis. IDL Powerful visualization. Interactive

More information

Integration strategy

Integration strategy C3-INAD and ESGF: Integration strategy C3-INAD Middleware Team: Stephan Kindermann, Carsten Ehbrecht [DKRZ] Bernadette Fritzsch [AWI] Maik Jorra, Florian Schintke, Stefan Plantikov [ZUSE Institute] Markus

More information

Development of Earth Science Observational Data Infrastructure of Taiwan. Fang-Pang Lin National Center for High-Performance Computing, Taiwan

Development of Earth Science Observational Data Infrastructure of Taiwan. Fang-Pang Lin National Center for High-Performance Computing, Taiwan Development of Earth Science Observational Data Infrastructure of Taiwan Fang-Pang Lin National Center for High-Performance Computing, Taiwan GLIF 13, Singapore, 4 Oct 2013 The Path from Infrastructure

More information

GOAPP Data Management Policy

GOAPP Data Management Policy Global Ocean-Atmosphere Prediction and Predictability Network GOAPP Data Management Policy Prepared by: GOAPP Data Management Committee (DMC) Jason Chaffey (DFO), Bill Merryfield (GOAPP Investigator),

More information

OGC at KNMI: Current use and plans Available products

OGC at KNMI: Current use and plans Available products OGC at KNMI: Current use and plans Available products Maarten Plieger KNMI R&D Ernst de Vreede, John van de Vegte, Wim Som de Cerff, Ian van der Neut, Raymond Sluiter, Jan Willem Noteboom Contents 1. ADAGUC

More information

NCDC Strategic Vision

NCDC Strategic Vision NOAA s National Climatic Data Center World s Largest Archive of Climate and Weather Data Presented to: Coastal Environmental Disasters Data Management Workshop September 16, 2014 Stephen Del Greco Deputy

More information

SAMOS, GOSUD. and the Observing System Monitoring Center. Second Joint GOSUD/SAMOS Workshop Seattle, Washington, USA 10-12 June 2008

SAMOS, GOSUD. and the Observing System Monitoring Center. Second Joint GOSUD/SAMOS Workshop Seattle, Washington, USA 10-12 June 2008 SAMOS, GOSUD and the Observing System Monitoring Center Second Joint GOSUD/SAMOS Workshop Seattle, Washington, USA 10-12 Steve Hankin NOAA/PMEL (presenter) Derrick Snowden NOAA/OCO Kevin O Brien JISAO/UW

More information

Enhanced Research Data Management and Publication with Globus

Enhanced Research Data Management and Publication with Globus Enhanced Research Data Management and Publication with Globus Vas Vasiliadis Jim Pruyne Presented at OR2015 June 8, 2015 Presentations and other useful information available at globus.org/events/or2015/tutorial

More information

NOMADS. Jordan Alpert, Jun Wang NCEP/NWS. Jordan C. Alpert jordan.alpert@noaa.gov. where the nation s climate and weather services begin

NOMADS. Jordan Alpert, Jun Wang NCEP/NWS. Jordan C. Alpert jordan.alpert@noaa.gov. where the nation s climate and weather services begin An Application for High Availability NOMADS Jordan Alpert, Jun Wang NCEP/NWS Jordan C. Alpert jordan.alpert@noaa.gov DMIT Workshop, Silver Spring, MD 8/13-14//2009 where the nation s climate and weather

More information

Research Data Collection Data Management Plan

Research Data Collection Data Management Plan Research Data Collection Data Management Plan This document records the whole process of the data collections, including pre- preparation data,

More information

CMIP5 Data Management CAS2K13

CMIP5 Data Management CAS2K13 CMIP5 Data Management CAS2K13 08. 12. September 2013, Annecy Michael Lautenschlager (DKRZ) With Contributions from ESGF CMIP5 Core Data Centres PCMDI, BADC and DKRZ Status DKRZ Data Archive HLRE-2 archive

More information

A DISTRIBUTED CATALOG AND DATA SERVICES SYSTEM FOR REMOTE SENSING DATA

A DISTRIBUTED CATALOG AND DATA SERVICES SYSTEM FOR REMOTE SENSING DATA A DISTRIBUTED CATALOG AND DATA SERVICES SYSTEM FOR REMOTE SENSING DATA Ramachandran Suresh *, Liping Di *, Kenneth McDonald ** * NASA/RITSS 4500 Forbes Blvd, Lanham, MD 20706, USA suresh@rattler.gsfc.nasa.gov

More information

Globus Research Data Management: Introduction and Service Overview. Steve Tuecke Vas Vasiliadis

Globus Research Data Management: Introduction and Service Overview. Steve Tuecke Vas Vasiliadis Globus Research Data Management: Introduction and Service Overview Steve Tuecke Vas Vasiliadis Presentations and other useful information available at globus.org/events/xsede15/tutorial 2 Thank you to

More information

Globus Auth. Steve Tuecke. The University of Chicago

Globus Auth. Steve Tuecke. The University of Chicago Globus Auth Enabling an extensible, integrated ecosystem of services and applications for the research and education community. Steve Tuecke The University of Chicago Cloud has transformed how platforms

More information

Big Data and the Earth Observation and Climate Modelling Communities: JASMIN and CEMS

Big Data and the Earth Observation and Climate Modelling Communities: JASMIN and CEMS Big Data and the Earth Observation and Climate Modelling Communities: JASMIN and CEMS Workshop on the Future of Big Data Management 27-28 June 2013 Philip Kershaw Centre for Environmental Data Archival

More information

Data Products via TRMM Online Visualization and Analysis System

Data Products via TRMM Online Visualization and Analysis System Accessing Global Precipitation Data Products via TRMM Online Visualization and Analysis System (TOVAS) Zhong Liu Center for Spatial Information Science and Systems (CSISS), George Mason University and

More information

Integrate with Library Workflows & Processes Custom Harvest Tool & Ingest Processes

Integrate with Library Workflows & Processes Custom Harvest Tool & Ingest Processes Rosetta installed in March 2012 Production Environment Development / Testing Environment Integrate with Library Workflows & Processes Custom Harvest Tool & Ingest Processes Ingest various types of collections:

More information

McIDAS-V: An open source data analysis and visualization tool for multiand hyperspectral satellite data ITSC-XVI, Angra do Reis, Brazil, 7 May 2008

McIDAS-V: An open source data analysis and visualization tool for multiand hyperspectral satellite data ITSC-XVI, Angra do Reis, Brazil, 7 May 2008 McIDAS-V: An open source data analysis and visualization tool for multiand hyperspectral satellite data ITSC-XVI, Angra do Reis, Brazil, 7 May 2008 Tom Achtor, Tom Rink, Tom Whittaker, Dave Parker, Gail

More information

Unidata s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud

Unidata s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud Unidata s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud The WWOSC 2014 17 August 2014 Montreal, CA Dr. Mohan Ramamurthy Unidata Program Center University Corporation

More information

Science Gateway Services for NERSC Users

Science Gateway Services for NERSC Users Science Gateway Services for NERSC Users Shreyas Cholia NERSC User Group Meeting October 7, 2009 Science Gateways at NERSC Web access methods to NERSC resources Much is possible beyond yesterday s ssh+pbs

More information

Canadian National Research Data Repository Service. CC and CARL Partnership for a national platform for Research Data Management

Canadian National Research Data Repository Service. CC and CARL Partnership for a national platform for Research Data Management Research Data Management Canadian National Research Data Repository Service Progress Report, June 2016 As their digital datasets grow, researchers across all fields of inquiry are struggling to manage

More information

Flexible and modular visualisation and data discovery tools for environmental information

Flexible and modular visualisation and data discovery tools for environmental information 21st International Congress on Modelling and Simulation, Gold Coast, Australia, 29 Nov to 4 Dec 2015 www.mssanz.org.au/modsim2015 Flexible and modular visualisation and data discovery tools for environmental

More information

Performance Analysis of a Numerical Weather Prediction Application in Microsoft Azure

Performance Analysis of a Numerical Weather Prediction Application in Microsoft Azure Performance Analysis of a Numerical Weather Prediction Application in Microsoft Azure Emmanuell D Carreño, Eduardo Roloff, Jimmy V. Sanchez, and Philippe O. A. Navaux WSPPD 2015 - XIII Workshop de Processamento

More information

NOAA Environmental Data Management

NOAA Environmental Data Management NOAA Environmental Management Report to Unidata Policy Cmtee 2013-05-15 Jeff de La Beaujardière, PhD NOAA Management Architect jeff.delabeaujardiere@noaa.gov +1 301-713-7175 Jeff.deLaBeaujardiere@noaa.gov

More information

The STAR Algorithm Integration Team (AIT) Research to Operations Process

The STAR Algorithm Integration Team (AIT) Research to Operations Process The STAR Algorithm Integration Team (AIT) Research to Operations Process Presented by Tom King 1 The Problem Scientists write code that needs to go to operations, but:» Many scientists often prefer interpreted

More information

PIT adoption for Climate Data Management

PIT adoption for Climate Data Management PIT adoption for Climate Management The German Climate Computing Center (DKRZ) 5th RDA Plenary Stephan Kindermann Overview DKRZ: A climate science service provider services for the international climate

More information

Unidata Cloud-Related Activities. Unidata Users Committee Meeting September 2014 Ward Fisher

Unidata Cloud-Related Activities. Unidata Users Committee Meeting September 2014 Ward Fisher Unidata Cloud-Related Activities Unidata Users Committee Meeting September 2014 Ward Fisher Overview Three ongoing efforts, broadly speaking. Unidata developers are incorporating the cloud and cloud-based

More information

Progress in Creating a Global Polar Metadata Interoperability Network

Progress in Creating a Global Polar Metadata Interoperability Network Progress in Creating a Global Polar Metadata Interoperability Network Julie E. Friddell Canadian Cryospheric Information Network/Polar Data Catalogue Department of Geography, University of Waterloo Canada

More information