Facilitate ES Data Interoperability using the SCIDIP-ES Data Virtualization Toolkit

Transcription

1 Facilitate ES Data Interoperability using the SCIDIP-ES Data Virtualization Toolkit Jinsongdi Yu, Peter Baumann Shirley Crompton PV Nov 2013

2 SCIDIP-ES Project An EC FP7 funded project under the Grant Agreement Work packages to be completed by 16 beneficiaries Project time frame October 2011 September 2014 Antecedent projects CASPAR, SHAMAN ESRIN/PV2013

3 Data Preservation More than keeping the data safe the bits are retrievable and are the same as the original Keeping the data usable the data can be understood and reused at a later date Slough (SL051) : to SLOUGH SL Manual Edited fof2 M3000F2 0.1 MHz Ionosonde Data Retreival results (CEDA) IIWG format, URSI Codes

4 SCIDIP-ES Objectives Deliver generic sustained services for long-term preservation and usability as part of the data infrastructure for e-science. Harmonize data preservation policies, approaches and tools in the Earth Science Domain. ESRIN/PV2013

5 SCIDIP-ES OAIS Conformant e-infrastructure Unfamiliar and unusable to familiar and usable Expand preserved data capabilities to be interoperable For present and future and among different disciplines

6 Motivation for DVT Continuous access to digital materials Mitigate evolution in technology and domain Software/hardware obsolescence Heterogeneous Earth Science (ES) data Common and domain independent models How to: abstraction for unified data analytics Veracity: beyond 3V s -> 4V s More! Stories continue ESRIN/PV2013

7 The state of art Bit stream interpretation e.g. DDR, EAST, DFDL, DRB and etc. Interpret bits into data Geospatial information standardization e.g. ISO, OGC and etc. Enable meaningful exploitation of the data A unified access layer is not available ESRIN/PV2013

8 OAIS Representation information How does RI look like for my archives? Let me give you a story ESRIN/PV2013

9 Example OAIS RepInfo Network for DVT ES Data Semantic GML COV Structure Other Other Adds meaning to DRB Desc DRB Lib XQuery OAIS RepInfo ESRIN/PV2013

10 Data Virtualization Toolkit Structural representation editing component (SREC) Bit stream interpretation component (BSIC) Mapping component (MC) Visualization component (VC) ESRIN/PV 2013

11 NERC MST USE CASE The Natural Environment Research Council (NERC) Mesosphere-Stratosphere- Troposphere (MST) NERC MST Radar are 46.5 MHz pulsed Doppler radars for studies of atmospheric winds, waves and turbulence. Explored by ncdump, ncbrowse and etc., m An MST sample water vapor density record or DVT (0,0) 86115s

12 DVT Use case DVT Facilitate ES data Interoperability by virtualization (SCIDIP-ES Toolkit) Stepwise Step 1 Step 2 Step 3 Step 4 Each step building on the previous one Describe NetCDF data using DRB Extract data values Convert to CSV/GMLCOV TopCAT Visualization Preserved Data Structural RepInfo Other RepInfo Applications NERC MST DRB description XQuery (for CSV, GMLCOV and etc.), Software (GAEL ESRIN/PV2013 DRB API) tabular, Image, OGC GMLCOV, and etc.

13 Structural description * Hierarchy * Functional definition * Intensive definition * Enumeration:

14 Extensive result in a table

15 More table Results DRB description Header values 1D data 2D data ESRIN/PV2013

16 Intensive result in OGC GMLCOV

17 2D visualization with TOPCAT ESRIN/PV 2013

18 3D visualization with TOPCAT ESRIN/PV 2013

19 Summary Allows to inspect content and structure Standard-based ES information extraction Preview a preservation archives Abstract diverse archives through a unified data access layer Use a common OGC model to facilitate interoperability ESRIN/PV2013

20 Outlook Bit stream to standard based ES services, e.g. OGC WCS, WFS and etc. Promote standard-based frameworks and paradigms to access and manipulate heterogeneous repositories ES Data Semantic GML COV Structure Other Other Adds meaning to DRB Desc DRB Lib XQuery ESRIN/PV2013

21 Useful Links SCIDIP-ES DVT Download DVT Demo video Data Request Broker - DRB API

22