Lesley Wyborn and Ben Evans. nci.org.au. IEEE Big Data in the Geosciences Workshop, Santa Clara National Computational Infrastructure 2015

Transcription

1 Integra(ng Big Geoscience Data into the Petascale Na(onal Environmental Research Interoperability Pla<orm (NERDIP): Successes and Unforeseen challenges Lesley Wyborn and Ben

2 The Australian Research Funding Schemes Two main tranches of funding: Na(onal Collabora(ve Research Infrastructure Strategy (NCRIS) $542M for ($75 M for cyberinfrastructure) Super Science Ini(a(ve $901 million for ($347M for cyberinfrastructure) Annual Maintenance funding of around $180M pa since All programmes were designed ensure that Australian research con(nues to be compe((ve and rank highly on an interna(onal scale. Millions of Dollars Compute Data Tools Networks

3 We went from nothing.. to...

4 The Research Data Storage Infrastructure Progress on Data Ingest as of 16 October, 2015: ~43 Petabytes in 8 distributed nodes 10 PBytes Source: h\ps://

5 Integrated World-class Scien(fic Compu(ng Environment at NCI Data Services THREDDS Server-side analysis and visualization VDI: Cloud scale user desktops on data 10PB+ Research Data Web-time analytics software

6 Na(onal Environment Research Data Collec(ons (NERDC) 1. Climate/ESS Model Assets and Data Products 2. Earth and Marine Observa(ons and Data Products 3. Geoscience Collec(ons 4. Terrestrial Ecosystems Collec(ons 5. Water Management and Hydrology Collec(ons Data Collec(ons CMIP5, CORDEX ACCESS products LANDSAT, MODIS, VIIRS, AVHRR, INSAR, MERIS Digital Eleva(on, Bathymetry, Onshore Geophysics Seasonal Climate Bureau of Meteorology Observa(ons Bureau of Meteorology Ocean-Marine Terrestrial Ecosystem Reanalysis products Approx. Capacity 2 Pbytes 3.3 Pbytes 2 Pbytes 400 Tbytes 600 Tbytes 400 Tbytes 220 Tbytes 290 Tbytes 175 Tbytes

7 10+ PB of Data for Interdisciplinary Science Astronomy (Optical) 200 TB CMIP5 3PB Earth Observ. 2 PB Atmosphere 2.4 PB Water Ocean 1.5 PB Weather 340 TB Bathy, DEM 100 TB Marine Videos 10 TB GA CSIRO ANU Other Na;onal Interna;onal Geophysics 300 TB

8 Managing 10+ PB of Data for Scalable In-situ Access Combined and integrated, the NCI collec(ons are too large to move bandwidth limits the capacity to move them easily the data transfers are too slow, complicated and too expensive even if our data can be moved, few can afford to store 10 PB on spinning disk We need to change our focus to: moving users to the data (for sophis(cated analysis) moving processing to data having online applica(ons to process the data in-situ Improving the sophis(ca(on of users with our help We called for a new form of system design where: storage and various types of computa(on are co-located systems are programmed and operated to allow users to interac(vely invoke different forms of analysis in-situ over integrated large-scale data collec(ons

9 Rethinking Hardware Architectures for Data-intensive Science Work at NCI has also highlighted the need for balanced systems to enable Dataintensive Science including: Interconnec(ng processes and high throughput to reduce inefficiencies The need to really care about placement of data resources Be\er communica(ons between the nodes I/O capability to match the computa(onal power Close coupling of cluster, cloud and storage NCI s Integrated High Performance Environment

10 My take is that Big Data is not just about the V s 1. Volume: data at rest 2. Velocity: data in mo(on (streaming) 3. Variety: many types, forms and structures (or no structures) 4. Veracity: trustworthiness, provenance, lineage, quality 5. Validity: data that is correct 6. Visualiza(on: data in pa\erns 7. Vulnerability: data at risk 8. Value: data that is meaningful 9. V????? 10. V?????

11 Big Data is a rela(ve term where the volume, velocity and variety of data exceed an organisa(ons storage or compute capacity for accurate and (mely decision making Big Data vs High Performance Data We define High Performance Data (HPD) as data that is carefully prepared, standardised and structured so that it can be used in Data-Intensive Science on HPC (Evans et al., 2015) To get on top of the Data Tsunami, we need to convert Big data collec(ons into HPD by Aggrega(ng data into seamless pre-processed data products Crea(ng hyper-cubes and self describing data arrays h\p:// 1964: 1KB = 2m of tape or ~ 20 cards 2014: a 4 GB Thumb drive = ~8000 Km of Tape or ~83 million cards 2014: 20 PB of modern storage = ~ 32 trillion metres of tape ~ 320 trillion cards

12 Crea(ng HPD collec(ons: eg the Landsat Cube The Landsat cube arranges 636,000 Landsat Source scenes spa(ally and temporally, to allow flexible but efficient large-scale in-situ analysis The data is par((oned into spa(ally-regular, (me-stamped, band-aggregated (les which are presented as temporal stacks. Temporal Stack Spa(ally par((oned (les

13 Current Landsat Holdings Reforma\ed as HPD 4M Spa(ally-Regular Time- Stamped Tiles (0.5 PB) 636,000 Landsat Source Scene Datasets (~52 x Pixels)

14 High-Res, Mul(-Decadal, Con(nental-Scale Analysis Water Detec;on from Space 15 Years of data from LS5 & LS7( ) 25m Nominal Pixel Resolu(on Approx. 133,000 individual source scenes in approx. 12,400 passes En(re archive of 1,312,087 ARG25 (les => 21x10 12 pixels can be processed in ~8 hours

15 Data Pla<orms of today need to scale down to small users

16 Scaling down to the smaller users Do we enable individual scenes to be downloaded for locally hosted small scale analysis? Or do we facilitate small scale analysis, in-situ on data sets that are dynamically updated?

17 Biodiversity & Climate Change VL Climate & Weather Systems Lab Introducing the Na(onal Environmental Data Interoperability Research Pla<orm (NERDIP) emast Speddexes ereefs AGDC VL All Sky Virtual Observatory Workflow Engines, Virtual Laboratories (VL s), Science Gateways VGL Globe Claritas VHIRL Open Nav Surface Ferret, NCO, GDL, GDAL, GRASS, QGIS Models Fortran, C, C++, MPI, OpenMP Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS TERN AuScope Data. gov.au Digital Bathymetry & Eleva=on Tools Data s Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

18 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: Enabling Mul(ple Ways to Interact with the Data emast AGDC All Sky Virtual Climate Infrastructure & Weather Speddexes ereefs to Lower Barriers VL Observatory VGL to Entry Globe VHIRL Systems Lab Claritas Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users Fortran, C, C++, MPI, OpenMP Tools Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS Data Discovery TERN AuScope Data s Data. gov.au Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer Data Library Layer 1 netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Climate/Weather/Ocean Data Pla<orm Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

19 Biodiversity & Climate Change VL NERDIP: Enabling Mul(ple Ways to Interact with the Data emast AGDC All Sky Virtual Climate Infrastructure & Weather Speddexes ereefs to Lower Barriers VL Observatory VGL to Entry Globe VHIRL Systems Lab Claritas Workflow Engines, Virtual Laboratories (VL s), Science Gateways Open Nav Surface Ferret, NCO, GDL, GDAL, GRASS, QGIS Models Ace Users Fortran, C, C++, MPI, OpenMP Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA TERN AuScope Data. Digital Bathymetry AODN/IMOS Data s gov.au & Eleva=on Tools Data s Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer Data Library Layer 1 netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Climate/Weather/Ocean Data Pla`orm Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

20 Pla<orms Free Data from the Prison of the s s are for visi;ng, pla`orms are for building on s present aggregated content in a way that invites explora(on, but the experience is pre-determined by a set of decisions by the builder about what is necessary, relevant and useful. Pla<orms put design decisions into the hands of users: there are innumerable ways of interac(ng with the data Pla<orms offer many more opportuni(es for innova(on: new interfaces can be built, new visualisa(ons framed, ul(mately new science rapidly emerges Tim Sherra\ h\p://

21 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: Enabling Ace Users to Interact with the Data emast AGDC All Sky Virtual Climate Infrastructure & Weather Speddexes ereefs to Lower Barriers VL Observatory VGL to Entry Globe VHIRL Systems Lab Claritas Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users Fortran, C, C++, MPI, OpenMP Tools Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS Data Discovery TERN AuScope Data s Data. gov.au Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Pla<orm Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

22 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: Enabling Ace Users to Interact with the Data emast AGDC All Sky Virtual Climate Infrastructure & Weather Speddexes ereefs to Lower Barriers VL Observatory to Entry VGL Globe VHIRL Systems Lab Claritas Workflow Engines, Virtual Laboratories (VL s), Science Gateways Fortran, C, C++, MPI, OpenMP Tools Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS Data Discovery TERN AuScope Data s Data. gov.au Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Pla<orm Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

23 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: Enabling Applica(on Developers to Interact with the Data Infrastructure to Lower Barriers to Entry Climate & Weather Systems Lab APPLICATION emast AGDC All Sky Virtual Speddexes ereefs VL Observatory Workflow Engines, Virtual Laboratories (VL s), Science Gateways VGL Globe Claritas Fortran, FOCUSSED C, C++, Python, R, Visualisa;on ANDS/RDA AODN/IMOS TERN MPI, OpenMP AuScope Data. Ace Users DEVELOPERS MatLab, IDL Drish; gov.au Tools VHIRL Data Discovery Data s Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS DATA MANAGEMENT WCS WPS SOS W*S Metadata Layer Data Library Layer 1 netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Pla<orm FOCUSSED Libgdal DEVELOPERS Airborne Climate/Weather/Ocean FITS Geophysics SEG-Y Line data LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

24 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP Territorial Wars: Applica(on Developers vs Data Managers Infrastructure to Lower Barriers to Entry emast AGDC All Sky Virtual Climate & Weather Speddexes ereefs VL VGL Globe VHIRL Open Nav Observatory Systems Lab Claritas Surface Workflow APPLICATION Engines, Virtual Laboratories (VL s), Science Gateways Fortran, C, C++, Python, R, Visualisa;on ANDS/RDA AODN/IMOS TERN MPI, OpenMP MatLab, IDL AuScope Data. Digital Bathymetry Ace Users Data Discovery Drish; gov.au & Eleva=on Tools Data s Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) FOCUSSED DEVELOPERS Services Layer (expose data models Fast whole-of-library & seman=cs) catalogue Direct Access RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 Data Pla<orm DATA MANAGEMENT HDF5 MPI-enabled FOCUSSED DEVELOPERS Lustre HDF5 Serial Other Storage (op;ons)

25 NERDIP: Applica(ons Replica(ng Ways of Interac(ng with the Data Biodiversity & Climate Change VL Climate & Weather Systems Lab emast Speddexes ereefs AGDC VL All Sky Virtual Observatory VGL Globe Claritas VHIRL Open Nav Surface Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ferret, NCO, GDL, GDAL, GRASS, QGIS Models Fortran, C, C++, MPI, OpenMP Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS TERN AuScope Data. gov.au Digital Bathymetry & Eleva=on Tools Data s Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

26 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: Loosely coupling Applica(ons and Data via a Services Layer Infrastructure to Lower Barriers to Entry Climate & Weather Systems Lab APPLICATION emast AGDC All Sky Virtual Speddexes ereefs VL Observatory Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users VGL Globe Claritas Fortran, FOCUSSED C, C++, Python, R, Visualisa;on DEVELOPERS ANDS/RDA AODN/IMOS TERN MPI, OpenMP AuScope Data. MatLab, IDL Drish; gov.au Tools VHIRL Data Discovery Data s Open Nav Surface Digital Bathymetry & Eleva=on Services Layer (expose data models & seman=cs) Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Direct Access SERVICES INTERFACE Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. DATA MANAGEMENT Data Library Layer 1 Climate/Weather/Ocean Libgdal Data FITS Pla<orm Airborne Geophysics Line data SEG-Y FOCUSSED DEVELOPERS LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

27 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: Loosely coupling Applica(ons and Data via a Services Layer Infrastructure to Lower Barriers to Entry Climate & Weather Systems Lab APPLICATION emast AGDC All Sky Virtual Speddexes ereefs VL Observatory Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users VGL Globe Claritas Fortran, FOCUSSED C, C++, Python, R, Visualisa;on DEVELOPERS ANDS/RDA AODN/IMOS TERN MPI, OpenMP AuScope Data. MatLab, IDL Drish; gov.au Tools VHIRL Data Discovery Data s Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer Data Library Layer 1 HP Data Library Layer 2 netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Climate/Weather/Ocean DATA MANAGEMENT HDF5 MPI-enabled Lustre Libgdal Data FITS Pla<orm Airborne Geophysics Line data SEG-Y FOCUSSED DEVELOPERS LAS LiDAR HDF5 Serial BAG Other Storage (op;ons)

28 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: the Metadata Layer emast AGDC All Sky Virtual Climate Infrastructure & Weather Speddexes ereefs to Lower Barriers VL Observatory VGL to Entry Globe VHIRL Systems Lab Claritas Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users Fortran, C, C++, MPI, OpenMP Tools Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS Data Discovery TERN AuScope Data s Data. gov.au Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

29 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models NERDIP: The Data Layers emast AGDC All Sky Virtual Climate Infrastructure & Weather Speddexes ereefs to Lower Barriers VL Observatory VGL to Entry Globe VHIRL Systems Lab Claritas Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users Fortran, C, C++, MPI, OpenMP Tools Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS Data Discovery TERN AuScope Data s Data. gov.au Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

30 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models emast AGDC All Sky Virtual Climate Infrastructure & Weather Speddexes ereefs to Lower Barriers VL Observatory VGL to Entry Globe VHIRL Systems Lab Claritas Workflow Engines, Virtual Laboratories (VL s), Science Gateways Fortran, C, C++, MPI, OpenMP NERDIP: Infrastructure to Lower Barriers to Entry Ace Users Tools Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS Data Discovery TERN AuScope Data s Data. gov.au Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer Data Library Layer 1 netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Climate/Weather/Ocean Data Pla<orm Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

31 Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models Climate & Weather Systems Lab NERDIP: Infrastructure to Lower Barriers to Entry AGDC VL VGL Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users Fortran, C, C++, MPI, OpenMP Tools Python, R, MatLab, IDL emast Speddexes ereefs Visualisa;on Drish; ANDS/RDA All Sky Virtual Observatory AODN/IMOS Globe Claritas VHIRL Data Discovery TERN AuScope Data s Data. gov.au Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Open Nav Surface Digital Bathymetry & Eleva=on Metadata Layer Data Library Layer 1 Direct Access netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. NetCDF-4 Climate/Weather/Ocean Fast whole-of-library catalogue NetCDF-4 RDF, LD Libgdal WMS WFS Data Pla<orm [FITS] WCS WPS Airborne Geophysics Line data SOS [SEG-Y] Open DAP Services Layer (expose data model & seman=cs) BAG LAS LiDAR HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (options)

32 NERDIP: Infrastructure to Lower Barriers to Entry Biodiversity & Climate Change VL Ferret, NCO, GDL, GDAL, GRASS, QGIS Models Climate & Weather Systems Lab AGDC VL Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ace Users Fortran, C, C++, MPI, OpenMP Tools Python, R, MatLab, IDL emast Speddexes ereefs Visualisa;on Drish; ANDS/RDA All Sky Virtual Observatory AODN/IMOS VGL Globe Claritas VHIRL Data Discovery TERN AuScope Data s Data. gov.au Open Nav Surface Digital Bathymetry & Eleva=on Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

33 NERDIP: Enabling Mul(ple Ways to Interact with the Data Biodiversity & Climate Change VL Climate & Weather Systems Lab emast Speddexes ereefs AGDC VL All Sky Virtual Observatory VGL Globe Claritas VHIRL Open Nav Surface Workflow Engines, Virtual Laboratories (VL s), Science Gateways Ferret, NCO, GDL, GDAL, GRASS, QGIS Models Fortran, C, C++, MPI, OpenMP Python, R, MatLab, IDL Visualisa;on Drish; ANDS/RDA AODN/IMOS TERN AuScope Data. gov.au Digital Bathymetry & Eleva=on Tools Data s Na=onal Environmental Research Data Interoperability PlaDorm (NERDIP) Services Layer (expose data models & seman=cs) Direct Access Fast whole-of-library catalogue RDF, LD Open DAP WMS WFS WCS WPS SOS W*S Metadata Layer netcdf-cf HDF-S ISO 19115, RIF-CS, DCAT, etc. Data Library Layer 1 Climate/Weather/Ocean Libgdal FITS Airborne Geophysics Line data SEG-Y LAS LiDAR BAG HP Data Library Layer 2 HDF5 MPI-enabled Lustre HDF5 Serial Other Storage (op;ons)

34 Key Messages on Big Data in the Geosciences Data at scales of today have to be built as shared global facili(es based around na(onal ins(tu(ons. Domain-neutral interna(onal standards for data collec(ons and interoperability are cri(cal for allowing complex interac(ons in HP environments both within and between HPD collec(ons No one can do it alone. No one organisa(on, no one group, no one country has the required resources or the exper(se. Shared collabora(ve efforts such as Research Data Alliance, the Earth Systems Grid Federa(on (ESGF), the Belmont Forum, EarthServer, the Oceans Data Interoperability Pla<orm (ODIP), EarthCube, G and OneGeology are needed to realise the full poten(al of the new data intensive science infrastructures For it now takes a village of partnerships to raise a HPD data center in a Big Data World h\p:// h\ps:// 2010/06/iStock_ XSmall.jpg