NCAS Research Forum on Data Archiving and Visualisation York 08/09/2015 Philip Stier, Nick Schutgens, Duncan Watson-Parris Department of Physics, University of Oxford Bryan Lawrence, Philip Kershaw Centre for Environmental Data Archival Analysis, NERC/STFC Climate Processes Group
Maximising observational constraints Intercomparison of data and models crucial for all aspects of climate and earth system science: Climate Processes Group
Maximising observational constraints Intercomparison of data and models crucial (but difficult) for all aspects of climate and earth system science: Evaluation of aerosols in HadGEM-UKCA: MODIS satellite imager (irregular lat/lon, HDF) AATSR/ESA CCI satellite imager (irregular lat/lon, netcdf) CALIOP satellite lidar (irregular lat/lon/alt, HDF) AERONET sunphotometer network (lat/lon, ASCII) In-situ aircraft data (irregular lat/lon/alt/pres, ASCII, netcdf,..) In-situ surface networks (irregular lat/lon, ASCII, ) Climate Processes Group
Maximising observational constraints Intercomparison of data and models crucial for all aspects of climate and earth system science: Context for in-situ aircraft measurements: MODIS satellite imager (irregular lat/lon, HDF) AATSR/ESA CCI satellite imager (irregular lat/lon, netcdf) CALIOP satellite lidar (irregular lat/lon/alt, HDF) AERONET sunphotometer network (lat/lon, ASCII) HadGEM-UKCA simulations (lat/lon, hybrid height, netcdf) In-situ surface networks (irregular lat/lon, ASCII, ) Climate Processes Group
CIS: open source python toolbox Generic tool to read, colocate, aggregate, analyse, visualise datasets Handling of complex gridded and ungridded data in many formats Simple command line syntax with many options Flexible approach through plug-ins, e.g. for new data sources Open source software & deployed for community use on JASMIN Read Plug- ins for gridded and ungridded data Analyse Colocation, Aggregation, Statistics, Algebra Output Plots, statistics, data in netcdf CEDA Database Web interface
Global Aerosol Synthesis and Science Project synthesized aircraft data in GASSP netcdf: Read-in and plotting of many datasets reduced to one simple command GASSP Data Collection & Harmonisation BAMS paper Reddington et al. (in prep.)
GASSP aircraft data cis plot ALT:*.nc --nasabluemarble Complex analysis: Read model and AERONET Colocate spatially and temporally Difference colocated data Plot Requires only 6 lines of CIS commands
GASSP aircraft data GASSP aircraft cis plot ALT:*.nc --nasabluemarble MODIS AOD CloudSat SEVIRI CALIOP Plug- in interface for new data Community Intercomparison Suite
Colocation Colocation method: 1. Specify searchbox Horizontal distance Vertical distance Time separation 2. Specify operation Nearest neighbour (time) Nearest neighbour (space) Average User plug-in CIS col <native file> <native variable>:<native file>:<colocation method> - o <file> This file provides the new spatio- temporal sampling This file provides the data that will be resampled Nearest neighbour or linear interpolation Output (netcdf)
Colocation of UKCA model to GASSP aircraft data: ICEALOT CCN measurements Simulated CCN Difference Colocation of hybrid-height model coordinates and data on height/pressure CIS produces netcdf output for subsequent analysis
Community Intercomparison Suite
CIS user workshop Draft agenda Welcome and showcase 9:00 9:30 Basic interface and workflow introduction 9:30 10:00 Hands- on with CIS session I: Plotting 10:00 10:45 Break 10:45 11:15 Collocation sampling considerations 11:15 11:45 Collocation command introduction 11:45 12:00 Hands on with CIS session II: Collocation 12:00 13:00 Lunch 13:00 14:00 CIS as a Python library 14:00 14:30 Contributing to CIS 14:30 14:40 Plugin development 14:40 15:40 Break 15:40 16:00 Hands on with CIS session III: Plugin development 16:00 17:30
CIS: open source python toolbox Generic tool to read, colocate, aggregate, analyse, visualise datasets Handling of complex gridded and ungridded data in many formats Simple command line syntax with many options Flexible approach through plug-ins, e.g. for new data sources Open source software & deployed for community use on JASMIN Read Plug- ins for gridded and ungridded data Analyse Colocation, Aggregation, Statistics, Algebra Output Plots, statistics, data in netcdf CEDA Database Web interface
CIS: open source python toolbox Generic tool to read, colocate, aggregate, analyse, visualise datasets Handling of complex gridded and ungridded data in many formats Simple command line syntax with many options Flexible approach through plug-ins, e.g. for new data sources Open source software & deployed for community use on JASMIN Read Plug- ins for gridded and ungridded data Analyse Colocation, Aggregation, Statistics, Algebra Output Plots, statistics, data in netcdf CEDA Database Web interface
Conclusions Summary: Status: CIS is an open source python toolbox to read, colocate, aggregate, analyse, visualise datasets CIS is available as open source software & deployed for community use on JASMIN CIS is generic and applicable to a wide range of user communities CIS has been developed on intermittent funding from STFC and NERC CIS 1.2 and webpage are ready and will be shortly announced to community CIS is being used by RAL for EUMETSAT projects and in the NERC IMPALA and CLARIFY projects CIS is currently unfunded and as capability not suitable for NERC funding routes We need funding to further develop CIS and to support community Future work: Robust integration with elastic search of data archived at CEDA/CEMS Need for a project webpage toolkit integrating CIS and elastic search