Releasing ESO Public Survey Data through the Phase 3 Jörg Retzlaff European Southern Observatory Data Management and Operations Division, Archive Science Group (ASG)
Outline Releasing ESO Public Survey Data through the Phase 3 1 2 3 4 Why? The context ESO Public Surveys: projects & progress Phase 3 Policies How? Who does what? Process & responsibilities Types of products & ESO Science Data Products Standard System components User support Monitoring Phase 3 Results Status of Public Survey Phase 3 Data Submissions Catalogue Facility ESO Science Archive Facility Data Access Conclusions Interoperability and the role of VO standards 2
ESO Public Surveys VISTA: 6 surveys started Apr 2010 (P85) VHS - 20000 deg 2 YJHKs (Ks<20 AB) VIKING - 1500 deg 2 ZYJHKs (Ks<21.2 AB) VIDEO - 3 Deep Extragalactic Fields Ultra-VISTA - Ultra-deep ZYJHKs + NB118 in the COSMOS field VVV - Variability study of 520 deg 2 in bulge+plane plus multi-color map VMC - Magellanic Survey VLT Survey Telescope: 3 surveys, started 15 Oct 2011 (P88) VST-Atlas 4500 deg 2 UVRIz, like SDSS KIDS 1500 deg 2 UVRI, 2.5 mag deeper than SDSS VPHAS+ 1800 deg 2 UVHaRI in the Southern Galactic Plane Spectroscopic Surveys: started 1 Jan 2012 PESSTO: 30+60n on NTT (SOFI+EFOSC) Gaia-ESO: 30+30n on VLT-UT2 (FLAMES) 3
Progress of Public Surveys Courtesy: M. Arnaboldi/ESO Survey Team Ø Max. fraction of completion: 56% for VHS, 55% for VST-ATLAS Ø Significant progress for all surveys (though at a different rate) Ø 30+ publications resulting from PSs; 11 ESO PRs; see also: Survey Science Workshop @ESO/Garching in October 2012. 4
ESO Phase 3 Policies The ESO archive is the collection point for the survey products and the primary point of publication/ availability of these products to the ESO community. (Cou104, Dec 2004) Phase 3 PIs of ESO observing programmes return data products to ESO Storage in the ESO Archive (long-term data preservation) Data publication/distribution to the scientific community ESO s policies governing Phase 3 are specific to the type of observing programme. Phase 3 is mandatory for ESO Public Surveys and for ESO Large Programmes since period 75; available also for other ESO observations(!) Further allocation of telescope time for ESO PSs is subject to the completion of the Phase 3. 5
Phase 3 Process & Responsibilities Phase 3 denotes the process of preparation, submission, validation and ingestion of science data products for storage in the ESO Science Archive Facility, and subsequent data publication to the scientific community. 1. Data preparacon 8. Data publicacon 2. User s data validacon 7. Archival storage 3. Data release definicon 6. Content validacon 4. Data transfer to ESO 5. AutomaCc release validacon Closing the data release P.I. Data provider The survey P.I. is responsible for the quality of the reduced data products and the associated documentation ( data release description ). ESO defines the required data format ( ESO Science Data Products Standard ), provides dedicated tools, user documentation and direct support for Phase 3 data providers. http://www.eso.org/sci/observing/phase3.html 6
Types of Data Products Survey Tile Image Astrometrically & photometrically calibrated FITS image with associated confidence map; Quality params. (limiting magnitude, PSF size, etc.) Generally: processing provenance (keyword PROVi) to trace back to the original (raw) data. 1-dim. extracted wavelength-calibrated spectrum FITS binary table format Support for 2d spectral frames as ancillary files Exposure map of the VISTA survey tile. Compliant to IVOA Spectral Data Model (v1) Survey Source List Single-band source catalogues directly extracted from the (tile) image associated to its originating image (provenance keyword PROVi) Based on nightly calibrations, degenerate w.r.t. physical sources Science Catalogues Homogeneous merged multi-band catalogue for each survey (possibly per region) Global astrometry/photometry; cross-calibrated using overlapping tiles and across bands Multiple detections merged, i.e. unique entries (ultimately) Uniform tabular structure including content descriptors (employing UCDs) Supports variability surveys: multi-epoch photometric catalogues (i.e. light curves) Tile-by-tile scheme supported for data delivery Dedicated query interface: ESO Catalogue Facility 7
ESO Science Data Products Standard European Organisation for Astronomical Research in the Southern Hemisphere Data Management and Operations Division Prepared: Phase 3 User Documentation ESO Science Data Products Standard Doc. No.: GEN-SPE-ESO-33000-5335 Issue: 5 Date: 11/01/2013 J. Retzlaff, N. Delmotte Name Date Signature Approved: M. Arnaboldi Released: Organisation Européenne pour des Recherches Astronomiques dans l Hémisphère Austral Name Date Signature M. Romaniello Name Date Signature ESO, Karl-Schwarzschild-Str. 2, 85748 Garching bei München, Germany Europäische Organisation für astronomische Forschung in der südlichen Hemisphäre Identification of product types and their data formats depending on instrument/mode. Definition of relevant keywords for data characterization, quality, provenance etc. Issue 1: EDP Standard, Date: 27/11/2010 Evolution driven by: the data to be handled (e.g. VISTA, VST, SPS s, catalogues), i.e. by the data release schedule for PSs; Archive services to be offered (example: Catalogue Facility); Led to Issue 2 (07/03/2011), Issue 3 (22/05/2012), Issue 4 (15/10/2012) Current Issue: #5, Date 11/01/2013 covering all PS data products: imaging & spectroscopy http://www.eso.org/sci/observing/phase3/p3sdpstd.pdf 8
Role of the ESO Science Data Products Standard Content validacon ESO DICD p3ck Metadata interface P3 User support P3 User doc. ESO/SDP Standard P3 web pages Catalogue facility Validator OCA rules IngesCon tool Basis for the data provider (pipeline development) Implemented in the Phase 3 data flow in terms of programmed rules (OCA) Many dependencies of infrastructure components Central role to Phase 3 operations; Any change is expensive ESO/SAF user Query forms 9
Phase 3 System Components The Phase 3 Validator is a command- line applicacon that verifies the data s compliance with the format standard and the validity of the FITS header keywords against predefined rules. Start of operacons: March 2011 The Phase 3 Release Manager is a web applicacon that allows the P.I. to define data colleccons and releases and to manage the Phase 3 delegacon to co- invescgators. hwp://www.eso.org/rm Any FTP client like l[p, filezilla etc. Note: the requirement for SSL/TLS was dropped (Jun 2012). The P.I. or a delegate transfers the data via standard FTP to the dedicated staging area at ESO headquarters. 10
Workshop for Phase 3 users Documentation Phase 3 Tools http://www.eso.org/sci/observing/phase3.html Helpdesk Phase 3 User Support 11
Monitoring the Phase 3 Progress The ESO Archive Science Group is monitoring the Phase 3 process and reports to Public Survey Panels/OPC 12
ESO Public Survey Data Products PHASE 3 RESULTS
Status of 1 st Phase 3 Data Submissions for VISTA Public Surveys The VISTA Public Survey data products released in 2011/2012 cover almost 2500 square degrees of the Southern Hemisphere. VHS- green VVV- red VMC- yellow VIDEO- pink UltraVISTA- black (EnCre survey footprints are shown in light blue.) Total data volume: ~5.5 TB 112,296 files http://archive.eso.org/wdb/wdb/adp/phase3_main/form 14
Catalogue Facility Query Interface 15
ESO Phase 3 Data Release Documentation Example: Ultra-VISTA Release descrip7on Provide short broad overview of the program, with an overview/layout of the observacons EssenCal input for data content validacon. Release content - Extended liscng for each sky posicon, filters, exposure Cmes, seeing Release notes ReducCon method used, calibracon procedures, data quality Data format DescripCon of files in this data release, associated files, and naming convencons Acknowledgements Bibliographic reference to be included when using these data. p.16 Phase 3 Process 36 th UC MeeCng, ESO Garching, 24.04.2012
Current Status of Public Survey Phase 3 Submissions VISTA PSs 2 nd Phase 3: in total ca. 6.5 TB of data uploaded. q VIDEO: 80 GB, feedback on content validation q VMC: 30 GB, to be archived q VHS: 4.6 TB, archiving q VVV: 1.4 TB, data archived q VIKING: 477 GB uploaded, to be archived q Ultra-VISTA agreed to submit in 2013 VST/OmegaCAM started operations on Oct 15, 2011. First Phase 3 submissions for VST Public Surveys (Mar-Apr 2013): VST-ATLAS: 3.09 TB KIDS: 718 GB VPHAS+: 766 GB In total: ca. 4.6 TB - data content validation is currently taking place. 18
ESO Science Archive Facility Data Access Statistics (1) Courtesy: M. Arnaboldi/ESO Survey Team/ASG 19
Data Access Statistics (2) Number of files downloaded per data product type Ø 5.6 TB of data products, >2x10 4 files downloaded from the ESO SAF since December 2011. 20
Impact of Data Products Source: ESO Telescope Bibliography http://telbib.eso.org Ø In 2012: 874 publicacons based on ESO data, Ø 25% thereof using archival data. 21
Internal Data Products UC36.R.3: The ESO data archive should contain calibrated data where at least instrumental signatures are removed to increase the value of the archive for the ESO users. Science-grade DPs to be produced and integrated for seamless access of internal and external data products through the ESO archive user interfaces. Timeline: Publication of UVES Echelle data in Q4 2013 (backlog+stream of new data), then (very preliminary): X-Shooter-Echelle, FLAMES- MEDUSA, and HAWK-I and VIMOS imaging (UK in-kind contribution) 22
Conclusions q The currently on-going eleven ESO public survey projects are progressing very well. q The ESO Science Archive Facility is being populated with resulting science-grade data products including calibrated images, spectra, and catalogues. q Start of the regular production and publication of pipelineprocessed science-grade data products from UVES/Echelle in 2013/Q4. q The Phase 3 process, implementing ESO s policies for public surveys, relies on the ESO Science Data Products Standard. q VO standards are being considered when applicable (examples: UCDs, spectrum data format è A. Micol s talk). q Collaborations on s/w being developed in the VO-context (VOview, OpenCADC libs). 23