Datamanagement at the European Southern Observatory: Strategies and Challenges. Michael Sterzik, ESO Datamanagement and Operations Division

Transcription

1 Datamanagement at the European Southern Observatory: Strategies and Challenges Michael Sterzik, ESO Datamanagement and Operations Division

2 European Southern Observatory - builds and operates state-of-the-art ground-based astronomical facilities most productive Observatory world-wide inter-governmental organisation supported by 16 member states involvement of ~1/2 of the astronomical community world-wide

3 Data Challenges in Astronomy! diversity of data collections! multi-mission, multi-messenger! multi-wavelength! time-domain! Astronomy becomes more and more data intensive! increasing volume and/or increasing complexity! depends on Infrastructure! to store, access, preserve and share! to process, analyse, synthesise! requires a (widely accepted) ecosystem! data standards, representations ( FITS )! metadata definitions! interoperability protocols ( VO )! associated SW tools! information services

4 Data Policy of ESO Council doc. on VLT/VLTI Science Operations Policy (2004)! access to the facility (general observing time, Guaranteed Time)! data rights: proprietary period (in general one year)! data access: public through an electronic archive! data products: level 0 (quality control) and 1 (science grade) by ESO! data analysis: higher level by community! necessary for defining a data management plan

5 Research cycle Astronomy

6 mapping research into a work/data flow Phase II Observing Block Preparation Phase I Long Term Schedule Short Term Schedule Program Preparation Metadata OB Execution Quality Control data products Data Enhancement Archive I/O Phase III raw data

7 Business model of Astronomy periodic (6mo) review of observing proposal: oversubscription rate 3-5 best ideas/proposals win observing time after peer-review Principal Investigators enjoy proprietary data rights for one year after: data go public, pressure to publish increases

8 Volume Volume versus Value! raw-data are use-less for direct scientific analysis! data sets must be complete, calibratable, and pass quality control! data processing ( reduction ) to create data products! quality control, master calibrations (Level 0)! transformation from instrumental to physical units (L 1)! combination of observations: deep and wide maps or spectra (L 2)! catalogs, high level data products after scientific analysis (L 3) Effort, Value

9 Strategies for the Generation of Content: Science Data ESO! In-house generation of Data Products (IDPs):! quality ensured through a standardised process of data acquisition for science and calibration data! near-real time quality control process ensures certified master calibrations (L0)! un-attended processing through certified pipelines! goal: science grade data for all popular instruments (L1)! External Data Products (EDPs):! provided by public surveys and large programs (deliverables)! programs specifically selected for high legacy value! most use dedicated (non-eso) user-pipes (eg CASU)! goal: advanced products (L2, L3)! perspective: user community at large contributes EDPs quality assurance: published datasets only? acknowledgement: specific DOI?

10 Phase 3 process for EDPs 1.! Data! prepara(on! 2.! User s!data! valida(on! 3.! Data! release! defini(on! 4.! Data! transfer!to! ESO! Closing the data release P.I. Data provider 8.! Data! publica(on! 7.! Archival! storage! 6.! Content! valida(on! 5.! Automa(c!! release! valida(on! ESO defines the required data format, provides dedicated tools, user documentation and direct support for Phase 3 data providers. The data provider i.e. survey P.I. is responsible for the quality of the reduced data products and the associated data release documentation.

11 Data Collections: Phase 3 Releases

12 ESO Data Portal: acknowledgements DOIs?

13 Linking Publications and Data! Scientific Return is a KPI for on Research Infrastructure! data is the primary scientific return! bibliometrics widely used as a proxy for quantity (number) quality (citations) merit (quality/cost)! managerial tool assess impact of science policies, implementation, operations,! Community Access to Publications and Data! enables active archive research! allows to recognise the authorship (in principle)! the impact/value of data archive research is not always appreciated (no much data-metrics)

14 way, for instance by number of observing hours, by actual share of data used in the papers (as many scientific articles use data from more than one observatory), or by budget (telescope construction costs and maintenance). Comparative(Inter-facility) Bibliometrics When comparing publication statistics among different observatories, it is essential to assess the selection criteria applied by each observatory. To the best of our knowledge, the observatories shown in this graph include only papers that actually use observational data from their facilities (as opposed to merely referencing them). All papers were published in refereed! journals. No. of publications Publications of major observatories by year 2000! The statistics shown in Fig. 3 and Table 3 were obtained as follows: U. Grothkopf et al., ESO total HST! Spitzer!VLT/VLTI! NRAO Chandra XMM Swift La Silla Keck Gemini ING (may contain duplicates)!subaru!eso survey tel. APEX (ESO) ALMA (Eur.) Fig. 3: Refereed publications by ESO and other observatories (as of Feb. 2015) Thick lines: ESO facilities. Thin lines: other ground-based facilities. Dashed lines: space-based facilities. useful Please note to estimate that selection the criteria global for inclusion performance exclusion of papers of a vary facility among observatories (Observatory) but biased, depends on how librarians count the contributions

15 Intra-facility bibliometrics useful to asses the relative performance of different components

16 Intra-facility bibliometrics 2.0 All Large Programs UT Productivity Number of Publications per Program Large GTO DDT TOO Guaranteed Time Average Target of Opportunity Discretionary Time Years between Program scheduled and Publication useful to asses the science policy and its implementation

17 Links to data and publications telbib.eso.org

18

19 All original data archive.eso.org

20 Content aggregation and exploration cdsportal.u-strasbg.fr

21 Archival data enabled publication fraction start of facility operations start archive population with DP archive services interoperability U. Grothkopf et al.,

22 Archival data enabled publication fraction HST start of facility operations start archive population with DP archive services interoperability U. Grothkopf et al.,

23 and costs? as a fraction of total RI operations costs! data archive operations! archive infrastructure TCO (1PB, 3 safe copies) 0.3-1%! content management (production, curation, assurance) ~10%! data generation! facility time for calibrations ~4%! lacking resources for data archive developments! apply existing frameworks, standards as much as possible!! do not re-invent archive infrastructure!

24 Summary! astronomy as data-intensive science! diversity of datasets with respect to quality, quantity, and origin! international data standards largely applied (IAU)! mapping the research process into data-flow process! allows a systemic approach to reach consistent data quality! allows to add-value through the generation of data products! allows to add-value through the sharing of data products! astronomical data archives! are an essential science resource! benefit (= science return + ) > costs! challenges ahead! recognise content, services, interoperability as priority! incentives for data providers