Usage statistics and archiving process of VizieR data in the VO context

Usage statistics and archiving process of VizieR data in the VO context

VO Implementation status VO Implementation status Application VOTable (1.1 1.2 1.3) Semantic Data Access Layer Data Model MOC SAMP UCD1 UCD1+ Simple Cone Search TAP 1.1 SIA (V1) SSA (V1) ObsTAP Photometry Model (IVOA note) ObsCore Available through Aladin and to query tables using a MOC Need some arrangements to facilitate client access! beta release planned before the end of the year Providing Photometric Data Measurements Description in VOTables (S.Derriere) mandatory items only VizieR catalogues and the TAP VizieR service are in registries IVOA 2015 (VizieR) - Gilles Landais 2/10

VO Output statistics VOTable output statistics Output type evolution CDS Xmatch API Number of queries per mounth (log) 1,00E+08 1,00E+07 1,00E+06 1,00E+05 1,00E+04 1,00E+03 1,00E+02 VOTable TSV HTML 700 IP/day Number queries average (VOTable) 2014 ~410,000 queries/day 2015 ~190,000 queries/day 92% of VOTable output comes from Simple Cone Search queries Date Output type repartition 35000000 Cone search in VOTable output 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% HTML TSV VOTable 30000000 Cone Search 25000000 VOTable 20000000 15000000 10000000 5000000 0 2014/03 2014/07 2014/11 2015/03 2014/01 2014/05 2014/09 2015/01 2015/05 Date IVOA 2015 (VizieR) - Gilles Landais 3/10

VO Output statistics TAPVizieR statistics TAPVizier contains all VizieR tables (except obsolete catalogues) Statistics 2015, without bots and registry queries (~26,000 queries/day): Num ber of queries 8000 7000 6000 5000 4000 3000 2000 1000 sync Number of TAP queries per mounth async 10 IP/day ~150 queries/day 7 10% queries/day contain ADQL geometrical functions 0 Need some arrangements to facilitate client access! Difficulties to work with TAPVizieR, because: Custom use of schema due to the the important number of columns Columns/tables names which need quotes. To provide quoted names in TAP_SCHEMA and VOSI output? IVOA 2015 (VizieR) - Gilles Landais 4/10

The VO visibility The VO visibility The VO is an adapted framework to provide data in the preservation context VO standards (protocols, formats, registries (OAI-PMH)) guaranty reusable data Matches with definition of the Access layer of OAIS (Open Archive Information system) OAIS architecture VO framework IVOA 2015 (VizieR) - Gilles Landais 5/10

The VizieR contents Assigning UCD The most popular UCD in VizieR Usage number CDS documentalists pay particular attention to UCD attribution A distribution with a long tail Main UCD (position, magnitude) are well assigned But, important usage of generic UCD and sometimes not optimum 50% columns 40000 35000 30000 25000 20000 15000 10000 5000 0 The difficulty to have a perfect matching! 75% UCD attribution repartition 90% sorted by ranking and restricted to the 500 most popular UCD rank others stat.error meta.record meta.id;meta.main meta.note meta.id pos.eq.ra;meta.main pos.eq.dec;meta.main meta.code.error meta.code meta.number meta.ref.url time.epoch phot.mag;em.opt.v phys.abund stat.fit.param spect.line.eqwidth meta.ref phot.mag;em.opt.b phot.mag;em.opt.i meta.ref;pos.frame others 434,867 columns 3350 different UCD 6,5% columns with no UCD IVOA 2015 (VizieR) - Gilles Landais 6/10

The VizieR contents The UCD assignment CDS documentalists set UCD1 for each columns using a UCD1 builder UCD1+ is constructed from UCD1 and other meta-data The reason to assign UCD1: The Simple Cone Search needs UCD1 for main positions Easier to work with a simple and restricted list than to construct UCD1+ UCD1 photometry is used to describe filters of the magnitude columns example: UCD1 Filter PHOT_JHN_B PHOT_COUS_I PHOT_HST_F170W PHOT_WLRV_W Johnson, B Cousins I HST/WFPC2, F814W Walraven, W IVOA 2015 (VizieR) - Gilles Landais 7/10

Providing and preserving data for the VO To provide and preserve through the VO Data preservation : the original data in input containing meta-data (ex: FITS) and the data provided (ex: VOTable) Author, journals ASCII table, FITS?VOTable ASCII, FITS, HTML VOTable IVOA 2015 (VizieR) - Gilles Landais 8/10

Providing and preserving data for the VO To accept Votable in input? Currently no VOTable are stored in the VizieR repository FITS headers are not well standardized VOTable containing rich metadata could improve the pipelines To provide the VOTable in their original format using SIA, SSA To promote VO standards in input VizieR encourage today the space agencies to provide VO standards for the VizieR logs pipeline (B/* catalogues updated weekly..) Nextly, VOTable will be soon accepted in VizieR for Associated data (spectra/time series/images using Saada and indexed with ObsCore) Note: Saada extract the metadata from simple VOTable IVOA 2015 (VizieR) - Gilles Landais 9/10

Providing and preserving data for the VO Maintenance/actions needed to preserve and provide rich meta-data Adding new mandatory items in output DM requires significant efforts for CDS (documentalists, engineer) to search the informations. In particular for old catalogue So Please, we would like to have the possibility of notset or noinfo In Input, rich meta-data in VOTable (DML, utype or other): Needs libraries for authors/instruments for the VOTable generation Need to continue to provide classicals header for current clients Increases the maintenance for preservation because: The evolution of the VO standards Obsolete VOTable needs migration which includes obsolete search processing, format migration and search adding information IVOA 2015 (VizieR) - Gilles Landais 10/10