CASU Processing: Overview and Updates for the VVV Survey Nicholas Walton Eduardo Gonalez-Solares, Simon Hodgkin, Mike Irwin (Institute of Astronomy)
Pipeline Processing Summary Data organization (check images, headers, rename files, ingest into raw data archive) Data processing (dark correction, linearity, flat field, sky, destripe, stack jitters, cataloguing, astrometric and photometric calibration) Produces one stacked pawprint image for each OB (+ associated confidence map and catalogue) Mosaic (tile) the stacked pawprint for each OB (+ associated confidence map and catalogue) [in progress] Post processing checks 2
Processing Status: Mar 2012 Processing is proceeding nominally P88: All data to 22 Feb 2012 reduced and will be released in the next few days. 3
VISTA Survey Progress http://casu.ast.cam.ac.uk/vistasp/overview as of 14 Dec 2011 4
5
VVV data volumes are significant VVV 6
Turning VISTA focalthis plane the VISTA Focal Plane 7
to this e.g Orion M78 as seeing by VIRCAM in JHK 8
Pipeline Overview 9
generating Science Data Products products consist of: calibrated single exposure images shifted average stack frames (pawprints) + conf maps calibrated stacked pawprint catalogues filled area tile images + confidence maps calibrated tile object catalogues sky background images, flats, darks, bad pixel mask science products are MEF (multi-extension FITS) files (images Rice-compressed) all QC parameters are stored in MEF headers 10
Image processing steps: removing the instrumental signature Reset correction (debias - inline) Dark correction Linearity correction Flat field correction Sky background correction Destripe - controller level pickup Crosstalk, persistence and fringing corrections are not necessary 11
VISTA data flow - I raw data transfers on USB disk (Rice-compressed MEFs) ingest & verification raw data archive create off-line tape backups update calibration files monthly (flats, linearity, masks) parallel nightly processing at OB-level (darks updated) stacked pawprint images instrumental signature removed catalogue generation from pawprint images & conf maps astrometric & photometric calibration header updates pawprint OB-level science products check derived QC info & sample of images processing web page updates and progress tracking: http://casu.ast.cam.ac.uk/surveys-projects/vista 12
Nightly Quality Control 13
VISTA data flow - II detector level monthly photometric zpt updates illumination correction tables mosaic OB-level tile image and create confidence map tile image cataloguing (Tangent Plane projection) nebulosity filter 6 component pawprints mosaic and correct for sky levels and distortion generate tile catalogue grout tiles to fix PSF and detector zpt variations check derived QC info & sample of images (cf. OB grade) ingest to post-processing database enables checks: FITS header contents, file size, provenance and calibration files, exploration of long-term trends, survey progress, data access http://casu.ast.cam.ac.uk/vistasp/imgquery/search 14
Survey progress overview QC plots summarise: astrometry; seeing; stellar ellipticity; sky brightness; magnitude zeropoint trends 15
Mag ZP Monitoring Silvered mirror Aluminised mirror Photometric ZP variation Cryostat window cleaned P85 P86 P87 P88 16
Tile Creation 1 tile = 6 stacked paws each with 16 detector level images Tiles and confdence maps @ 1 GB ~ 100GB (25GB compressed) Tile Confidence Map 17
Tile Generation & Corrections http://casu.ast.cam.ac.uk/surveys-projects/vista/technical/tiles A number of subtleties in extracting the catalogues from the files PSF variations across the 96 detectors in each time PSF variations increased during poorer seeing periods GROUTING removes a number of these issues For any object computes a photometric correction by reference to the aperture correction at that object weighted by the confidence map Example magnitude and colour-colour diagrams for tiles before (left) and after (right) grouting 18
Astrometric Calibration 2MASS - VISTA WCS - ZPN projection NB. tiles are TAN projection Linear solution per detector rms < 100 mas Tabulated systematics from stacked residuals sys < 25 mas 19
Photometric Calibration colour equations convert 2MASS to instrumental system 2MASS s:n > 10 in J,H,Ks and NIR ~100-1000 standards per pointing 0 < (J-Ks) < 2 & 0 < (J-Ks)_o < 1 (extinction corr) 0 < (J-Ks) & (J-Ks)_o < 1 & (J-Ks) < 0 (update extcorr) no restriction required to be stellar and unsaturated Zpt + error per pointing; can compare with FS fields monitor long term Zpt behaviour average monthly detector Zpt offsets illumination corrections 20
VIRCAM Stripes cleaning and correcting the data 21
Removing the (time varying) background sky Tilesky double pass combination of all observations in tile(s) Pawsky single pass combination of all observations in a pawprint with object masking iterated dynamically Pawsky with object mask as above, but the mask is defined beforehand using e.g. deep stacked tiles Offset sky use a sky taken nearby (spatially & temporally) Pawsky and half tilesky - minus **** 22
Before And After Background Correction 23
Flat Field Holes 24
Sky Frame Blobs (not stars) 25
Software modules nebuliser mosaicer CASU tiling software developed for VISTA grouter removes complex background variations enhanced object detection & parameterisation applied to tile catalogues to remove the effect of PSF variations and photometric throughput (+ MJD column) psf ers under development automatically generates detector-level PSFs and performs PSF photometry 26
Background Subtraction Nebulosity filter applied to stacks before mosaicking 27
Background Subtraction 28
Background Subtraction example: use of nebuliser M17/ K-band 29
Improved object detection after removing background nebulosity 30
Issues with tiles Imperfect sky subtraction pawprint matching : low level discontinuous artifacts Variable PSF across single pawprints : each detector has different PSFs Variable seeing conditions : each pawprint has different PSFs Variable extinction during the observations : variation of ZP over tile Astrometric distortion : photometric distortion Interpolation options : varying correlated noise patterns 31
PSF in each detector... 96 different PSFs go into a tile... 32
PSF fitting Moffat fit is a good representation 33
Pixel Scale Variation 34
Photometric Distortion over a Tile Inherent illumination correction Tile exposure map 35
CASU Software Release http://casu.ast.cam.ac.uk/surveys-projects/software-release 36
CASU Software Release: imcore http://casu.ast.cam.ac.uk/surveys-projects/software-release 37
Known Issues http://casu.ast.cam.ac.uk/surveys-projects/vista/technical/known-issues Bad pixels in detector number 1 Point-like objects in flat and sky frames Detector 16 variable QE Bad channel in detector 6 Filter wheel out of position Detector #4 Spurious detections around bright stars Images with large ellipticities Persistance Tile images variable depth Aperture2 fluxes now fixed (Aper3 is default) 38
CASU new features under development Improved PSF Fitting Tailored for high density fields, thus VVV Improved handling of multi-epoch data Improved data transfers over the EVALSO link Implementation of CASU pipeline to run in 'quick look' mode at Paranal Generation of Reflex workflows for processing VIRCAM data Currently using Version 1.2 of the processing software (since 1 May 2011) Always check the change log at http://casu.ast.cam.ac.uk/surveys-projects/vista/data-processing/version-log 39
http://casu.ast.cam.ac.uk 40