How To Process Data From A Casu.Com Computer System

Transcription

1 CASU Processing: Overview and Updates for the VVV Survey Nicholas Walton Eduardo Gonalez-Solares, Simon Hodgkin, Mike Irwin (Institute of Astronomy)

2 Pipeline Processing Summary Data organization (check images, headers, rename files, ingest into raw data archive) Data processing (dark correction, linearity, flat field, sky, destripe, stack jitters, cataloguing, astrometric and photometric calibration) Produces one stacked pawprint image for each OB (+ associated confidence map and catalogue) Mosaic (tile) the stacked pawprint for each OB (+ associated confidence map and catalogue) [in progress] Post processing checks 2

3 Processing Status: Mar 2012 Processing is proceeding nominally P88: All data to 22 Feb 2012 reduced and will be released in the next few days. 3

4 VISTA Survey Progress as of 14 Dec

5 5

6 VVV data volumes are significant VVV 6

7 Turning VISTA focalthis plane the VISTA Focal Plane 7

8 to this e.g Orion M78 as seeing by VIRCAM in JHK 8

9 Pipeline Overview 9

10 generating Science Data Products products consist of: calibrated single exposure images shifted average stack frames (pawprints) + conf maps calibrated stacked pawprint catalogues filled area tile images + confidence maps calibrated tile object catalogues sky background images, flats, darks, bad pixel mask science products are MEF (multi-extension FITS) files (images Rice-compressed) all QC parameters are stored in MEF headers 10

11 Image processing steps: removing the instrumental signature Reset correction (debias - inline) Dark correction Linearity correction Flat field correction Sky background correction Destripe - controller level pickup Crosstalk, persistence and fringing corrections are not necessary 11

12 VISTA data flow - I raw data transfers on USB disk (Rice-compressed MEFs) ingest & verification raw data archive create off-line tape backups update calibration files monthly (flats, linearity, masks) parallel nightly processing at OB-level (darks updated) stacked pawprint images instrumental signature removed catalogue generation from pawprint images & conf maps astrometric & photometric calibration header updates pawprint OB-level science products check derived QC info & sample of images processing web page updates and progress tracking: 12

13 Nightly Quality Control 13

14 VISTA data flow - II detector level monthly photometric zpt updates illumination correction tables mosaic OB-level tile image and create confidence map tile image cataloguing (Tangent Plane projection) nebulosity filter 6 component pawprints mosaic and correct for sky levels and distortion generate tile catalogue grout tiles to fix PSF and detector zpt variations check derived QC info & sample of images (cf. OB grade) ingest to post-processing database enables checks: FITS header contents, file size, provenance and calibration files, exploration of long-term trends, survey progress, data access 14

15 Survey progress overview QC plots summarise: astrometry; seeing; stellar ellipticity; sky brightness; magnitude zeropoint trends 15

16 Mag ZP Monitoring Silvered mirror Aluminised mirror Photometric ZP variation Cryostat window cleaned P85 P86 P87 P88 16

17 Tile Creation 1 tile = 6 stacked paws each with 16 detector level images Tiles and confdence 1 GB ~ 100GB (25GB compressed) Tile Confidence Map 17

18 Tile Generation & Corrections A number of subtleties in extracting the catalogues from the files PSF variations across the 96 detectors in each time PSF variations increased during poorer seeing periods GROUTING removes a number of these issues For any object computes a photometric correction by reference to the aperture correction at that object weighted by the confidence map Example magnitude and colour-colour diagrams for tiles before (left) and after (right) grouting 18

19 Astrometric Calibration 2MASS - VISTA WCS - ZPN projection NB. tiles are TAN projection Linear solution per detector rms < 100 mas Tabulated systematics from stacked residuals sys < 25 mas 19

20 Photometric Calibration colour equations convert 2MASS to instrumental system 2MASS s:n > 10 in J,H,Ks and NIR ~ standards per pointing 0 < (J-Ks) < 2 & 0 < (J-Ks)_o < 1 (extinction corr) 0 < (J-Ks) & (J-Ks)_o < 1 & (J-Ks) < 0 (update extcorr) no restriction required to be stellar and unsaturated Zpt + error per pointing; can compare with FS fields monitor long term Zpt behaviour average monthly detector Zpt offsets illumination corrections 20

21 VIRCAM Stripes cleaning and correcting the data 21

22 Removing the (time varying) background sky Tilesky double pass combination of all observations in tile(s) Pawsky single pass combination of all observations in a pawprint with object masking iterated dynamically Pawsky with object mask as above, but the mask is defined beforehand using e.g. deep stacked tiles Offset sky use a sky taken nearby (spatially & temporally) Pawsky and half tilesky - minus **** 22

23 Before And After Background Correction 23

24 Flat Field Holes 24

25 Sky Frame Blobs (not stars) 25

26 Software modules nebuliser mosaicer CASU tiling software developed for VISTA grouter removes complex background variations enhanced object detection & parameterisation applied to tile catalogues to remove the effect of PSF variations and photometric throughput (+ MJD column) psf ers under development automatically generates detector-level PSFs and performs PSF photometry 26

27 Background Subtraction Nebulosity filter applied to stacks before mosaicking 27

28 Background Subtraction 28

29 Background Subtraction example: use of nebuliser M17/ K-band 29

30 Improved object detection after removing background nebulosity 30

31 Issues with tiles Imperfect sky subtraction pawprint matching : low level discontinuous artifacts Variable PSF across single pawprints : each detector has different PSFs Variable seeing conditions : each pawprint has different PSFs Variable extinction during the observations : variation of ZP over tile Astrometric distortion : photometric distortion Interpolation options : varying correlated noise patterns 31

32 PSF in each detector different PSFs go into a tile... 32

33 PSF fitting Moffat fit is a good representation 33

34 Pixel Scale Variation 34

35 Photometric Distortion over a Tile Inherent illumination correction Tile exposure map 35

36 CASU Software Release 36

37 CASU Software Release: imcore 37

38 Known Issues Bad pixels in detector number 1 Point-like objects in flat and sky frames Detector 16 variable QE Bad channel in detector 6 Filter wheel out of position Detector #4 Spurious detections around bright stars Images with large ellipticities Persistance Tile images variable depth Aperture2 fluxes now fixed (Aper3 is default) 38

39 CASU new features under development Improved PSF Fitting Tailored for high density fields, thus VVV Improved handling of multi-epoch data Improved data transfers over the EVALSO link Implementation of CASU pipeline to run in 'quick look' mode at Paranal Generation of Reflex workflows for processing VIRCAM data Currently using Version 1.2 of the processing software (since 1 May 2011) Always check the change log at 39

40 40