Archive and data packaging

Transcription

1 Archive and data packaging Or, why there are so many empty sub-directories in the Cycle 0 data packages! North American ALMA Science Center Mark Lacy Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Large Baseline Array 1

2 Getting ALMA data Public data: SV data linked off the ALMA Science Portal ( under the Data link (also some datasets for which CASAGuides were written can be directly downloaded from the relevant CASAGuides wiki) Public archival data, also linked from Data Proprietary data: Cycle 0: link sent to PI Cycle 1: PI or delegate logs into portal and downloads from the archive. (Most Cycle 0 datasets can also now be obtained this way.) 2

3 ALMA data products Raw ALMA data is archived in the form of an ALMA Science Data Model (ASDM). The unit of data collection is the execution block (EB). The first task in data processing is to read the EBs into CASA (or the pipeline, eventually) to convert them from ASDM to CASA Measurement Set format (MS) format. The MSes are run through a calibration script. 3

4 ALMA data products -2 FITS images are then generated The results are then packaged (packages are different for Cycle 0 and Cycle 1) The packages are then ingested into the archive as one or more tar files. To download a package, you need to download every tar file in it. 4

5 ALMA data Cycle 1 will start to see the implementation of the ALMA data processing hierarchy Key concept is the ObsUnitSet defined as a set of data that can be processed together by the pipeline. Three levels of ObsUnitSet (OUS). Science Goal OUS (accounting purposes) Group OUS (combining e.g. 12m, ACA and TP) Member OUS (single dataset, typically from one Scheduling block) 5

6 The ALMA data hierarchy Observing view: Science Goal Scheduling block Execution block Scheduling block Execution block Data processing view: Science Goal OUS Group OUS Member OUS Group OUS Member OUS Execution block Execution block 6

7 Cycle 0 packages Cycle 0 packages contain: Raw data measurement set with Tsys and WVR corrections applied Calibrated data measurement set with all calibration applied Calibration tables Scripts Logs QA information Products (FITS images and cubes) 7

8 Cycle 0 packages directory structure Although the full data hierarchy was not implemented in Cycle 0 there are subdirectories in the packages for each level. <Project code>/sg_ouss_id/group_ouss_id/ member_ouss_id/<your data here!> Cycle 1 deliveries will have directory names corresponding to the actual OUS status IDs. 8

9 Cycle 1 packages Cycle 1 packages (one per OUS)will contain Calibration tables Scripts Logs QA information Products (FITS images and cubes) Calibrated measurement sets may be produced by running a script on the raw data, available through the archive. 9

10 The Cycle 0/Cycle 1 scripts Cycle 0 and early Cycle 1 packages contain calibration and imaging scripts. The calibration script (raw data to calibrated MS) is consistent and reliable. Imaging scripts were usually at the discretion of the data reducer in Cycle 0. Script is specific to the version of CASA that data were reduced in. Bug fixes/upgrades may be available in later versions of CASA, but you may need to change the script slightly to work in a different CASA version. (See knowledgebase articles.) Examples are better primary beams in CASA 3.4, a bug in shadow flagging in 3.4 and 4.0.0, better planet calibration models in CASA 4.0. Effect on results should only be at few% level though. 10

11 Reimaging The most likely thing you will want to do is to reimage the data. Start with the calibrated.ms file in the calibrated directory. Self-calibration often improves images significantly if they contain enough flux for selfcal to work. 11

12 Using the ALMA Science Archive (ASA) The ASA is linked from the Science Portal (under the Data menu). Currently two options a full archive query tool, and a delivery list If you know which project you are interested in, use the delivery list or the project view on the query interface. If you wish to search for data, use the raw view on the query interface. 12

13 Archive queries The SESAME name resolver searches SIMBAD and NED Select more intents to search for calibrators of various types Mouse over fields to show dialog boxes Raw data view shows every observation matching a search. Project view shows all the data deliveries containing those data 13

14 Results table (raw data view) Click on column headings to sort Check boxes to select data to download Enter filter in dialog box e.g. >120 14

15 Live demo Go to almascience.nrao.edu and follow links to the archive query tool. To find observations near Cen A: Enter Cen A in box Change search radius to 20 Select first dataset returned Hit download data & follow RH instructions To find observations calibrated on Callisto: Select Amplitude in the intents menu Type Callisto into Source Name (ALMA) box Etc 15

16 Request handler Once you have selected data to download, the request is passed to another piece of software, the request handler. If you are an anonymous user, this will check if the data are public. If you are PI of the project for which you are requesting data (or have been delegated by the PI in his/her ALMA user profile) and are logged into the portal the RH will also allow you to download you data within the proprietary period. Default download is via a direct internet stream, wget script and (coming soon!) hard media options also available. 16

17 Science Verification data Science verification data is not (currently) stored in the ASA. Instead, it is available by selecting the Science verification item in the Data menu on the Portal. 17

18 Summary Lots of good data from SV and publicly available in the ASA, with much more becoming public in the future. Data processing and delivery changing as software construction is progressing. ALMA data hierarchy is complicated but for good reasons! Most of the time you will not need to worry about it. 18