The NOAO Science Archive and NVO Portal: Information & Guidelines Mark Dickinson, 14 March 2008 Orig. document by Howard Lanning & Mark Dickinson Thank you for your help testing the new NOAO Science Archive (NSA). We are inviting selected PIs of KPNO and CTIO 4m MOSAIC and NEWFIRM observing programs to help us test an early version of the NSA by accessing and retrieving their raw data. Your feedback will be help us to understand the performance of the system in real world conditions, and to improve the interfaces, instructions and general usability of the system. In particular, we would like to know: whether you, as a new user, can easily find and retrieve your data; how long it takes to retrieve your data via ftp over the network to your home institution, and whether that is acceptable for your purposes; your impressions of the archive interface (the NOAO NVO Portal) and documentation; any comments about the FITS data products that you receive (which, for raw data, should be identical to those produced by the instrument data taking system at the mountain, except for the filenames and some additional header information introduced by the archive system). Additionally, we are asking MOSAIC observers to examine data products produced by the new NOAO MOSAIC data reduction pipeline. This pipeline will be automatically reducing MOSAIC data from both 4m telescopes for delivery to the observers and (eventually) for archival storage. We are now in the science verification stage for the MOSAIC pipeline, and we need feedback from scientists about the quality of the pipeline data reductions. Later in the 2008A semester, we plan to do similar verification testing for the NEWFIRM data reduction pipeline, now under development. Here is a brief description of the process that you will follow to register as an archive user, then find and retrieve your data. This is followed by a more detailed set of set-by-step notes to guide you through the process. Should you have any questions or need assistance, please contact us at vohelp@noao.edu.
You may also find useful information in the new NOAO Data Handbook. This is still in draft form, and you may obtain a copy by anonymous ftp from ftp.noao.edu/pub/dpp/noao_dhb/. The Data Handbook includes information about the NOAO Science Archive and NVO Portal. Another chapter gives information about data from the MOSAIC imager, and about the MOSAIC data reduction pipeline and data products. If you are PI for a MOSAIC program and are advised that pipeline-reduced data will be available for you to retrieve, the MOSAIC chapter of the Data Handbook will help you to understand your data products. A note on browsers: Currently, only the Firefox browser is fully supported for use with the NOAO NVO Portal interface. Therefore, we request that you use Firefox for all activities including registering with the VO, confirming your registration, logging into the NVO and accessing your data or performing data searches. Registration: In order to retrieve proprietary data from the NOAO Science Archive, you must first register with the National Virtual Observatory (NVO) to get a user ID and password. PIs for selected NOAO observing (e.g., you) will receive an e-mail invitation from archive-noreply@noao.edu to register with the NVO. This e-mail may be received prior to your observing run, or, in some cases, may apply to a run that has already taken place. You will register with the NVO and create a username and password to access data from the NOAO Science Archive. At that point, you can log in to the NOAO NVO Portal to find and access your data. If you have already registered in a previous observing semester, then no action should be needed your new proposal ID should automatically be linked to your existing user ID. Steps 1 through 3 below take you through the process of registering and logging into the NOAO NVO Portal. Querying data: To retrieve your data from the NOAO archive, you must first submit a query to find the data. Step 4 below takes you through this query process, which is done from the Query pane that can be selected from the toolbar at the top of the NVO Portal. Queries are made via the Astronomical Query Data Language (ADQL), using a query-building tool, and in principle, quite sophisticated queries are possible. In practice, however, you will probably want to use the 'Simple Query' option, which makes it easy to build a basic query for the data taken for a given observing program and/or from a particular range of observing dates. Retrieving data: Once you have found the data that you wish to retrieve, then you will add them to your 'shopping cart', much as you would shop for products at an on-line store. You will then switch to the Access/Analyze pane via the toolbar at the top of the NVO Portal page. "Check-out" from the store involves staging the
selected data sets for ftp retrieval. Step 5 below takes you through this process. Note that the Portal currently imposes an upper limit of 500 data sets that can be staged at one time. Eventually, we expect many users to be accessing the system, some of whom might try to access large volumes of non-proprietary data, and we need to impose some way to limit the staging volumes; this 500 image limit is our initial means of doing so. Please let us know if you find this upper limit to be restrictive when you retrieve your data! For high data-volume programs, e.g., with NEWFIRM, you may have to stage and retrieve your data in several batches. File sizes/volumes, particularly for NEWFIRM, are large. The NOAO NVO Portal staging area must, therefore, be cleaned on a regular basis. Staged files more than 7 days old are subject to automatic deletion; it is the user's responsibility to retrieve their data within this time frame. At any time, users may re-stage their data to restart their 7-day period. Retrieving pipeline-reduced data products: If you are the PI for a MOSAIC program (and, later in 2008A, we expect for NEWFIRM as well), you may also be receiving data automatically reduced by new pipelines at NOAO. Pipelinereduced data are not yet being archived, and you will not query them by the same process you use for raw data. However, the pipeline products will automatically be staged for you to retrieve via the NOAO NVO Portal. Step 6 below describes how to get your pipeline reductions. Archived data availability: Data from the NOAO 4m telescopes are archived daily. Currently, the archive metadata are updated during the following periods: KPNO data: after 1:15PM before 3:00PM (MST) CTIO/CP data: after 10:15AM before 2:00PM (MST) Data from the previous night s observations should always be available in the archive after the later times above, but may often be available earlier.
A Step-by-Step Guide to Accessing Your Data PIs with selected NOAO proposals will receive an email invitation to register with the NVO. This email may be received prior to the execution of the observations. The PI will then Register with the VO and create a username/password. The specific steps in this process, and for querying, staging, and retrieving your data from the archive are outlined here. Tutorials: Within each pane of the NOAO NVO Portal (e.g., Query or Access/Analyze ), a link at the upper right will take you to a Tutorial that gives more detailed instructions and examples. 1) Browser preparation/setup a) Bring up Firefox (1.5 or later) browser. Currently only Firefox is supported for use with the NOAO NVO Portal! b) Your browser must set to allow pop-ups. Select Preferences/Content and deselect Block pop-up windows 2) Registration/Log-in a) If you are already registered as an NOAO//NVO archive user (e.g., from a previous semester of NOAO archive use, or for any other reason), you need not do anything. Your new program ID should automatically be linked to your existing NVO ID. b) If you are registering for the first time: View invitation e-mail from archive-noreply@noao.edu, Subject: NOAO Science Archive Registration Invitation Copy the URL from the Invitation email into your Firefox browser. You will then be taken to a pre-populated VO registration form. 1. Complete the registration information in the NVO Login Registration Portal form 2. Select your username and password Click Submit on the registration form. 1. You will next be taken to registration confirmation page. In the interim, you will receive an e-mail with a Confirmation Token. View Confirmation Token email 1. Subject: NVO Registration 2. Access the link in the email or cut & paste the token into the Confirmation Form on your browser
c) Log into the VO Use your username and password The browser will automatically take you to the NOAO NVO Portal upon submission of your selected username and password. d) More information about the registration process, including the NVO single sign-on authentication, may be found at http://nvo.noao.edu/noaonvo/nvosinglesignon.shtml 3) Successive log-ins following initial registration a) You may access the NOAO NVO Portal at http://nvo.noao.edu b) Click on the Portal image or the link in the panel bar on the left c) Sign-in by entering your registered username and password 4) Finding your proprietary data (raw data) a) Select the Query pane on the top Portal menu bar b) Select the Simple Query tab c) Select Search My Data from the pulldown menu at right. d) For Simple Queries, you may search by Date, Proposal ID, or both. By Proposal ID: Input your proposal id. The % sign is used to indicate a wildcard. For example, 2008A% indicates observations for your 2008A observing programs. (Note that sometimes program IDs hold over from previous semesters.) % refers to all of your programs with archived data. The LIKE operator is usually appropriate for querying by proposal ID. Set Date to % to find all data for your program(s). (See below for instructions on how to search by date.) Click Update Query to form the ADQL query (shown at left). Click Submit to activate query By Date: The observation Start Date used for searches is the local calendar date (not UT date!) of the start of a given night of observations. Each observing day runs from noon to noon. i. Select Date using the pop-up calendar or via manual entry. Operators are available to search for data from a specific date (=), before (<, <=) or after (>, >=) a date, or in a range of dates (BETWEEN) ii. Set Proposal ID as above, or to % for all of your programs. iii. Click Update Query
iv. Click Submit to activate query e) Select images from the list of returned data (the Data Grid, which may extend over many pages if the query returns >20 images). i. Select individual images by clicking on box next to image ii. Select + to mark and add all images (up to 20) on the currently displayed page of the Data Grid iii. Select * to mark and add all images returned by the query Note: currently, a limit of 500 images is imposed when adding to the Cart. If your query returns more than 500 images (e.g., from several nights of NEWFIRM observing), you will e.g., to stage them in several batches. In this case, we recommend trying to query and retrieve data by date, one or more nights at a time as needed in order to keep each retrieval within the 500 file limit. f) Click Add selected rows to cart 5) Download data a) Select the Access/Analyze pane from the Portal top menu bar b) Select images to download (i.e. stage to ftp) (As noted previously, you may select images by clicking in the box to the left of the image or using the + or * signs) c) Select Stage Data d) Select Show Staging Status to view the current status of each file being staged (currently updates every 10 seconds) Staging large volumes of data may take up to a few hours You may log out of your session and log back in again later to view the staging status. e) Select the Staging directory link in the Retrieval Information pane to view files downloaded to NVO ftp area. f) You may also connect by ftp to the node and directory location specified under Retrieval Information using the username and password provided g) Transfer files to your local area via ftp h) If you plan to query/stage more data, we suggest that you delete data you have already staged first, to avoid confusion. Currently, the only way to do this is within your ftp transfer session. E.g., if you are using command line ftp, you can do something like this: ftp> prompt Interactive mode off. ftp> mdel NSAR3*fits.gz
Note that there may be other files in your ftp staging area, such as an xml file that records information about the staging results. We recommend that you not delete those (e.g., do not use mdel * ). The next version of the portal will have a remove from staging area feature that will simplify this process. i) Also, if you plan to query/stage more data, we suggest emptying your access cart first to avoid confusion about what you have already staged and what you haven t. Within the Access Cart: Select images to remove (e.g. with + or * ) Click Remove from Cart. 6) Accessing Pipeline Processed Mosaic Images Please see the MOSAIC chapter of the NOAO Data Handbook for information about the MOSAIC data reduction pipeline and its products. a) Select the Access/Analyze pane from the Portal top menu bar b) Select Show Staging Status. Under Retrieval Information, select Staging directory: staging/<user_name>, or ftp by other means (e.g., from a command line) using the address, username, and password given. c) Select Reduced link in list of staged files d) Select Data to access tar files of observations for download e) Select Html to access tar files containing pipeline review pages. These are web pages with accompanying png image files that provide a handy way to look at your processed data, along with diagnostic information about the data sets. Retrieve these tarfiles and unpack them on your local disk, then point your web browser at the relevant directory to view the pipeline review information. MOSAIC PIs will receive another document describing the structure and contents of these pipeline review web pages. f) Note that at present it is necessary to query some raw data and add it to your cart in order to see the Staging Status link in the Analyze/Access Data pane. Therefore, if you are retrieving reduced MOSAIC data, you must first follow this procedure for some raw data. You need not stage the raw data for retrieval (although we encourage you to try this as well) just adding some data to the Cart is sufficient to let you see your Staging Status area. This procedure will be changed in a future software release.
Exercising the NVO Portal In addition to serving as the archive interface for retrieving NOAO data, the NOAO NVO Portal also serves many other functions. The Portal provides a sophisticated way to find and access other data sets from public, VO-compliant archives around the world, using the NOAO Sky graphical interface (which lets you find and select data sets from a map of the celestial sphere), the NOAO-Time interface (which lets you search in the time domain), and with other search tools. Currently accessible data include SDSS DR5, Chandra, XMM, HST, and GALEX, and the Isaac Newton Telescope. Reduced data products from previous NOAO Survey Programs can also be retrieved, and are displayed in red on the NOAO Sky view. You may query on a specific target position or an input list of targets to view available images. You may also run cross-matches, view images with Aladin and VOPlot, and, you may run WESIX to obtain catalog data on selected images. You may also access data in the various archives directly via the SIA Live search. We would like to encourage you to take some time to exercise the various capabilities and options of the NOAO NVO Portal. Here, we list some of those capabilities/tools currently available, but do not describe them in any detail. The on-line documentation should help guide you in using these features. Note that the archived data available through the Portal content are cached for efficiency. The Portal cache is expected to be updated twice-yearly, and therefore the most recent data from an archive may not always be accessible. Some Portal Features: NOAO-Sky Double-click on wireframes/zoom View preview images for selected wireframe searches (e.g., SDSS) Add Marker o manual input o input list o Xmatch Specify Object Name or Sky Location o Name Resolver, Zoom Display and Search by Archive o with Cart selected, view Pushpins on NOAOSky (designates positions of images which have been added to the Cart)
Data Grid Select data to add to Cart View selected rows in VOPlot See Data Grid as VOTable Toggle coordinate format between decimal degrees and sexagesimal format Columns may be sorted. NOAO-Time Pan through calendar Select/Update Archives View images for specific days/months/years (from calendar) - Select images from bullets in TIME - View Data Grid for selected images Select new date via Date Center pop-up calendar SIA Input Object Name/Resolve Input Size (required) Select Archive o select desired parameters if non-default Select datasets for Adding to Cart ACCESS/ANALYZE (Cart) Select images individual by clicking on box or selecting + for visible images page or * for all images returned View images via Aladin View images via VOPlot Extract sources (WESIX) into catalogs View images list as VOTable