OMI trending and in-flight calibration facility highlevel system description and requirements

Transcription

1 Page 1 of 1 OMI trending and in-flight calibration facility highlevel system description and requirements Summary: This document gives a high-level system description of the OMI trending and in-flight calibration facility. Date Author: 18 November 2002 Checked: P. Levelt 18 November 2002 B. van den Oord R. Dirksen Archive: R. Noordhoek 18 November 2002

2 Page 2 of 2 Distribution list: M. Dobber KNMI R. Dirksen KNMI / SRON P. Levelt KNMI B. van den Oord KNMI P. Veefkind KNMI Change status: Issue Date Comments Affected pages 1 18 November 2002 First issue All

3 Page 3 of 3 Reference documents OMIS Nominal Operations Baseline, RP-OMIE-KNMI-336, issue 1, 26 April OMI in-flight calibration plan, PL-OMIE-KNMI-393, issue 1, 14 November Keydata definition (OPF description), RP-OMIE-7300-DS-391, issue 2, 18 September OMI level 1b product format specification, SE-OMIE-0562-DS/02, issue 1 (draft 7), 14 August OMI data flow document

4 Page 4 of 4 OMI trending and in-flight calibration facility high-level system description and requirements Introduction and purpose of the OMI trending and in-flight calibration facility Once in flight, the OMI system will generate various output data types: 1) Level-0 data 2) Level-1b radiance data product (output of 0-1b data processing) 3) Level-1b irradiance data product (output of 0-1b data processing) 4) Level-1b calibration data product (output of 0-1b data processing) 5) Level-2 data products (output of 1-2 data processing) 5.1) Ozone total columns data product 5.2) NO2 total columns data product 5.3) Aerosol data product 5.4) Ozone profile data product 5.5) Cloud data product In order for the level-2 data product to yield correct and accurate results, it is important that the level- 1b data products, that serve as inputs for the level-2 data product calculations, are properly calibrated. This requires that the initial 0-1b calibration parameters, as determined on ground, are accurate and sufficient, as well as the changes to the 0-1b calibration parameters that occur in flight after launch. This is one of the main tasks for the OMI trending facility and in-flight calibration system, which is the subject of the present document. A complete list of all tasks for the OMI trending facility and in-flight calibration system is: 1) Maintain the 0-1b calibration of the OMI instrument in flight. 2) Deliver new OPF s which represent the status of the OMI instrument performance and calibration for use in the 0-1b data processor. 3) Build and maintain an in-flight calibration database, partly automatically, partly semiautomatically (after user request). 4) Allow to an operator investigation of the OMI in-flight instrument performance and calibration by accessing the in-flight calibration database and produce the required output, visual or otherwise. 5) Accessing the in-flight calibration database to generate the new OPF entries. 6) Generate automatically the required outputs (files, plots, prints, reports,...) to assess the status of the OMI instrument in flight in terms of performance and calibration. 7) Store all incoming level-1b irradiance and calibration data, as well as all incoming engineering / housekeeping data automatically. 8) Reduce automatically the incoming data from 7 by appropriate averaging. Store these reduced data automatically. 9) Perform restricted quality assurance (QA) tasks on the data from 7 and 8. 10) Provide a possibility for investigating level-0 and level-1b radiance, irradiance, and calibration data, even though not all this data is available directly on the trending and in-flight calibration facility. The way the 0-1b calibration of the instrument is maintained is implemented by using all available inflight calibration data to keep the Operational Parameter File (OPF) up to date. The OPF is used by the 0-1b data processor. One of the main tasks of the trending and in-flight calibration system is therefore to produce regularly (e.g. once per month) a new OPF, which represents the status of the OMI instrument performance and calibration. In order to perform the task of maintaining the 0-1b calibration of the instrument in flight various inflight calibration measurements have been defined. A complete list and description of these calibration measurements is given in refs?? These measurements have to be handled, stored and processed properly to yield the necessary information to maintain the 0-1b calibration of the instrument in-flight

5 Page 5 of 5 and to provide output (plots, reports, etc.) that allows users to easily view the 0-1b calibration status of the instrument. The process of updating the so-called Operational Parameter File (OPF), which contains all required information of the 0-1b calibration to be used by the 0-1b data processing, has to be defined and (semi-)automated, which is also addressed in this document. System description for the OMI trending facility and in-flight calibration The complete system which we discuss here is schematically depicted in figure 1. The hardware configuration and some of the most important interfaces are shown in figure 2. The OMI instrument in flight generates the following data: 1) Level-0 radiance data. 2) Level-0 irradiance data. 3) Level-0 calibration data. 4) Engineering data. 5) Housekeeping data. These data are processed and (partly) stored on US side in the SIPS/DAAC system (TBC). The engineering data and housekeeping data is available on a GSFC server, from which it is flowing to the KNMI IST system via the KNMI firewall and the KNMI PDR server (TBC). The Level-0 data is processed in the US by the 0-1b data processor, which uses the OPF as input, to produce: 6) Level-1b earth radiance data product. 7) Level-1b solar irradiance data product. 8) Level-1b calibration data product. Level-1b radiance and irradiance data products are stored at KNMI on the Mass Storage System (MOS), to be further processed afterwards. The level-1b calibration data are also stored on a specific location on the MOS. The engineering and housekeeping data are also stored on the MOS. A dedicated in-flight version of the visualisation tool DATOS, which has been used also during the onground performance verification and on-ground calibration activities, is able to read both level-0 data and level-1b data products on offline analysis computers. Level-1b data reduction software (S/W-2 running on H/W-1) The level-1b calibration and irradiance data has a data volume of about 150 MBytes per orbit. There are 56 different types of calibration measurements, which have been divided over 11 different orbit types, as follows (reference document nominal operations baseline): Orbit type Number of calibration measurement types Nominal scenario N1 4 Daily scenario D 6 Weekly scenario W1 4 Weekly scenario W2 8 Weekly scenario W3 8 Monthly scenario M1 6 Monthly scenario M2 12 Monthly scenario M3 1 Nominal scenario N2 4 Nominal scenario N3 3 Nominal scenario N4 5 Table 1: measurement types per orbit type.

6 Page 6 of 6 Each of the 56 different calibration measurement types is used for a specific calibration purpose. When a specific measurement takes time t to complete, and the measurement is repeated n times to obtain a total measurement time for this calibration measurement of T=n*t, the level-1b calibration output product contains various entries for the same calibration observation. The first step in the data reduction process is to calculate the average and standard deviation of such measurements by a socalled data reduction software package (S/W-2). The data reduction software looks to the data storage computer, and upon arrival of new measurement data, performs the required calculations which result in the data products which are used subsequently in the process. The level-1b data reduction software: 1) Looks continuously to the storage area (MOS) and takes action upon the arrival of new level- 1b calibration data or engineering or housekeeping data. 2) Calculates the required averages and standard deviations of the data from 1 (this is called the reduced calibration data). The format of the results is TBD. 3) Runs on a dedicated computer (computer-1, H/W-1), configuration TBD. 4) Stores the results from 2 (formats TBD) on the computer from 3. 5) Reduces the data volume, typically by a factor 10. 6) Performs limited quality assurance (QA) checks on completeness of the data, saturation, etc. The data reduction software will know the orbit type pertaining to incoming level-1b calibration data, and it will therefore know which calibration measurement types are contained in the data. This system will need to be able to cope with the incoming data rates and volumes, be robust with respect to system and calculational stability, and contain sufficient data storage capacity. The expected data rate is typically 150 MBytes / 100 minutes. The delay in the availability of the data from the moment of acquisition is 2 days (TBC). OMI makes about 5260 orbits per year. Thus the level-1b calibration volume is about 790 GBytes per year, equalling about 4.0 TBytes for 5 years. The data can be stored according to the OrbitNumber and the EquatorCrossingTime and EquatorCrossingDate (available from the level-1b metadata, see reference level-1b product format specification??). A database or index system can be kept to find back the reduced measurement data quite fast. The idea is that the reduced level-1b data contains in the filenames: <date>_<time>_<orbit number>_<orbit type>_<data type>.<reduction output type> <date>: yyyymmdd. <time>: hhmmssps (accurate to tenths of a second), hh from [00..23]. <orbit number>: 5 digit positive integer [ ]. <orbit type>: {n1,n2,n3,n4,da,w1,w2,w3,m1,m2,m3} (see above). <data type>: {rad,irr,cal} for radiance, irradiance,calibration. <reduction output type>: {ave,std} for average, standard deviation. An example: _104307p5_00024_w1_cal.ave for an average file of a calibration orbit w1 obtained on 15 February 2004 on a time 10:43:07.5 in orbit number 24. All orbits fom 1 month (about 432) can be stored in one directory, the directories (60 after 5 years) can be named using the date and time of the first orbit that is contained in it. A more detailed description of this software is given in reference document?? Level-1b data reduction software (S/W-2 running on H/W-1) inputs: - Level-1b calibration data. - Level-1b irradiance data. - Engineering data. - Housekeeping data. Level-1b data reduction software (S/W-2 running on H/W-1) outputs: - Reduced level-1b calibration data from level-1b measurement data (irradiance data, calibration data) and from engineering and housekeeping data. i. Averaged data.

7 Page 7 of 7 ii. Standard deviations. The standard deviations can be used to provide a first indication on the quality of the calibration measurements. This check will be performed using predefined limits. Trending and long-term monitoring software and hardware (S/W-3 running on H/W-2) After that, a second computer system (computer-1, H/W-2) probes continuously whether or not new reduced calibration data have arrived, and if so, performs the next steps in the process. The steps have been depicted schematically in figure 3. The data reduction software (S/W-3) will know the orbit type pertaining to incoming level-1b calibration data, and it will therefore know which calibration measurement types are contained in the data. Using the so-called in-flight calibration database from the previous orbit, the newly acquired reduced calibration data are combined into the existing in-flight calibration database n in order to obtain the new calibration database n+1 by using a number of trending and long-term monitoring algorithms that operate on the reduced calibration data. This can be either fully automated (running by some kind of script), or semi-automated, i.e. by using a piece of prepared software that runs automatically, but only at the request of a user. The software running on this computer shall be able to read the in-flight calibration database (S/W-6 running on computer-3 (H/W-3)) and produce output (results, files, plots) at the request of a user to investigate the in-flight behaviour of the OMI instrument. In this way it shall be possible to investigate / plot the complete instrument history, or for a selected time interval, of all, or a selected set, of calibration and performance parameters contained in the in-flight calibration database. This system block constitutes the heart of the OMI trending and in-flight calibration system, and it will need to be detailed out in much more detail in a separate document to cover all interfaces and inputs and outputs, functionality, hardware specifications separately. The in-flight calibration database, described in reference document??, is configuration controlled. It is stored on computer-2 (H/W-2). Based on a reduction factor of about 10 the data volume is about 80 GBytes per year, equalling about 400 GBytes for a mission duration of 5 years. A more detailed description of this software is given in reference document?? Trending and long-term monitoring software and hardware (S/W-3 running on H/W-2) inputs: - Reduced level-1b calibration data from level-1b measurement data (irradiance data, calibration data) and from engineering and housekeeping data. i. Averaged data. ii. Standard deviations. - In-flight calibration database n (related to OPF). Trending and long-term monitoring software and hardware (S/W-3 running on H/W-2) outputs: - In-flight calibration database n+1 (related to OPF). - Updated OPF parameters ready for implementation in OPF by dedicated OPF S/W. - Trending and long-term monitoring output results for inspection of in-flight OMI instrument performance and calibration (files, plots, reports, etc.). In-flight calibration database analysis software (S/W-6 running on H/W-3) The in-flight calibration database constitutes the heart of the OMI trending and in-flight calibration facility. It contains those parameters that will be monitored in-flight with the following purposes: 1) Obtain an overview of the performance of certain parameters. 2) Obtain an overview of the calibration of certain parameters. 3) Investigate trends to decide whether or not instrument settings will have to be changed. 4) Calculate new OPF entries based on the most recent measurement data. The replacement process, however, requires the use of the OPF manipulation software (S/W-4) and approval of a user.

8 Page 8 of 8 Not all of the OPF entries can or will be replacable in flight. Only those ones, that can be monitored in flight and that can potentially be changed, will be tracked in the in-flight database. The in-flight calibration database also contains additional parameters which are not contained in the OPF. With these parameters trends and developments in the instrument performance and calibration can be tracked. In principle the in-flight calibration database will be changed every orbit upon the arrival of new level- 1b calibration data and engineering and housekeeping data. Configuration control needs to be applied, as well as having the possibility of accessing old versions of the database. This, combined with the fact that the contents of the database can be quite large (in the order of 50 MBytes), results in the danger that the data volumes and tracking systems will explode in terms of size and complexity. In the more detailed documentation great care will need to be taken to avoid this and organise the data structures, search mechanisms and configuration control properly. The idea is to run the fully-automated analysis procedures on one computer system (computer-2, H/W-2), whereas the semi-automated analysis procedures (the ones started by a user) will run on another computer system (computer-3, H/W-3). In this way all automated software can run on H/W-2, where no user analyses will be performed, whereas the semi-automated software, as well as other user software, will run on H/W-3, which thereby becomes a user-analysis computer, which does not influence the automatic data analysis flow. A more detailed description of this software is given in reference document?? OPF manipulation software (S/W-4 running on H/W-2) and OPF The Operational Parameter File (OPF) is used by the 0-1b data processing software. This file contains all required instrument performance and calibration data needed for 0-1b processing. The estimated OPF filesize is about 70 MBytes. At launch the OPF is completely filled with data determined on the ground. In order to keep the OMI instrument properly calibrated the OPF will have to be updated when necessary in flight. To this end the trending and long-term monitoring software (S/W-3) updates OPF entries based on old and newly available calibration data. A user decided whether or not to update an old OPF entry with a new one. For this purpose specially available OPF manipulation software (S/W-4 running on H/W-2) is available in order to make sure that this is performed properly. After the changes the new OPF, which is under configuration control, is made available to the 0-1b data processor via I/F 9. History copies of all OPF s used during the mission are kept under configuration control at both computer-2 (H/W-2) and the SIPS/DAAC. The OPF is expected to be updated with a frequency of no more than once per month, when there is a need to do so. This will result in a total of no more than 60 OPF s for a mission lifetime of 5 years (4.2 GBytes). A more detailed description of this software is given in reference document?? DATOS+ (S/W-5 running on H/W-3) A dedicated in-flight version DATOS+ of the visualisation tool DATOS, which has been used also during the on-ground performance verification and on-ground calibration activities, is able to read both level-0 data and level-1b data products on offline analysis computers. The idea is to have at least one dedicated offline analysis computer (computer-3, H/W-3) available with IDL and DATOS+ running on it. A more detailed description of this software is given in reference document?? Data types and data products The following data types and data products are identified: 1) Level-1b calibration data. 2) Level-1b irradiance data. 3) Engineering data. 4) Housekeeping data.

9 Page 9 of 9 5) Reduced level-1b calibration data from level-1b measurement data (irradiance data, calibration data) and from engineering and housekeeping data. i. Averaged data. ii. Standard deviations. 6) In-flight calibration database. 7) OPF. Software functional blocks The following software functional blocks are identified: 1) S/W-1: Level-1b and engineering data and housekeeping data storing on MOS. 2) S/W-2: Level-1b data reduction software. 3) S/W-3: Trending and long-term monitoring software. 4) S/W-4: OPF manipulation software. 5) S/W-5: DATOS+. 6) S/W-6: In-flight calibration database analysis software. Interfaces The following interfaces are identified: 1) I/F-1: Between GSFC server for engineering and housekeeping data and KNMI computer (PDR server). 2) I/F-2: Between SIPS/DAAC and KNMI computer (PDR server) for access to / storage of level- 1b irradiance and calibration data. 3) I/F-3: Between PDR server KNMI and IST. 4) I/F-4: Between PDR server KNMI and MOS. 5) I/F-5: Between MOS and computer-1 (H/W-1). 6) I/F-6: Between computer-2 (H/W-2) and user to decide whether or not to update OPF parameter entry. 7) I/F-7: Between user and computer-2 (H/W-2) to update OPF parameter entry. 8) I/F-8: Between computer-1 (H/W-1) and computer-2 (H/W-2). 9) I/F-9: Between computer-2 (H/W-2) and SIPS/DAAC for OPF exchange. 10) I/F-10: Between level-0 data and DATOS+ (e.g. computer-3 (H/W-3)). 11) I/F-11: Between MOS and computer-3 (H/W-3) for use of DATOS+ on level-1b radiance, irradiance and calibration data, as well as on engineering data and housekeeping data. 12) I/F-12: Between computer-2 (H/W-3) and computer-3 (H/W-3) for in-flight calibration database analysis software. Hardware systems The following hardware systems are identified: 1) H/W-1: Computer-1 to perform data reduction. 2) H/W-2: Computer-2 to perform: a. Trending and long-term monitoring analysis. b. Calculate updated OPF entries. c. Exchange OPF entries. d. Interface to the DAAC/SIPS to make the new OPF available for the 0-1b dataprocessing. e. Storage and configuration control of all OPF n. f. Storage and configuration control off all in-flight calibration databases n. g. Produce in-flight calibration database n+1 from n. h. Produce trending and long-term monitoring results, plots, print results / status reports, make results available for use on internet.

10 Page 10 of 10 3) H/W-3: Computer-3 for offline analyses: a. DATOS+. b. In-flight calibration database analysis software (S/W-6). Trending and in-flight calibration facility (TF) high-level system requirements The following high-level requirements can be identified for the trending and in-flight calibration facility (TF). Automatically implies that no user or operator interaction is required to realise a certain action. TF-1 TF-2 TF-3 TF-4 TF-5 TF-6 TF-7 TF-8 TF-9 The TF shall be able to ingest level-1b irradiance and level-1b calibration data from the OSIPS automatically. The TF shall be able to ingest engineering and housekeeping data from / to the Instrument Support Terminal (IST) at KNMI automatically. The TF shall be able to store automatically the received level-1b irradiance and level-1b calibration data, as well as the received engineering and housekeeping data in a database. This data shall be available for the complete mission of OMI, with an access time of less than 1 minute. The TF shall be able to perform automatic trend analysis based on the level 1b irradiance and calibration data and instrument engineering and housekeeping data. The TF shall be able to perform semi-automatic trend analysis based on the level 1b irradiance and calibration data and instrument engineering and housekeeping data. This means that analysis software is available, but operator action is required. An in-flight calibration database shall be part of the TF. The TF shall be able to store the trend analysis results from TF-4 and TF-5 automatically (TF- 4) and semi-automatically (TF-5) in the in-flight calibration database. It shall be possible to read, analyse and change parts of the in-flight calibration database or the complete in-flight database automatically or semi-automatically (by operator interaction). The in-flight calibration database shall contain (access to) the trend analysis results of the complete mission of OMI, with an access time of less than 1 minute in order to allow sufficiently fast semi-auomatic replotting and retrending calculations on the complete dataset in the in-flight database as far back as the launch of EOS-AURA. TF-10 Parts of the results of the trend analysis software contained in the in-flight database shall be plotted in graphs for visual inspection. These plots are generated and updated either automatically, or semi-automatically (by operator request). TF-11 The trend analysis visual output software shall be flexible in order for the operator to change plot ranges, plot sizes, hide plots, show plots, print plots, etc. TF-12 The TF shall generate automatically once per day a limited status report on instrument calibration and performance of all orbits executed that day (daily overview). TF-13 The TF shall generate automatically once per week, after completion of the weekly calibration orbits, a status report on instrument calibration and performance (weekly overview). TF-14 The TF shall generate automatically once per month, after completion of the monthly calibration orbits, an extended status report on instrument calibration and performance (monthly overview). TF-15 The TF shall generate updates of the OPF for the level-1b data processor in the OSIPS and the ONRTS. These OPF updates shall reflect an accurate calibration status of the instrument at that moment in time. TF-16 The TF shall test the updated OPF s on format and completeness before delivery to the US TLCF and Sodankyla (via the PI SCF). TF-17 A prototype 0-1b data processor, which is representative for the 0-1b data processor running on the OSIPS, shall be available and running on the TF for offline use. TF-18 The TF shall be able to test the updated OPF s with the use of the raw images contained in the level-1b irradiance and calibration data and a prototype 0-1b data processor, which is representative for the 0-1b data processor running on the OSIPS, in order to investigate the improvement in data and calibration quality before it is decided to update the OPF.

11 Page 11 of 11 TF-19 The TF shall be able to read, manipulate (exchange parameters) and write OPF s automatically or semi-automatically. TF-20 The TF shall be able to interface with the OSIPS in order to supply new versions of the OPF. TF-21 All used versions of the OPF shall be kept in the TF. TF-22 The OPF s in the TF shall be configuration controlled in the TF. TF-23 A change log of the different OPF versions shall be kept in the TF. TF-24 The working OPF in the TF shall be an exact copy of the working copy of the OPF in the OSIPS at all times. TF-25 The TF shall contain an in-flight version DATOS+ of the visualisation tool which has been used before for the performance verification and on-ground calibration phases. This DATOS+ software package shall be able to read both level-0 data and level-1b radiance, irradiance and calibration data products. TF-26 Each functional block of software, hardware and the interfaces contained in the TF shall be properly documented with a user manual and software/hardware detailed description, including an input/output format and contents specification, a software/hardware test plan and a software/hardware test results document. TF-27 Each functional block of software, hardware and the interfaces contained in the TF shall be properly software/hardware tested before integration into the overall TF system. TF-28 The integral TF system, containing all elements, shall be completely tested (software, hardware, interfaces) no later than one month before lauch of the EOS-AURA satellite. A detailed test plan shall be written for this purpose, as well as a document describing the results of the test. TF-29 The TF shall perform limited quality assurance (QA) tasks on the level-1b irradiance and calibration data and on the engineering / housekeeping data. TF-30 A validated version of the instrument simulation software, representative for the OMI PFM performance and calibration at the time of launch, shall be running and available on the TF for offline use.

12 Page 12 of 12 Documentation to be written: OMI TF: Overall system software/hardware/interface test plan OMI TF: Overall system software/hardware/interface test results document OMI TF: User manual. OMI TF: Level-1b data reduction software: User requirements and input/output definition document Software architecture description and hardware description document Software / hardware test plan Software / hardware test results document User manual OMI-TF: Trending and long-term monitoring software and hardware: Algorithm specification document (to a large extent the OMI in-flight calibration plan can be used for this purpose) User requirements and input/output definition document Software architecture description and hardware description document Software / hardware test plan Software / hardware test results document User manual OMI-TF: In-flight calibration database description: User requirements and input/output definition document Software architecture description and hardware description document Software / hardware test plan Software / hardware test results document User manual OMI-TF: OPF read/write/replace software description: User requirements and input/output definition document Software architecture description and hardware description document Software / hardware test plan Software / hardware test results document User manual OMI-TF: Datos+ description: User requirements and input/output definition document Software / hardware test plan Software / hardware test results document User manual OMI-TF: Data types and data formats description. OMI-TF: Interface descriptions. OMI-TF: Hardware description. Open points: - OMI instrument simulation software. - Reprocessed level-1b irradiance and calibration data.