IRS Level 2 Processing Concept Status Stephen Tjemkes, Jochen Grandell and Xavier Calbet 6th MTG Mission Team Meeting 17 18 June 2008, Estec, Noordwijk Page 1
Content Introduction Level 2 Processing Concept Description of Modules Proxy data Conclusion Page 2
MTG-IRS mission objective Primary mission objective is to monitor small scale water vapour structures in support of regional and global NWP A secondary objective is to support the monitoring of atmospheric dynamics through e.g. the provision of clear sky wind, global instability,.. A further objective is to support emerging applications regarding chemical weather and air quality Page 3
Thus primary mission objective calls for accurate moisture and to a lesser degree temperature profiles from MTG-IRS observations. The development of operational L2 processing scheme is presented next. How this will be used for the implementation of an operational scheme is TBD. Page 4
L2 Processing Concept Input: radiometrically and spectrally calibrated spectra with geometrical information appended auxilliary data Processing split into Pre-processing to determine the type of scene: clear/cloudy, dusty, fire, cloud parameter determination. Quick processing: retrievals using a fast statistical technique like EOF linear regression, neural networks, etc Processing: Physical retrieval (optimal estimation). Post-processing: formating, gridding, QA, Output Profiles of moisture and temperature + error co-variance Page 5
Processing: The Challenge Retrievals can be performed for all scenes (though the quality depends on presence of clouds) Example iasi L2 (processing 235 channels per spectrum): On current system (2CPU IBM power 4 processor): 0.008333 min/spectrum IRS: >7 10 6 spectra / BRC => 60 000 min to process one BRC Note that this is with respect to IBM power 4 Machines (several years old technology). Current state of the art machines (e.g. Power 6) are significant factors faster. Still we need to explore all possibilities to keep the L2 processing affordable. So we need to look at Implementation: Efficient codes (e.g. different RTM) Parallelisation Processing Not all scenes => need for scenes analysis All should be done without compromising quality. Page 6
End-to-End L2 Processing Chain Radiative transfer modeling Scenes Analysis Inversion Page 7
RTM Need for an efficient radiative transfer module Scene Inversion PCRTM OSS RTTOV Community? + ++ Maintenance? + Performance?? ++? Accuracy??? Static Application - + - Page 8
RTTOV vrs OSS: two different approaches RTTOV: Statistical model for mean transmission Multiple scattering (TBC) Includes Jacobians Difficulties wide band imagers OSS: Course resolution line-by-line Multiple scattering (Not yet procured) Jacobians PC scores could be implemented Page 9
RTTOV-OSS Comparison 5801 diverse profiles Sea surface emissivity T, q, O3 from ECMWF Rest: climatology RTTOV-9 GENLN2, HITRAN 2000, CKD2.4 OSS LBLRTM V11.3, HITRAN2004, MT_CKD 1.0 LBLRTM V11.3 as reference (for 500 profiles) Page 10
Timing Results (in sec/profile) Direct only Direct + Jacobians RTTOV 9 1.11 11.42 OSS - 0.67 Averaged over 5308 profiles, on IBM power 2: xlf90 O3 q64 Page 11
Comparison to IASI First OSS Second RTIASI Averaged results over 500 cases Red line mean Blue standard deviation Page 12
OSS Page 13
RTIASI Page 14
Comparison to LBLRTM RTTOV SAD were generated using GENLN2 OSS SAD were generated using LBLRTM Difference in performance could be result of difference in GENLN2 and LBLRTM. Single profile Red line OSS Blue line RTTOV Page 15
Page 16
Summary For hyperspectral applications: Performance: OSS = RTTOV/16 Accuracy: OSS compares favourable to reference Not shown: For Imager applications: Performance OSS = 2*RTTOV Accuracy: OSS compares favourable to reference We will use OSS for our development of IRS L2 processing Page 17
Expert Note: Reduce the number of mono-chromatic calculations by OSS significantly through the so-called global training, as opposed to the localised training applied here (up to factor 10) There is room to improve OSS efficiency, by how much will depend on application (hyperspectral, imagery) Page 18
Pre-processing: Scenes Analysis At Day-1 process only cloud free FOV (+ maybe Low Level clouds) Implemented the SCE by Watts & McNally for AIRS/IASI. Page 19
Scenes Analysis: Method Page 20
Cloudy - clear radiances (provided by phil watts) Page 21
Cloudy - clear radiances (provided by phil watts) Small difference Big difference Clear radiances *RTTOV- 6 (Matricardi et al.) *Ecmwf T,Q,O 3 *Model noise (H.B.H T + F) Cloudy radiances *RTTOV-6 + (Chevallier et al.) *Ecmwf T,Q,O 3,CLW *Meas. Noise (O) (AIRS Flight Model) Page 22
Pressure ranking: all channels (228 NESDIS NRT) Model noise (in 6 μm band) Measured - Model (K) Cloud emissivity effect Page 23
Clear-channel id 1: Low-pass filter δbt(j) LP[δBT(j)] 10-20 ch Measured - Model (K) Detect gross cloud signal (+ or -) * Proceed > higher until LP gradient & signal small Declare channels with smaller index clear Ranked channel index Page 24
Scenes Analysis: results applied to IASI Page 25
Page 26
Page 27
IASI L2 cloud mask Page 28
Quantitative Comparson IASI L2 Cloud Mask No data Cloudy Clear ECMWF cloud detection (ECD) No data 1818 0 0 Cloudy 66 1571 84 Clear 4 105 73 Page 29
Conclusion Successfully implemented Not yet an extensive validation Need to apply to MTG-proxy data Integrate into end-to-end development chain Replace current rtm with OSS CO2 slicing alternative methods to be considered Gives cloud top pressure and cloud fraction Page 30
CO2 slicing results: cloud fraction Page 31
CO2 slicing results: cloud top pressure Page 32
CO2 slicing results: cloud top pressure Page 33
Inversion Page 34
IRS L2 Prototype Processor tests IRS L2 Prototype Processor running with: IASI real data converted to IRS with IASI2IRS tool IRS spectra have the original noise coming from IASI RTM: RTTOV-9.1+IASI2IRS Clear sky over ocean selection scenes as for IASI (threshold test method only) Bias corrected Optimal estimation First guess from EOF retrieval Background from Chevallier Optimal measurement covariance from OBS-CALC Page 35
IRS L2 OBS-CALC Bias and STDV Page 36
IRS and IASI EOF retrievals Page 37
IRS and IASI Physical retrievals Page 38
IRS L2 Inversion: future Improve fast retrieval method -> Neural networks? Introduce faster radiative transfer model -> OSS Keep on verifying with real IASI data correct scene classification Introduce realistic noise into IRS synthetic measurements Introduce pseudo-noise diffraction effects into IRS synthetic measurements using proxy data Determine minimal set of channels for with Inversion will be applied Analyse co-registration errors Analyse correlated noise in observations Analyse spectral calibration errors Apply to field experiements (jaivex), proxy data Compare to independent methods Page 39
Other components Surface emissivity retrieval Improve accuracy through exploitation of the time domain (e.g. Kalman filter) Page 40
Proxy Data For end-to-end processing chain For technical studies like compression, error budgets Source IASI Synthetic based on models run by SSEC Met Office Page 41
IASI as proxy for IRS IASI L1C can be converted into IRS Select LWIR or MWIR Generate Interferogram De-apodise interferogram Truncate Interferogram Convert into spectrum Tool to do this is available upon request Can be applied to Observations, simulations and jacobians Page 42
IRS proxy data from WRF In support of GOES-R H. Huang, T. Greenwald and xx generated two case studies based upon WRF Consider here the European simulation Hope to get a sample tape soon, to see if we can transfer data using LTO III data-tapes Data needs to be converted into IRS (and possible other candidate mission) radiances 16-17 August 2006 Domain File Size per Output Time (GB) Total Dataset Size (TB) Spatial Resolution (km) 00-09 UTC 09-15 UTC 15-00 UTC Temporal Resolution (minute) Full disk 103 GB 16 TB 3 15 5 15 WRF model output data volume and spatial and temporal resolution for the NCSA MSG simulation. Page 43