Hybrid-DA in NWP. Experience at the Met Office and elsewhere. GODAE OceanView DA Task Team. Andrew Lorenc, Met Office, Exeter.

Transcription

1 Hybrid-DA in NWP Experience at the Met Office and elsewhere GODAE OceanView DA Task Team Andrew Lorenc, Met Office, Exeter. 21 May 2015 Crown copyright Met Office

2 Recent History of DA for NWP 4DVar was used by most leading NWP centres in ~2005 Its main scientific weakness was the use of climatological error covariances instead of Errors Of The Day. ECMWF Workshop on "Flow dependent aspects of data assimilation" June DVar is technically difficult and expensive Some (e.g. Kalnay 2007) advocate using EnKF instead. WWRP/THORPEX Workshop on 4D-Var and Ensemble Kalman Filter Inter-comparisons. Buenos Aires - Argentina, November 2008 Several different hybrid methods developed: Using localised ensemble perturbations directly in VAR (Lorenc 2003, Buehner 2005) Using a covariance model modified by a recent ensemble (ECMWF, Météo-France, Purser) Averaging increments from 4DVar & EnKF (Penny 2014, Bonavita 2015) Crown copyright Met Office Andrew Lorenc 2

3 Response to a single T ob Basic 3D-Var 3D-Var + 1 bred mode Dale Barker EOTD expts. Mark Dubal GCT expts. Adrian Semple, 2001: Meteorological Assessment of the Geostrophic Co-ordinate Transform and Error Breeding System When used in 3D Variational Data Assimilation. NWP TechRep357. Andrew Lorenc 4, from ECMWF Workshop on "Flow dependent aspects of data assimilation" June 2007.

4 Vision ideal Global DA for NWP, using quasi-linear methods Best estimate DA of known scales (~12km), using 4D-Var because of: Desire to treat all scales together; Desire to make best use of satellite obs e.g. by bias correction, using high-resolution. Hybrid ensemble to carry forward error information from past few days. May still be scope for nested regional systems to give more rapid running and higher resolution. N.B. This vision is good for perhaps a decade, while we are restricted to well-known scales, so the KF theory of a best estimate + a covariance description of uncertainty is useful. WWRP/THORPEX Workshop on 4D-Var and Ensemble Kalman Filter Inter-comparisons. Buenos Aires - November 2008 Crown copyright Met Office Andrew Lorenc 5

5 Crown copyright Met Office Andrew Lorenc 6

6 Crown copyright Met Office Andrew Lorenc 7

7 Crown copyright Met Office Andrew Lorenc 8

8 Crown copyright Met Office Andrew Lorenc 9

9 u increments fitting a single u ob at 500hPa, at different times. 4D-Var at start of window at end of 6-hour window Hybrid 4D-Var Unfilled contours show T field. Clayton et al. 2013

10 Hybrid-4DVar Adding ensemble covariances from the MOGREPS system to 4DVar gave ~1% improvement (Clayton et al. 2013) But we do not want to rely on 4DVar : Increasingly expensive on MPP computers; We are already developing a - New dynamical core (GungHo) - In a new software system (LFRic). Nevertheless current hybrid-4dvar + ENDGAME model form a world-class system which will be good for ~5 years. Crown copyright Met Office Andrew Lorenc 11

11 Hybrid-4DVar (Operational at the Met Office since July 2011) (MOGREPS-G, based on localised ETKF. Currently 44 members.) Background error covariance at beginning of window: B B B 2 c c 2 e e B C f e P e Climatological covariance Ensemble covariance Spatial localisation covariance Raw ensemble covariance

12 Hybrid-4DVar (Operational at the Met Office since July 2011) (MOGREPS-G, based on localised ETKF. Currently 44 members.) B implicitly propagated by a linear Perturbation Forecast (PF) model: ~100 PF + adjoint forecasts run serially. But PF model doesn t scale well. And difficult to keep PF model in line with forecast model. We need an alternative scheme for future supercomputers that excludes the PF model

13 Hybrid-4DEnVar No PF model, but much more IO required to read ensemble data: 11 times faster with 22 N216 members and 384 PEs. (IO around 30% of cost) Analysis consists of two parts: A 3DVar-like analysis based on the climatological covariance B c A 4D analysis consisting of a linear combination of the ensemble perturbations. Localisation is currently in space only: same linear combination of ensemble perturbations at all times.

14 Crown copyright Met Office Andrew Lorenc 15

15 Crown copyright Met Office Andrew Lorenc 16

16 Met Office trial of 4DEnVar Lorenc et al. (2015) hybrid-4dvar 3.6% better hybrid-4denvar 0.5% better hybrid-3dvar = hybrid-3denvar We expected hybrid-3dvar and hybrid-3denvar to perform equally. Adding the time-dimension in hybrid-4dvar made a big improvement, whereas adding it in hybrid-4denvar gave much less benefit. Crown copyright Met Office Andrew Lorenc 17

17 Conclusions from 4D analysis increment study 1. The main error in our hybrid-4denvar (v hybrid- 4DVar) is that the climatological covariance is used as in 3DVar. 2. 3D localization not following the flow is not an important error for our 1200km localization scale and 6hour window, but does become important for a 500km scale. Crown copyright Met Office Andrew Lorenc 18

18 Improving 4DEnVar The maintenance and running costs of hybrid-4dvar are larger, so there is an incentive to improve hybrid-4denvar. We need to reduce the weight on climatological B relative to the ensemble covariance. We must first improve the ensemble covariances: a bigger ensemble; better ensemble generation; better filtering of ensemble covariance, e.g. localization. Encouraging progress has been made in all of these. Crown copyright Met Office Andrew Lorenc 19

19 4DEnVar in Operational NWP Implemented in Canada (Buehner 2015) Easier for us because Met Office s 4DVar was better and Ensemble worse Implemented at NCEP (Kleist 2014) only had to beat 3DVar. Under development in Météo-France (Desroziers 2014) - strategic decision not to use 4DVar at convective scale. Planned for Met Office s new LFRic software system & GungHo model in ~5 years. Crown copyright Met Office Andrew Lorenc 20

20 Where to get the ensemble? Separate EnKF ensemble system? recentred about best estimate deterministic NWP? EDA of perturbed observation VAR assimilations. Being researched: Mean-Pert ensemble of perturbed-ob 4DEnVars. EVIL use of VAR Hessian. Crown copyright Met Office Andrew Lorenc 21

21 Why not use EnKF? EnKF is easiest way to get reasonable results quickly. But NWP centres are striving for excellence. Localisation is essential because of NWP s BIG models. EnVar localises in model space, so it is better for applying scale-dependent and balance-aware localisation methods. EnVar costs are not very dependent on number of observations; EnVar is better for dense observations, non-local observations and correlated errors. Crown copyright Met Office Andrew Lorenc 22

22 Ensemble covariance filtering B is big! We need a large ensemble PLUS clever filtering to reduce sampling noise, based on 2 ideas: Assume local homogeneity apply smoothing: horizontal, rotational, and time Assume some correlations are near zero, & localise: horizontal, vertical, spectral, between transformed variables Two approaches to hybrid covariances, using these ideas: 1. Train a covariance model using recent ensembles 2. Augment B by using localised ensemble perturbations Crown copyright Met Office Andrew Lorenc 23

23 covariance 1.5 Horizontal localisation Errors in sampled ensemble covariances N= distance (km) From Lorenc (2003)

24 covariance The Schur Product 1.5 Curve C chosen such that covariances go to zero at distance. e.g. compactly supported (4.10) from Gapsari and Cohn (1999) n=100 * compact support 1 This gives: Ensemble covariances modified to be 0 at distance. Covariance function slightly narrower than ideal distance (km) From Lorenc (2003)

25 Convective-scale ensemble s.d. of humidity at 945hPa Large ensemble (84 members) Small ensemble (6 members) Horizontally filtered small ensemble Using the AROME ensemble (Ménétrier et al. 2014). Crown copyright Met Office Andrew Lorenc 26

26 Sampled raw ensemble s.d. Crown copyright Met Office Andrew Lorenc 27

27 s.d. after spectral localization Crown copyright Met Office Andrew Lorenc 28

28 Column cross-correlations between: divergence (up) & relative humidity (across). Raw ensemble Horizontally, vertically & spectrally localized ensemble Inter-variable localized ensemble Crown copyright Met Office Andrew Lorenc 29

29 Ensemble covariance filtering: Conclusions The two approaches to hybrid covariances: 1. Train a covariance model using recent ensembles 2. Augment B by using localised ensemble perturbations start from different ends; 1 starts from a climatological covariance model, then adds ensemble derived coefficients, 2 starts from a raw ensemble then filters the covariances. Eventually they might meet in the middle. There is less scope for these methods in the EnKF, other than simple spatial localisation. Crown copyright Met Office Andrew Lorenc 30

30 References Crown copyright Met Office Andrew Lorenc 31

31 Questions and answers Crown copyright Met Office