A verification score for high resolution NWP: Idealized and preoperational tests

Transcription

1 Technical Report No. 69, December 2012 A verification score for high resolution NWP: Idealized and preoperational tests Bent H. Sass and Xiaohua Yang HIRLAM - B Programme, c/o J. Onvlee, KNMI, P.O. Box 201, 3730 AE De Bilt, the Netherlands

2 Contents A verification score for high resolution NWP : Idealized and preoperational tests. Abstract Introduction Definition of SWS Idealized tests of SWS a Idealized tests in one dimension Test cases in two dimensions Defi nition of cases and setup Results of SWS a Implementation of SWS algorithm for verification of operational NW models Data extraction from model and observation Computation of SWS with model- and observation data Evaluation of relative performance of a high resolution model in comparison to a coarse resolution model Sensitivity to upscaling Alternatively defi ned event criteria Discussion and outlook Acknowledgments References

3 2

4 A verification score for high resolution NWP : Idealized and preoperational tests Bent H Sass and Xiaohua Yang Danish Meteorological Institute August 2012 (revised November 2012) Abstract Significant Weather Score (SWS), a new score for use in numerical weather prediction (NWP) verification, is defined and tested in idealized cases and in preoperational conditions using data from high resolution limited area models run at the Danish Meteorological Institute (DMI), and from the global NWP model run at the European Centre for Medium range Weather Forecasts (ECMWF). The verification with SWS is user oriented and designed for high resolution numerical weather prediction. SWS verifies model results against weather observations considered `significant to users. It typically defines, as indicator of significant weather, high values and low values of selected meteorological parameters to be verified over a certain geographical model domain, making use of spatial upscaling principles to take into account effects of phase errors in space and time when comparing model results with point observations from a synoptic network. The scheme has been found easy to implement in an operational NWP environment. The preliminary test results indicate that the SWS score has a good potential to show the added value of high resolution NWP against lower resolution models. 1. Introduction Ever since the beginning of numerical weather prediction (NWP) it has been relevant to ask the question: how is the quality of the model forecast of a given meteorological parameter when compared with observations? One way to address the question in practice is to compare NWP forecast products with weather observations in a subjective manner. This approach which has been used by meteorologists for many years, e.g. in connection with case studies of high impact weather, is useful but not sufficient for model developers who need to design objective methods that may be used in a highly systematic fashion to reveal model weaknesses. Formally the verification problem is very complex as has been described e.g. in a World Meteorological Organization (WMO) survey of verification methods in meteorology by Stanski et al. (1989). The review includes six attributes of a weather forecast which combine to the total quality, namely reliability, accuracy, skill, resolution, sharpness and uncertainty. Moreover, a conclusion of the survey is that no simple verification measure provides complete information about the quality of a product. In another WMO survey by Bougeault (2003) on verification it is concluded that `as high resolution models are expected to provide results in direct relevance to user needs, there is a growing pressure to develop "user-oriented" verifications. The present article defines a `user-oriented verification scheme, `Significant Weather Score (SWS) aiming at verifying meteorological events that are relevant to the users of the model forecast. The scheme is currently kept simple by defining selected high and low values of a parameter in daily 3

5 forecasts as indicator of `significant weather for verification. In this process, the performance of a high resolution model may be compared to a coarser resolution model, in order to investigate the relative skill of the models with regard to the forecasting of significant weather events. The concepts of the SWS scheme are defined in section 2. A particular challenge related to verification has been realized during the last two decades as the model resolution of NWP has steadily increased. This problem has been called `the Double Penalty Issue, e.g. discussed by Mass et al. (2002) and also pointed out by Bougeault (2003) and others. It becomes pronounced when predicting small scale meteorological events that are verified using point observations. This comparison is a challenge as the observed meteorological parameter exhibits large oscillations for models to forecast correctly: a high resolution model often predicts the spatial structure of the weather event, but at a somewhat wrong location when examined at a specific observation point. First the high resolution forecast may be penalized for not forecasting the event at the observation point, and next penalized for predicting the oscillation at a wrong location. In contrast, a low resolution model would often be penalized only once for not predicting the event. Traditional scores, e.g. root mean square error (RMS), will in these situations often assign a better score to the model that does not predict the oscillation. The shortcomings of traditional verification in observation points have also been mentioned by others, e.g. by The Applied Meteorological Unit of NASA 1 stating that `traditional objective techniques that evaluate NWP model performance based on point error statistics and precipitation threat scores can no longer accurately represent the skill of mesoscale NWP models as resolutions continue to increase along with available computer power. Hence in recent years more `fair verification methods, e.g. fuzzy verification methods, have been developed (Ebert 2008). An example of the new types of spatial verification schemes is provided by Roberts and Lean (2008) defining the Fractions Skill Score comparing an observed and a forecasted distribution of a meteorological parameter in a certain area. The SWS verification described here includes a simple upscaling principle similar in spatial concept to Atger (2001), in which forecasts within a certain size of neighborhood of the observation point are examined to determine model skills. SWS examines model prediction about `significant weather within a given tolerance limit. Adding `significant weather to the verification framework is expected to be beneficial for showing the potential of high resolution models, because high resolution models are expected to capture extremes more often than lower resolution models. When defining significant weather we are faced with a dilemma: on the one hand it is desirable to focus on rather extreme events, e.g., for precipitation it is relevant to study events above a high threshold. Such events will occur only for quite limited time periods, and verification statistics has to be computed over long periods of time. On the other hand it is desirable to be able to produce a verification number regularly. The default choice of SWS as defined in section 2 enables that the score can be computed in connection with each forecast. In previous studies, idealized test cases have been defined for the purpose of benchmarking new verification methods, e.g. Gilleland et. al (2010), Gilleland et. al (2009). In section 3, the SWS scheme is examined with idealized tests following those defined in Ahijevych et al (2009). 1 ) In National Aeronautics and Space Administration (NASA) contractor report no. CR

6 Implementation and tests of SWS for high resolution models at the Danish Meteorological Institute, DMI, namely HIRLAM 1 operated at a grid size of 3 km and HARMONIE 2 operated at 2.5 km grid, are documented in section 4. Finally, section 5 contains a discussion and outlook related to the SWS are methodology documented in operational in section practice. 4. Finally, section 5 contains a discussion and outlook related to the SWS methodology in operational practice Definition of of SWS SWS In In this this paper paper we we design design the `Significant the `Significant Weather Weather Score (SWS) Score to (SWS) be a simple to be framework a simple for framework for verifying high resolution mesoscale models, with an aim to show benefits of such models in verifying high resolution mesoscale models, with an aim to show benefits of such models in comparison to a competing (reference) model which typically is at a lower resolution. The score is comparison to a competing (reference) model which typically is at a lower resolution. The score is designed to have the following virtues and characteristics: designed to have the following virtues and characteristics: 1) It may be computed for any specific parameter, or a combination of parameters, which are both 1) It may be computed for any specific parameter, or a combination of parameters, which are both forecasted and observed. forecasted and observed. 2) It allows forecast and observation to be compared up to a maximum spatial distance. Typically it 2) It allows forecast and observation to be compared up to a maximum spatial distance. Typically it verifies a finite area which may represent the suitable predictable scales. verifies a finite area which may represent the suitable predictable scales. 3) It is `robust as a function of time compared to some other scores since two models are verified 3) It is `robust as a function of time compared to some other scores since two models are verified under the same conditions, and the score is computed as a fraction between the two model results. under the same conditions, and the score is computed as a fraction between the two model results. 4) It allows flexible implementation, e.g., by selection of a certain number of the most extreme 4) It allows flexible implementation, e.g., by selection of a certain number of the most extreme observations over the chosen verification area of suitable size, to show added value with regard to observations over the chosen verification area of suitable size, to show added value with regard to prediction of extremes. prediction of extremes. 5) The concepts behind the score are easy to explain to e.g. public authorities unlike many advanced 5) The concepts behind the score are easy to explain to e.g. public authorities unlike many advanced verification methods. verification methods. The score, SWS, is defined as: The score, SWS, is defined as: K K SWS=(1+ F meso )/( 1 + F ref ) (1) (1) k=1 k=1 where the F meso and F ref measure the success of the prediction with mesoscale model and with where the F meso and F ref measure the success of the prediction with mesoscale model and with reference model, respectively, and K denotes the number of comparisons between observation and reference model, respectively, and K denotes the number of comparisons between observation and prediction in consideration. F meso and F ref are assigned values between 0 and 1. A value of zero is prediction in consideration. F meso and F ref are assigned values between 0 and 1. A value of zero is assigned to a forecast failure and a value of 1 to a perfect forecast. Increasing success is reflected by assigned to a forecast failure and a value of 1 to a perfect forecast. Increasing success is reflected an increasing fraction between 0 and 1. This concept may also be applied in the context of by an increasing fraction between 0 and 1. This concept may also be applied in the context of probabilistic forecasting ( see section 5 ) probabilistic forecasting ( see section 5 ) In the framework of SWS, a successful prediction of a significant weather event implies a correct prediction In the framework by the of model SWS, a within successful a given prediction error of margin, a significant both weather in magnitude event implies and in a correct space, the latter typically prediction being by the a model multiple within of grid a given distances error margin, of the both model. in magnitude The principle and in of space, upscaling the latter is applied here, taking typically into being account a multiple a reasonable of grid distances tolerance of the limit model. for inaccuracies The principle of of the upscaling NWP is prediction, applied here, e.g. due to phase taking errors. into account a reasonable tolerance limit for inaccuracies of the NWP prediction, e.g. due to phase errors. With (1), the SWS score is always well defined, with positive values less than infinity. A neutral impact 1 of a high resolution model is characterized by a SWS- value of 1. Positive and negative impacts ) 2 of high resolution will get scores larger than 1 or smaller, respectively. The new score is dimensionless ) because it is computed as a fraction between a result of one model (nominator) and 5 3

7 With (1), the SWS score is always well defined, with positive values less than infinity. A neutral impact of a high resolution model is characterized by a SWS- value of 1. Positive and negative impacts of high resolution will get scores larger than 1 or smaller, respectively. The new score is dimensionless because it is computed as a fraction between a result of one model (nominator) and the other one one (denominator). The The two two model model verifications verifications are done are done in same in way, same e.g. way, using e.g. the using same the same set the of of other observations. one (denominator). The two model verifications are done in same way, e.g. using the same set of observations. Since SWS is is only only a measure a measure of relative of relative skill, it skill, may it be may desirable be desirable to compute to a compute related absolute a related absolute Since measure SWS SWS is a given a only given by a by measure (2). (2). In this In of case this relative the case denominator the skill, denominator it may contains be desirable contains the total number the to compute total of number events a related K. of events absolute K. measure Hence SWS a will a will a given become become by a number (2). a In number this between case between 0 the (zero) denominator 0 and (zero) 1. A small and contains 1. value A represents small the total value a number poor represents forecast of events a poor K. skill Hence forecast while SWS skill a high a while will value become a high represents value a number high represents quality. between high SWS0 a quality. is (zero) a measure SWS and of a 1. the is A a total measure small number value of of the represents defined total number a poor of events defined forecast that events skill are while forecasted that are a high forecasted successfully. value represents successfully. high quality. SWS a is a measure of the total number of defined events K that are forecasted successfully. SWS a =(1 + F K meso )/( 1 + K) (2) (2) SWS a =(1 + k=1 F meso )/( 1 + K) (2) k=1 Likewise, one can also define a corresponding quantity, SWS b for the second model in the Likewise, comparison: one one can can also also define define a corresponding a corresponding quantity, SWS quantity, b for the SWS second b for model the in second the model in the comparison: K SWS b =(1 + F K ref )/( 1 + K) (3) SWS b =(1 + k=1 F ref )/( 1 + K) (3) (3) k=1 The scores SWS a and SWS b as expressed in (2) and (3) are, however, likely to be more prone to The variations scores during SWS a a and year SWS than b as the expressed score given in by (2) (1). and Experience (3) are, however, from operational likely to be verification, more prone e.g. to The scores SWS a and SWS b as expressed in (2) and (3) are, however, likely to be more prone to shown variations in quarterly during a year verification than the reports score given at DMI, by (1). indicates Experience that natural from operational weather variations verification, during e.g. a variations during a year than the score given by (1). Experience from operational verification, e.g. year shown are in reflected quarterly in verification a similar way reports in two at DMI, different indicates forecast that models. natural This weather tends variations to make a during fraction a shown in quarterly verification reports at DMI, indicates that natural weather variations during a year between year are scores reflected of two in a different similar way models in two more different robust than forecast the absolute models. scores This tends defined to make by (2) a and fraction (3). are between reflected scores in a similar of two way different in two models different more forecast robust models. than the absolute scores defined by (2) and (3). This tends to make a fraction between scores of two different models more robust than the absolute It remains to be defined how F ref and F meso are computed for a given application of SWS. Various scores defined by (2) and (3). options It remains could to be be defined used when how considering F ref and F meso verification are computed over a for given a given area. application An obvious of SWS. choice Various would be options to estimate could area be used integrated when considering values to compare, verification - observation over a given derived area. and An model obvious derived. choice However would It this be remains to is estimate difficult to be area defined to do integrated how practice F ref values and without F meso to compare, special computed assumptions. - observation for a given derived application and of model SWS. derived. Various However options this is difficult could be to used do when in practice considering without verification special assumptions. over a given area. An obvious choice would be to In estimate the scheme area integrated described values below to for compare, operational - observation use, it is derived suggested and model to compare derived. a However given observation this is with In difficult the model scheme to grid do described in point practice values below without within for special operational a given assumptions. distance use, it from is suggested that observation to compare site. a The given default observation design of with SWS model involves grid point identification values within of a number a given of distance observed from maxima that observation and minima site. within The the default verification design In area of the SWS to scheme identify involves described `significant identification below weather for of operational a number related use, of to observed it the is suggested parameter maxima to compare considered. and minima a given This within observation makes the verification the scheme with area suited model to for identify use grid on point `significant a daily values basis. within weather When a given considering related distance to the from extreme parameter that observation weather considered. events site. The such This default as makes warning design the weather scheme of with suited SWS high for involves values use on identification to a daily be predicted basis. of a When number (e.g., considering gale, of observed storm extreme etc.), maxima it is weather and suggested minima events for within simplicity such the as verification warning to consider weather all area observations with to high identify values satisfying `significant to be predicted the weather threshold (e.g., related gale, definition to the storm parameter etc.), of the it considered. is meteorological suggested This for makes simplicity event. the scheme Similarly to consider when all suited predicting observations for use extreme on satisfying a daily low basis. the values When threshold of considering a meteorological definition extreme of weather parameter, meteorological events e.g. such as temperatures warning event. weather Similarly below when some with threshold, predicting high values it extreme is suggested to be predicted low to values include (e.g., of gale, all a meteorological observations storm etc.), it is satisfying suggested parameter, the for definition e.g. simplicity temperatures to of consider the meteorological below some all threshold, event. observations it is satisfying suggested the to threshold include all definition observations of the meteorological satisfying the event. definition Similarly of the when meteorological predicting event. extreme low values of a meteorological parameter, e.g. temperatures below some threshold, Another question it is suggested occurs to with include respect all observations to the consideration satisfying the of definition different of forecast the meteorological parameters: should event. Another these be computed question occurs separately with or respect be combined to the consideration into the same of SWS, different i.e. making forecast the parameters: sum of individual should scores these be represent computed forecasting separately of or events be combined involving into different the same weather SWS, i.e. parameters making the? sum It is of suggested individual to Another implement scores represent question the score occurs forecasting separately with respect of for events to individual the involving consideration parameters different of different and weather to compute forecast parameters parameters: afterwards? It should is a total suggested score (if to these needed) implement be computed by the computing score separately separately a weighted or be for combined average individual into from parameters the components same SWS, and of i.e. to individual compute making the afterwards parameters. sum of individual a total score (if scores needed) represent by computing forecasting a weighted of events average involving from different components weather of parameters individual? It parameters. is suggested to implement The score the of a score given separately parameter for may individual be computed parameters according and to compute to the following afterwards steps: a total score (if needed) The score by of computing a given parameter a weighted may average be computed from components according of individual to the following parameters. steps: I ) Define the total area over which the verification should be done. I ) Define the total area over which the verification 6 should be done. 4 4

8 The score of a given parameter may be computed according to the following steps: I ) Define the total area over which the verification should be done. II) Specify the meteorological application of SWS, e.g. the parameter(s) or the event to be verified II) Specify the meteorological application of SWS, e.g. the parameter(s) or the event to be verified III) Choose a number of the most extreme observations to be considered in relation to the III) meteorological Choose a number event. of the most extreme observations to be considered in relation to the meteorological event. IV) Define a method to identify the extreme observations at a given time. The observation times IV) considered Define a will method depend to on identify current the practice extreme and on observations the available at model a given data time. for the The models observation times considered compared. will depend on current practice and on the available model data for the models compared. V) Choose a threshold distance between observation and model grid points contributing to the V) verification Choose a (see threshold discussion distance below between about Fig.1). observation and model grid points contributing to the verification (see discussion below about Fig.1). VI) Within the threshold distance the maximum of the model prediction will be compared with VI) the Within selected the observation threshold from distance the identified the maximum observed of the maxima. model This prediction comparison will results be compared in with the selected a score observation contribution from (F) which the is identified a number observed between 0 maxima. and 1 measuring This comparison the rate of success results of in a score contribution the prediction. (F) The which low is value a number (0) is used between to assign 0 and failure 1 measuring of the forecast the rate to of predict success the event. of the prediction. The The low high value value (0)(1) is is used to to assign indicate failure a forecast of the success. forecast Partly to predict successful the event. forecasts The may high be value (1) is used defined to indicate by assigning a forecast a value success. of F between Partly successful 0 and 1. Similarly, forecasts may low values be defined of selected by assigning a value of observations F between the 0 lowest and 1. model Similarly, prediction for inside low the values threshold of selected distance observations is compared to the lowest model prediction observation inside and the a score threshold contribution distance 0 is F compared 1 is assigned. to the Default observation criteria, and currently a score defining contribution 0 F individual 1 is assigned. score contributions Default F criteria, for temperature, currently wind defining and precipitation, individual respectively, score contributions are as F for temperature, follows: wind and precipitation, respectively, are as follows: a) Lowest and highest temperatures F = 1 if T fc T ob 1.5 C, otherwise 0 b) Lowest and highest 10m wind speeds : V ob 10 m/s: F = 1 if V fc V ob < 2 m/s, otherwise 0 V ob > 10 m/s: F = 1 if V fc V ob < (V ob -10) m/s, otherwise 0 c) Lowest and highest accumulated 12 hour precipitation: P ob 0.1 mm: F=1 if P fc 0.2 F=1 - (2.5 P fc ), if 0.1 < P fc 0.5 F=0 if P fc >0.5 P ob > 0.1 mm: F =0 if 0 P fc 1/3 P ob F = 3 P fc /P ob -1 if 1/3 P ob < P fc 2/3 P ob F = 1 if 2/3 P ob < P fc 4/3 P ob F =15/11 3/11 P fc /P ob if 4/3 P ob < P fc 5 P ob F = 0 if P fc < 5 P ob `fc ~forecast, `ob ~observation, T : 2m-temperature, V: 10m wind, P: accumulated precipitation. The definitions according to a), b) and c) are 7 used throughout this paper unless otherwise stated. 5

9 Fig. 1 For a given forecast observations are selected that are `signifi cantly high and `signifi cantly low according to the specifi cation of the users. Circular forecast areas representing upscaling around each of the selected observations are verifi ed The definitions according to a), b) and c) are used throughout this paper unless otherwise stated. It is seen from b) and c) that the quality assessment for wind and precipitation depends on the magnitude of the observed parameter. For wind it is argued that it is more difficult to predict high wind speed with the same absolute accuracy as for low wind. This is also reflected in the well known Beaufort wind scale using larger intervals at high wind speed. The specific choice of constants in a) c) have been chosen as a result of experience from operational verification and experimentation for Danish area. For precipitation, it is much more difficult to predict high amounts with the same absolute accuracy as low amounts, and the observed highly nonlinear frequency distribution, with many observations close to zero, should be accounted for. Hence, a continuous and asymmetric function is suggested for verifying precipitation, and F is then defined as a fraction between 0 and 1. Furthermore, observations are normally sparse and cannot be expected to represent the true variability of the precipitation field in space. This issue is mentioned further in section 5. The functional form defined in c) is non-symmetric in order to give some credit to larger values which are not very close to the observation. The function has been chosen to be piecewise linear. F=1 is defined for an interval around the observed value. The amplitude of this interval is 1/3 of the observed value. At larger values a linear decrease is defined, and F=0 at forecasts above 5 times the observed value. The functional form is shown fin Fig 2a-b. Fig. 2 (a-b) shows in graphical form the precipitation score SWSa as defi ned in VI-c of section 2. The top fi gure (2a ) shows the computation of F for small precipitation amounts while 2b illustrates the functional form of F in the general case with larger amounts. 8

10 It is argued that it is very hard for a forecast model to predict correctly the complex structure of maxima and minima within the specified tolerance limits day after day with different magnitudes of observed extremes. A high quality forecast model e.g. describing a realistic spatial variation and having a small bias, will be rewarded in a statistical sense compared with a poor model which cannot be expected to pick up the spatial structure of minima and maxima correctly. This is illustrated from Fig.1 displaying structures of minima and maxima which will be hard to forecast unless the model is highly skilful. This issue is mentioned quantitatively in section 3 describing idealized tests. VII) Finally compute the SWS score according to (1) for the given type of meteorological event. The models verified could be, for example, a high resolution mesoscale model and the lower resolution global forecast model at ECMWF. It is worth stressing here that SWS as defined in (1) measures only relative skills between two models. For any verified forecast parameter, a SWS value larger or smaller than 1 only means that the higher resolution model has a higher or lower skill respectively, relative to the reference model, and the value itself does not exclude that both models have been good or bad in predicting a given case. If both models are equally good or poor according to the threshold definition, a SWS value of 1 is assigned. 3. Idealized tests of SWS a In recent years, it has been a common practice in the international community on verification of NWP models to test newly proposed scores against some benchmark cases to examine the characteristics of these scores. When performing idealized tests one should be aware, that examining specific idealized cases cannot give a full picture of how a verification scheme behaves in operational conditions where the effect of many different forecasts and situations will be accumulated in the statistics of the score. In this chapter, we first present basic test examples in one spatial dimension. As a second step idealized tests in 2 dimensions from the international community are described and commented on. Since SWS is defined as a relative skill between two models, it is considered appropriate to choose the subcomponent of the proposed score, namely SWS a, for such examination. 3.1 Idealized tests in one dimension It is possible to illustrate the properties of SWS a as defined in section 2 for precipitation by simple one-dimensional test cases. One may, for example, imagine a 1D prediction (FC) to be compared with an observed precipitation (OBS) as illustrated in the 4 cases of Figs. 3a-d. These figures illustrate 10 points along a x-axis with observed values (OBS) at locations 0.5, 1.5, 2.5, 3.5,,9.5. We define the forecast (FC) to be constant between the integers, e.g. for 0 x < 1, 1 x< 2,, 9 x < 10. In these tests upscaling is not in focus, but one may imagine a selection of small symmetric intervals of a dimension small enough to reside inside the specified intervals. Since the function is defined as being piecewise constant in these intervals the computation of SWSa will not be sensitive to upscaling in these experiments. However, these tests illustrate how SWS a behaves as a function of bias and standard deviation of the forecasts as shown in Fig. 3a-d. For simplicity, only one minimum value (point 3) and one maximum value (point 8) on the abscissa are considered in the computation of SWS a. Fig.3a illustrates a negatively biased forecast with minimum and maximum precipitation in correct points (locations) but with a too small variation. The minimum forecast value equals 0 which will imply 9

11 Fig.3a Fig.3b Fig.3c Fig.3d Fig.3: Idealized 1-D tests of SWS a as defi ned in section 3.1 for precipitation. The fi gures display results of simple test cases in 1 dimension where forecasts (FC) of accumulated precipitation in 10 points along the abscissa are compared with observations. Only one minimum (point 3) and one maximum (point 8) are compared with forecast values. Fig.3a illustrates a negatively biased forecast with little variation. Fig.3b shows a forecast with reduced bias. Fig.3c shows a forecast with no bias, but with too little variation. The forecast of Fig.3d has zero bias and increased variability (see text for details). a score contribution equal to 0, according to VI-c of section 2. The contribution of the maximum forecast (point 8) equals 2/9 for a single forecast. The second and third cases, Fig.3b and Fig.3c respectively, illustrate that reducing bias provides a better result. In the second case the minimum and maximum values contribute with score contributions of 0.75 and 0.44 respectively. The third case is free of bias but it receives a smaller combined score due to a smaller contribution (0.25) from point 3. The maximum point contributes with value In the fourth case the variation is increased while the bias is still zero. This leads to an improved score. Hence the chosen cases illustrate that the SWS a defined for precipitation is sensitive to bias and spatial variance in a reasonable way. Regarding the actual values of SWS a associated with these idealized cases 1 4 they are respectively 0.153, 0.616, and It is then assumed that the score computation is based on 10 forecasts of the individual types shown in the figures. If the computation is based on 100 forecasts the corresponding numbers are 0.116, 0.599, and respectively. As mentioned in section 2 the computation of SWS a (and SWS) will reward forecasts which are able to predict minima and maxima within specified tolerance interval. For the present 1D test cases one may ask how easy it is for a random forecast to select that a minimum exist in point 3 and a maximum exist in point 8. A selection process of minimum followed by maximum gives 10 options ( 1,2,3 10) for selecting minimum point, followed by a choice for maximum point among the 9 remaining points. Hence the probability that the right pair is chosen is only 1 out 90 = 1.1 % 10

12 It thus appears quite difficult for a random forecast or another forecast without much skill to be rewarded fully in a computation of SWS a even if only one or few sets of minima and maxima are selected in the verification. In such situations however the agreement between forecast and observation in other points are not measured. The disagreement in such points may be large. If quality assessment is to be done for an individual forecast an additional computation of SWS a can be made using all observations rather than the small amount connected to minima and maxima. If such additional computation is done for the 1D cases 1-4 (points 1, 2, 3,,10), assuming statistics based on 10 forecasts, the SWS a for cases 1 4 are 0.190, 0.603, 0.893, respectively. The most remarkable difference between these numbers and the corresponding ones based on verifying only one minimum (point 3) and one maximum (point 8) applies to the third case having no bias but a small variation. Considering the abscissa as a spatial dimension case 3 is similar to a larger scale prediction with small variation which does not capture the extremes very well. The properties of the verification then becomes rather similar to a standard scheme looking at mean absolute error. We are left with the dilemma that, on one hand it is natural to have good assessment of forecast accuracy in all points, but on the other hand this verification strategy does not give a good indication of the model s ability to capture extremes. Using the arguments above it is to be expected that measures such as SWS a or SWS are adequate on long time series to measure the model s ability to forecast extremes in a statistical sense. For evaluation of individual forecasts an SWS-type of computation should be done also with more observational data selected, and preferably supplemented with other verification measures to make sure that the general forecast accuracy can be estimated well. 3.2 Test cases in two dimensions The properties of the proposed SWS schemes are studied in the following for test cases as defined in Ahijevych et al. (2009). These so called geometric cases are defined from analytical expressions used on a 601*501 grid. The grid size is assumed to be 4 km. The analyzed and forecasted precipitation for the test cases are defined by isolines of observed/forecasted medium - and high accumulation of precipitation respectively (Fig.4). As discussed by Ahijevych et al. (2009) the geometric cases illustrate errors due to displacement, frequency bias and aspect ratio, respectively. Traditional verification schemes have difficulties to diagnose errors of precipitation structures due to displacement and aspect ratio Defi nition of cases and setup The observed field of all the 5 geometric forecast cases is shown in Fig. 4a. The green and blue ellipses show observed precipitation exceeding 12.7 mm and 25.4 mm respectively. A major part of the grid area has zero precipitation. Fig.4b-f show the observation field together with forecast fields displaying the same intervals as given for the observed field. (but light red signifies 12.7 mm and dark red the 25.4 mm isoline ). In case 1 the forecasted structure is displaced horizontally by 50 grid points ( Fig.4b ) compared to the observed structure. In case 2 the displacement is 200 grid points in the same direction (Fig.4c). Case 4 (Fig.4e) illustrates the effect of a different aspect ratio of the elliptic structures but without increased forecast bias. In cases 3 and 5 (Fig.4d and Fig.4f respectively) the forecast structure is larger than the observed one implying positive precipitation bias of the forecasts. In case 5, the forecast of the medium precipitation amount overlaps the corresponding area of observed precipitation, and it is the only case where forecast- and observation areas overlap! The standard tests of SWS a described below have been carried out using 10 high and 10 low observations respectively, randomly selected over the relevant areas associated with the test cases. 11

13 Fig. 4a (observed precipitation) Fig. 4b (case 1) Fig. 4c (case 2) Fig. 4d (case 3) Fig. 4e (case 4) Fig. 4f (case 5) Fig. 4 : Idealized tests according to Ahijevych et al. (2009): Fig.4a shows observed precipitation structure used in the idealized tests. Dark blue corresponds to high precipitation class and green is associated with medium size precipitation. Fig.4b-f show observed and forecasted structures corresponding to cases 1-5 respectively. Color of the forecasted high precipitation class is dark red and medium size is light red (for details: see text). Sensitivity tests using more observations have been performed showing only a modest differences of few percent in the computed values of SWS a. A random number generator has been used to select integers (grid points with observation) used in the verification. Also SWS a specifies a high and a low precipitation threshold. Fig.4a shows the verification results of all 5 cases obtained using a selection of 10 points with high observed values and 10 points with low observed values. The high observed values are equal to 25.4 mm and the low value is 0 mm. The 5 columns of each experiment apply to different degrees of upscaling ( 0 km, 40 km, 80 km, 160 km and 320 km respectively). These results are shown in Fig5a Results of SWS a In Ahijevych et al. (2009) some traditional scores have been computed for each of the 5 cases (their Table 2). The scores behave quite differently for these cases, partly because they address rather 12

14 Fig. 5a. SWS a for the 5 cases using the specifi cations valid for each experiment and various sizes of upscaling as shown in the legend. Fig. 5b. A non-zero forecasted precipitation occurs over fraction f of the forecast area. Fig. 5c. SWS a computation for forecast values > 0 over fraction f of area, and observed values=0 everywhere. different aspects of the forecast field. Root mean square error (RMS) makes use of all field values while most measures focus on the prediction of the precipitation occurrence larger than zero. This implies that the frequency bias (FBI) associated with cases 1, 2 and 4 is 1.0 while cases 3 and 5 are associated with increasing values of FBI ~ 4 and 8 respectively. On the other hand case 5 is given a high score when computing `probability of detection =0.88 compared with a value of zero for the other cases. Also the Hanssen-Kuipers discriminant computing ( Hits / Obs Yes ) - ( False alarms / Obs No ) gives a high score (0.69) to case 5 while other cases are given a slightly negative value of the score. This is because the forecasted non-zero precipitation field overlaps the observed non-zero field in case 5 contrary to the other cases. New spatial verification measures such as SWS a show some new properties when applied to the forecast fields defined in these cases as will be shown below. As in Ahijevych et al. (2009) the focus here is on addressing the features `displacement error, `bias and `aspect ratio. a) Effect of displacement This feature may be examined from results of case 1 and case 2. The treatment of upscaling dimension as a measure of the smallest scales that are predictable in the model is clearly seen in the results of Fig 5a. Case 1 shows a gradually improved score as the upscaling dimension is increased. The effects are mainly seen between 80 km and 320 km where SWS a increases from 0.71 to 1. However, the precipitation displacement associated with case 2 is displaced so far from the observed field that only the effect of the successful forecasting of zero precipitation is seen. This applies to all the investigated sizes of upscaling. The results of these cases appear reasonable from a subjective point of view. It may be emphasized that traditional verification scores based on point verification can not distinguish between cases 1, 2 and 4. With regard to other non-local verification measures, 13

15 e.g. classified as `neighborhood methods, `scale separation methods, `features based methods and `field deformation methods (Gilleland et al. 2010), it may be stated that most schemes belonging to these groups are in one way or another sensitive to displacement errors. According to Ahijevych et al. (2009) the fractions skill score (Roberts and Lean 2008) exhibits similar characteristics as SWSa for case 2: The forecasted precipitation pattern is too far away from the observed one to be accepted as a contribution to forecast skill unless unreasonably large upscaling sizes are used. b) Bias features In Ahijevych et al. (2009) it is emphasized that the precipitation forecasts of case 3 and 5 are associated with increased frequency bias. Most verification methods are in one way or another sensitive to forecast bias. Case 5 with the largest bias is therefore to some extent considered a poor forecast. However, it is interesting that different conclusions may be reached depending on the focus. In their paper it is stated that to be fair, a hydrologist may actually prefer the forecast of geom005 (case 5), even if it is considered to be extremely poor by modelers and other users. Other scores such as `probability of detection and the `Hanssen-Kuipers quality measure give quite high scores for case 5. An important reason for this is that the forecasted precipitation area overlaps the observed area of precipitation. Another argument for not considering the forecast of case 5 as being poor is the following: consider that the main user criterion is to have a right forecast of whether precipitation occurs or not in the location of the user. Considering the entire forecast domain, this implies that about 80 percent of the users will receive a correct forecast. This is under the assumption of a constant user density (population) per area in the model domain. Furthermore, case 5 is the only forecast which predicts precipitation over a large fraction of the area where it is observed. SWS a is having an emphasis on both observations in the area of observed precipitation and in the precipitation free area. As a consequence, SWS a is rather high for case 5. In Fig.5a SWS a increases from 0.61 to 0.76 as the size of upscaling is increased. This is a bit higher than the SWS a values of case 3 reaching from 0.52 to When assessing SWS a of case 5 two features contribute with different sign. On one hand the big overlap of the forecasted and observed non-zero precipitation affects the score in a positive way which is reasonable. On the other hand it is affected negatively by the fraction of the forecasted precipitation which is not observed. Since this area of wrongly forecasted precipitation is a rather small fraction of the entire area with observed dry conditions its negative effect is still rather limited in case 5. However, if a modified case is considered where the observed precipitation free area is reduced such that the verification area is identical to the area with either forecasted or observed non-zero precipitation, one would get a much lower SWS a (0.33). For such modified situation none of the observed dry spots have been forecasted correctly and the total area bias of the forecasted precipitation is now higher than in the original case with larger dry areas. This means that the forecast is `punished for a positive bias of precipitation appearing in the observed dry area. It thus appears that SWS a computation reacts in a reasonable way. In addition, it is possible to illustrate bias effects using SWS a in a simple way for 2 dimensions as shown in Fig.5 b-c. These figures illustrate the effect of non-zero forecasts ( >0.5 mm ) over a certain fraction f of the grid and 0 mm over the remaining part. It is assumed that the observed precipitation is zero over the entire grid. When carrying out the computation of SWS a for this situation we see that the score goes from 1 to zero as the forecast fraction F goes from 0 to 1. The effect of upscaling is shown in the figure using observation points in the entire grid. It is noted that the SWS a of case 3 is smaller than that of case 1 if upscaling is used. Decreasing score 14

16 is a combined effect of positive bias over the observed precipitation free area in combination with a small reduction in forecasted precipitation close to the observed precipitation area. c) Aspect ratio Finally, the aspect ratio of the precipitation pattern is addressed in case 4 where the original forecast pattern of case 1 is stretched. This changes the aspect ratio, e.g. the zonal width is 4 times too large, and the meridional extent is too narrow. Interestingly we see that the SWS a of case 4 also gives lower score compared to the results of case 1, provided that the upscaling radius is larger than 40 km. According to Ahijevych et al (2009) only the field deformation approach is truly capturing errors in aspect ratio. The well known features-based method SAL (Wernly et al. 2009) is insensitive to this type of error, but other methods are to some extent able to diagnose it (Ahijevych et. al 2009) 4. Implementation of SWS algorithm for verification of operational NW models 4.1 Data extraction from model and observation To test the application of SWS in assessment of relative merit of mesoscale vs coarse resolution NWP models, a first trial of the SWS scheme for model products has been made using in-situ surface synoptic observation data. In order to facilitate such comparison, relevant model and observation data need to be extracted to observation locations. Recently, for the HIRLAM post-processing tool box GL, extension has been made to facilitate verifications applying upscaling principles such as the computation of maxima, minima as well as averaged values within given upscaling distances. Using this facility, data from several NWP models are extracted for pre-defined upscaling distances of 15, 45 and 90 km, respectively, which are used to compute SWS in this study. The smallest (15 km) and the biggest (90 km) sizes selected here are believed to represent the smallest resolvable scale of the mesoscale models and the global ECMWF model, respectively. In addition, an intermediate size of 45 km is tested. Extraction of model data for upscaling requires access to original model output, and in practice it is most conveniently done in real time. Unfortunately, in the time series of model data used for this study, there have been occasional data gaps due to unavailability of data from the archive, which explains occasional discontinuity in the plotted time series below in figs For observations, the data extracted from the archive with the HIRLAM utility GL can be readily used for SWS computation. For Danish territory, observation data from approximately 50 regularly operating Danish synoptic stations reporting surface 2m temperature and 10m wind are used in this study. For precipitation, measurement data from 230 stations, including those from additional local networks such as those collected by the Danish waste water management authorities, is used. 4.2 Computation of SWS with model- and observation data After extracting model and observation data for a given station list, SWS scores may be computed for specified weather events within chosen distances of upscaling. In the current implementation of SWS, the selected observed maxima are compared with maxima of the model data computed from grid points around the observation location and within the upscaling distance. Similarly, the selected observed minima are compared with minima of the model grid point values inside the area of upscaling and centered around the observation point. These comparisons are performed with regard to selected threshold criteria. Obviously, such procedure emphasizes on the model skill to produce significant weather events as characterized by observed extremes in the form of maxima and minima. The computation involves the following steps: I) Read observation data for certain synoptic time points. At present, surface temperature- 15

17 and wind observations every 6-hours are used. For precipitation, only 12h accumulated total precipitation at 06 and 18 UTC is used. For a specified area or station list (`Denmark, `Scandinavia, `All available, etc.), sorting is done for relevant parameters (wind, temperature and precipitation) to select a given number (3, 5, ) of observation stations where either highest or lowest observations are found. Note that only one observation is selected within each defined upscaling radius. In this study, we report findings with the use of a Danish station list, and the default computations are obtained using top 3 `extreme observations for each case of observation selection. II) III) IV) Read extracted data from both models, e.g., one from high resolution model and another from coarse resolution model. For each model, each verification parameter and each selected observation station, find within the radius of the upscaling distance the maximum or minimum values, and determine the F -value as defined in section 2 according to the fit within or beyond threshold between observation and model (0 F 1). Sum up, and compute SWS, SWS a and SWS b as defined in (1)-(3) to obtain values for a given parameter, or a combination of parameters. Statistics is then made using such obtained time series. 4.3 Evaluation of relative performance of a high resolution model in comparison to a coarse resolution model The first test of SWS in an operational environment has started at DMI during 2011 and The high resolution models considered are the operational HIRLAM model at DMI (referred to as SKA) based on HIRLAM 7.3 (see and the HARMONIE pre-operational model DN1 based on IFS cycle 36 (see These models are compared with corresponding forecast from the operational global ECMWF model (`ECH ). The basic characteristics of the models are summarized in Table 1. The pre-operational status of DN1 should be emphasized, especially with regard to data-assimilation. None of the high resolution models assimilate radar data (reflectivities or rain rate) or data from surface GNSS. Also the satellite information does not include IASI-data. Table 1: Model characteristics Model HIRLAM (SKA) HARMONIE (DN1) ECMWF (ECH) Domain focus Scandinavia Denmark Global Horizontal grid ~3 km ~2.5 km ~15 km Vertical resolution 65 levels 65 levels 91 levels Results with fi xed upscaling distance Figures 6 and 7 show SWS scores applied to selected weather parameters such as precipitation, surface 10m-wind and 2m-temperature for the first 8 months of 2012 using an upscaling distance of 15 km. Fig.6 shows time series of SWS comparing HARMONIE-DN1 with ECMWF-ECH forecasts as initiated at 00 UTC and valid at 18 UTC, i.e., with a forecast length of 18h. As observation, data from Danish surface stations are used. The Figures 6 a-c are those for individual parameters and Fig.6d applies to the combined result computed as a weighted sum. Note that due to unavailability 4 ) HIRLAM and HARMONIE models both have a model top at 10 hpa while the ECMWF model top is at 0.01 hpa. Below 10 hpa the vertical resolution of the models is comparable 16

18 Fig. 6a Fig. 6b Fig. 6c Fig. 6d Fig.6: Time series of SWS on the basis of surface observations in Denmark. Harmonie-DN1 (2.5 km grid spacing) is compared with ECMWF forecasts as initiated at 12 UTC and valid at 06 UTC for second day, with a forecast length of 18h. Results apply to the period from January - August a) accumulated precipitation for the past 12 hr; b) 2m- temperature; c) 10m-wind, d) combined result for all three parameters with equal weight. An upscaling distance of 15 km is used. For precipitation 230 stations are used (see text). The green curves show running mean values for 30 day periods. This averaging period builds up over the fi rst 30 days. of data archive the time series of Fig. 6 contain a gap for the beginning of March In Fig. 7, corresponding time series comparing HIRLAM-SKA with ECH model are shown. In these figures, the fits between the predicted values of the models and the 3 highest and lowest observed values, within a 15 km upscaling radius, are compared. Note that, at an upscaling radius of 15 km, data from the ECMWF model is not upscaled. The SWS values at individual forecast times, as shown in the figures, vary inherently following daily changes of relative skills between models for predicting extreme values, but averages over periods of a month (or more), as depicted in the figures by solid green lines, appear to give a good indication of trends. Accordingly, the SWS output may be used to monitor verification trends over longer periods of time. The results as shown in Figures 6 and 7 indicate that, even though the verification is still done in a traditional manner by an inter-comparison of model data with in-situ observations on a relatively sparse network, thanks to the definition of SWS and an upscaling, the added value of the mesoscale models relative to the coarser resolution model is seen rather clearly. The results of Fig. 6a indicate an increased skill of the mesoscale DN1- model relative to ECH with regard to the prediction of precipitation during the summer period of 17

19 Fig. 7a Fig. 7b Fig. 7c Fig. 7d Fig.7: same as Fig.6, but for SWS scores comparing HIRLAM-SKA to corresponding ECMWF model results (see text). July-August. A similar feature is to some extent visible in Fig 7a for model SKA. The results of the two mesoscale models differ somewhat depending on the parameter compared but the combined result, Figs. 6d and 7d respectively, is rather similar when averaged over the total period. grid spacing) is compared with ECMWF forecasts as initiated at 12 UTC and valid at 06 UTC for second day, with a forecast length of 18h. Results apply to the period from January - August a) accumulated precipitation for the past 12 hr; b) 2m- temperature; c) 10m-wind, d) combined result for all three parameters with equal weight. An upscaling distance of 15 km is used. For precipitation 230 stations are used (see text). The green curves show running mean values for 30 day periods. This averaging period builds up over the first 30 days Sensitivity to upscaling The sensitivity of SWS scores to upscaling radius is demonstrated in Fig.8 - Fig.11 valid for both high resolution models and ECMWF-ECH model. Fig.8 a-c illustrate the benefits of upscaling the precipitation forecast for all models, DN1, SKA, and ECH, compared to the case with no upscaling. This is obvious from the SWS a and SWS b in Fig.8 a-b. Interestingly, all models achieve improved hit with upscaling up to 90 km, confirming the potential of this technique for compensation of model errors, e.g. phase errors. This also applies to longer forecast length (not shown). 18

20 Fig. 8a Fig. 8b Fig. 8c Fig.8: Time series of SWS a validating model accumulated precipitation for different upscaling distances. Fig. 8a-b is valid for DN1 and SKA respectively (no upscaling, 15 km, 45 km and 90 km), using surface observations in Denmark. Model forecasts are initiated at 0 UTC and valid at 18 UTC (January - August 2012). Fig. 8c applies to corresponding results from ECMWF model. Solid lines in the fi gures are cumulative means. 19