the Manual on the global data processing and forecasting system (GDPFS) (WMO-No.485; available at

Transcription

1 Gudelne on the exchange and use of EPS verfcaton results Update date: 30 November 202. Introducton World Meteorologcal Organzaton (WMO) CBS-XIII (2005) recommended that the general responsbltes for a Lead Centre for Verfcaton of EPS be added to the Manual on the global data processng and forecastng system (GDPFS) (WMO-No.485; avalable at and that the Presdent desgnate RSMC Tokyo (Japan Meteorologcal Agency; JMA) as the Lead Centre. JMA hosts two Internet stes for the Lead Centre works; a Web ste and a FTP ste. The FTP ste s used for EPS producng centres to upload ther statstcs of EPS verfcaton and the Web ste s used to publsh them. The Internet stes enable EPS producers to make ther own statstcs of EPS verfcaton open to the WMO Members. The Internet stes also enable the WMO Members to obtan not only the statstcs but also ther vsualzed mages. Ths gudelne s organzed as follows. Secton 2 descrbes defntons of the EPS verfcaton. Secton 3 descrbes how to record and read the statstcs on the Internet stes for EPS producers and the WMO Members, respectvely. Secton 4 descrbes the rule of the report, and secton 5 descrbes how to obtan the EPS verfcaton scores. Secton 6 descrbes how to post the statstcs on the FTP ste only for EPS producers. 2. Defnton of standard EPS verfcaton It s defned at the Attachment II.7, Table F, Secton III of the Manual on the GDPFS (See Appendx A) that standard verfcaton of an EPS are exchanged monthly to measure the forecast skll of ensemble mean, spread and probablty wth respect to the analyss and/or clmatology. The factors and defntons of the measure are also explaned n Table. 3. The statstcs of EPS verfcaton 3. The verfcaton table of ensemble mean and spread The verfcaton table, called the determnstc table here, s defned as follows: A header lne, whch starts wth #, should be put at the head of the determnstc table. The header lne should consst of 5 elements as follows;

2 # <Center name> <Element> <Year> <Month> <Area> where <term> ndcates a keyword whch s lsted up n Table 2. The frst, second, thrd, fourth and ffth column of the table followng the header lne ndcates the number, forecast tme, anomaly correlaton coeffcent of the ensemble mean, root-mean-square (RMS) error of the ensemble mean and spread (standard devaton), respectvely. Some spaces, at least one, are necessary among each column as a separator. A comment lne, whch starts wth!, can be put anywhere. 3.2 Relablty table The table s compled wth Standardzed Verfcaton System (SVS) for long-range forecasts (LRF) (Table 6 n Attachment II.8 of the Manual on the GDPFS). The table s quoted as Table 3 n the document. A header lne, whch starts wth #, should be put at the head of the relablty table. The header lne should consst of 7 elements as follows; # <Center name> <Element> <Threshold> <Year> <Month> <Area> <Forecast tme> where <term> ndcates a keyword whch s lsted up n Table 2. The frst, second, thrd, fourth and ffth column of the table followng the header lne ndcates the bn number, the number of members forecastng occurrence of the event, the number of members forecastng non occurrence of the event, observed occurrences and observed non-occurrences, respectvely. Some spaces, at least one, are necessary among each column as a separator. A comment lne, whch starts wth!, can be put anywhere. 3.3 The verfcaton table of contnuous rank probablty score (CRPS) The verfcaton table, called the CRPS table here, s defned as follows: A header lne, whch starts wth #, should be put at the head of the CRPS table. The header lne should consst of 5 elements as follows; # <Center name> <Element> <Year> <Month> <Area> where <term> ndcates a keyword whch s lsted up n Table2.

3 The frst, second, thrd and fourth column of the table followng the header lne ndcates the number, forecast tme, CRPS for EPS forecast dstrbuton, CRPS for the control forecast n EPS, respectvely. CRPS for determnstc forecast s equal to the mean absolute error. If CRPS for the control forecast n EPS s not beng reported, the fourth column has to be set - nstead. Some spaces, at least one, are necessary among each column as a separator. A comment lne, whch starts wth!, can be put anywhere. 4. Rule of the report on the verfcaton table The relablty table, the CRPS table and the determnstc table should be reported wth an asc fle(s)(see Appendx B, C and D). The fle name of the verfcaton table should comply wth the followng rule; <Center name>_<classfcaton>_<year><month>.txt where <term> ndcates a keyword whch s lsted up n Table 2. If EPS producer would lke to add and/or update the verfcaton table after reportng t, she/he has to overwrte t usng the same fle name. 5. How to get the verfcaton table JMA operates a Web ste for both exchangng the statstcs and publshng them. The name and address of the Web ste are EPSV and respectvely. 6. How to post the verfcaton table JMA operates a FTP ste whch s used for EPS producers to report ther statstcs of EPS verfcaton. The name and address of the ftp ste are FTPEPSV and ftp://ftpepsv.kshou.go.jp, respectvely. 6. Regstraton The ftp ste s allowed to access only from the regstered ste. The EPS producer should notfy the admnstrator of the global IP address(es) (for example, ) of her/hs own ste. JMA wll send the EPS producer the logn nformaton of FTPEPSV, such as user name, at least n one month after the notfcaton. 6.2 Post

4 The EPS producer s able to logn FTPEPSV wth the user name and the password provded. She/He put the monthly verfcaton table on her/hs home drectory of FTPEPSV. The verfcaton table s avalable by accessng EPSV (see secton 5) n a day after they are put on FTPEPSV. And, old date verfcaton tables on FTPEPSV may be deleted due to the lmtaton of dsk resources.

5 Table : Factors and defntons used n the EPS verfcaton Area Grd Ensemble mean Spread (Standard Northern hemsphere extratropcs (90N-20N) Tropcs (20N-20S) Southern hemsphere extratropcs (20S-90S) Verfyng analyss and clmatology are the centre s on a lattude-longtude grd 2.5x2.5,.5x.5, or.25x.25 -Note that the next Manual on the GDPFS wll specfy only.5x.5 as s currently descrbed for the determnstc scores. -Note also that the clmatology provded by the LC-DNV s on the.5x.5 grd. ~ F m F, j m j m ~ ( ) Devaton) F, j F Root Mean Square Error m j Mean Absolute Error e Anomaly Correlaton Coeffcent Brer Score n n 2 e ( f f ) ( a a ) n 2 2 ( f f ) cosϕ ( a a ) cosϕ ( P O Contnuous Rank Probablty Score [ P ( x) Oˆ ( x) ] 2 ) 2 cosϕ ˆ 2 where F: forecast value, A: correspondng verfyng value, C: correspondng clmatologcal value, P: probablty, O: {observed:, or not-observed:0}, m : the ensemble sze, n : the number of grd ponts n the verfcaton area, cos ϕ : cosne of lattude of grd pont dx E.g. H. Hersbach, 2000: Decomposton of the contnuous ranked probablty score for ensemble predcton systems. Wea. Forecastng, 5,

6 . cos cos, ) ( ) ( ) ( ), ( ) ( ˆ ), ( ) ( ˆ,,,, < n n m j j X X x x x H A x H x O F x H m x P C A a C F f A F e ϕ ϕ

7 Table 2 : Lst of keywords Element <Center name> <Classfc aton> <Element > <Threshol d> Keyword (Bold) BOM,CMA,CMC,CPTEC,EC MWF,JMA,KMA,MF,NCEP,R UMS,UKMO RELTBL, ACCERRSPD or CRPS Z500anm T850anm PMSLanm Z500 T850 PMSL W850 U850 V850 U250 V250 PR24 Z500anm, T850anm or PMSLanm W850 U850, V850, U250 or V250 PR24 gtsd,gt.5sd, gt2sd,lt-sd,lt -.5sd or lt-2sd gt0mps,gt5 mps or gt25mps gt0p, gt25p,gt75p or gt 90P gtmm,gt5m m, gt0mm or gt25mm Abbrevaton of EPS producer RELTBL ACCERR SPD The relablty table The scores of ensemble mean forecast and the spread CRPS The contnuous rank probablty scores of ensemble forecast and determnstc forecast Z500anm T850anm PMSLan m Anomaly for 500hPa geopotental heght (unt:m) Anomaly for 850hPa temperature (unt:k) Anomaly for mean sea-level pressure (unt:hpa) Z hPa geopotental heght (unt:m) T hPa temperature (unt:k) PMSL Mean sea-level pressure (unt:hpa) W hPa wnd speed (unt:m/s) U(V) hPa u(v) wnd component (unt:m/s) U(V) hPa u(v) wnd component (unt:m/s) PR24 24-hour accumulated precptaton (unt:mm) gt Abbrevaton of greater than lt Abbrevaton of less than sd Abbrevaton of Standard devaton mps Abbrevaton of m/s P Abbrevaton of percentle pont of clmatologcal chance (%) mm Unt of precptaton

8 <Year> 2003, 2004, (Numerc) Target year <Month> 0, 02, 03, 04, 05, 06, 07, 08, (Numerc) Target month 09, 0, or 2 <Area> NH, SH or TR NH Northern Hemsphere extratropcs (20N-90N) SH Southern Hemsphere extratropcs (20S-90S) TR Tropcs (20S-20N) <Forecast tme> 24, 48, 72, 96, 20, (Numerc) forecast tme (unt:hour)

9 Table 3: The defnton of relablty table (Source: SVS for LRF of the Manual on the GDPFS)

10 Appendx A : Latest standard verfcaton measures of EPS (Source: the Attachment II.7, Table F, Secton III of the Manual of the GDPFS) (To be modfed to reflect the latest revson) III STANDARD VERIFICATION MEASURES OF EPS EXCHANGE OF SCORES Monthly exchanges: Ensemble mean For verfcaton of ensemble mean, the specfcatons n ths table of the attachment for varables, levels, areas and verfcatons should be used. Spread Standard devaton of the ensemble averaged over the same regons and varables as used for the ensemble mean. Probabltes Probablstc scores (excludng the CRPS) are exchanged n the form of relablty tables. Detals of the format of the exchange of verfcaton data are provded on the webste of the Lead Centre for verfcaton of EPS. Lst of parameters PMSL anomaly ±, ±.5, ± 2 standard devaton wth respect to a centre-specfed clmatology. Verfed for areas defned for verfcaton aganst analyss. Z500 wth thresholds as for PMSL. Verfed for areas defned for verfcaton aganst analyss. 850 hpa wnd speed wth thresholds of 0, 5, 25 m s. Verfed for areas defned for verfcaton aganst analyss. 850 hpa u and v wnd components wth thresholds of 0 th, 25 th, 75 th and 90 th percentle ponts wth respect to a centre-specfed clmatology. Verfed for areas defned for verfcaton aganst analyss. 250 hpa u and v wnd components wth thresholds of 0 th, 25 th, 75 th and 90 th percentle ponts wth respect to a centre-specfed clmatology. Verfed for areas defned for verfcaton aganst analyss.

11 T850 anomales wth thresholds ±, ±.5, ± 2 standard devaton wth respect to a centre-specfed clmatology. Verfed for areas defned for verfcaton aganst analyss. Precptaton wth thresholds, 5, 0, and 25 mm/24 hours every 24 hours verfed over areas defned for determnstc forecast verfcaton aganst observatons. Observatons for EPS verfcaton should be based on the GCOS lst of surface network (GSN). Verfcaton of precptaton may alternatvely be aganst a proxy analyss.e. short range forecast from the control or hgh-resoluton determnstc forecast, e.g. 2-36h forecast to avod spn-up problems. NOTE: Where thresholds are defned wth respect to clmatology, the daly clmate should be estmated. Scores Brer Skll Score (wth respect to clmatology) (see defnton below*) Relatve Operatng Characterstc (ROC) Relatve economc value (C/L) dagrams Relablty dagrams wth frequency dstrbuton Contnuous Rank Probablty Score (CRPS) NOTES: Annual and seasonal averages of the Brer Skll Score at 24, 72, 20, 68 and 240 hours for Z500 and T850 should be ncluded n the yearly Techncal Progress Report on the Global Data-processng System. In the case of CRPS, centres are encouraged to submt ths for both EPS and the determnstc (control and hgh-resoluton) forecast as well CRPS for determnstc forecast s equal to the mean absolute error.

12 Appendx B : A sample of the content of a determnstc table --- start --- # JMA PMSL NH ! comment # JMA T NH end ---

13 Appendx C : A sample of the content of a relablty table --- start --- # JMA PMSLanm gt.5sd NH # JMA PMSLanm gt.5sd NH

14 end ---

15 Appendx D : A sample of the content of a CRPS table --- start --- # JMA Z NH !comment # JMA Z SH end ---