Cloud Model Verification at the Air Force Weather Agency

Size: px

Start display at page:

Download "Cloud Model Verification at the Air Force Weather Agency"

Gervais Chandler
8 years ago
Views:

1 2d Weather Group Cloud Model Verification at the Air Force Weather Agency Matthew Sittel UCAR Visiting Scientist Air Force Weather Agency Offutt AFB, NE Template: 28 Feb 06

2 Overview Cloud Models Ground Truth Verification Technique Sample Statistics MET Output 2

3 Cloud Models Three are currently run at AFWA Advect Cloud (ADVCLD) Quasi-Lagrangian advection using global model winds Diagnostic Cloud Forecast (DCF) Statistical relation based on recent performance of mesoscale model Stochastic Cloud Forecast Model (SCFM) Statistical relation based on long-term performance of GFS model 3

4 Cloud Model Comparison Model Domain Model Run Frequency Forecast Time Step Maximum Forecast Hour Grid Spacing Vertical Layers ADVCLD Hemispheric 3-hourly 1 hour 12 hours 16 th mesh 5 DCF Theater 6-hourly 3 hours 72 hours 16 th mesh 5 SCFM Hemispheric 6-hourly 3 hours 84 hours 45, 15 km 9 4

5 Cloud Model Outputs Total Cloud Amount Cloud Base Height Cloud Top Height Cloud Type (DCF only) 5

6 Cloud Model Outputs Total Cloud Amount Cloud Base Height Cloud Top Height Cloud Type (DCF only) 6

7 Total Cloud Amount ADVCLD and SCFM forecast cloud amount to the nearest 1%. DCF does not 7

8 DCF Total Cloud Cloud Amount Coded Value 0% 0% 1-20% 13% 21-40% 33% 41-60% 53% 61-80% 73% % 93% SCFM and ADVCLD total cloud forecasts are converted to this categorical scheme when comparing to DCF. 8

9 Ground Truth: WWMCA WWMCA = World Wide Merged Cloud Analysis Run hourly Northern and Southern Hemisphere Total cloud (resolution to nearest 1%), cloud base and top heights 16 th mesh grid (~788,000 usable points) 9

10 Ground Truth: WWMCA Geostationary Data Polar Orbiting Data NOGAPS Upper Atmos. Temp Surface Observations Snow Analysis Resolution: 25 nm Obs: Surface, SSM/I Freq: Daily, 12Z World-Wide Merged Cloud Analysis (WWMCA) Hourly, global, real-time, cloud Surface Temp Analysis Resolution: 25 nm Obs: IR imagery, SSM/I Temp Freq: 3 Hourly Total Cloud and Layer Cloud data supports National Intelligence Community, cloud forecast models, and global soil temperature and moisture analysis. 10

11 WWMCA Components Geostationary Satellites Polar Orbiting Satellites Surface Temperature Analysis Snow Depth Analysis Upper Air Temperature Data Surface Observations Manual QC 11

12 A Perfect WWMCA All satellites functioning properly No problems with satellite data transmission All satellite data received at AFWA correctly/on time Satellite data conversion is problem-free Availability of specialized analyses Decision process is correct (e.g., snow vs. cloud) Error-free observational data Correct manual QC 12

13 WWMCA Timeliness Hemispheric analyses are not snapshots! Age limits are applied No data older than 120 minutes are used in verification 13

14 WWMCA Data Counts Percent Data Availability Run Date (YYYYMMDDCC) On average, 82% of WWMCA global data points are usable (~1.29 million data points per run). 14

15 Verification Technique Determine model-observation pairs ADVCLD and SCFM are already co-located with WWMCA ground truth data points DCF points depend on domain s map projection When ADVCLD or SCFM is compared to DCF, use nearest neighbor to map ADVCLD, SCFM and WWMCA to the DCF domain WWMCA is dumbed down to the 6 categories when compared to DCF Data counts for total cloud contingency table categories (6 for DCF, 101 for ADVCLD, SCFM) are archived for long-term statistics calculations 15

16 Cloud Verification Statistics Root Mean Square Error Mean Absolute Deviation Forecast Bias Index 16

17 Cloud Verification Statistics Root Mean Square Error Mean Absolute Deviation Forecast Bias Index 17

18 20-20 Index Percent of model-observation data points with error 20% or less For each i of n forecast pairs: Forecast and observation expressed as a percentage ranging from 0 to is best, 0 is worst 18

19 Domain-Wide Statistics 50 June 2009 DCF RMSE Average RMSE Forecast Hour 19

20 Domain-Wide Statistics 50 June 2009 DCF MAE Average MAE Forecast Hour 20

21 Domain-Wide Statistics 0 June 2009 DCF Bias Average Bias Forecast Hour

22 Domain-Wide Statistics 0.9 June 2009 DCF Index 0.89 Average Index Forecast Hour 22

23 Sample WWMCA Distribution June 30, 00Z (both hemispheres combined) Almost 70% of the data points are 0 or 100%. This is a typical amount. Count Total Cloud Percentage 23

24 Sample Contingency Tables 24-hour total cloud forecasts CONUS domain 18Z model run 30 day totals: June 1-30, DCF cloud categories = 6x6 table 24

25 June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% 0% 336,315 84,209 35,369 26,641 22,889 92,600 13% 46,839 31,483 17,711 14,392 13,511 57,593 33% 36,207 15,335 8,842 7,484 7,471 43,730 53% 35,750 14,314 7,806 6,500 6,047 30,874 73% 19,769 10,312 7,249 6,236 6,692 42,949 93% 75,845 49,645 36,367 37,284 43, ,603 25

26 June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% Total 0% 336,315 84,209 35,369 26,641 22,889 92, ,023 13% 46,839 31,483 17,711 14,392 13,511 57, ,529 33% 36,207 15,335 8,842 7,484 7,471 43, ,069 53% 35,750 14,314 7,806 6,500 6,047 30, ,291 73% 19,769 10,312 7,249 6,236 6,692 42,949 93,207 93% 75,845 49,645 36,367 37,284 43, , ,388 Total 550, , ,344 98, , ,349 1,812,507 26

27 June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% Total 0% 336,315 84,209 35,369 26,641 22,889 92, ,023 13% 46,839 31,483 17,711 14,392 13,511 57, ,529 33% 36,207 15,335 8,842 7,484 7,471 43, ,069 53% 35,750 14,314 7,806 6,500 6,047 30, ,291 73% 19,769 10,312 7,249 6,236 6,692 42,949 93,207 93% 75,845 49,645 36,367 37,284 43, , ,388 Total 550, , ,344 98, , ,349 1,812,507 Hit Rate = HSS =

28 June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% 0% 336,315 84,209 35,369 26,641 22,889 92,600 13% 46,839 31,483 17,711 14,392 13,511 57,593 33% 36,207 15,335 8,842 7,484 7,471 43,730 53% 35,750 14,314 7,806 6,500 6,047 30,874 73% 19,769 10,312 7,249 6,236 6,692 42,949 93% 75,845 49,645 36,367 37,284 43, ,603 Let s simplify to a 2x2 contingency table cloud vs. no cloud 28

29 2x2 : Cloud vs. No Cloud WWMCA (Observation) DCF (Forecast) 0% Non-Zero 0% 336, ,708 Non-Zero 214,410 1,000,074 29

30 2x2 : Cloud vs. No Cloud WWMCA (Observation) DCF (Forecast) 0% Non-Zero 0% 336, ,708 Non-Zero 214,410 1,000,074 Hit Rate = (was for 6x6 table) HSS = (was for 6x6 table) POD = FAR = CSI =

31 Using MET MODE MET = Model Evaluation Tools MODE = Method for Object-Based Diagnostic Evaluation Tool How does MODE perform with cloud forecasts? 31

32 MET MODE Example Total Cloud Cover Sample Event: July 15, Z Model Run, 6-hour forecast 15 km CONUS DCF vs. 16 th mesh WWMCA (~24 km) WWMCA is re-mapped to exactly match the DCF domain for use in MODE 32

33 Resolving Objects: Threshold DCF is already limited to 6 categories Non-zero cloud amounts are dominated by 100% cases All 100% cases are coded as 93% in DCF Threshold is the 93% DCF category (81-100% cloud) Used ge81.0 for both raw forecast and observation value in the configuration file 33

34 DCF Total Cloud Forecast 34

35 WWMCA Ground Truth 35

36 IR Satellite Image 36

37 WWMCA Ground Truth Satellite Pass Boundary Terrain? 37

38 WWMCA Objects, > 0 gs (Default) 38

39 MODE Defaults Area Threshold for Objects: 0 grid squares (gs) Convolution Radius: 4 grid units (gu) Is there any benefit to changing these? 39

40 WWMCA Objects, > 50 gs 40

41 WWMCA Objects, > 100 gs 41

42 Convolution Radius = 4 gu (Default, Objects set to 50 gs) 42

43 Convolution Radius = 2 gu 43

44 Convolution Radius = 1 gu 44

45 MODE Summary Plot (using Defaults) 45

46 MODE Summary Plot (using Defaults) 46

47 Diagnosing DCF Performance How is MODE best used for cloud model verification? Domain-wide summaries? dominated by large objects? Noisy WWMCA adds to the challenge Geographic subregions? Persistent objects (e.g., Coastal stratus) 47

Estimating Firn Emissivity, from 1994 to1998, at the Ski Hi Automatic Weather Station on the West Antarctic Ice Sheet Using Passive Microwave Data

Estimating Firn Emissivity, from 1994 to1998, at the Ski Hi Automatic Weather Station on the West Antarctic Ice Sheet Using Passive Microwave Data Mentor: Dr. Malcolm LeCompte Elizabeth City State University