2d Weather Group Cloud Model Verification at the Air Force Weather Agency Matthew Sittel UCAR Visiting Scientist Air Force Weather Agency Offutt AFB, NE Template: 28 Feb 06
Overview Cloud Models Ground Truth Verification Technique Sample Statistics MET Output 2
Cloud Models Three are currently run at AFWA Advect Cloud (ADVCLD) Quasi-Lagrangian advection using global model winds Diagnostic Cloud Forecast (DCF) Statistical relation based on recent performance of mesoscale model Stochastic Cloud Forecast Model (SCFM) Statistical relation based on long-term performance of GFS model 3
Cloud Model Comparison Model Domain Model Run Frequency Forecast Time Step Maximum Forecast Hour Grid Spacing Vertical Layers ADVCLD Hemispheric 3-hourly 1 hour 12 hours 16 th mesh 5 DCF Theater 6-hourly 3 hours 72 hours 16 th mesh 5 SCFM Hemispheric 6-hourly 3 hours 84 hours 45, 15 km 9 4
Cloud Model Outputs Total Cloud Amount Cloud Base Height Cloud Top Height Cloud Type (DCF only) 5
Cloud Model Outputs Total Cloud Amount Cloud Base Height Cloud Top Height Cloud Type (DCF only) 6
Total Cloud Amount ADVCLD and SCFM forecast cloud amount to the nearest 1%. DCF does not 7
DCF Total Cloud Cloud Amount Coded Value 0% 0% 1-20% 13% 21-40% 33% 41-60% 53% 61-80% 73% 81-100% 93% SCFM and ADVCLD total cloud forecasts are converted to this categorical scheme when comparing to DCF. 8
Ground Truth: WWMCA WWMCA = World Wide Merged Cloud Analysis Run hourly Northern and Southern Hemisphere Total cloud (resolution to nearest 1%), cloud base and top heights 16 th mesh grid (~788,000 usable points) 9
Ground Truth: WWMCA Geostationary Data Polar Orbiting Data NOGAPS Upper Atmos. Temp Surface Observations Snow Analysis Resolution: 25 nm Obs: Surface, SSM/I Freq: Daily, 12Z World-Wide Merged Cloud Analysis (WWMCA) Hourly, global, real-time, cloud analysis @12.5nm Surface Temp Analysis Resolution: 25 nm Obs: IR imagery, SSM/I Temp Freq: 3 Hourly Total Cloud and Layer Cloud data supports National Intelligence Community, cloud forecast models, and global soil temperature and moisture analysis. 10
WWMCA Components Geostationary Satellites Polar Orbiting Satellites Surface Temperature Analysis Snow Depth Analysis Upper Air Temperature Data Surface Observations Manual QC 11
A Perfect WWMCA All satellites functioning properly No problems with satellite data transmission All satellite data received at AFWA correctly/on time Satellite data conversion is problem-free Availability of specialized analyses Decision process is correct (e.g., snow vs. cloud) Error-free observational data Correct manual QC 12
WWMCA Timeliness Hemispheric analyses are not snapshots! Age limits are applied No data older than 120 minutes are used in verification 13
WWMCA Data Counts 100 90 80 Percent Data Availability 70 60 50 40 30 20 10 0 Run Date (YYYYMMDDCC) On average, 82% of WWMCA global data points are usable (~1.29 million data points per run). 14
Verification Technique Determine model-observation pairs ADVCLD and SCFM are already co-located with WWMCA ground truth data points DCF points depend on domain s map projection When ADVCLD or SCFM is compared to DCF, use nearest neighbor to map ADVCLD, SCFM and WWMCA to the DCF domain WWMCA is dumbed down to the 6 categories when compared to DCF Data counts for total cloud contingency table categories (6 for DCF, 101 for ADVCLD, SCFM) are archived for long-term statistics calculations 15
Cloud Verification Statistics Root Mean Square Error Mean Absolute Deviation Forecast Bias 20-20 Index 16
Cloud Verification Statistics Root Mean Square Error Mean Absolute Deviation Forecast Bias 20-20 Index 17
20-20 Index Percent of model-observation data points with error 20% or less For each i of n forecast pairs: Forecast and observation expressed as a percentage ranging from 0 to 100 1 is best, 0 is worst 18
Domain-Wide Statistics 50 June 2009 DCF RMSE 49 48 47 Average RMSE 46 45 44 43 42 41 40 6 12 18 24 30 36 42 48 Forecast Hour 19
Domain-Wide Statistics 50 June 2009 DCF MAE 49 48 47 Average MAE 46 45 44 43 42 41 40 6 12 18 24 30 36 42 48 Forecast Hour 20
Domain-Wide Statistics 0 June 2009 DCF Bias -0.5-1 Average Bias -1.5-2 -2.5-3 -3.5-4 Forecast Hour 6 12 18 24 30 36 42 48 21
Domain-Wide Statistics 0.9 June 2009 DCF 20-20 Index 0.89 Average 20-20 Index 0.88 0.87 0.86 0.85 6 12 18 24 30 36 42 48 Forecast Hour 22
Sample WWMCA Distribution 25000 20000 15000 June 30, 00Z (both hemispheres combined) Almost 70% of the data points are 0 or 100%. This is a typical amount. Count 10000 5000 0 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 Total Cloud Percentage 23
Sample Contingency Tables 24-hour total cloud forecasts CONUS domain 18Z model run 30 day totals: June 1-30, 2009 6 DCF cloud categories = 6x6 table 24
June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% 0% 336,315 84,209 35,369 26,641 22,889 92,600 13% 46,839 31,483 17,711 14,392 13,511 57,593 33% 36,207 15,335 8,842 7,484 7,471 43,730 53% 35,750 14,314 7,806 6,500 6,047 30,874 73% 19,769 10,312 7,249 6,236 6,692 42,949 93% 75,845 49,645 36,367 37,284 43,644 476,603 25
June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% Total 0% 336,315 84,209 35,369 26,641 22,889 92,600 598,023 13% 46,839 31,483 17,711 14,392 13,511 57,593 181,529 33% 36,207 15,335 8,842 7,484 7,471 43,730 119,069 53% 35,750 14,314 7,806 6,500 6,047 30,874 101,291 73% 19,769 10,312 7,249 6,236 6,692 42,949 93,207 93% 75,845 49,645 36,367 37,284 43,644 476,603 719,388 Total 550,725 205,298 113,344 98,537 100,254 744,349 1,812,507 26
June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% Total 0% 336,315 84,209 35,369 26,641 22,889 92,600 598,023 13% 46,839 31,483 17,711 14,392 13,511 57,593 181,529 33% 36,207 15,335 8,842 7,484 7,471 43,730 119,069 53% 35,750 14,314 7,806 6,500 6,047 30,874 101,291 73% 19,769 10,312 7,249 6,236 6,692 42,949 93,207 93% 75,845 49,645 36,367 37,284 43,644 476,603 719,388 Total 550,725 205,298 113,344 98,537 100,254 744,349 1,812,507 Hit Rate = 0.478 HSS = 0.270 27
June, 2009 Total Cloud WWMCA (Observation) DCF (Forecast) 0% 13% 33% 53% 73% 93% 0% 336,315 84,209 35,369 26,641 22,889 92,600 13% 46,839 31,483 17,711 14,392 13,511 57,593 33% 36,207 15,335 8,842 7,484 7,471 43,730 53% 35,750 14,314 7,806 6,500 6,047 30,874 73% 19,769 10,312 7,249 6,236 6,692 42,949 93% 75,845 49,645 36,367 37,284 43,644 476,603 Let s simplify to a 2x2 contingency table cloud vs. no cloud 28
2x2 : Cloud vs. No Cloud WWMCA (Observation) DCF (Forecast) 0% Non-Zero 0% 336,315 261,708 Non-Zero 214,410 1,000,074 29
2x2 : Cloud vs. No Cloud WWMCA (Observation) DCF (Forecast) 0% Non-Zero 0% 336,315 261,708 Non-Zero 214,410 1,000,074 Hit Rate = 0.737 (was 0.478 for 6x6 table) HSS = 0.394 (was 0.270 for 6x6 table) POD = 0.611 FAR = 0.438 CSI = 0.414 30
Using MET MODE MET = Model Evaluation Tools MODE = Method for Object-Based Diagnostic Evaluation Tool How does MODE perform with cloud forecasts? 31
MET MODE Example Total Cloud Cover Sample Event: July 15, 2009 06Z Model Run, 6-hour forecast 15 km CONUS DCF vs. 16 th mesh WWMCA (~24 km) WWMCA is re-mapped to exactly match the DCF domain for use in MODE 32
Resolving Objects: Threshold DCF is already limited to 6 categories Non-zero cloud amounts are dominated by 100% cases All 100% cases are coded as 93% in DCF Threshold is the 93% DCF category (81-100% cloud) Used ge81.0 for both raw forecast and observation value in the configuration file 33
DCF Total Cloud Forecast 34
WWMCA Ground Truth 35
IR Satellite Image 36
WWMCA Ground Truth Satellite Pass Boundary Terrain? 37
WWMCA Objects, > 0 gs (Default) 38
MODE Defaults Area Threshold for Objects: 0 grid squares (gs) Convolution Radius: 4 grid units (gu) Is there any benefit to changing these? 39
WWMCA Objects, > 50 gs 40
WWMCA Objects, > 100 gs 41
Convolution Radius = 4 gu (Default, Objects set to 50 gs) 42
Convolution Radius = 2 gu 43
Convolution Radius = 1 gu 44
MODE Summary Plot (using Defaults) 45
MODE Summary Plot (using Defaults) 46
Diagnosing DCF Performance How is MODE best used for cloud model verification? Domain-wide summaries? dominated by large objects? Noisy WWMCA adds to the challenge Geographic subregions? Persistent objects (e.g., Coastal stratus) 47