VERIFICATION OF PRECIPITATION FORECASTS ASSOCIATED WITH MID-LATITUDE CYCLONES ACROSS THE EASTERN UNITED STATES.

Transcription

1 VERIFICATION OF PRECIPITATION FORECASTS ASSOCIATED WITH MID-LATITUDE CYCLONES ACROSS THE EASTERN UNITED STATES. A THESIS PRESENTED TO THE DEPARTMENT OF HUMANITIES AND SOCIAL SCIENCES IN CANDIDACY FOR THE DEGREE OF MASTER OF SCIENCE By JESSICA WILLIAMS NORTHWEST MISSOURI STATE UNIVERSITY MARYVILLE, MISSOURI NOVEMBER, 2013

2 VERIFYING WEATHER FORECASTS Verification of Precipitation Forecasts Associated With Mid-Latitude Cyclones over the Eastern United States Jessica Williams Northwest Missouri State University THESIS APPROVED Thesis Advisor, Dr. Ming-Chih Hung Date Dr. Yi-Hwa Wu Date Mr. Jeffrey Bradley Date Dean of Graduate School, Dr. Gregory Haddock Date

3 Verification of Precipitation Forecasts Associated with Mid-Latitude Cyclones across the Eastern United States Abstract The accuracy of weather forecasts is crucial in preparing the community for potentially dangerous weather. Geographic Information Systems (GIS) are used to display information spatially in order to assist in decision making processes. Forecast Meteorologists use much data and information in making decisions to provide the most accurate forecast possible. One important piece of information used in the forecast process is how well Numerical Weather Prediction (NWP) models are performing, or verifying. Currently, NWP model verification information is not displayed spatially, with only descriptive statistical scores being computed operationally. Displaying this verification information spatially will provide more useful information to help in the forecast process decisions, thereby increasing the likelihood of more accurate forecasts. This research used GIS to verify a sample of 32 Global Forecast System (GFS) NWP model 24 hour precipitation forecasts associated with mid-latitude cyclones over the Eastern US. The research methodology produced spatial verification maps which show model errors spatially related to the mid-latitude cyclones alongside quantitative verification scores for each day and overall. A hypothesis test was performed to determine if the GFS total precipitation over the study area is statistically different from the observation during mid-latitude cyclones. The hypothesis test results infer with 95% confidence that the GFS total precipitation means are statistically different from the iii

4 observation total precipitation means. Based on the study of 32 sample days for the 2013 winter and spring season, the spatial verification maps as well as the quantitative scores reveal that the GFS model has a tendency to over forecast precipitation coverage associated with mid-latitude cyclones over the Eastern US. Finally, GIS models were built to investigate the possibility of having a near real-time automated process to provide spatial verification maps which could potentially be used in forecast operations. iv

5 TABLE OF CONTENTS ABSTRACT... iii LIST OF FIGURES... vi LIST OF TABLES... vii LIST OF ABBREVIATIONS... viii CHAPTER 1 INTRODUCTION Background Research Objectives...7 CHAPTER 2 LITERATURE REVIEW Incorporation of GIS into NWS GIS for Spatial Forecast Verifications Review of Verification Methods on NWP models Rationale for using GIS to Generate Spatial Verification of QPF...23 CHAPTER 3 CONCEPTUAL FRAMEWORK AND METHODOLOGY Weather Pattern, Geographical Area, and Time Frame Studied Data and Data Sources Research methodology Data Preparation Spatial Verification Descriptive Statistical Verification Inferential Statistics on Forecast GIS Models for Semi-Automation Forecast Verification...49 CHAPTER 4 ANALYSIS RESULTS AND DISCUSSION Spatial Analysis Descriptive Statistics Inferential Statistics...77 CHAPTER 5 CONCLUSIONS Review of Research Rationale for Spatial Verification Accomplishment of Research Objectives Potential Contributions Further Research Suggested...88 REFERENCES...91 v

6 LIST OF FIGURES Figure 1 Geographical Study Area...27 Figure 2 Data Preparation Flowchart...35 Figure 3 Spatial Verification Flowchart...39 Figure 4 Attribute Table for a Sample Error Grid...43 Figure 5 Descriptive Statistical Verification Flowchart...44 Figure 6 Inferential Statistical Verification Flowchart...49 Figure 7 GIS Model 1 Net2Ras...50 Figure 8 GIS Model 2 Final...51 Figure 9 Final Flow Chart for all Verification Results...53 Figure 10 Final Spatial Verification Maps...72 Figure 11 Histogram of Forecasted Total Precipitation Samples...80 Figure 12 Histogram of Observed Total Precipitation Samples...80 vi

7 LIST OF TABLES Table 1 Verification Dates...32 Table 2 List of Categories in the Verification Maps...40 Table 3 Contingency Analysis Table...41 Table 4 Descriptive Verification Statistics...75 Table 5 24-Hour Total Precipitation Amounts (mm)...78 Table 6 Precipitation Totals Statistics...79 Table 7 Results of Two Sample Difference of Means Hypothesis Test...81 vii

8 LIST OF ABBREVIATIONS NOAA NWS NCEP GFS QPF JWGFVR GIS NWP National Oceanic Atmospheric Administration National Weather Service National Center For Environmental Prediction Global Forecast System weather model Quantitative Precipitation Forecast Joint Working Group on Forecast Verification Research Geographic Information System Numerical Weather Prediction viii

9 CHAPTER 1: INTRODUCTION Accurate weather forecasts minimize the impacts severe weather can have on the local and national economy by reducing the death toll and protecting property. This is the primary mission of the National Oceanic Atmospheric Administration s (NOAA) National Weather Service (NWS) (National Weather Service 2011). With enough lead time, accurate forecasts allow people to take necessary precautions such as protecting their property and evacuating the area if needed, thereby reducing the risk of property damage and deaths that could result from the severe weather. Weather forecasts, especially precipitation forecasts, have a significant impact on the economy and daily commerce. To name just a few, weather forecasts significantly impact emergency management and preparation, search and rescue efforts, business decisions, homes and real estate, energy forecasting, hydrology and water resources, agriculture, forensics, aviation and mere public convenience. Bad weather forecasts have a negative, costly, and sometimes deadly impact on any one of these areas. Therefore, accurate weather forecasts are critical and highly sought after by all. The ultimate goal of this research is to improve weather forecasts by spatially verifying weather model precipitation forecasts associated with mid-latitude cyclones, which bring heavy precipitation to populated areas. 1.1 Background In order to understand the relevance of this research and the problem it s solving, a basic background on forecasting, weather models, and weather model verification is 1

10 necessary. This section will first explain the weather forecasting process and how it can be improved. Then it will explain the basics of Numerical Weather Prediction (NWP) models and the need for these models to be verified, specifically the quantitative precipitation forecasts (QPF) generated by these models, in order to obtain more accurate forecasts. Finally it will discuss the current operational verification methods used by NOAA and the lack of spatial verification results. The weather forecasts we receive on television, internet, or radio do not come directly from automated weather models. They are generated by Forecast Meteorologists who analyze in-situ data such as surface and upper-air observations, radar and satellite imagery, the Numerical Weather Prediction (NWP) models, and statistical guidance. In combination with the models and data, the Meteorologists also apply climatology, pattern recognition, local geography, learned rules of the weather patterns, and their own forecasting experiences to create a forecast for a specific area of their responsibility that is disseminated to the public (Ahrens 2000). A majority of this information they ingest in making their forecast is displayed geographically on information processing systems. The forecasts they create include many variables, some of which are precipitation amounts, wind speed and direction, dew points, and cloud cover. These forecast variables are created geographically on a map of their area of responsibility in grid format, so to create continuity of their forecast throughout the area. The forecast grids created by the meteorologists for their area of responsibility are meshed with all the different NWS forecast center areas to create one national grid for each forecast variable and time frame; the final products are part of the National Digital Forecast Database (NDFD). The accuracy of their forecast is dependent upon all of the sources they use and 2

11 their experience and understanding in weather behavior and patterns in their area of responsibility. One of the best ways a meteorologist can improve their weather forecast is by knowing more about the NWP models and how well they are verifying. The forecasts that are more than a couple hours out weigh most heavily on the NWP models. These models are mathematical models which contain many equations that approximate how atmospheric variables, such as temperature and pressure, will change over time (Ahrens 2000). Since the atmosphere and its changes over time contain many variables in 4 dimensions (X, Y, Z, and time) and are extremely complex and not fully understood, these numerical models can never be exact. This is why there is always room for more research and forecasting improvement. These NWP models are run on super computers at model processing centers across the world in order to solve the mathematical calculations. In the United States, NOAA s National Center for Environmental Prediction (NCEP) runs a few well known global and national models, specifically the Global Forecast System (GFS) and the North American Mesoscale Model (NAM). They ingest observational data around the globe and run this data through the model equations for a set of initial conditions at time t = 0, then solve for each increment of future time, which depends on the models temporal resolution. The model solutions are grid points at a set distance apart based on the models spatial resolution. The computer interpolates the grid points to create contours or surfaces/fields of atmospheric variables which are output in charts on a map for the surface and at varying vertical levels of the atmosphere for each future time. Each model forecast chart is called a prognostic chart. 3

12 With computers today, the model initial conditions and prognostics are viewed, analyzed, and combined with other atmospheric or topographic data in information processing systems. Within NOAA, forecasters currently use the Advanced Weather Interactive Processing System (AWIPS) to display model data. These meteorological processing systems used by the NWS are similar to Geographic Information Systems (GIS) in that they are now capable of overlaying several layers and changing colors and transparency for easier analysis, but they do not provide the processing and analysis capabilities of GIS. With over a handful of weather model solutions to chose from, each with their own differing set of dynamic equations, specializing in one area or another, and each with different spatial and temporal resolutions, meteorologists have a lot of decisions to make on which models to rely on most on a particular day. Verification of the NWP models is a key factor in the forecast process, as verification will allow the meteorologist to know how well these models are doing and how heavily to rely on each model solution (Ebert & McBride 2000). They can adjust their forecasts according to the verification results, and provide a more accurate forecast for the public. The Joint Working Group on Forecast Verification Research (JWGFVR) stated that verification of forecasts is needed in order to monitor forecast quality, compare the forecast quality of different weather models, and most importantly to improve forecast quality (JWGFVR 2012). Verification of NWP models is comparing both objectively and subjectively the forecast parameter with the observed value at that same forecast time and over the same location. The verification can be objectively, meaning that the forecast and observation are compared to provide statistical scores or even inferences, or even a new spatial map of model error. Or it can be subjectively, 4

13 meaning the forecaster visually looks or overlays model data with observed data, notes the differences, and takes this into account when using the model for their forecast that day. Model verification skills are also used in determining whether a models equations need to be updated to improve the model performance. This decision is highly dependent on the Quantitative Precipitation Forecast (QPF) skills from the model. Since QPF is generated from equations that involve many other variables it is a good estimator of model performance (Hamill 1999). Forecast accuracy, measured by error and statistical scores, is more important to society when the forecast has value. A forecast has high value if users of the forecast are relying on it to make important decisions, like whether to evacuate or not (JWGFVR 2012). Verification studies on forecasts suggest the degree to which the forecasts can be relied upon in making these decisions (Goddard et al. 2003). Precipitation forecasts are therefore of high value to the public, and this research will focus on the verification of Quantitative Precipitation Forecasts (QPF), a total amount of precipitation forecasted to fall over a set amount of time. Precipitation location and amounts are also good indicators of the strength and position of mid-latitude cyclones and associated cold and warm fronts. According to the NWS Instruction on Verification (NWS 2011), the operational QPF verification process compares each QPF to the observation for that time and space, measures the forecast error, and calculates statistics to help assess the forecast quality. Most real-time verification methods currently produced by the NWS and NCEP are based on point observations and are also not computed in real-time. They only provide an error amount or statistics on forecast quality or at most display the forecast and observation 5

14 grids side by side (Strassberg & Settelmaier 2010). Point verifications only verify the forecast at those points, current performance statistics do not show where the model needs to be improved spatially in relation to weather systems, and displaying the forecast and observed grids side by side provide only subjective analysis, making it difficult to tell exactly where the model is not performing well. QPF verification can also reveal different types of errors, such as in the position of a weather system, the shape and size of the rain pattern, and the magnitude or intensity of the rain (Ebert & McBride 2000). Therefore, spatial verification of QPFs can reveal much more about the types of errors present in each QPF verified. The results of current verification methods are mainly quantitative, providing descriptive statistical skill numbers to assess the quality of a NWP model for each verification period on a national or global level. Current verification methods do not consist of operational spatial verification maps of QPF nor are they based on weather systems (Zhu 2007; Ebert & McBride 2000). While the verification statistics currently generated will help a forecaster determine how well a model is doing for a particular forecast parameter for that time period, it does not help them determine how the model is performing spatially in relation to a particular weather system or geographical area. GIS software has the ability to quickly perform spatial verification by creating error grids from forecast and observed grids, therefore displaying statistical verification information geographically. The importance of this is that geographical verification will allow forecasters to quickly see where model biases are in relation to certain areas and weather patterns allowing them to have a forecast opportunity where the forecaster has a better chance of improving upon the model s forecast (Strassberg & Settelmaier 2010). 6

15 1.2 Research Objectives The main research objective is to produce spatial verification maps alongside descriptive statistics on forecast precipitation error associated with mid-latitude cyclones in the Eastern US for the GFS day two 24-hour QPF. The purpose of this is to provide a spatial distribution of hits, misses and false alarms for precipitation in relation to midlatitude cyclones, allowing forecasters to easily detect timing and location errors of the model and improve upon the model. A sample of 32 different random forecast days were verified to provide a greater understanding of model tendencies and errors spatially associated with mid-latitude cyclones. Meteorologists can use this knowledge to improve their forecasts accordingly when relying heavily on the GFS model forecast during midlatitude cyclones. Alongside the spatial verification maps, descriptive statistics were generated to have a quantitative objective analysis on how well the model performed for each day and for the entire sample. Inferential statistics were also computed to answer if the day two QPF forecasts are significantly different from the observations to determine the need for the improvement of forecasts over the models. A secondary research objective was to investigate the use of GIS in producing spatial verification results in near real-time for daily use by forecast meteorologists. The ultimate goal of using GIS to verify NWP model forecasts and provide the results spatially is to show where the model needs improvement in relation to weather systems, thereby giving forecasters the information to improve the accuracy of weather forecasting accordingly. To do this, GIS models were made to semi-automate the process of producing the spatial verification results. 7

16 CHAPTER 2: LITERATURE REVIEW Utilizing GIS to improve weather forecasts has been a topic of much research over the past decade. This section will first discuss the ways GIS has been slowly incorporated into NWS operations. Then it will discuss some previous research using GIS to study weather patterns spatially as well as using GIS to provide spatial verification of forecasts. Next, previous research and operational methods of NWP model forecast verification will be reviewed. Finally, a critical analysis of previous research and rationale for this research will be discussed. 2.1 Incorporation of GIS into NWS Weather data is useless if it is not associated with a geographic location. To best visualize and use weather data, it is displayed onto maps in points, contours, polygons or fields. Before computers, meteorologists would hand draw on maps observations and contours to analyze the weather patterns and create forecasts. Meteorology has come a long way since the introduction of computers, information processing systems, and GIS. GIS is well known for being a decision support tool in many fields and for many applications. This is true also in the field of meteorology and weather prediction, as well as emergency management during severe weather events. Customized and open source GIS applications, as well as standard GIS desktop applications are being incorporated into the NWS operations and research (Dobson et al. 2009). According to Waters (2006), the NWS issues about 30,000 short term storm warnings each year. These warnings include tornado, severe thunderstorm, flash flood, 8

17 and marine warnings. Prior to 2007 the NWS issued these warnings for entire counties, which put many people under a warning when they did not need to be, especially for larger counties. With the advance of GIS in the NWS, they switched to issuing polygon shaped warnings based on the actual storm size and predicted short term locations affected, which dramatically reduces the false alarm rate (Waters 2006). According to Stellman et al. (2009), in September 2007 NOAA began using Google Earth Pro and Google Maps Application Programming Interface (API) for their operations. Using the web mapping capabilities through the Maps API, the NWS can show the storm warning polygons on Google maps, interact with the map, and determine where the storm is compared to businesses, addresses and schools. This also improves verification of the storm warnings, since they can contact potential businesses or spotters in the area to inquire for storm reports. This short time verification helps the forecaster immediately as they can use this information to continue the warnings as the storm moves on or not (Stellman et al. 2009). The Google software also allows meteorologists and hydrologists to overlay many weather data layers onto political or geographic layers such as terrain or industry. Using these multiple layers in one map improves situational awareness for the forecasters when looking at many datasets for making their decision on warnings and forecasts. Using Google Earth s GIS has also enabled better collaboration and data sharing between offices and emergency managers. The web mapping application also allows for better understanding in post storm analysis of storm tracks and photographs, and aids in verification of storm warnings (Stellman et al. 2009). The Advanced Weather Interactive Processing System (AWIPS) currently used by NWS now has the capability to read GIS 9

18 format files. Therefore, GIS data can add to the benefits of AWIPS by combining other datasets involving demographics, infrastructure, transportation, etc., to the weather information, which leads to more informed decision making in the forecast and warning process (Dobson et al. 2009). Open source GIS tools such as API s have also been incorporated into some NWS offices to save time in searching for weather data and to display several data types into one internet based map. The NWS office in Greenville, SC created what is called a Mash-Up in open source GIS using the Google API, Google Maps, and a database of available webcams in their area. The map provides links to the webcam and other information about the webcams, as well as other layers of meteorological data and weather stations. This assists in quickly viewing weather conditions across the area through cameras and in-situ observations all while sitting at the desk in the office. It helps to verify weather events, and in the prediction of weather for the day (Dobson et al. 2009). NWS employees and the meteorological research community are beginning to use desktop GIS software to perform studies on large samples of weather data. This ranges from using GIS to understand tropical cyclones in relation to heavy precipitation areas, in the verification of tornado or severe thunderstorm warnings, and even in the verification of NWS and some NWP model forecasts. 2.2 GIS for Spatial Forecast Verifications Some research has been conducted using GIS for the analysis and processing of meteorological data to provide spatial verification of weather forecasts from models, the 10

19 NWS forecasts, and climatology forecasts. The advantage of spatial verification provides discernment in model biases over geographic regions in given weather patterns. This enables the forecaster to identify where and when models perform a certain way in relation to weather systems and geography, and adjust their forecasts accordingly (Strassberg & Settelmaier 2010). It also allows users of the forecasts to weigh in the reliability of the forecast in their geographical region in deciding whether or not to use the forecast in making decisions (Goddard et al. 2003). This section will discuss research using GIS to produce real-time or operational forecast verifications spatially in relation to weather patterns. GIS is also being incorporated into many meteorological studies of past weather events to better understand these events and the relationship among the storms and impacts, in order to better forecast events. One example of this is in a study Matyas (2010) completed on precipitation data associated with tropical storm and hurricane centers at landfall. The study was done in a GIS in order to better understand where precipitation is heaviest in relation spatially to the tropical cyclone for the 24 hours after landfall, when the most damage is typically done. The study focused on daily precipitation totals for a sample of 43 different tropical cyclones of varying intensities from 1995 to The motivation for understanding where the heaviest precipitation falls in relation to the tropical cyclone center for 24 hours after landfall was to give more advance and accurate warning for flooding associated with these storms in the future. Heavy convective precipitation area extents and their relation to the tropical cyclones center were determined based upon other meteorological variables associated with the storm, such as intensity, storm motion, vertical wind shear, and extra-tropical transition, 11

20 using the GIS software. The precipitation areas were converted into polygons for easier analysis of location shifts. The study found areas of heaviest precipitation occurred on the right front quadrant of the tropical cyclone center, but shifted towards the front of the cyclone as the storm became either extra-tropical or weakened. The area of precipitation became larger if the storm became extra-tropical, but smaller if it weakened. This study is an example of how GIS gives a better understanding of weather patterns associated with storms, based on a large sample of storms in the past, to help us better forecast weather impacts with future storms. Though this research does not verify forecasts, it studies the tendencies of severe weather such as flooding and how it is spatially related to the tropical cyclones. It is similar to this research in that this research will show model tendencies and error in spatial relation to low pressure systems for the forecaster to take into consideration and improve upon when forecasting for these events. Feidas et al. (2007) examined the possibility of developing a fully automatic GIS model for real-time qualitative and quantitative verification of precipitation forecasts from the NWP Bologna Limited Area Model (BOLAM) in Greece. The GIS model was developed using programming and scripting in ArcGIS, and named the Precipitation Forecasts Evaluator (PFE). The research emphasized the importance of this type of model in short-range forecasting of weather events associated with heavy rain, and tested the PFE for a 2-day heavy rain event that occurred in In the PFE model, precipitation observation estimates extracted from satellite data as well as rain gauge data were used to verify one of the Greek weather model QPF forecasts, the Bologna Limited Area Model (BOLAM). The authors stated the ideal case would be to use rain gauge data for the verification of forecasts, but since it is rarely available there until 12 hours later, to 12

21 use rain gauge data for a real-time model of verification of the current situation would be difficult. Because of this, the precipitation estimates extracted from satellite data are used, since they are available at near real-time over the country of Greece. The GIS model runs the observed precipitation estimates and forecasts through GIS processing so the spatial and temporal resolutions match up and the projections are the same. A pixel by pixel comparison is done to reveal a difference grid between the forecast and observation for the 3-hour total precipitation as well as a final contingency map revealing hits, misses, and false alarms spatially. The automated tool was designed so forecasters can quickly get a spatial error grid and its relation to the weather pattern for use in making their future forecasts. Not only did the outcome provide a spatial error grid, but also numerical statistics calculated from a statistical contingency table of hits or misses. The research of Feidas et al. (2007) greatly influenced this research in some of the methodology for verifying NWP models. Although the main purpose behind the research in Greece was to provide a real-time operational GIS tool for forecasters to use, whereas the main purpose for this research is to study the GFS error and behavior associated with low pressure systems across the US by running a similar model through a large sample of past events. Knowing the spatial behavior of a particular model and its performance during a specific weather pattern will allow forecasters to take this into consideration as yet another tool to use in the forecasting process. Meteorologists with the NWS have been exploring GIS to verify weather forecasts for the past couple years. The NWS have been maintaining performance statistics on their forecasts for many parameters. However, these statistics are not graphically based or displayed. Settelmaier (2009) stated GIS can be used to display old 13

22 verification information to help understand a month s weather patterns on forecast performance. For this purpose, he used ESRI s ArcGIS software to display the average 7 day forecast errors in morning low temperatures for April 2008, displayed by NWS office County Warning Area (CWA). Settelmaier (2009) went a step further and displayed forecast error of the NWS low temperature forecast compared to high resolution analysis of low temperature observations to display forecast error spatially across a 5X5km grid for one day. Due to the spatial presentation of forecast temperature error, the highest error was notably along and behind a cold front. Therefore, the result of displaying forecast error geographically allows the user to see where the forecast error is in relation to weather patterns, such as a cold front. Strassberg and Settelmaier (2010) then went a step further to run a regularly scheduled GIS process to create near real time difference grids between the model forecast and the human forecast relative to the observed data for the NWS office in Miami, FL. In this case the regularly scheduled error grids are generated for morning low temperature forecasts. In order to run this automated process, the 5km Continental United States (CONUS) forecast grids from the NDFD and the observation grids are automatically downloaded, and basic difference grids (forecast minus observation) for the past 7 days are created using the geoprocessing tools available in ESRI s ArcGIS. These difference grids are run at the NWS southern region headquarters, and downloaded at the NWS Miami office automatically for additional processing and writing to an internal webpage for display and used by forecasters as well as archived for future analysis. It is stated if this kind of implementation and use is achieved nationwide, temperature forecasts are likely to improve. 14

23 Based on the work of the NWS so far, it shows that GIS is in the beginning stages of being incorporated into NWS verification techniques for their forecasts. However, there is still much research and work left to be done. For example, the NWS incorporation of GIS into daily operations has been in the verification of their forecasts, mainly in temperature forecasting at only some offices, and not in the verification of NWP models. While accurate temperatures are important, precipitation has a much greater impact on daily operations, so precipitation forecasts are extremely important especially during heavy precipitation events affecting large areas. The research done in Greece (Feidas et al. 2007) on implementing real-time verification tools for NWP precipitation forecasts are a few steps ahead. Although no studies have been done thus far on large samples of data to infer how a specific model behaves during a selected weather pattern, the daily verification of precipitation can still be used to improve forecasts. 2.3 Review of Verification Methods on NWP models Operational verification of NWP models within the NWS are performed using statistical methods to produce descriptive forecasting skill numbers to show how well a model is performing. Skill numbers are a scalar quantity described by single numbers that indicate a measure of how well a forecast performed (Goddard et al. 2003). This section will discuss current descriptive statistical verification methods and explain what the descriptive statistical skill numbers mean and how they are affected. Then it will discuss some research on methods of inferring statistical differences between model forecasts. 15

24 Finally, it will discuss research which presents new methods of verifying NWP spatial QPFs. The NOAA Hydro-meteorological Prediction Center (HPC) started calculating verification statistics for NWP QPF s in 1971, all of which are posted on the HPC website (NWS 2011). Precipitation is considered a dichotomous forecast, meaning the forecast predicts whether an event will happen or not, oftentimes based on categorical thresholds of the event, like greater than 1 inch of precipitation (JWGFVR 2012). The most common and preferred way of statistically verifying dichotomous forecasts, to include QPF, in operations and the meteorological research community is using a two by two contingency table for the observed and forecast categories chosen. The two by two contingency tables produce four outcomes: hits when the model correctly forecasts rain, false alarms when the model forecasts rain but no rain is observed, misses when the model does not forecast rain but rain is observed, and correct negatives when no rain is forecasted or observed (Zhu 2007; JWGFVR 2012; Hamill 1999). This is considered an objective evaluation of global precipitation forecast models as it provides comparative skill numbers on forecast performance. Feidas et al. (2007) used GIS to generate statistical scores based on the 2X2 spatial contingency tables generated, in addition to the spatial verification grids. In the PFE tool created in GIS, summary statistics on the models QPF performance for each of the 3-hour forecasts would pop up on the screen alongside the difference and contingency grids to provide the user with descriptive statistics as well as a spatial analysis of forecast performance. Zhu (2007) performed a four year study on the verification of the GFS model QPFs using different verification techniques and different spatial and temporal 16

25 resolutions to determine the impacts these have on model skill numbers. According to the research, the skill numbers improve over the years as the model equations representing the physical processes are updated. Although these descriptive verification skill numbers are a great way to compare forecasts and determine forecast performance, the numbers themselves are highly dependent on the resolution and time period of the verified forecasts. Skills have shown to be higher with lower resolution or larger area coverage verified, and also higher over a longer time period (Zhu 2007). A longer time period or larger area verified gives the model more room in missing precipitation by a short amount of time or by only a small distance, therefore the skill will be higher. Many descriptive statistics can be calculated based on a 2X2 contingency table, all of which tell something different about model performance and can be used in combination with each other to describe the performance best. Two verification skill numbers that are considered most useful and common are the Bias and Equitable Threat Score (ETS) (Hamill 1999; Zhu 2007). The BIAS is a ratio that measures the frequency of forecasted events to observed, so it is a measure of whether the model forecasts tend to under forecast the event (BIAS<1) or over forecast (BIAS>1). The ETS measures the fraction of forecast events that were correctly predicted, and takes into consideration hits associated by random chance. A perfect score is 1.0 indicating a perfect forecast, and 0.0 indicates a completely random forecast (JWGFVR 2012; Hamill 1999). Some other useful verification numbers that can be used alongside the BIAS and ETS, and to help understand performance are the False Alarm Rate (FAR), the Hit Rate or Probability of Detection (POD), and the Accuracy. The Accuracy measures the fraction of the forecasts which were correct, but can be misleading since it can be influenced by the most common 17

26 category of correct negatives if the spatial area studied is large compared to the precipitation falling in that area (JWGFVR 2012). There are several other skill numbers that can be computed from a 2X2 contingency table of verification; however based on the usefulness and commonality of the above skill numbers mentioned, this research will provide the Accuracy, BIAS, ETS, and FAR on yes or no precipitation forecasts of the GFS NWP model. The skill numbers based on model verification are descriptive statistics that help forecasters determine how well a model is performing. Verification of precipitation forecasts using formal hypothesis testing to infer whether NWP model forecasts are significantly different from each other or from the observations are rarely performed (Hamill 1999). Because of this, Hamill (1999) explored using hypothesis tests for evaluating differences between different NWP model forecasts. The hypothesis tests were performed mainly on descriptive skill numbers to determine if better numbers from one model were statistically significant. The issues associated with performing hypothesis testing on model forecasts and comparing these forecasts was discussed in detail, to include whether the samples were normally distributed, and how the spatial correlation would impact the testing. Perhaps the reason why hypothesis testing lacks in the verification of weather forecasts is due to these complications. Hamill did investigate using the two sample difference of means test, but did not perform this test on total precipitation amount forecasts for competing forecasts, or to compare it to the amount in the observations. Climatology forecasts, if they are reliable, can be of great value in the decision making process to management in industries such as energy, agriculture, and water 18

27 resources. Therefore, the verification of climatology forecasting is also important to both forecasters and users of the forecasts. Goddard et al. (2003) generated spatial maps of the overall forecast skill for the International Research Institute for Climate Prediction (IRI) net assessment forecasts. Forecast skill is a scalar quantitative measurement of the reliability of a forecast based on verification techniques. There are several different forecast skills, but this research utilized the Rank Probability Skill Score (RPSS) because of its effectiveness in verifying probabilistic forecasts, and the IRI net assessment forecasts are probabilistic. The net assessment forecasts contain predictions for seasonal, or 3-month, mean temperatures and precipitation. The precipitation forecasts are probabilistic forecasts for 3 categories: above, below, or at climatologically normal precipitation for the predicted season. The norm was established based on a 30-year historical distribution for that season, where above or below normal predictions would be in the upper or lower 15 th percentile of the distribution. So a 30% probability of having above normal precipitation would be double what climatology would suggest, and could assist in preparations necessary for a wetter than normal season over a particular region. The verification data used in this study was a combination of 1200 rain gauge observations and satellite observations in a 2.5 by 2.5 degree latitude and longitudinal grid over the entire globe. The spatial layout of the RPSS numbers produced global maps showing the spatial distribution of how well the net assessment forecasts performed, where positive RPSS numbers indicated the forecasts were better than climatology, zero were equal to climatology, and below zero was worse than climatology forecasting. The study was performed for each of the four seasons over a four year period from Some 19

28 conclusions of the study stated that the overall skill numbers were higher in wet seasons for particular geographical areas. More importantly, the results with a spatial global display of forecast skill show where patterns of positive skills exist, in these geographical locations, forecast users can rely more on the forecast for input in decision making (Goddard et al. 2003). Although the climatology prediction verification study did not use GIS to generate spatial maps of forecast skill, the final result served two purposes to first show users where and how much to rely on forecasts as input in their decision making, and second to show forecasters geographically where they can improve upon. The difference is in long term or seasonal forecast verification where the maps of forecast skill relate forecast skill to seasons and geographical areas, whereas more short-term forecast spatial verification maps relate forecast skill to weather patterns and geographical areas. The net assessment forecasts were verified using a combined rain gauge and satellite derived dataset, due to the study area being global and lack of rain gauges in many regions. Since the verification in the net assessment study was both categorical and probabilistic, as opposed to the deterministic forecasting of rain or no rain in this study, the skill measure used is different and there were no spatial maps of contingency analysis shown. Several new methods for the verification of NWP precipitation forecasts have been proposed due to the complexity of precipitation forecast verification and the importance of accurate precipitation forecasts. Ebert and McBride (2000) proposed an object-oriented approach in the verification of NWP QPFs. The objects of verification were defined contiguous rain areas, denoted by CRA s in the research. The CRA s were generated based on isopleths of a set amount of rainfall forecasted or observed, and 20

29 typically represented weather systems. The goal was to produce three types of errors for QPFs, the first two of which are spatial in nature: displacement errors, pattern errors, and rain volume errors. The verification was performed on 24-hour accumulated rainfall from an NWP Model in Australia over a four year period from The results showed 50% of the errors were in a displacement of the rain area, 45% in the spatial weather pattern, and only 5% in the actual volume of rain which fell in the 24 hours. Interpretation of the results based on the CRA s, or weather systems, were able to show model tendencies over particular geographical areas, such as a tendency to place the weather system too far west during particular months, or too far northward along the south coast of Australia, often times indicating the model moving the system too fast or slow if it was a directional movement coinciding with the directional movement of the storm system. Tartaglione et al. (2005) used the CRA approach proposed by Ebert and McBride (2000) in a case study verifying forecasted rainfall with a rain gauge observational network over Cyprus Island in the Mediterranean Sea. The NWP model forecasts were from the BOLAM model, which is the same model that Feidas et al. (2007) studied in presenting their use of GIS in the verification of QPF. They verified 24 hour precipitation forecasts with a rain gauge network from the time period of 06 UTC to 06 UTC. However, their case study only analyzed one precipitation event. The conclusions stated that while the CRA verification does indicate systematic errors in quantitative precipitation forecasts, the evaluation of precipitation for small areas, such as the island of Cyprus or in oceanic regions, may not be as effective due to the size of the area compared to the forecasted rain field, which mainly falls over the ocean. 21

30 Casati (2004) presented a new verification technique for spatial precipitation based on an intensity scale, called the intensity-scale verification approach on precipitation over England. This approach separated the verification of precipitation by differing intensities of precipitation, such as a convective precipitation event, drizzle/light rain event, or heavy precipitation associated with a main weather system. The forecasting skill, or numerical value representing the effectiveness of the forecast, was a function of a spatial scale of forecast error and intensity of the event. The precipitation forecasts evaluated were 3-hour precipitation forecast accumulations with a 3-hour lead time. The presentation of this verification method was based on the findings of previous research concluding that precipitation fields, or areas, are largely impacted by the presence of spatial weather patterns, such as mid-latitude cyclones. The results showing a loss of forecasting skill was mainly due to small horizontal displacement of intense precipitation events and the misplacement of the large-scale storms. The results from Casati (2004) agreed with the object-oriented verification approach presented by Ebert and McBride (2000) as they both suggested precipitation errors are mainly spatial errors in the placement of the precipitation areas. Although neither approach utilized GIS in their methodology, nor did they present geographical maps with precipitation errors, the results proved that the errors were indeed mostly spatial in nature. To make the best use of precipitation verification, GIS can more efficiently and quickly determine the spatial errors on large datasets as well as display them in a way that forecasters can better understand the errors and improve upon them. With a more efficient and faster way to verify QPF s, systematic tendencies on different 22

31 NWP models over different regions, and in different weather patterns, can be deduced, and then improved upon. 2.4 Rationale for using GIS to Generate Spatial Verification of QPFs While the NWS is incorporating the benefits of GIS into their everyday operations for situational awareness, sharing data and providing more interactive display of their data, and even using it to verify temperature forecasts at some local offices, they have yet to utilize the geo-processing capabilities of GIS operationally for the verification of precipitation forecasts. In addition, much research has been conducted utilizing GIS to study weather and its impacts in spatial relation to particular weather systems. For example, the work Matyas (2010) did to relate heavy precipitation location to tropical cyclone storm centers. However, there is a lack of research in the meteorological community to determine NWP model performance behavior spatially in relation to specific weather systems in the United States. While there is currently operational real time descriptive statistical scores used for the verification of NWP model performance, these scores are not associated with a spatial representation of model errors. Also, there is a lack of hypothesis testing to research statistically significant differences between forecasts and observations. This research is intended to study the use of GIS to fill in the gaps and limitations in the current verification methods of NWP model forecasts and research studies. The methodology of this research in conducting spatial verification of model error as well as statistical scores could potentially be used operationally in the US, provided the NWS has the manning and funding to utilize GIS software to produce these spatial verification 23

32 maps. The goal of this research is to model how GIS can be used to study a sample of NWP model forecasts for a specific weather pattern in an effort to determine the population model errors spatially associated with that specific weather pattern. A final part of this research is to present hypothesis testing to ultimately infer whether there is a statistically significant difference between the forecast and observation for a specific weather pattern and geographical location. For this research the forecast parameter of QPF is used based on earlier statements on the importance of precipitation forecasts. The rationale for the research intentions stated above can be found in the lack of research in forecast verification spatially and the importance of forecast verification on forecast accuracy, and therefore how weather impacts the community. 24

33 CHAPTER 3: CONCEPTUAL FRAMEWORK AND METHODOLOGY Weather forecast verification can only produce reliable results if the area, weather patterns, and data are matched up appropriately and the methodologies used compare the proper forecast and observation data. This chapter will first describe the geographical study area and weather pattern for the samples studied, and the forecast time frame selected. Then it will give a description of the forecast and observed data and sources. Finally, the methodology used in creating spatial and statistical verification on NWP models that was used on the sample of data in this research will be discussed. 3.1 Weather Pattern, Geographical Area, and Time Frame Studied In order to make conclusive inferences about NWP model behavior based on verification results, a large sample needs to be analyzed for the same type of weather pattern in the same geographical area, and for the same forecast time frame. This section will discuss the reasoning behind the weather pattern, geographical area, and forecast time frame studied in this research. Weather forecast accuracy varies depending on the weather pattern. For instance, under high pressure precipitation forecast accuracy is much higher since precipitation is less likely to occur, and under low pressure precipitation forecast accuracy tends to be lower due to a greater chance for precipitation. Early weather forecasters studied precipitation and its causes; they found that precipitation was generally accompanied by areas of lower or falling pressure. Scientists from Norway studied patterns of lower pressure and developed models of low pressure systems which formed and moved along 25