Forecasting relatively rare business events like

Transcription

1 Subjective Probability Forecasts or Recessions EVALUATION AND GUIDELINES FOR USE By Kajal Lahiri and J. George Wang Kajal Lahiri is distinguished proessor o economics and director o the Econometric Research Institute at the University at Albany, SUNY. He has been a visiting scholar at the Social Security Administration, U.S. Department o Transportation, and the International Monetary Fund. He has received grants and contracts rom the National Science Foundation, World Bank, and many ederal and New York state agencies. He earned a B.A. rom Calcutta University, and a Ph.D. rom the University o Rochester. J. George Wang is an assistant proessor o inance at the business department o the College o Staten Island (CSI) o the City University o New York. Prior to joining CSI, he was a lead analyst o AT&T Bell Labs. He holds a Ph.D. in economics rom the State University o New York at Albany and a M.A. and a B.A. in economics rom Peking University in China. Probabilistic orecasts are oten more useul in business than point orecasts. In this paper, the joint subjective probabilities or negative GDP growth during the net two quarters obtained rom the Survey o Proessional Forecasters (SPF) are evaluated using various decompositions o the Quadratic Probability Score (QPS). Using the odds ratio and other orecasting accuracy scores appropriate or rare event orecasting, we ind that the orecasts have statistically signiicant accuracy. However, compared to their discriminatory power, these orecasts have ecess variability that is caused by relatively low assigned probabilities to orthcoming recessions. We suggest simple guidelines or the use o probability orecasts in practice. Forecasting relatively rare business events like recessions or major stock market corrections is inherently risky, resulting in requent misses and alse signals. However, when uncertainty about uture events is epressed in terms o probabilities (e.g., the probability o a recession net year is 3 percent), these orecasts are more inormative and useul than purely categorical orecasts (e.g., recession or no recession) in that the probabilities can be used in the calculation o various measures o interest such as epected payos and downside risks. Also, because more inormation is imbedded in probability orecasts, there may be more scope to improve prediction. The ailure o point orecasts rom large scale structural macro and VAR models or rom proessional surveys (e.g., Blue Chip, OECD, Survey o Proessional Forecasters, National Association or Business Economics, etc.,) in predicting or even timely recognition o postwar recessions is well documented. 1 Admittedly, recessions that are caused by eternal shocks cannot, by deinition, be predicted. However, the trans- 1 See, or instance, Filardo (1999, 4), Fintzen and Stekler (1999), and Juhn and Loungani (). 6 Business Economics April 6 Subjective Probability Forecasts or Recessions

2 mission o the eogenous shocks through the economy can take some time to generate a ull-ledged recession. Moreover, anti-inlationary monetary policies, that have oten caused recessions in the U.S. economy, take quarters to take eect. Thus, the basic challenge is whether one can identiy, at least probabilistically, an impending recession by understanding the structure o the transmission mechanism. Not surprisingly, in recent years, economists have developed advanced macroeconomic models to generate probability orecasts or business cycle turning points. However, one such model the dynamic single inde model developed by Stock and Watson (1993) could not identiy its irst two out-o-sample recessions, viz., those o 199 and 1. Since the Stock-Watson model is built on one o the strongest scientiic oundations ound in the literature and on etensive use o time series data, the ailures o their recession indees represent a signiicant challenge or today s business cycle researchers. In eplaining orecast ailures, Stock and Watson (3) painully all back on Leo Tolstoy in Anna Karenina, Happy amilies are all alike; every unhappy amily is unhappy in its own way. That is, econometric models typically ail to predict recessions because each recession is special in its own novel way. For eample, while the decline o the stock market gave some advance warning o the 1 recession, it was not otherwise reliable during the 198s and the 199s. In short, the structure o the economy changes sometimes abruptly and no single model speciication or a set o variables can do justice to all orthcoming recessions. Yet, recessions inlict enormous costs to society, the eact etent o which we have just begun to eplore. For instance, Bangia, et al. () showed that the economic capital required to capitalize a bank during a recession year is about 5-3 percent higher than that during an epansion year. Carey () ound that losses o a typical bank portolio during a recession are about the same as losses in the.5 percent tail during an epansion. Human costs due to lay-os and stock market declines are well known, and need no elaboration. In this paper, we will study the useulness o subjective probability orecasts obtained rom the Survey o Proessional Forecasters (SPF) as predictors o cyclical downturns. Since these probability orecasts are generated rom no speciic models or variables but are based on subjective probability heuristics o proessional economists, there may be certain advantages in using them over For eamples o models generating probability orecasts, see Diebold and Rudebusch (1991), Hamilton (1989), Stock and Watson (1991, 1993), and Zellner, et al. (1991). models based macro orecasts (Kahnemann and Tversky, 1973). Even though the probability orecasts are available since 1968, and have drawn some media attention, 3 very little systematic analysis has been conducted to look into their useulness as possible business cycle indicators. 4 Fortunately, there is a rich history o probability orecasts o rare events in meteorology, psychology, and geophysics, see or eample, Murphy (1991), Doswell, et al. (199), and Ogata, et al. (1994). We will utilize veriication methodologies developed in these disciplines to see i the SPF probability orecasts have any value and then eplore ways o reading these orecasts or monitoring cyclical downturns. The plan o this paper is as ollows: In the net sections, we will introduce the data, eplain the set up, and evaluate the probability orecasts using procedures developed in other disciplines. We will also suggest simple ways to monitor and interpret time series movements in the data in terms o odds ratios and other accuracy score measures appropriate or rare-event orecasting. Finally, concluding remarks will be summarized. The SPF Forecasts and the Joint Probability Predictor Thanks to the ingenuity o Victor Zarnowitz, one o the world's leading scholars on business cycles, indicators, and orecast evaluation, the Survey o Proessional Forecasters (SPF) 5 has been collecting subjective probability orecasts o real GDP/GNP declines during the current and our subsequent quarters since its inception in At the end o the irst month o each quarter, the individual orecasters in SPF orm their orecasts. The survey collects probability assessments or a decline in real GDP in the current quarter, in the net quarter conditional on the growth in the current period, and so on. The number o respondents has varied between 15 and 6 over the quarters. In this study, we use probabilities averaged over individuals. The joint probability o GDP 3 The New York Times columnist David Leonhardt (September 1, ) calls the one-quarter-ahead GDP decline probability the Anious Inde. 4 Notable eceptions include Braun and Yaniv (199), Wang (1993), Graham (1996), and Stock and Watson (3). However, these studies emphasized dierent aspects o the probability orecasts. Baghestani (5) suggests a way o improving interest rate orecasts available in SPF. 5 Formerly the surveys were done under the auspices o American Statistical Association and National Bureau o Economic Research (ASA-NBER). Since 199, the Federal Reserve Bank o Philadelphia has conducted the survey. See Croushore (1993) or an introduction to SPF. 6 The deinition o real output in the survey has changed rom real GNP to real GDP since 199:1. Subjective Probability Forecasts or Recessions Business Economics April 6 7

3 declining in both the current and in the net quarter can be obtained by multiplying the two probabilities using Bayes rule o conditional probability. Note that the product o the current and irst quarter orecasts is eectively a two-quarter ahead orecast o real GDP. Even though two consecutive quarters o negative GDP growth is a popular deinition o a recession, the NBER deines recession using a number o monthly indicators, and thus the two do not match eactly. Using the July revisions o the real-time GDP growth, during our sample period there were si episodes o negative GDP growth in two or more consecutive quarters those beginning 1969:4, 1974:1, 1981:4, 198:3, 199:4 and 1:1. These si separate episodes o two or more consecutive quarters o real GDP declines match with ive o the si NBER-deined U.S. recessions over this period. The 198 NBER recession ehibited negative real GDP growth only in one quarter, and hence did not match with our deinition o two consecutive quarters o negative real GDP growth as a cyclical downturn. Otherwise, the two deinitions are very close. It is now well accepted that the currently available (benchmark) revised data should not be used in evaluating orecasts. (Diebold and Rudebusch, 1991). We used the annualized real-time, real GDP growth issued every July as the orecasting target in this analysis, against which the orecasting perormance o the proposed predictor will be evaluated. We also considered the 3-day FIGURE 1 JOINT PROBABILITY OF TWO CONSECUTIVE (Q-Q1) GDP DECLINES: Q Q 4 Joint Probability (%) Real GDP Growth (%) preliminary announcements as the target variable. Ecept or the substantial revisions during the last recession, these two data vintages do not make much dierence so ar as SPF probabilities are concerned. The SPF recession probability and the real time real GDP growth are depicted in Figure 1. The shaded bars represent the NBERdeined recessions. We ind that the joint probabilities rise sharply and contemporaneously during quarters with negative real GDP growths. Validity Tests or Recession Probability Forecasts Consider the joint distribution o the probability orecasts and observations p(, ), where is the probability orecast, and the negative GDP growth indicator. A conventional way to evaluate probability orecasts is to calculate the mean square error (MSE), or hal Brier s T 1 QPS = ( t ) t where and are the orecasts and the (, T t = 1 1) event variable, respectively. The QPS varies between and 1, with implying perect orecasts. In our case, the QPS was calculated to be.71 suggesting impressive accuracy. However, a more meaningul measure o perormance is the skill score (SS), which measures the QPS accuracy relative to a chosen benchmark. We calculated 8 Business Economics April 6 Subjective Probability Forecasts or Recessions (1) NBER Recession Joint Probability Real Time Real GDP Growth SS(, ) = 1 [ MSE(, ) / MSE( µ, )] to be.95, where MSE (µ, ) is the accuracy associated with the constant base rate prediction at our sample average value µ =.11. The base rate is deined as the proportion o quarters with two or more consecutive quarters o negative real GDP growth in our sample (i.e., 16/143). It may be noted that using the historical average as the base rate presumes substantial knowledge on the part o the orecaster and is more demanding than the use o no-change orecast in our sample. It is interesting that the average orecast probability o a recession in our sample (µ ) was.7 which is considerably less than µ =.11, suggesting under-conidence. In contrast, the rare events like earthquakes or snowstorms, are typically over-predicted (Murphy, 1991). Thus, the cost/loss structure in recession orecasting must be quite dierent rom that o weather orecasting. Note that MSE (µ, ) = σ =.1 and σ =.1 in our sample.

4 Murphy Decomposition It is widely recognized that a single measure like QPS is grossly inadequate or evaluating the goodness o probability orecasts, particularly o rare events (Lahiri and Wang, 1994). 7 There are several eatures that characterize good probability orecasts. Murphy (197) decomposed the MSE or the Brier score into three components () (, ) = σ + E ( µ ) E ( µ MSE µ ) The irst term o the right-hand-side (RHS) o equation () is the variance o the observations and can be interpreted as the MSE o constant orecasts equal to the base rate. It represents orecast diiculty. The second term on the RHS o equation () is the calibration or reliability o the orecasts, which measures the dierence between the conditional mean o the occurrence in a probability group and the orecast probability. The third term on the RHS o equation () is a measure o the resolution or discrimination that requires signiicant subtleties in interpretation (Yates, 1994). In general, resolution implies that it is desirable or the relative requency o occurrence o an event to be larger (smaller) than the unconditional relative requency o occurrence when is larger (smaller). Even though calibration is a natural eature to have, it is resolution that makes the orecasts useul in practice. Thereore, equation () can be written as: (3) Accuracy o the orecasts = Uncertainty + Calibration - Resolution The calibration and resolution reer to two distinct attributes o the orecast. A sequence o probability orecasts is said to be perectly calibrated i, or all orecast values, the relative requency o occurrence o the event or these observations associated with a particular orecast probability, p( = 1 ), is equal to that probability value. The magnitude o any dierence between the orecast probability and the requency o the occurrence would indicate the degree o miscalibration. For perectly calibrated orecasts, µ = and µ = µ, and the resolution term equals the variance o the orecasts, σ. Resolution or discrimination (or sharpness) reers to the marginal or predictive distribution o the orecasts p(). A sample o probability orecasts is said to be completely resolved i the probability only takes values zero and one. Thus, completely reined orecasts would be miscalibrated due to the inability o the orecasters to predict the uture with certainty. 7 Diebold and Rudebusch (1989) introduced this measure and the Murphy decomposition in economics. Conversely, well-calibrated probability orecasts generally ehibit only a moderate degree o reinement. Thus, a possible trade-o between the calibration and resolution eists to minimize MSE. Forecasts possess positive skill when resolution reward eceeds the miscalibration penalty. The distributions o p( ) and p() are depicted in Figures and 3. In Figure, µ is plotted against, and this is reerred to as the attributes diagram. The calculations are eplained in Table 1. In contrast, Figure 3 depicts the marginal or predictive distribution o the orecasts p(), in which p() is plotted against. Figure indicates the relationship between µ and or the relevant sample o the orecasts and observations and also contains several reerence or benchmark lines. The straight 45 line or which µ = represents perectly calibrated orecasts. The horizontal line represents completely unresolved orecasts, or which µ = µ or all F. The dotted line equidistant between the 45 line and the horizontal lines represents orecasts o zero skill in terms o SS. To the right (let) o the vertical auiliary line at = µ and above (below) the zero-skill line, skill is positive; and skill is negative below (above) it. Thereore, Figure permits qualitative evaluation o resolution and skill as well as TABLE 1 CALIBRATION CALCULATIONS N = number o χ = number o µ = relative = probability the observations realizations requency Total calibration or individual values o F. An eamination o the empirical curves in Figure indicates that the joint probability orecast is generally well calibrated. Most points on the empirical curves all in regions o positive skill. In Figure 3, the distribution o p() indicates that low probability values or probability values near the historical base rate value (.11) are used Subjective Probability Forecasts or Recessions Business Economics April 6 9

5 FIGURE ATTRIBUTES DIAGRAM Observed Relative Frequency Forecast Probability µ No Resolution Perect Cailbration Zero SS FIGURE 3 PREDICTIVE DISTRIBUTION Relative Frequency [p()] 1% 9% 8% 7% 6% P() 5% 4% 3% % 1% % Forecast Probability () much oten than high probability values. There is some accumulation o probability mass at point.5 that is associated with negative real GDP growth. In Figure 4 this graph is split into two conditional likelihood distributions given = 1 (recession) and = (no recession). For these two conditional distributions, the means and variances were calculated to be (.4,.1) or = and (.9,.4) or = 1, respectively. Good discriminatory orecasts will give two largely nonoverlapping marginal distributions; and, in general, their vertical dierences will be large. Due to the overuse o low probabilities during regime = 1, the two lines overlap. However, the vertical dierence at each probability value and the dierence in their means (.4 vs..9) are suggestive o reasonable sharpness. Note that the dierence between two conditional means, the so-called orecast slope, relects the orecaster s ability to respond properly to cues that are predictive o the target event and to ignore cues that are not predictive o that event. It approaches one in a strong predictive situation. Numerical values o the three components in the Murphy decomposition o QPS in equation (3) were ound to be.1,.13 and.4 respectively. We ind that the overall orecast perormance, as measured by MSE, is improved by about 3 percent over the constant relative requency orecast (CRFF) (rom.1 to.71). The major contributor or the improvement in MSE is resolution, which helps reduce the baseline MSE (CRFF) by about 4 percent. On the other hand, the miscalibration increases CRFF by 1 percent. As indicated by the attributes igure (Figure ) and the overlapping o the p( = 1) distribution over p( = ) (Figure 4), the SPF probabilities are conservative in assigning high probability to the quarters when recession occurs. This also suggests 3 Business Economics April 6 Subjective Probability Forecasts or Recessions

6 that distinguishing between occurrences and non-occurrences, and assigning higher probabilities to the quarters when recession occurs, can possibly improve the resolution o the orecasts. It may be noted that the assignment o lower probability or rare events is not unusual. It is quite common in weather orecasting. When the diagnostic inormation or cue is not adequate enough to make the orecast, the tendency or the orecaster is to assign a low base-rate probability. Yates Decomposition In a series o inluential papers, Yates (198) and Yates and Curley (1985) showed that calibration and resolution components in the Murphy decomposition are not independent o each other and suggested a covariance decomposition o MSE that can shed additional light on the characteristics o probability orecasts. It is written as: (4) LIKELIHOOD DIAGRAM Likelihood [P( )] 1% 9% 8% 7% 6% 5% 4% 3% % 1% % MSE Since the variance o is µ (1- µ), (4) can be transormed into a more revealing decomposition: (5) MSE FIGURE Forecast Probability () (, ) = σ + σ + ( µ µ ) σ, P( =) (, ) = µ (1 µ ) + σ + σ,min + ( µ µ ) σ,.9 1 reerence or the interpretation o MSE. It can be shown that the Brier s score is ully determined by this term when an unskilled orecaster makes a constant orecast by setting the constant probability to the relative requency o the outcome (i.e., the base rate). It is also an important reerence point when comparing dierent orecasters perormance, because it indicates the degree o the diiculties o the target being orecasted. The conditional minimum orecast variance σ,min relects the double role that the variance o the orecast plays in orecasting perormance. On the one hand, minimized σ will help reduce the MSE; on the other hand, minimized orecast variance can be achieved only when the constant orecast is oered. The constant orecast leads to zero covariance o the orecast and event, which would increase the MSE. So the solution is to minimize the orecast variance given the covariance. This strategy demonstrates the undamental orecast ability o the orecasters. The conditional minimum value o orecast variance is achieved when the orecaster has perect oresight such that he or she could ehibit perect discrimination o the instances in which the event does and does not occur. Since σ = σ - σ,min, the term may be considered as the ecess variability in the orecasts. I the covariance indicates how responsive the orecaster is to inormation related to event s occurrence, σ might reasonably be taken as a relection o how responsive the orecaster is to inormation that is not related to event s occurrence. Another variation o Yates decomposition is as ollows: P( =1) (6) MSE (, ) = µ (1 µ ) + Scat + σ, min + ( µ µ ) σ, where Scat = ( N σ + N σ 1 = 1 = ) / N, N 1, i =,1 is the number o the periods associated with the occurrence (i =1) and non-occurrence (i = ), N 1 +N = N. So the term is the weighted mean o the conditional orecast variances. Using the SPF probability orecast data, the components in equation (5) were computed and are presented below in parentheses as: where, σ ( µ µ ) µ (1 ), σ, min = = 1 = µ The outcome inde variance σ provides a benchmark = σ σ,min (7) MSE (, ) = µ (1 µ ) + σ + σ,min + ( µ µ ) σ, (.71) = (.1) + (.1) + (.6) + (.) - (.49) Bias µ µ =.4 µ µ. 48 = = 1 = = Subjective Probability Forecasts or Recessions Business Economics April 6 31

7 The overall MSE value.71, which is less than constant relative requency orecast variance.1, demonstrates the skillulness o the SPF joint probability orecasts. The primary contributor to the perormance is the covariance term that helps reduce the orecast variance by almost 5 percent. The covariance relects the orecaster s ability to make distinctions between individual occasions in which the event might or might not occur. It assesses the sensitivity o the orecaster to speciic cues that are indicative o what will happen in the uture. It also shows whether that cue responsiveness is oriented in the proper direction. As noted beore, the orecasts ehibit some degree o overall bias (under conidence) as evidenced by µ µ =.4. The conditional minimum orecast variance σ,min is.6. Compared to the overall orecast variance.18, the observed variability o SPF probability orecasts is about three times the variability that is necessary, given the dierence in conditional means µ = 1 µ = =.48. This means the subjective probabilities are scattered unnecessarily around µ = 1 and µ =. The Yates decomposition shows that the primary reason or an ecess MSE over the CRFF variance is the variance o the orecasts. As seen above, the orecast variance.18 is about three times greater than what is necessary as measured by the minimum orecast variance σ,min =.6. The ecess variation, as measured by Var() =.1 increases the CRFF variance by about 1 percent. The choice o relatively low probabilities when the event actually occurred seems to be the root cause o the inlated MSE. During 5 percent o the quarters (8 out o 16) when GDP growth was negative, the probabilities assigned were below percent. In contrast, 9 percent o the quarters (117 out o 17) when GDP growth did not go down, the probabilities assigned were correctly below 15 percent. That eplains why Var( = 1) is much bigger, about our times, than Var( = ). Overall, both the Murphy and Yates decompositions support the useulness o the SPF probability as a predictor o the two consecutive quarters o negative real GDP growth and suggest ways o improving the orecasts. The probabilities embody eective inormation related to the occurrence o the event, and the overall average orecast probabilities are close to the relative requency o the occurrence o the event. However, improvement can be made by urther distinguishing actors related to the occurrence o recessions, while keeping the sensitivity o the orecasts to inormation that is actually related to the occurrence o GDP declines. This would imply a reduction o unnecessary variance o orecasts, particularly during GDP declines, thereby increasing resolution urther. Ogata s AIC Dierence as Skill Score Similar to the concept o skill score that measures the improvement o QPS associated with a particular set o orecasts over average base rate orecasts, Ogata (1995) developed a orecast improvement measure using the Akaike Inormation Criterion (AIC). For the constant orecast ( = µ ), (8) AIC and or the SPF orecast ( i ), (9) AIC AIC measures how close the orecast is to the occurrence o the event, so the orecast with smaller AIC is considered to be a better it. The dierence AIC = AIC 1 -AIC measures the quality o the SPF orecast perormance over the base rate orecasts with (1) where n = ( ) { log + (1 )log(1 )} i i i= 1 n = ( ) { i log + (1 ) log(1 )}. 1 i i i i = 1 AIC = ( ) n Q i i= 1 (11) Qi = i log( i / ) + (1 i )log{(1 i) /(1 )} indicates the size o gain or loss o the SPF probability orecast against the constant relative requency orecast or each i. Over our sample, it was calculated to be AIC = AIC 1 - AIC = = showing that the SPF probability orecasts improve the orecast quality over the constant relative requency orecast in a signiicant way as evidenced by reducing the AIC by over 39 percent. This is very similar to what we ound earlier using the conventional QPS-based skill score and has not been used in economics beore. How to Use and Interpret the Probability Forecasts One important issue or the users o the probability orecasts o rare business events, such as recessions or stock market crashes, is how to use and interpret the probabilities. Given the inrequent nature o the event and the lack o strong diagnostic inormation or cues, the probabilities assigned to these events are seldom very high. Psychologists have shown that individuals have a propensity to bias their estimated probabilities towards an anchor, the base rate in this case, particularly when they ace diicult orecast situations. That is, in diicult ore- 3 Business Economics April 6 Subjective Probability Forecasts or Recessions

8 cast situations, individuals do not adjust enough to new inormation, making the value o the anchor very critical. This is the theory o anchoring and adjustment due to Kahnemann and Tversky (1973). Whatever may be the reason, some users may tend to ignore such relatively modest probability values. Murphy (1991) suggested using odds in addition to the probabilities to communicate the probability orecasts to the users. For eample, i the relative requency o a recession during the observed sample is.13, and the probability orecast or the occurrence is.39, the occurrence o the event is less likely than its nonoccurrence or this occasion. However, it is considerably more likely this period than its relative requency. Speciically, it is three times more likely than the average base rate. The odds or risk o an event is the ratio o the probability that the event occurs to the probability that the event does not occur. Thus, a recession in this case has an odds o.39/ (1-.39) =.64 (or.6 to 1 on/or in bookmaker s language). One interesting property o odds is that the odds or the complement o the event is the reciprocal o the odds or the event. Forecast inormation imbedded in subjective orecasts can be judged by comparing the odds o a recession using SPF probabilities to the odds o a recession corresponding to the base rate. Base rate odds (BO) in avor o the event in this case is BO = µ /(1 µ ) =.13/(1-.13) =.15. The orecast odds (FO) in avor o the event is FO = /(1-) =.39/(1-.39) =.64. The odds ratio (OR) in avor o the event is OR = FO/BO =.64/.15 = 4.8. Thus, the orecast odds in avor o the event is more than our times greater than the base rate odds. According to Murphy (1991), looking at these odd ratios may overcome the problem o relatively modest orecast probabilities, and enable users to identiy those occasions where the risk o the rare event is suiciently high to warrant taking appropriate precautionary measures. We plotted the odds ratios (OR) against real GDP growth over in Figure 5. As epected, during the cyclical downturns, OR sharply increases and gives useul signals. By studying its historical behavior beore two-consecutive GDP declines, we ound that a threshold value in ecess o 1.5 suggests a cyclical downturn until it FIGURE 5 ODD RATIO OF JOINT PROBABILITY Q Q 4 Odd Ratio Real GDP Growth (%) NBER Recession Odd Ratio Real Time Real GDP Growth comes down below 1.4. Using these thresholds, we ound that OR provided timely and dependable warnings or negative GDP growth periods over the sample. Using real time GDP igures (July revisions), OR identiied ive o the si episodes o two or more consecutive quarters o negative real GDP growth in our sample period those beginning: 1969:4, 1974:1, 1981:4, 198:3, and 199:4; o these, on two occasions (1981:4 and 198:3) OR had a lead o one quarter, on one occasion (1969:4) it lagged the onset by one quarter and on the other two occasions (1974:1 and 199:4), OR gave signals coincidentally. It should be pointed out that one-quarter lag or the twoquarter-recession beginning 1969:4 means the signal came in the middle o the episode in January 197. Apparently, SPF could not oresee the 1 recession; the signal came with a lag in 1:4 when the negative growth period had already passed (Stock and Watson, 3). Thus, using the July-revised real time GDP data, SPF not only missed the 1 recession, but the lagged signal has to be considered as a alse signal. There are two additional alse signals, one beginning 198:1 and 1975:. The 198:1 alse signal can be eplained by the act that 198:1-198:3 is a NBER deined recession even though it was not characterized by a two-consecutive quarterly all in real GDP. We will see that i we use the real time GDP igures based on the initial 3-day announcements as the true SPF target, then the alse Subjective Probability Forecasts or Recessions Business Economics April 6 33

9 TABLE PROBABILITIES OF A QUARTERLY DECLINE IN REAL GDP FROM THE SURVEY OF PROFESSIONAL FORECASTERS Target Date Forecasts Made In Quarter Actual Growth 1 3-day Preliminary Revised Q3 Q4 Q1 Q Q3 Q4 Q % 4% 1 Q % 11% 37% 1 Q % 17% 3% 3% 1 Q % 19% 3% 9% 35% 1 Q % 18% 3% 6% 8% Q % 18% % 49% Q % 16% 7% Q % 18% Note: Forecast entries are the probability that real GDP growth will be negative, averaged across SPF orecasters. The orecasted probability that growth will be negative in the quarter ater the orecast is made (that is, the one-quarter-ahead orecast) appears in bold. Table is adapted rom Stock and Watson (3). alarm o 1975: and the missed signal o the latest recession will disappear. During the ive-quarter recession beginning 1974:1, SPF continued to give a high recession probability even in 1975: when real GDP was no longer negative according to the July-revised real time data. But according to the 3-day preliminary data, the growth was negative in 1975:. 8 I the SPF orecasters were predicting the 3-day revised GDP growth data, the probabilities and the outcomes would match better with these than those with Julyrevised growth data. There is direct evidence that SPF respondents target the growth rates based on the initial 3-day GDP announcements. In terms o QPS scores, the SPF probabilities o negative GDP growth during net si months eplains the real GDP declines better when the target is deined in terms o the 3-day real time data (QPS =.56) compared to those based on July revisions (QPS =.71). This result is not entirely unepected in view o the act that when SPF respondents orm their orecasts, they have the 3-day announcements as the most recent available inormation on GDP. Also, in real time, the 3-day GDP announcements are possibly more important to actual market analysts than the revised data or the NBER recession chronologies that are more academic in nature. In Table, we have reproduced Table o Stock and Watson (3) where they argued that SPF recession probabilities missed the last recession. Their GDP growth 8 Graham (1996) noted that momentum ollowing, that is, repeating the same orecast in a number o quarters consecutively, was not a problem with SPF data over shorter horizons. TABLE 3 CONTINGENCY TABLE Forecast/ Occur Not Occur Total Event Yes X = 1 Z = 8 X+Z = No Y = 4 W = 119 Y+W = 13 Total X+Y = 16 Z+W = 17 X+Y+Z+W = 143 igures were the revised data as o February 8, 3. These were, however, very similar to our July revisions. We have augmented their table with real GDP growth data that were available in the real time one month ater the end o the quarter (i.e., the 3-day announcements). As is well known, and can be seen rom Table, these two actual quarterly growth rate series are remarkably dierent during the last recession in that there was only one quarter (1:3) during :4-:3 in which the GDP growth was negative i we ollow the 3-day preliminary data. However, the subsequent revisions showed that during the three-quarter period 1:1-1:3. GDP growth was negative. Thus, i we use the 3-day preliminary real time data, we would conclude that SPF orecasters were correct in not issuing high probability assessments or a recession. However, the anemic growth during the period was signaled as the probability jumped rom around ten percent in :4 to 37 percent in 1:1 and stayed that way throughout 1. The high current quarter probability o 8 percent in 1:4 (and hence a alse signal) can be eplained by the act that the survey orecasters were 34 Business Economics April 6 Subjective Probability Forecasts or Recessions

10 responding within a month o the 9/11 attack, and the high etraordinary GDP growth during the :1 can be eplained by the eect o 9/11 during the last quarter o 1. Admittedly, catastrophic events like 9/11 can adversely aect judgmental orecasts, and model-based orecasts could do better in such situations. Based on the OR values and the thresholds, we constructed a contingency table to study the predictive content o the OR. Counts o the correct classiications, the misses, and the alse alarms are presented in Table 3 with notations. The most common summary measure o veriication skill using contingency tables is the so-called Kuipers Perormance Inde (Granger and Pesaran, ). This is a requently used measure o skill that is obtained by taking the dierence between the hit rate and the alse alarm rate, and can be computed rom the contingency table as: (1) Kuipers ' score = ( w yz) /[( + y)( z +. w)] However, in evaluating rare events where one epects the dominance o the occurrence o non-events, this inde is subject to hedging behavior, (i.e., deviating rom the orecaster s true belies in order to increase the veriication score). Doswell, et al. (199) argued that Kuiper s Inde is an improper scoring rule or rare event orecasting. It should be emphasized that or rare events like recessions, the orecast value comes rom correctly orecasting the rare events and not the nonevents. Following Doswell, et al. (199), we use the Heidke skill score that is deined as the proportion o correct classiications compared to that obtained under no-skill random orecasts and can be easily calculated using numbers rom the contingency table as (13) Heidke Skill Score = ( w yz) /[ y + z + w + ( y + z)( + w)] It also ranges rom 1 to +1. Stephenson () has suggested a skill score based on a comparison o odds o making a good orecast (a hit) to the odds o making a bad orecast (a alse alarm). In other words, he suggests using the odds ratio θ = (H/1-H)/ (F/1-F) where H = /(+y), the hit rate, and F = z/ (z + w), the alse alarm rate. This statistic has a long history in medical diagnosis. From the contingency table, this was easily calculated as θ = w/yz = A simple skill score ranging rom 1 to +1 can be obtained rom the odds ratio θ by the transormation (14) Odds Ratio Skill Score (ORSS) = (θ 1)/ (θ + 1) Based on data in Table 3, the Kuipers, Heidke s and Odds Ratio skill scores were calculated to be.687,.619, and.956 respectively. All these values suggest impressive orecast skill. Kuipers score seems to be slightly inlated compared to Heidke score. ORSS value shows that the OR-based predictions have ecellent skill score. Stephenson () also suggested the use o the odds ratio to test the statistical signiicance o the skill. It is also well known that the log (odds ratio) = (log + log w log z log y) is approimately normally distributed with standard error 1/(n j ) 1/ where 1/n j = (1/)+(1/z)+(1/y)+(1/w). Based on Table 3, the log odds ratio was calculated as with standard error.683. Thus the value is more than 1.96 standard errors away rom zero implying that there is less than ive percent chance the positive skill ound using the odds ratio approach could be due to pure chance. Conclusions In this paper we have evaluated the subjective probability orecasts or real GDP declines during The Survey o Proessional Forecasters record probability orecasts or real GDP declines during the current and net our quarters. Using the current and the one-quarterahead orecasts we generated orecasts or GDP declines during the net two quarters. By using orecast evaluation methodologies developed in meteorology, psychology, and other disciplines, we studied the quality o these probability orecasts in terms o calibration, resolution, and alternative variance decompositions. We ound conclusive evidence that these orecasts possess signiicant skill and are acceptably calibrated and resolved. These results are similar in sprit to those ound in Graham (1996), even though Graham s study was based on a small subset o the complete data set. We also ound evidence that the SPF targets the initial 3-day preliminary GDP growth igures, and not the subsequently revised igures. The variance o the orecasts, particularly during cyclical downturns, was ound to be three times more than necessary. This result implies that orecasters are responsive to cues or predictors that are not related to the occurrence o negative GDP growth. Thus, orecast improvement is possible by urther distinguishing actors related to the event rom those that are not, while keeping the sensitivity o the orecasts to correct inormation. However, this may not be an easy task in practice. Similar to the record o subjective orecasts o rare events in other disciplines, the recession probabilities seldom rose very high and were muted. Following recent climatological literature, we used odds ratios to identiy signals in these orecasts compared to base line orecasts. Based on historical data, we ound that an odds ratio in Subjective Probability Forecasts or Recessions Business Economics April 6 35

11 ecess o 1.5 signals two consecutive GDP declines quite successully. By using skill score measures that are appropriate or rare event orecast evaluation, we ound that the odd ratios have statistically signiicant orecasting power. It may be noted that over-conidence in judgment is requently reported in the meteorology and psychology literature o probability orecasts o rare events. This reminds one o a dictum o the 17th century French moralist La Rocheoucauld, Everybody complains about their memory but no-one complains about their judgment. Our results, however, indicate that the recession probability orecasts are under-orecasted and lack conidence. The mean orecast probability or two consecutive declines in real GDP was only.73, which is about 35 percent less than the relative requency o the occurrence o the event, which was.1119 in our sample. This under-orecasting may be a result o a dierent loss unction in the business world compared with that in meteorology and psychology. In meteorology, rare events such as tsunamis or earthquakes occur suddenly and pass very quickly, but the damage they leave behind can be tremendous. Thereore, the cost o the missing target would be much higher than that o alse signals. By contrast, a rare business event such as a recession occurs more gradually and lasts longer. Also, the eects or the course o a recession can be negated or changed by government policies. Thus, the cost o missing the target (and not reducing shipment orders, work orce, etc. immediately) could be much less than the cost o a alse signal (dislocations due to mistakenly laying o workers, reducing orders when it is unnecessary, etc.). So the basic costs/incentives patterns seem to be dierent in business situations rom predicting natural disasters. The decomposition methodologies introduced in this paper have much broader application in evaluating model it in Logit, Probit and other limited dependent variable models. These models generate probabilities o discrete events as model predictions. Again, oten in economics, we try to identiy events that are relatively rare. Usually the model it criteria based on maimized likelihood look ecellent, but the estimated model hardly identiies the small but special population o interest. By using the evaluation methodology o probability orecasts analyzed in this paper, one can study the true value o many estimated limited dependent variable models or out-o-sample predictive purposes. The application o the concept o the odds ratio to recession probabilities provides a simple but powerul monitoring scheme or impending recessions. Considering the act that the chronologies o NBER recessions are usually determined long ater the recession is over, negative GDP growth in two consecutive quarters is probably a more realistic and practical metric or tracking business cycles in real time. We have ound conclusive evidence that SPF subjective probability orecasts are useul in this regard. ACKNOWLEDGEMENT We are grateul to Richard Cohen, Nigel Harvey, Thad Mirer, Herman Stekler, Kenneth Wallis, Arnold Zellner, two anonymous reerees, and the editor or many helpul comments and suggestions. However, we are solely responsible or any remaining errors and omissions. REFERENCES Baghestani, H. 5 Improving the Accuracy o Recent Survey Forecasts o the T-bill Rate. Business Economics. 4:, pp Bangia, A., F. X. Diebold, A. Kronimus, C. Schagen, and T. Schuermann.. Ratings Migration and the Business Cycle, with Application to Credit Portolio Stress Testing. Journal o Banking and Finance. 6, pp Braun, P. and I. Yaniv A Case Study o Epert Judgment: Economists Probabilities versus Base Rate Model Forecasts. Journal o Behavioral Decision Making. 5, pp Carey, M.. A Guide to Choosing Absolute Bank Capital Requirements. Journal o Banking and Finance. 6, pp Croushore, D Introducing: The Survey o Proessional Forecasters. Federal Reserve Bank o Philadelphia Business Review. November/December, pp Diebold, Francis X. and G. D. Rudebusch Scoring the Leading Indicators. Journal o Business. 64, pp Turning Point Prediction with the Composite Leading Inde: An E Ante Analysis, in K. Lahiri and G.H. Moore (eds.), Leading Economic Indicators: New Approaches and Forecasting Records. Cambridge: Cambridge University Press, pp Doswell, C. A., R. Davies-Jones, and D. L. Keller On Summary Measures o Skill in Rare Event Forecasting Based on Contingency Tables. Weather and Forecasting. 5, pp Filardo, A. J How Reliable Are Recession Prediction Models? Federal Reserve Bank o Kansas City Economic Review, pp The 1 US Recession: What Did the Recession Prediction Models Tell Us? Bank o International Settlements, BIS Working Paper. No 148, March. Fintzen, D. and H. O. Stekler Why Did the Forecasting Fail to Predict the 199 Recession? International Journal o Forecasting. 15, pp Graham, H. R Is a Group o Forecasters Better Than One? Than None? Journal o Business. 69:, pp Granger, C. W. and M. H. Pesaran.. Economic and Statistical Measures o Forecast Accuracy. Journal o Forecasting. 19, pp Hamilton, J. D A New Approach to the Economic Analysis o Nonstationary Time Series and the Business Cycle. Econometrica. 57, pp Juhn, G. and P. Loungani.. Further Cross-Country Evidence on the Accuracy o the Private Sector Output Forecasts. IMF Sta Papers. 49, pp Kahnemann, D. and A. Tversky On the Psychology o 36 Business Economics April 6 Subjective Probability Forecasts or Recessions

12 Prediction. Psychological Review. 8, pp Lahiri, K and J. G. Wang Predicting Cyclical Turning Points with Leading Inde in a Markov Switching Model. Journal o Forecasting. pp Murphy, A Scalar and Vector Partitions o the Probability Score: Part I. Two-state Situation. Journal o Applied Meteorology. 11, pp Murphy, A. H Probabilities, Odds, and Forecasters o Rare Events. Weather and Forecasting. 6, pp Ogata, Y Evaluation o Probability Forecasts o Events. International Journal o Forecasting. 11, pp Ogata, Y., T. Utsu, and K. Katsura Statistical Features o Foreshocks in Comparison with Other Earthquake Clusters. Geophysical Journal International. 11, pp Stephenson, D.B.. Use o the Odds Ratio or Diagnosing Forecast Skill. Weather and Forecasting. 15, pp Stock, J. H. and M. W. Watson A Probability Model o the Coincident Economic Indicators in K. Lahiri and G.H. Moore (eds.), Leading Economic Indicators: New Approaches and Forecasting Records. Cambridge: Cambridge University Press, pp A Procedure or Predicting Recessions with Leading Indicators: Econometric Issues and Recent Eperience. in J.H. Stock and M.W. Watson (eds.), New Research on Business Cycles, Indicators, and Forecasting, Chicago: University o Chicago Press, pp How Did Leading Indicator Forecasts Perorm During the 1 Recession? Federal Reserve Bank o Richmond Economic Quarterly. 89: 3, pp Wang, J. G On the Use o Markov Regime-Switching Model in Forecasting Business Cycles. Unpublished PhD dissertation, University at Albany-SUNY. Yates, J. F Eternal Correspondence: Decompositions o the Mean Probability Score. Organizational Behavior and Human Perormance. 3, pp Subjective Probability Accuracy Analysis, in G. Wright and P. Ayton (eds.), Subjective Probability. Chichester, UK: John Wiley, pp Yates, J. F. and S. P. Curley Conditional Distribution Analysis o Probabilistic Forecasts. Journal o Forecasting. 4, pp Zellner, A., C. Hong, and C-K. Min Forecasting Turning Points in International Growth Rates Using Bayesian Eponentially Weighted Autoregression, Time Varying Parameter, and Pooling Techniques. Journal o Econometrics. 49, pp Subjective Probability Forecasts or Recessions Business Economics April 6 37