USING OUT-OF-SAMPLE MEAN SQUARED PREDICTION ERRORS TO TEST THE MARTINGALE DIFFERENCE HYPOTHESIS. Todd E. Clark Federal Reserve Bank of Kansas City

Transcription

1 USING OUT-OF-SAMPLE MEAN SQUARED PREDICTION ERRORS TO TEST THE MARTINGALE DIFFERENCE HYPOTHESIS Todd E. Clark Federal Reserve Bank of Kansas City Kenneth D. West University of Wisconsin

2 Example: predicting monthly changes in Swiss franc/dollar exchange rate Null model: random walk (predicts value of 0 for next month s change in the exchange rate) Alternative model: use interest differential to predict y t+1 = % change in franc/dollar rate in month t+1 i t -i * t = U.S. interest rate - Swiss interest rate, monthly Predict using OLS estimates of y t+1 = b 0 +b(i t -i * t ) + e t+1

3 To estimate y t+1 = b 0 +b(i t -i * t ) + e t+1, we use 10 year rolling regressions ( R = 120 months), with the first sample beginning in 1975: estimate 1975:1-1984:12, predict 1985:1 estimate 1975:2-1985:1, predict 1985:2... estimate 1993: :9, predict 2003:10 So predictions run from 1985:1-2003:10 (number of predictions P = 226) Figure (plot of ^b t, t=1984: :9)

4 Mean squared prediction error (MSPE) from random walk model : 9 t = 1985:1 y2 t+1 /226 = MSPE from interest parity model = G 2 t 003: 9 = 198 5:1 [y t+1 - ^b 0t - ^b t (i t -i * t )]2 /226 = As in many earlier studies, random walk model has smaller MSPE (i.e., 12.27<12.33). We argue: one can nonetheless reject at the 5% level the null the random walk model predicts as well as the interest parity model by a MSPE criterion, against the alternative that the interest parity model predicts better.

5 Intuition: If the random walk model is right, the alternative model (the interest parity model) introduces noise into the forecasting process: the alternative model attempts to estimate parameters b 0 and b that are zero in population. In finite samples, use of the noisy estimate of the parameter will raise the estimated MSPE of the interest parity model relative to the random walk model. So if the random walk model is right, the random walk MSPE should not only be smaller but be smaller by the amount of estimation noise.

6 That noise is estimable (details below), and one can and should adjust MSPE comparisons for the estimated noise. In this example, the estimate happens to be So ^F 1 2 = MSPE for random walk = ^ F2 2 = MSPE for interest parity = adj. = adjustment for estimation noise = 0.96 ^ F2 2 -adj. = MSPE for interest parity, adjusted for estimation noise = = Point estimate: 11.37<12.27, i.e., adjusted MSPE for interest parity model is less than that of random walk model One can construct in familiar fashion a standard error for a test of H 0 : After adjustment for estimation noise, MSPE for interest parity = MSPE for random walk (Diebold and Mariano (1995), West (1996))

7 Previous literature on MSPE comparisons for nested models: West (1996) notes specifically that his procedures maintain a rank condition that is not satisfied when models are nested; a similar rank condition is implicit in Diebold and Mariano (1995) McCracken (2000) and Clark and McCracken (2001, 2003) present analytical and simulation evidence to argue vigorously that standard critical values should not be used when models are nested Some applied papers therefore use simulations to get critical values (e.g., Mark (1995), Kilian and Taylor (2003)) Other applied papers use standard critical values, sometimes with apology (Clarida et al. (2003), Cheung et al. (2003)) by construction, would not reject null in Swiss franc example, since MSPE for random walk < (unadjusted) MSPE for interest parity

8 Some theoretical papers have proposed alternative tests for nested models Chao, Corradi and Swanson (2003) propose an encompassing test Giacomini and White (2003) propose a certain conditional test Our procedure - adjust the alternative model s MSPE for bias from estimation, then compare to the null model using familiar procedures - implicitly transforms the MSPE comparison to an encompassing test, though not the test proposed by Chao et al. (2001)

9 I. Introduction and background II. This paper s procedure III. Simulation evidence IV. Empirical example V Conclusions

10 II. This paper s procedure Specifically, (2.1) y t = e t (model 1: null model), (2.2) y t = X t N$+e t (model 2: alternative model). (2.4) (2.5) ^ F 2 1 / P-1 G T t=t-p+1 y2 t+1 = MSPE from model 1, ^ F 2 2 / P-1 G T t=t-p+1 (y t+1 -X t+1 N ^$ t ) 2 = MSPE from model 2. Since we have y 2 t+1 - (y t+1 -X t+1 N ^$ t ) 2 = 2y t+1 X t+1 N ^$ t - (X t+1 N ^$ t ) 2 (2.6) ^ F ^F 2 2 = 2(P-1 G T t=t-p+1 y t+1 X t+1 N ^$ t ) - [P -1 G T t=t-p+1 (X t+1 N ^$ t ) 2 ].

11 (2.6) ^ F ^F 2 2 = 2(P-1 G T t=t-p+1 y t+1 X t+1 N ^$ t ) - [P -1 G T t=t-p+1 (X t+1 N ^$ t ) 2 ]. ==== 2(P -1 G T t=t-p+1 y t+1 X t+1 N ^$ t ). 0, - [P -1 G T t=t-p+1 (X t+1 N ^$ t ) 2 ] < 0 so under the MDS null we expect ^ F ^F 2 2 <0 or: we expect the MSPE from the MDS model to be less than that from the alternative model. We propose looking not at ^F ^F 2 2 but at ^F ^F 2 2 -adj., where ^ F 2 2 -adj. = ^F [P-1 G T t=t-p+1 (X t+1 N ^$ t ) 2 ].

12 ^ F 2 2 -adj. = ^F [P-1 G T t=t-p+1 (X t+1 N ^$ t ) 2 ] === In the Swiss franc data, ^ F 2 2 = [P -1 G T t=t-p+1 (X t+1 N ^$ t ) 2 ] = 0.96 ^ F 2 2 -adj. = < = ^F 2 1 = MSPE from random walk model

13 Figures 1 and 2

14 Inference (3.3) ^ f t+1 / y 2 t+1 - [(y t+1 -X t+1 N ^$ t ) 2 - (X t+1 N ^$ t ) 2 ]. Then one can compute (3.4) b f = P -1 G T t=t-p+1 Then ^ f t = ^F ^F 2 2 -adj. = ^F ^F [P-1 G T t=t-p+1 (X t+1 N ^$ t ) 2 ]. qp b f - A N(0,V), V consistently estimated by ^V = P -1 G T t=t-p+1 Perform t-test with qp b f / q ^V ^ f t 2.

15 Asymptotics rolling samples used to generate regressions (rolling sample size R=120 in the exchange rate example) P = number of predictions (P=168 in the exchange rate example) Total sample size T+1=R+P As T64, P64 but R stays fixed (Giacomini and White (2003)) versus West and McCracken (1998), McCracken (2000) and Clark and McCracken (2001, 2003): T64, P64, R64. Various mixing and moment conditions (Giacomini and White (2003)); conditional heteroskedasticity allowed; stationarity; parametric linear alternative extensions to allow moment drift and parametric nonlinear alternative are straightforward

16 III. Simulation evidence We examine MSPE-adjusted (our statistic) MSPE: normal MSPE: McCracken (same statistic as MSPE: normal, but different critical values) CCS: a certain encompassing test of Chao, Corradi and Swanson (2001)

17 Two DGPs Both assume alternative model is univariate (X includes a constant and a single stochastic variable x) x is highly persistent (AR(1) with parameter 0.95) y (=presumed martingale difference sequence) much more variable than x

18 DGP 1: intended to capture some central characteristics of monthly exchange rate data little contemporaneous correlation between innovation in x (= interest differential) and innovation in y (=exchange rate) when simulating under alternative, y t+1 = -2x t + shock DGP 2: intended to capture some central characteristics of monthly stock price data high negative correlation between innovation in x (=dividend/price ratio) and innovation in y (=stock return) conditional heteroskedasticity of y in some parameterizations when simulating under alternative, y t+1 =.365x t + shock

19 Summary of simulation results for nominal one-tailed10% tests, one step ahead forecasts MSPE-adjusted has size of 5-11%; across DGPs and various sample sizes, median size is 8% MSPE-normal has size of 0-7%; median size is 0% MSPE-McCracken has size of 2-21%; median size is 9% Chao et al. has size of 10-25%; median size is 13% Size-adjusted power comparable across tests

20 Tables 1-6, Figure 3

21 Unadjusted power (DGP 1, b=-2) (4.6) R=120, P=144 R=120, P=240 MPSE-adjusted MSPE: normal MSPE: McCracken CCS

22 IV. Empirical example Monthly bilateral exchange rates and interest differentials, U.S. vs: Canada, Japan, Switzerland, U.K. Rolling samples of size R=120 months First regression sample is 1975:1-1984:12 for Switzerland and U.K., 1980:1-1989:12 for Canada and Japan Last prediction for all four countries is 2003:12; number of monthly predictions P=166 for Canada and Japan, P = 226 for Switzerland and the U.K.

23 (Unadjusted) MSPE interest differential model is more than that for random walk in Japan, Switzerland, U.K.; interest differential model does beat random walk for Canada Adjusted MSPE for interest differential model is less than that for random walk in all four countries MSPE-adjusted test statistic rejects null in Switzerland and Canada at the 5% level.

24 MSPE- MSPE: adjusted normal ^ F1 2 ^ F2 2 ^ adj. F2 2 -adj ^F2 1 -( ^F 2 2 -adj.) ^F2 1 - ^F 2 2 Canada CCS (0.08) 1.78 ** Japan (0.43) * Swiss (0.48) 1.88 ** U.K (0.33)

25 V. Conclusions When testing the MDS hypothesis, MSPE comparisons should be adjusted for noise from parameter estimation Our procedure is straightforward and works reasonably well

26 Important extensions null model that uses estimated parameter vector to predict recursive (instead of rolling) regressions to generate predictions

27 1 Rolling Coefficient Estimates, Switzerland

28 5.0 Rolling Coefficient Estimates, Switzerland (with 90% confidence band)

29 Figure 1 Density of Simulation MSPEs Under the Null, R=120, P Varying, DGP 1 A. MSPE(1) - MSPE(2) P = 48 P = 144 P = 240 P = B. Average of (X*betahat)^2 P = 48 P = 144 P = 240 P = C. MSPE(1) - [MSPE(2) - average of (X*betahat)^2] P = 48 P = 144 P = 240 P =

30 Figure 2 Density of Simulation MSPEs Under the Null, R Varying, P=144, DGP 1 R = 60 R = 120 R = 240 A. MSPE(1) - MSPE(2) B. Average of (X*betahat)^2 R = 60 R = 120 R = C. MSPE(1) - [MSPE(2) - average of (X*betahat)^2] R = 60 R = 120 R =

31 Figure 3 Density of Simulation MSPE-Adjusted Test Statistic Under the Null, DGP A. R Varying, P= R = 60 R = 120 R = B. R=120, P Varying P = 48 P = 144 P = 240 P = 1200

32 Figure 4 Density of Simulation MSPE Test Statistic (Unadjusted) Under the Null, DGP A. R Varying, P= R = 60 R = 120 R = B. R=120, P Varying P = 48 P = 144 P = 240 P = 1200

33 Figure 1 Density of Simulation MSPEs Under the Null, R=120, P Varying, DGP 1 A. MSPE(1) - MSPE(2) P = 48 P = 144 P = 240 P = B. Average of (X*betahat)^2 P = 48 P = 144 P = 240 P = C. MSPE(1) - [MSPE(2) - average of (X*betahat)^2] P = 48 P = 144 P = 240 P =

34 Figure 2 Density of Simulation MSPEs Under the Null, R Varying, P=144, DGP 1 R = 60 R = 120 R = 240 A. MSPE(1) - MSPE(2) B. Average of (X*betahat)^2 R = 60 R = 120 R = C. MSPE(1) - [MSPE(2) - average of (X*betahat)^2] R = 60 R = 120 R =

35 Figure 3 Density of Simulation MSPE-Adjusted Test Statistic Under the Null, DGP A. R Varying, P= R = 60 R = 120 R = B. R=120, P Varying P = 48 P = 144 P = 240 P = 1200

36 Figure 4 Density of Simulation MSPE Test Statistic (Unadjusted) Under the Null, DGP A. R Varying, P= R = 60 R = 120 R = B. R=120, P Varying P = 48 P = 144 P = 240 P = 1200

37 Table 1 Empirical Size: DGP 1 Nominal Size = 10% A. R=60 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS B. R=120 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS C. R=240 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS Notes: 1. The predictand y t+1 is i.i.d. N(0,1); the alternative model s predictor x t follows an AR(1) with parameter 0.95; data are conditionally homoskedastic. In each simulation, one step ahead forecasts of y t+1 are formed from the martingale difference null and from rolling estimates of a regression of y t on X t = (1,x t-1 )N. 2. R is the size of the rolling regression sample. P is the number of out-of-sample predictions. 3. Our MSPE adjusted statistic, defined in (3.1) and (4.4), uses standard normal critical values. MSPE normal, defined in (4.5), refers to the usual (unadjusted) t-test for equal MSPE, and also uses standard normal critical values. MSPE-McCracken relies on the MSPE-normal statistic but uses the asymptotic critical values of McCracken (2000). 4. The number of simulations is 10,000. The table reports the fraction of simulations in which each test rejected the null using a one-sided test at the 10% level. For example, the figure of.078 in panel A, P=48, MSPE-adjusted, indicates that in 780 of the 10,000 simulations the MSPE-adjusted statistic was greater than 1.65.

38 Table 2 Empirical Size: DGP 2 Nominal Size = 10% A. R=60 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS B. R=120 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS C. R=240 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS Notes: 1. See the notes to Table 1.

39 Table 3 Empirical Size: DGP 2 with Conditional Heteroskedasticity Nominal Size = 10% A. GARCH P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS B. Multiplicative conditional heteroskedasticity P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE-normal MSPE-McCracken CCS Notes: 1. See the notes to Table 1. The regression sample size R is In the upper panel of results, the predictand y t+1 is a GARCH process, with the parameterization given in equation (4.4). In the lower panel, the predictand y t+1 has conditional heteroskedasticity of the form given in equation (4.5), in which the conditional variance at t is a function of x 2 t-1.

40 Table 4 Size-Adjusted Power: DGP 1 Empirical Size = 10% A. R=60 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE CCS B. R=120 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE CCS C. R=240 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE CCS Notes: 1. The DGP is defined in equation 4.3, with: b=-2; e t+1 ~ i.i.d. N(0,1); x t ~ AR(1) with parameter 0.95; data are conditionally homoskedastic. In each simulation, one step ahead forecasts of y t+1 are formed from the martingale difference null and from rolling estimates of a regression of y t on X t = (1,x t-1 )N. 2. In each experiment, power is calculated by comparing the test statistics against simulation critical values, calculated as the 90th percentile of the distributions of the statistics in the corresponding size experiment reported in Table The number of simulations is 10, See the notes to Table 1.

41 Table 5 Size-Adjusted Power: DGP 2 Empirical Size = 10% A. R=60 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE CCS B. R=120 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE CCS C. R=240 P=48 P=96 P=144 P=240 P=480 P=1200 MSPE-adjusted MSPE CCS Notes: 1. The coefficient b is set to Remaining details are as described in the notes to Table 4.

42 Table 6 Forecasts of Monthly Changes in U.S. Dollar Exchange Rates (1) (2) (3) (4) (5) (6) (7) (8) (9) MSPE- MSPEcountry prediction adjusted normal ^ ^ ^ sample adj. F 2 2 -adj ^F2 1 -(^F 2 2 -adj.) ^F2 1 -^F 2 2 CCS F 2 1 F 2 2 Canada 1990: :10 (0.08) 1.78 ** Japan 1990: :10 (0.43) * Switzerland 1985: :10 (0.48) 1.88 ** U.K. 1985: :10 (0.33) Notes: 1. In column (3), ^F 2 1 is the out of sample MSPE of the no change or random walk model, which forecasts a value of zero for the one month ahead change in the exchange rate. 2. In column (4), ^F 2 2 is the out of sample MSPE of a model that regresses the exchange rate on a constant and the previous month s cross-country interest differentials. The estimated regression vector and the current month s interest differential are then used to predict next month s exchange rate. Rolling regressions are used, with a sample size R of 120 months. 3. In column (5), adj. is the adjustment term P -1 G T t=t-p+1 (X t+1 N ^$ t ) 2, where: P is the number of predictions, P = 166 for Canada and Japan, P=226 for Switzerland and the U.K.; T=2003:10; X t+1 =(constant, interest differential at end of month t)n; ^$ t is the estimated regression vector. 4. In columns (7)-(9), standard errors are in parentheses and t-statistics (columns (7) and (8)) or a P 2 (2) statistic (column (9)) are in italics. Standard errors are computed as described in the text. * and ** denote test statistics significant at the 10 and 5 percent level, respectively, based on one sided tests using critical values from a standard normal (columns (7) and (8)) or chi-squared (CCS) distribution. In columns (7)and (8), and denote statistics significant at the 10 and 5 percent level based on McCracken s (2000) asymptotic critical values. 5. Data are described in the text. See notes to earlier tables for additional definitions.