Financial Risk Forecasting Chapter 8 Backtesting and stresstesting

Financial Risk Forecasting Chapter 8 Backtesting and stresstesting Jon Danielsson London School of Economics 2015 To accompany Financial Risk Forecasting http://www.financialriskforecasting.com/ Published by Wiley 2011 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 1 of 80

Financial Risk Forecasting 2011,2015 Jon Danielsson, page 2 of 80

Introduction Financial Risk Forecasting 2011,2015 Jon Danielsson, page 3 of 80

The focus of this chapter is on Backtesting Application of backtesting Significance of backtests Bernoulli coverage test Testing the independence of violations Joint test Loss-function-based backtests Expected shortfall backtesting Problems with backtesting Stress testing Financial Risk Forecasting 2011,2015 Jon Danielsson, page 4 of 80

Notation W T Testing window size T = W E +W T Number of observations in a sample η t = 0,1 Indicates whether a VaR violation occurs (i.e., η t = 1) υ i,i = 0,1 Number of violations (i = 1) and no violations (i = 0) observed in W T υ ij Number of instances where j follows i in W T Financial Risk Forecasting 2011,2015 Jon Danielsson, page 5 of 80

Backtesting Financial Risk Forecasting 2011,2015 Jon Danielsson, page 6 of 80

Forecasting VaR - Example Imagine you have 10 years of data, from 2004 to 2014 It is 2006 and you are trying to forecast risk for January 1st 2006 You will be using 2004 and 2005 to forecast daily VaR for the first day of 2006 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 7 of 80

The 500 trading days in 2004 and 2005 constitute the first estimation window W E is then moved up by one day to obtain the risk forecast for the second day of 2006, etc. Start End VaR forecast 1/1/2004 31/12/2005 VaR(1/1/2006) 2/1/2004 1/1/2006 VaR(2/1/2006)... 31/12/2012 30/12/2014 VaR(31/12/2014) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 8 of 80

What is backtesting? Backtesting is a procedure that can be used to compare various risk models It aims to take ex ante VaR forecasts from a particular model and compare them with ex post realized return (i.e., historical observations) Whenever losses exceed VaR, a VaR violation is said to have occurred Financial Risk Forecasting 2011,2015 Jon Danielsson, page 9 of 80

Usefulness of backtesting Useful in identifying the weaknesses of risk-forecasting models and providing ideas for improvement Not informative about the causes of weaknesses Models that perform poorly during backtesting should have their assumptions and parameter estimates questioned Backtesting can prevent underestimation of VaR and, hence, ensure that a bank carries sufficiently high capital However, it can also reduce the likelihood of overestimating VaR, which can lead to excessive conservatism Financial Risk Forecasting 2011,2015 Jon Danielsson, page 10 of 80

Assessment of VaR forecasts should ideally be done by tracking the performance of a model in the future using operational criteria However: as violations are only observed infrequently, a long period of time would be required Backtesting evaluates VaR forecasts by checking how a VaR forecast model performs over a period in the past Financial Risk Forecasting 2011,2015 Jon Danielsson, page 11 of 80

Estimation window (W E ): the number of observations used to forecast risk. If different procedures or assumptions are compared, the estimation window is set to whichever one needs the highest number of observations Testing window (W T ): the data sample over which risk is forecast (i.e., the days where we have made a VaR forecast) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 12 of 80

The entire sample size T is equal to the sum of W E and W T : t = 1 Entire data sample t = T Financial Risk Forecasting 2011,2015 Jon Danielsson, page 13 of 80

The entire sample size T is equal to the sum of W E and W T : t = 1 Entire data sample t = T t = 1 First estimation window t = W E VaR(W E +1) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 14 of 80

The entire sample size T is equal to the sum of W E and W T : t = 1 Entire data sample t = T t = 1 First estimation window t = W E VaR(W E +1) t = 2 Second estimation window t = W E +1 VaR(W E +2) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 15 of 80

The entire sample size T is equal to the sum of W E and W T : t = 1 Entire data sample t = T t = 1 First estimation window t = W E VaR(W E +1) t = 2 Second estimation window t = W E +1 VaR(W E +2) t = 3 Third estimation window t = W E +2 VaR(W E +3) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 16 of 80

The entire sample size T is equal to the sum of W E and W T : t = 1 Entire data sample t = T t = 1 First estimation window t = W E VaR(W E +1) t = 2 Second estimation window t = W E +1 VaR(W E +2) t = 3 Third estimation window t = W E +2 VaR(W E +3). Financial Risk Forecasting 2011,2015 Jon Danielsson, page 17 of 80

The entire sample size T is equal to the sum of W E and W T : t = 1 Entire data sample t = T t = 1 First estimation window t = W E VaR(W E +1) t = 2 Second estimation window t = W E +1 VaR(W E +2) t = 3 Third estimation window t = W E +2 VaR(W E +3). t = T W E Last estimation window t = T 1 VaR(T) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 18 of 80

VaR forecasts can be compared with the actual outcome: the daily 2001 to 2009 returns are already known Instead of referring to calendar dates (e.g., 1/1/2004), refer to days by indexing the returns, assuming 250 trading days per year: y 1 is the return on 1/1/1999 y 2,500 is the return on the last day, 31/12/2009 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 19 of 80

The estimation window W E is set at 500 days, and the testing window W T is therefore 2,000 days: t t +W E 1 VaR(t +W E ) 1 500 VaR(501) 2 501 VaR(502)... 1,999 2,499 VaR(2,500) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 20 of 80

VaR violation If the actual return on a particular day exceeds the VaR forecast, then the VaR limit is said to have been violated VaR violation: an event such that: η t = { 1, if y t VaR t 0, if y t > VaR t. υ 1 is the count of η t = 1 and υ 0 is the count of η t = 0, which are simply obtained by: υ 1 = Ση t υ 0 = W T υ 1. Financial Risk Forecasting 2011,2015 Jon Danielsson, page 21 of 80

Violation ratios The main tools used in backtesting are violation ratios, where the actual number of VaR violations are compared with the expected value. Violation ratio: VR = Observed number of violations Expected number of violations = υ 1 p W T If the violation ratio is greater than one the VaR model underforecasts risk If smaller than one the model overforecasts risk Financial Risk Forecasting 2011,2015 Jon Danielsson, page 22 of 80

One can make a judgment on the quality of the VaR forecasts by recording the relative number of violations or use more sophisticated statistical techniques We record the violations as η t, which takes the value 1 when a violation occurs and 0 otherwise The number of violations are collected in the variable υ, where υ 1 is the number of violations and υ 0 is the number of days without violations These two add up to make the testing period Financial Risk Forecasting 2011,2015 Jon Danielsson, page 23 of 80

Market risk regulations Financial institutions regulated under the Basel II Accords are required to set aside a certain amount of capital due to market risk, credit risk and operational risk It is based on multiplying the maximum of previous day 1%VaR and 60 days average VaR by a constant, Ξ, which is determined by the number of violations that happened previously: ( ) Market risk capital t = Ξ t max VaR 1% t,var 1% t +constant Financial Risk Forecasting 2011,2015 Jon Danielsson, page 24 of 80

VaR 1% t is average reported 1% VaR over the previous 60 trading days The multiplication factor Ξ t varies with the number of violations, υ 1, that occurred in the previous 250 trading days - the required testing window length for backtesting in the Basel Accords. This is based on three ranges for the number of violations, named after the three colors of traffic lights: 3, if υ 1 4 (Green) Ξ t = 3+0.2(υ 1 4), if 5 υ 1 9 (Yellow) 4, if 10 υ 1 (Red). Financial Risk Forecasting 2011,2015 Jon Danielsson, page 25 of 80

Estimation window length The W E length is determined by the choice of VaR model and probability level Different methods have different data requirements (EWMA requires c. 30 days, HS needs at least 300 for VaR 1%, and GARCH even more) When making comparisons, the estimation window should be sufficiently large to accommodate the most stringent data criteria Even within the same method, it may be helpful to compare different window lengths Financial Risk Forecasting 2011,2015 Jon Danielsson, page 26 of 80

Testing window length VaR violations are infrequent events: with a 1% VaR, a violation is expected once every 100 days, so that 2.5 violations are expected per year Therefore, the actual sample size of violations is quite small, causing difficulties for statistical inference We might need perhaps 10 violations for reliable statistical analysis, or 4 years of data Financial Risk Forecasting 2011,2015 Jon Danielsson, page 27 of 80

Violation ratios VR=1 is expected, but how can we ascertain whether any other value is statistically significant? A useful rule of thumb: If VR [0.8,1.2] it is a good forecast If VR < 0.5 or VR > 1.5 the model is imprecise Both bounds narrow with increasing testing window lengths As a first attempt, one could plot the actual returns and VaR together, thereby facilitating a quick visual inspection of the quality of the VaR forecast Financial Risk Forecasting 2011,2015 Jon Danielsson, page 28 of 80

Application of backtesting Financial Risk Forecasting 2011,2015 Jon Danielsson, page 29 of 80

Volatility and VaR: extreme example 140% EWMA $110 Volatility 100% 60% 20% 20% 60% 100% $90 $70 $50 $30 $10 VaR 0 500 1000 1500 2000 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 30 of 80

Volatility Volatility and VaR: extreme example 140% 100% 60% 20% 20% 60% 100% EWMA MA 0 500 1000 1500 2000 $110 $90 $70 $50 $30 $10 VaR Financial Risk Forecasting 2011,2015 Jon Danielsson, page 31 of 80

Volatility and VaR: extreme example Volatility 140% 100% 60% 20% 20% 60% 100% EWMA MA HS 0 500 1000 1500 2000 $110 $90 $70 $50 $30 $10 VaR Financial Risk Forecasting 2011,2015 Jon Danielsson, page 32 of 80

Volatility and VaR: extreme example Volatility 140% 100% 60% 20% 20% 60% 100% EWMA MA HS GARCH 0 500 1000 1500 2000 $110 $90 $70 $50 $30 $10 VaR Financial Risk Forecasting 2011,2015 Jon Danielsson, page 33 of 80

10% Volatility and VaR for S&P 500 EWMA $120 $100 Volatility 5% $80 0% $60 VaR 5% $40 $20 10% Jan 98 Jan 02 Jan 06 Jan 10 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 34 of 80

10% Volatility and VaR for S&P 500 EWMA MA $120 $100 Volatility 5% $80 0% $60 VaR 5% $40 $20 10% Jan 98 Jan 02 Jan 06 Jan 10 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 35 of 80

Volatility 10% 5% 0% Volatility and VaR for S&P 500 EWMA MA HS $120 $100 $80 $60 VaR 5% $40 $20 10% Jan 98 Jan 02 Jan 06 Jan 10 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 36 of 80

Volatility and VaR for S&P 500 Volatility 10% 5% 0% EWMA MA HS GARCH $120 $100 $80 $60 VaR 5% $40 $20 10% Jan 98 Jan 02 Jan 06 Jan 10 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 37 of 80

Volatility and VaR for S&P 500, focus on 2003-4 3% 2% EWMA $40 $35 Volatility 1% 0% 1% 2% 3% $30 $25 $20 $15 VaR Jan 03 Apr 03 Jul 03 Oct 03 Jan 04 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 38 of 80

Volatility and VaR for S&P 500, focus on 2003-4 Volatility 3% 2% 1% 0% 1% 2% EWMA MA $40 $35 $30 $25 $20 VaR 3% $15 Jan 03 Apr 03 Jul 03 Oct 03 Jan 04 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 39 of 80

Volatility and VaR for S&P 500, focus on 2003-4 3% 2% EWMA MA HS $40 $35 Volatility 1% 0% 1% 2% 3% $30 $25 $20 $15 VaR Jan 03 Apr 03 Jul 03 Oct 03 Jan 04 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 40 of 80

Volatility and VaR for S&P 500, focus on 2003-4 Volatility 3% 2% 1% 0% 1% EWMA MA HS GARCH $40 $35 $30 $25 VaR 2% 3% $20 $15 Jan 03 Apr 03 Jul 03 Oct 03 Jan 04 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 41 of 80

Volatility and VaR for S&P 500, focus on 2008 Volatility 10% 5% 0% EWMA $100 $80 $60 VaR 5% $40 10% Jul 08 Sep 08 Nov 08 Jan 09 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 42 of 80

Volatility and VaR for S&P 500, focus on 2008 Volatility 10% 5% 0% EWMA MA $100 $80 $60 VaR 5% $40 10% Jul 08 Sep 08 Nov 08 Jan 09 $20 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 43 of 80

Volatility and VaR for S&P 500, focus on 2008 Volatility 10% 5% 0% EWMA MA HS $100 $80 $60 VaR 5% $40 10% $20 Jul 08 Sep 08 Nov 08 Jan 09 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 44 of 80

Volatility and VaR for S&P 500, focus on 2008 Volatility 10% 5% 0% 5% EWMA MA HS GARCH $120 $100 $80 $60 $40 VaR 10% $20 Jul 08 Sep 08 Nov 08 Jan 09 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 45 of 80

Significance of Backtests Financial Risk Forecasting 2011,2015 Jon Danielsson, page 46 of 80

Focus on two issues: The number of violations (tested by the unconditional coverage) Clustering (tested by independence tests) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 47 of 80

Unconditional coverage property Ensures that the theoretical confidence level p matches the empirical probability of violation: For a 1% VaR backtest, we would expect to observe a VaR violation 1% of the time. If, instead, violations are observed more often, say 5% of the time, the VaR model is systematically underestimating risk at the 1% level In other words, the 1% VaR produced from the model is in reality the 5% VaR Financial Risk Forecasting 2011,2015 Jon Danielsson, page 48 of 80

Independence property Requires any two observations in the hit sequence to be independent of each other Intuitively, the fact that a violation has been observed today should not convey any information about the likelihood of observing a violation tomorrow If VaR violations cluster, we can predict a violation today if there was one yesterday, and therefore the probability of a loss exceeding 1% VaR today is higher than 1% In this case, a good VaR model would have increased the 1% VaR forecast following a violation Financial Risk Forecasting 2011,2015 Jon Danielsson, page 49 of 80

Pros and cons Unconditional coverage and independence are distinct and it is entirely possible that a VaR model satisfying one of them would not satisfy the other The main downside of these tests is that they rely on asymptotic distributions Given that violations are rare events, the effective sample size is relatively small and, therefore, tests may not be as robust as we would like One may be better off obtaining confidence bounds by means of simulations Financial Risk Forecasting 2011,2015 Jon Danielsson, page 50 of 80

Bernoulli coverage test η t is a sequence of Bernoulli-distributed random variables We then use the Bernoulli coverage test to ascertain the proportion of violations The null hypothesis for VaR violations is: H 0 : η B(p), where B stands for the Bernoulli distribution. Financial Risk Forecasting 2011,2015 Jon Danielsson, page 51 of 80

The Bernoulli density is given by: (1 p) 1 ηt (p) ηt,η t = 0,1. The probability p can be estimated by: ˆp = υ 1 W T The likelihood function is given by: L U (ˆp) = T t=w E +1 (1 ˆp) 1 ηt (ˆp) ηt = (1 ˆp) υ 0 (ˆp) υ 1 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 52 of 80

Denote this as the unrestricted likelihood function because it uses estimated probability ˆp in contrast to the likelihood function below where we restrict the probability value to p. Under the H 0, p = ˆp, so the restricted likelihood function is: L R (p) = T t=w E +1 = (1 p) υ 0 (p) υ 1 (1 p) 1 ηt (p) ηt Financial Risk Forecasting 2011,2015 Jon Danielsson, page 53 of 80

We can use an LR test to see whether L R = L U or, equivalently, whether p = ˆp: LR = 2(logL U (p) logl U (ˆp)) = 2log (1 ˆp)υu (ˆp) υ 1 (1 p) υ 0 (p) υ 1 asymptotic χ 2 (1) Choosing a 5% significance level for the test, the null hypothesis is rejected if LR > 3.84 The choice of significance level affects the power of the test: a low type I error implies a higher type II error and therefore a lower power for the test. Financial Risk Forecasting 2011,2015 Jon Danielsson, page 54 of 80

The Bernoulli coverage test is nonparametric in the sense that it does not assume a distribution for the returns and often provides good benchmarks for the assessment of accuracy of VaR models It does not have much power when sample sizes are small, like the one-year size specified in the Basel Accords Financial Risk Forecasting 2011,2015 Jon Danielsson, page 55 of 80

Testing the independence of violations It is also of interest to test whether violations cluster (i.e., whether all violations happen one after the other), indicating a sequence of losses since violations should theoretically spread out over time Calculate the probabilities of two consecutive violations (i.e., p 11 ) and the probability of a violation if there was no violation on the previous day (i.e., p 01 ) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 56 of 80

More generally, the probability that: p i,j = Pr(η t = i η t 1 = j) where i and j are either 0 or 1 The first-order transition probability matrix is defined as: ( ) 1 p01 p Π 1 = 01 1 p 11 p 11 The restricted likelihood function, where the transition matrix from the null hypothesis is used since the hit sequence is Bernoulli distributed, is: L R (Π 1 ) = (1 p 01 ) υ 00 p υ 01 01 (1 p 11) υ 10 p υ 11 11 where υ ij is the number of observations where j follows i. Financial Risk Forecasting 2011,2015 Jon Danielsson, page 57 of 80

Maximum likelihood (ML) estimates are obtained by maximizing L R (Π 1 ): υ 01 υ 00 ˆΠ 1 = υ 00 +υ υ 01 10 υ 00 +υ υ 01 11 υ 10 +υ 11 υ 10 +υ 11 Under the null hypothesis of no clustering, the probability of a violation tomorrow does not depend on today seeing a violation Financial Risk Forecasting 2011,2015 Jon Danielsson, page 58 of 80

Then, p 01 = p 11 = p and the estimated transition matrix is simply: ( ) 1 ˆp ˆp ˆΠ 0 = 1 ˆp ˆp where υ 01 +υ 11 ˆp = υ 00 +υ 10 +υ 01 +υ 11 The unrestricted likelihood function according to null hypothesis uses the estimated transition matrix and is: L U (ˆΠ 0 ) = (1 ˆp) υ 00+υ 10ˆp υ 01+υ 11 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 59 of 80

Likelihood ratio test The likelihood ratio test: ( LR = 2 logl U (ˆΠ 0 ) logl R (ˆΠ 1 ) ) asymptotic χ 2 (1) This test does not depend on true p and only tests for independence Financial Risk Forecasting 2011,2015 Jon Danielsson, page 60 of 80

The main problem with tests of this sort is that they must specify the particular way in which independence is breached However, there are many possible ways in which the independence property is not fulfilled: The test will have no power to detect departures from independence if the likelihood of VaR being violated today depends on whether VaR was violated 2 days ago not on yesterdays VaR being violated. Financial Risk Forecasting 2011,2015 Jon Danielsson, page 61 of 80

Testing S&P 500: comparison of the four VaR models January 30, 1998 to December 31, 2009 (W T = 3000) with crisis Model Coverage test Independence test Test statistic p-value Test statistic p-value EWMA 18.1 0.00 0.00 0.96 MA 81.2 0.00 7.19 0.01 HS 24.9 0.00 4.11 0.04 GARCH 16.9 0.00 0.00 0.99 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 62 of 80

Testing S&P 500: comparison of the four VaR models January 30, 1998 to November 1, 2006 (W T = 2000) - without crisis Model Coverage test Independence test Test statistic p-value Test statistic p-value EWMA 2.88 0.09 0.68 0.41 MA 6.15 0.01 2.62 0.11 HS 0.05 0.82 1.52 0.22 GARCH 1.17 0.28 0.99 0.32 Financial Risk Forecasting 2011,2015 Jon Danielsson, page 63 of 80

Joint test We can jointly test whether violations are significantly different from those expected and whether there is violation clustering by constructing the following test statistic: LR(joint) = LR(coverage)+LR(independence) χ 2 (2) The joint test has less power to reject a VaR model which only satisfies one of the two properties Individual tests should be used if the user has some prior knowledge of weaknesses in the VaR model. Financial Risk Forecasting 2011,2015 Jon Danielsson, page 64 of 80

Loss-function-based backtests An alternative backtest could be constructed based on a general loss function: l(var t (p),y t ) It would take into consideration the magnitude of VaR violation. An example of such a loss function is: l(y,x) = 1+(y x) 2,y x Otherwise, it takes the value zero If y indicates returns and x negative VaR, then we would typically calculate sample average loss as: ˆL = 1 W T T t=w E +1 l(y t,var t (p)) Financial Risk Forecasting 2011,2015 Jon Danielsson, page 65 of 80

Pros and cons Advantage of loss-function-based backtests The functional form of the loss function is flexible and can be tailored to address specific concerns Disadvantages of loss-function-based backtests In order to determine whether ˆL is too large relative to what is expectedan explicit assumption about P/L distribution needs to be made. A null hypothesis for ˆL should be mandatory Financial Risk Forecasting 2011,2015 Jon Danielsson, page 66 of 80

Given the uncertainty over the true P/L distribution, a finding that ˆL is too large relative to what is expected could be due to: Inaccuracy in the VaR model An inaccurately assumed distribution In other words, the test is a joint test of the VaR model and the P/L distribution. This is problematic Loss-function-based backtests may consequently be better suited for discriminating among competing VaR models than judging the absolute accuracy of a single model Financial Risk Forecasting 2011,2015 Jon Danielsson, page 67 of 80

Expected Shortfall Backtesting Financial Risk Forecasting 2011,2015 Jon Danielsson, page 68 of 80

It is harder to backtest expected shortfall ES than VaR because we are testing an expectation rather than a single quantile Fortunately, there exists a simple methodology for backtesting ES that is analogous to the use of violation ratios for VaR For days when VaR is violated, normalized shortfall NS is calculated as: NS t = y t ES t where ES t is the observed ES on day t Financial Risk Forecasting 2011,2015 Jon Danielsson, page 69 of 80

From the definition of ES, the expected Y t, VaR is violated, is: E[Y t Y t < VaR t ] ES t = 1 Therefore, average NS, denoted by NS, should be one and this forms our null hypothesis: H 0 : NS = 1 In what follows, we opted to implement just the EWMA and HS versions, since the other two (MA and GARCH) are quite similar Financial Risk Forecasting 2011,2015 Jon Danielsson, page 70 of 80

The reliability of any ES backtest procedure is likely to be much lower than that of VaR backtest procedures: With ES, we are testing whether the mean of returns on days when VaR is violated is the same as average ES on these days. Much harder to create formal tests to ascertain whether normalized ES equals one or not than the coverage tests developed above for VaR violations Hence, backtesting ES requires many more observations than backtesting VaR In instances where ES is obtained directly from VaR, and gives the same signal as VaR (i.e., when VaR is subadditive), it is better to simply use VaR Financial Risk Forecasting 2011,2015 Jon Danielsson, page 71 of 80

Problems with Backtesting Financial Risk Forecasting 2011,2015 Jon Danielsson, page 72 of 80

Backtesting assumes that there have been no structural breaks in the data throughout the sample period: But financial markets are continually evolving, and new technologies, assets, markets and institutions affect the statistical properties of market prices Unlikely that the statistical properties of market data in the 1990s are the same as today, implying that a risk model that worked well then might not work well today Financial Risk Forecasting 2011,2015 Jon Danielsson, page 73 of 80

Data mining and intellectual integrity: Backtesting is only statistically valid if we have no ex ante knowledge of the data in the testing window If we iterate the process, continually refining the risk model with the same test data and thus learning about the events in the testing window, the model will be fitted to those particular outcomes, violating underlying statistical assumptions Financial Risk Forecasting 2011,2015 Jon Danielsson, page 74 of 80

Stresstesting Financial Risk Forecasting 2011,2015 Jon Danielsson, page 75 of 80

The purpose of stresstesting is to create artificial market outcomes in order to see how risk management systems and risk models cope with the artificial event, and to assess the ability of a bank to survive a large shock The gap that stress testing aims to fill is model failure to encounter rare situations that could cause a severe loss, since backtesting relies on recent historical data Financial Risk Forecasting 2011,2015 Jon Danielsson, page 76 of 80

Scenario analysis The main aim is to come up with scenarios that are not well represented in historical data but are nonetheless possible and detrimental to portfolio performance We then revalue the portfolio in this hypothetical environment and obtain an estimate of maximum loss This procedure then allows the bank to set aside enough capital for such an eventuality Financial Risk Forecasting 2011,2015 Jon Danielsson, page 77 of 80

Examples of historical scenarios Scenario Period Stock market crash October 1987 ERM crisis September 1992 Bond market crash April 1994 Asian currency crisis Summer 1997 LTCM and Russia crisis August 1998 Global crisis 2007-2009 Eurozone crisis Since 2010 Scenarios can be classified into two broad types: Simulating shocks that have never occurred or are more likely to occur than historical data suggest Simulating shocks that reflect permanent or temporary structural breakswhere historical relationships do not hold Financial Risk Forecasting 2011,2015 Jon Danielsson, page 78 of 80

Issues in scenario analysis The results indicate that banks should set aside a large amount of capital to absorb the worst case loss, which is usually not practical But there are other actions that a bank can take such as buying insurance for the events in question and changing the composition of the portfolio to reduce exposure to a particular market variable The lack of a mechanism to reliably judge the probability of a stress test scenario since they are both highly subjective and difficult to pinpoint The potential for feedback effects, or endogenous risk since the focus is on institution-level risk, where feedback effects are disregarded Financial Risk Forecasting 2011,2015 Jon Danielsson, page 79 of 80

Scenario analysis and risk models It is straightforward to integrate scenario analysis with risk models in use. Assume a risk manager has assigned a probability ξ to a particular scenario and the potential loss arising from it. The probability can be purely a subjective judgment from the managers experience or could be derived from a model Financial Risk Forecasting 2011,2015 Jon Danielsson, page 80 of 80