Evaluating credit risk models: A critique and a new proposal

Transcription

1 Evaluatng credt rsk models: A crtque and a new proposal Hergen Frerchs* Gunter Löffler Unversty of Frankfurt (Man) February 14, 2001 Abstract Evaluatng the qualty of credt portfolo rsk models s an mportant queston for both banks and regulators. Lopez and Sadenberg (2000) suggest cross-sectonal resamplng technques n order to make effcent use of avalable data and to produce measures of forecast accuracy. We frst show that ther proposal dsregards crosssectonal dependence n smulated subportfolos, whch renders standard statstcal nference nvald. We proceed by suggestng another evaluaton methodology whch draws on the concept of lkelhood rato tests. Specfcally, we compare the predctve qualty of alternatve models by comparng the probabltes that observed data have been generated by these models. The dstrbuton of the test statstc can be derved through Monte Carlo smulaton. To explot dfferences n cross-sectonal predctons of alternatve models, the test can be based on a lnear combnaton of subportfolo statstcs. In the constructon of the test, the weght of a subportfolo depends on the dfference n the loss dstrbutons whch alternatve models predct for ths partcular portfolo. Ths makes effcent use of the data, and reduces computatonal burden. Monte Carlo smulatons suggest that the power of the tests s satsfactory. Key words: credt rsk, backtestng, model valdaton, bank regulaton JEL classfcaton: G2; G28; C52 *Correspondng author: Hergen Frerchs, Char of Bankng and Fnance, Unversty of Frankfurt (Man), P.O. Box , Frankfurt (Man), Germany. Phone: , facsmle: , e-mal: frerchs@ww.unfrankfurt.de. The paper s part of a jont research project wth Deutsche Bundesbank on modelng credt portfolo rsk. We wsh to thank Jan Peter Krahnen, Thlo Lebg, Ludger Overbeck, Peter Raupach, Mark Wahrenburg and semnar partcpants at the Unversty of Frankfurt (Man) for helpful comments.

2 1. Introducton In the lterature on credt rsk, t s commonplace to refer to the problems of evaluatng the qualty of credt portfolo rsk models. Several years after the frst models have been proposed, there s only one paper whch emprcally tests ther predctve ablty (Nckell, Perraudn and Varotto, 1999). One explanaton for ths scarcty of research are concerns that evaluaton procedures borrowed from the lterature on market rsk models have lttle power when appled to credt portfolos. To make effcent use of avalable data, Lopez and Sadenberg (2000) propose evaluaton methods for credt portfolo rsk models based on cross-sectonal smulaton. In the constructon of the tests they assume that predcton errors of portfolos resampled from the loss experence of one year are ndependent. As we demonstrate n ths paper, ths wll not be the case n a typcal settng. If the economy moves nto recesson, for nstance, credt losses wll be above average both n the entre sample and n randomly drawn subsamples. Ths renders standard statstcal nference nvald. The major part of ths paper s thus devoted to the development of relable and powerful test procedures for the evaluaton of credt rsk models. We propose to compare the predctve qualty of alternatve models by comparng the probabltes that observed data have been generated by these models. The test statstc s based on a rato of lkelhoods, even though t s not a lkelhood rato test n the tradtonal sense. Its dstrbuton can be derved usng Monte Carlo smulaton. To explot dfferences n cross-sectonal predctons of alternatve models, the lkelhood rato crteron s appled to subportfolos, and the test statstc s formed as a lnear combnaton of the subportfolo ratos. Unlke Lopez and Sadenberg (2000), we do not recommend to resample these portfolos randomly from the entre portfolo. Rather, the weght of a subportfolo depends on the dfference n the loss dstrbutons whch alternatve models predct for ths partcular portfolo. Ths makes effcent use of the data and reduces computatonal burden. To gan ntuton, consder a portfolo whose oblgors belong to one of two sectors. The correct default probablty s 1% for the frst, and 3% for the second sector. Now assume that a rsk analyst uses the default experence of ths portfolo to evaluate a model whch posts a unform default probablty of 2%. If the test s based only on the entre portfolo or random subsets, the nadequacy of the 2

3 model wll not be revealed as the expected default rate wll be 2% n ether case. We examne the power of our proposed test procedures usng Monte Carlo smulatons. The power appears to be satsfactory. Wth 10 years of data on annual defaults, for example, t s smlar to a market rsk settng where two rsk models are compared based on 60 observatons. For ease of exposton, the smulatons are conducted for smple two-state credt rsk models whch gnore mgraton and recovery rsk. However, the tests can easly be appled to more complex models, and they do not requre that the models under comparson are nested. Our test procedure follows recent papers from the market rsk lterature n that tests are based on the entre portfolo dstrbuton rather than on quantles (e.g. Berkowtz, 1999, or Crnkovc and Drachman, 1996). Related papers nclude Carey and Hrycay (2001), who dscuss varous resamplng strateges to construct expected loss dstrbutons from a default hstory. Sobehart, Keenan and Sten (2000) proposes technques for assessng the qualty of ndvdual default rate estmates, an mportant nput to credt rsk models. Löffler (2000) quantfes the effects of nput parameter uncertanty on the relablty of credt rsk models. A useful summary of avalable credt rsk models s gven n Crouhy, Gala and Mark (2000). To the best of our knowledge, our proposed approach s novel n two ways: frst, n the use of lkelhoods to evaluate credt portfolo rsk models, and second, n the judcous use of subportfolos to ncrease the tests power. The paper s organzed as follows. Secton 2 descrbes the methodology for the evaluaton of test procedures. Secton 3 dscusses the tests proposed by Lopez and Sadenberg (2000). Secton 4 presents our proposal and assesses ts power. Secton 5 contans senstvty analyses, and Secton 6 concludes. 2. Methodology for the evaluaton of test procedures A natural way for evaluatng the performance of test procedures s to employ a Monte Carlo study. We smulate a large number (10,000) of random default hstores whch are all generated by one specfc credt portfolo rsk model. We then state the null hypothess that the default hstory s governed by some model specfcaton, choose a sgnfcance level, and apply a statstcal test separately for each smulated default hstory. The performance of the test s judged by two crtera: If the H 0 -model s the 3

4 one that has generated the hstory, the rejecton frequency n the smulatons should be equal to the chosen sgnfcance level. If the H 0 -model s ncorrect, the rejecton frequency,.e. the power of the test, should be as large as possble. We consder credt rsk models whose output s a dstrbuton of the expected number of defaults wthn a portfolo. Thus, we neglect both mgraton rsk and recovery rate uncertanty, whch renders a dscrete dstrbuton of expected and realzed outcomes. The framework we apply s smlar to a two-state verson of CredtMetrcs 1. Default correlatons are modeled based on the asset value model of Merton (1974). There, a frm defaults f ts asset value falls below a crtcal level defned by the value of labltes. Correlatons of changes of asset values thus translate nto default correlatons. In a two-state world, the credt portfolo rsk models whch have been developed n the last few years are smlar n the underlyng structure and produce nearly dentcal outputs when parameterzed approprately. 2 For ths reason, we are confdent that our results are of general nterest even though we examne only one class of credt portfolo rsk models. In addton, the test procedures put forward n ths paper can also be used n more complex settngs, e.g. when mgraton rsk s added. To capture asset correlatons, asset value changes factor model: A are modeled through a K- K K Σ w, k Z k + 1 Σ k = 1 k = 1 A = w ε, (1) 2, k wth Z k denotng the systematc factors, and ε the dosyncratc rsk of oblgor. 3 The Z k and ε are assumed to be normally dstrbuted wth zero mean and a covarance matrx equal to the dentty matrx. As a consequence, asset value changes follow a standard normal dstrbuton. A also The factor senstvtes w determne asset correlatons. In the smplest form of (1), a one-factor model (K=1) wth constant factor senstvtes w =w, the mutual asset correlaton s equal to w². Default correlatons can be calculated va the bvarate 1 Cf. JP Morgan (1997) for a general descrpton of CredtMetrcs. 2 Cf. Fnger (1998), Koyluoglu and Hckman (1998), Gordy (2000), and Wahrenburg and Nethen (2000). 3 Cf. Fnger (1999), Koyluoglu and Hckman (1998), and Belkn, Suchower and Forest (1998b) for 4

5 normal dstrbuton. A borrower defaults whenever A 1 < Φ ( p ), where p s the uncondtonal default probablty and Φ denotes the cumulatve standard normal dstrbuton functon. Gven the realzatons of the systematc factors Z k, the condtonal default probablty p Z k equals K 1 Φ ( p ) Σ w, k Z k = Φ k= 1 p Z k. (2) K 2 1 Σ w, k k = 1 Expected loss dstrbutons are generated through a Monte Carlo smulaton nvolvng 1,000,000 scenaros for the number of defaults n the portfolo. 4 To avod zero probabltes we ft exponental trends to the number of defaults, separately for both tals of these dscrete dstrbutons. For the left tal of the dstrbuton, the regresson parameters are estmated over an nterval startng at the frst pont on whch at least 10 random scenaros fell, and endng at the frst pont wth at least 200 scenaros. All numbers of defaults wth less than 10 scenaros are replaced by the predcted trend. For the rght tal of the dstrbuton, we proceed n the same manner. In the parameterzatons used n ths paper, the R 2 of the trend regressons s always larger than 85%. In the base case, we assume that evaluators of credt rsk models observe 10 years of annual data on homogeneous portfolos of 10,000 borrowers. The portfolos are homogeneous n terms of constant model parameters and portfolo composton across tme. Portfolo weghts are rrelevant as the evaluators only examne the number of defaults. Throughout the paper, we set the uncondtonal annual default probablty equal to 1% for each oblgor. In the base case, we also assume that there s no seral correlaton of defaults across tme. The justfcaton s that a credt portfolo rsk model should elmnate any seral correlaton by condtonng on current nformaton. applcatons of ths model. 4 If uncondtonal default probabltes are equal across oblgors, a quck way to perform the smulatons s ) draw N(0,I)-dstrbuted random numbers for the factor realzatons, ) calculate the condtonal default probablty, and ) draw the number of defaults from a bnomal dstrbuton gven the number of loans and the condtonal default probablty. For descrptve statstcs of smulated expected loss dstrbutons see appendx. 5

6 3. The proposal of Lopez and Sadenberg The man problem when evaluatng credt rsk models s the scarcty of data n the tme dmenson. Lopez and Sadenberg (2000) suggest cross-sectonal resamplng technques to ncrease the power of evaluaton procedures. Gven a credt data set coverng T years of data for N loans, a large number R of subportfolos s randomly drawn for each year t n T. In drawng the borrowers for a partcular subportfolo, Lopez and Sadenberg suggest to draw wthout replacement. They also recommend to draw large subportfolos, but do not dscuss ths ssue n detal. For each subportfolo, the credt loss dstrbuton s forecasted and compared wth the subportfolo s observed number of defaults. In a sense, the number of observatons avalable for model evaluaton s thus multpled by the factor R. Lopez and Sadenberg propose tests for the predcted expected loss, predcted quantles and the predcted entre credt loss dstrbuton. In the dervaton of the tests, the authors assume that observed predcton errors are ndependent and that a model s accuracy can be examned usng standard testng procedures. As t turns out, ths assumpton s nvald. The lack of ndependence of subportfolo defaults results from the fact that any subportfolo defaults drawn from a one-year default experence mrror the realzaton of the systematc factors n ths specfc year. In a bad year, subportfolos wll have a relatvely hgh number of defaults on average. In a good year, there wll typcally be few defaults n the resampled portfolos. One consequence of ths dependence s that there wll be cases where there s no volaton of the predcted value at rsk across all subportfolos pertanng to one year. 5 In the followng, we conduct smulatons to demonstrate that the lack of ndependence can severely affect the performance of the test statstcs proposed by Lopez and Sadenberg. We proceed as follows: 1. Smulate 10, year default hstores based on a one-factor asset value model wth an uncondtonal default probablty of 1% and an asset correlaton of 5 In fact, cross-sectonal dependence arses even when we resample from a portfolo wth zero default correlaton. Consder a homogenous portfolo wth 1,000 oblgors, a default probablty of 0.01 and a zero default correlaton. If the chosen subportfolo sze s 500, the 90% quantle for subportfolo defaults s 10. Wth a probablty of 46%, however, the overall number of defaults n the entre portfolo s 9 or less. That s, n 49% of all cases you would not observe a volaton of the 90% quantle n any of the random subportfolos. 6

7 2 5% (.e. w = 0.05 for all ). 2. Draw 1,000 subportfolos for each year as proposed by Lopez and Sadenberg. We do ths for three dfferent subportfolo szes of 2,000, 5,000 and 8,000 borowers, respectvely. 3. Smulate expected credt loss dstrbutons of several credt rsk models, one of whch s equal to the one used for generatng the default hstores. Whle keepng the uncondtonal default rate constant at 1%, we vary the assumed asset correlatons (1%, 2.5%, 5%, 7.5%, and 20%). 4. Implement one of the test statstcs proposed by Lopez and Sadenberg and calculate the rejecton frequences for all models n queston. We concentrate on the quantle test proposed by Lopez and Sadenberg. 6 Under the assumpton that the predcted quantles are accurate and observed volatons of the quantles are ndependent, these volatons are draws from a bnomal dstrbuton. Whether or not the percentage of observed volatons αˆ s equal to the chosen confdence level α can be tested usng the lkelhood rato statstc y T R y y T R [ ( αˆ (1 αˆ) ) log( α (1 )] y LR( α ) = 2 log α), (3) where y s the number of volatons across the T*R subportfolos. In the smulatons, we examne the performance of a test usng the 90% quantle, usng a sgnfcance level of 10%. As a prelmnary analyss, we compute the frequency of 90%-quantle volatons across all smulated subportfolos. As we generate 10, year default hstores and draw 1,000 subportfolos for each year, there are 100,000,000 subportfolos n our sample. The results are summarzed n Table 1. 6 Lopez / Sadenberg (2000), p

8 Table 1: Smulated frequency of 90%-quantle volatons (correct correlaton assumpton s 5%) Correlaton Assumpton Frequency of 90%-quantle volatons for a subportfolo sze of 2,000 5,000 8, % 19.1% 20.5% 20.7% 2.5% 13.7% 14.6% 14.7% 5.0% 10.0% 9.8% 9.9% 7.5% 7.2% 7.3% 7.2% 20.0% 3.5% 3.2% 3.2% Snce the test s based on the 90%-quantle we expect the frequency of quantle volatons to be 10% for the correct model, whch has an asset correlaton of 5%. Ths s ndeed the case. (The small devatons from 10% are due to smulaton error.) Falsely assumng lower correlatons leads to a hgher frequency of volatons, and vce versa. The results demonstrate that the chosen models dffer consderably from each other. The subportfolo sze does not affect the average frequency of quantle volatons. We now calculate the lkelhood rato test statstc (3) separately for each of our 10,000 hstores. The test results are summarzed n Table 2. The frequency wth whch the null hypothess s rejected decreases wth ncreasng rsk (see column three). Wth a subportfolo sze of 20%, the rejecton frequency equals 95.8% for a credt rsk model wth 1% correlaton. Vewed n solaton, ths would ndcate that the test s powerful. For a null hypothess of 20% correlaton, the rejecton frequency equals 69.6% whch stll s a satsfactory result. Dsturbng, however, s the rejecton frequency of 88.1% for the 5%-correlaton model. As ths s the model whch has generated the hstores, the fgure should be 10% whch s the chosen sze of the test. The result suggests that the test s of lttle use snce the probablty that the true model wll be rejected s, frst, much hgher than the assumed sze and, second, hgher than the probablty of rejectng a model wth a false correlaton assumpton. A second problem of the quantle test s the large number of scenaros where there are no volatons at all (see column 4). In these cases the test statstc s not defned. The number of scenaros wthout any volatons rses as ether the correlaton parameter s ncreased relatve to the correlaton of the default hstory or a larger subportfolo sze s used. A hgher correlaton mples hgher rsk and a wder credt loss dstrbuton. Thus, the probablty of extreme events s hgher as well. The 8

9 Table 2: Smulated performance of the Lopez and Sadenberg quantle test (correct correlaton = 5%, sgnfcance level of quantle test = 10%) Subportfolo sze H 0 (Correlaton) Rejecton frequency of H 0 Runs wth no volaton Runs wth LR(False)>LR(True) 2, % 95.8% 0.2% 65.8% 2.5% 91.6% 1.2% 54.9% 5.0% 88.1% 2.4% 7.5% 85.6% 4.2% 59.0% 20.0% 69.6% 23.2% 72.9% 5, % 89.6% 2.0% 64.8% 2.5% 78.4% 6.6% 53.0% 5.0% 65.7% 15.4% 7.5% 57.6% 23.2% 43.4% 20.0% 37.7% 49.2% 60.0% 8, % 81.2% 4.2% 60.4% 2.5% 64.0% 12.3% 45.0% 5.0% 48.5% 24.2% 7.5% 39.3% 33.7% 29.8% 20.0% 20.5% 61.5% 50.3% second pattern arses because, for a gven realzed default rate, the probablty of drawng subportfolos wth a hgh default frequency goes down when the subportfolo sze s ncreased. Ths results from the fact that the number of defaults n the resampled subportfolos follows a hypergeometrc dstrbuton. The last column of Table 2 wll be nterestng when comparng the Lopez and Sadenberg quantle test wth the test statstc whch we wll propose n the next secton. It dsplays the frequency that the lkelhood rato statstc LR (α ) of the false model s hgher than that of the true one. Wth a subportfolo sze of 20% the frequency s always above 50% and ncreases wth the dstance to the true model. When the subportfolo sze s ncreased, the values drop below 50% for some correlaton assumptons. In the next secton, t wll be shown that the correspondng fgures for our test statstc are better. To sum up, as the R subportfolos are not cross-sectonally ndependent the standard test statstcs proposed by Lopez and Sadenberg cannot be used. Ths also holds f 9

10 one tested the complete loss dstrbuton nstead of the 90%-quantle, or ran Mncer- Zarnowtz regressons to examne unbasedness of the forecasted credt losses. Each of the tests proposed by Lopez and Sadenberg requres ndependent draws. One mght thnk of modfyng the test procedure by drawng subportfolos wth replacement. Ths would mtgate some of the problems, but the contemporaneous dependence of subportfolo losses would reman. Another possble soluton would be to remove the dependence by condtonng the forecasts of subportfolo losses on the default experence of those borrowers whch are not ncluded n ths specfc subportfolo. Whle ths mght be a vald and useful procedure n some cases, t would fal to detect false models n others. For example, t would not be possble to dscrmnate between models whch post that asset values are drven by one factor wth a constant factor senstvty w, and whch dffer only n the assumed senstvty w. Another remedy would be to smulate the dstrbuton of the test statstcs n the presence of cross-sectonal dependence. Ths s one element of the alternatve test procedures whch we wll present n the next secton. 4. Evaluatng models based on lkelhoods 4.1 Outlne of the methodology The basc dea of our approach s to examne the lkelhood that a gven default hstory has been generated by a partcular credt portfolo rsk model. Consder a hstory of T annual observatons on the number of defaults n a portfolo. Call ths emprcal dstrbuton h(t). Credt rsk model A produces an expected dstrbuton of the number of annual defaults. For each possble number of defaults x, t specfes p A (x,t), the probablty of the number of defaults beng equal to x n year t. The probablty of observng a default hstory h(t) under model A s thus L A = T t= 1 p A ( h( t), t) (4) Ths holds true f the annual losses are serally ndependent or any exstng dependence s reflected n the model s predcton of losses. 7 A frst nsght nto the ablty of ths lkelhood crteron to dscrmnate between dfferent specfcatons of credt portfolo rsk models can be gven by a smple 10

11 Fgure 1: Smulated frequency of observng lkelhood (true model) > lkelhood (alternatve model) for a one-factor model wth unform senstvtes 100% 75% 50% 25% 0% 0% 2% 4% 6% 8% 10% 12% 14% Correlaton assumpton n alternatve model (true = 5%) smulaton experment: 1. Generate 10, year default hstores based on a one-factor asset value model wth an uncondtonal default probablty of 1% and an asset correlaton of 5% (.e. 2 w = 0.05 for all ). 2. For each of the 10,000 hstores, calculate lkelhoods for dfferent correlaton assumptons whle keepng the default probablty constant. 3. Compute the frequency that the lkelhood of the 5%-correlaton model s larger than that of an alternatve model. Fgure 1 shows ths frequency for varous correlaton assumptons. In several of the mutual comparsons, the fgure s hgher than 75%. When settng a correlaton assumpton of 0% aganst the correct 5%, the observed lkelhood of the latter s n all smulatons larger than that of the former. These results ndcate that the lkelhood crteron s useful for dscrmnatng between alternatve models. Remember that we computed smlar statstcs for the Lopez and Sadenberg quantle test. There, the frequency fgures for a subportfolo sze of 2,000 ranged from 54.9% for the 2.5%- correlaton assumpton to 72.9% for the 20%-correlaton assumpton. Ths compares to 74% for the 2.5% correlaton and 94% for the 15% correlaton n Fgure 1. Whle there s some nformaton n a smple comparson of lkelhoods, t s not yet a test statstc for whch we can assess the power gven some sgnfcance level of the 7 Ths assumpton s tested n secton 5. 11

12 test. We therefore use the lkelhood crteron as part of a test statstc for the comparson of two competng credt portfolo rsk models A and B. Specfcally, we construct the followng lkelhood rato statstc: where LB λ = 2log, (5) LA L A s the observed lkelhood of model A. In standard statstcs, maxmum lkelhood estmaton nvolves settng up a lkelhood functon and maxmzng t for a set of parameters. The lkelhood rato test s used for hypothess testng n ths settng. Imposng a vald restrcton on the unrestrcted maxmum lkelhood should not lead to a large reducton n the log-lkelhood functon. Snce the unrestrcted lkelhood must always be larger than the restrcted lkelhood and both lkelhoods cannot be negatve, the lkelhood rato wll le between 0 and 1. Under regularty condtons, the large sample dstrbuton of the lkelhood rato test statstc λ s ch-squared wth degrees of freedom equal to the number of restrctons mposed. 8 In our settng, there are two sensble ways of usng a lkelhood rato test: 1. We perform an optmzaton wthn a specfed class of credt portfolo rsk models n order to obtan the model parameters whch maxmze the lkelhood (4). Then we test restrctons on ths optmum usng a lkelhood rato test. For example, we mght have another default correlaton estmate derved from equty correlatons (as s done n CredtMetrcs) and would lke to test the null hypothess that ths estmate s true gven a default hstory. 2. We have two sets of parameters specfed separately from the default hstory, and would lke to use the assocated lkelhood to dscrmnate between these parameterzatons. For example, t mght be the case that we nfer dfferent default correlatons from equty data and, alternatvely, from default rate volatltes (as s done n CredtRsk + ). 9 Even f one agreed on usng equty correlatons to nfer 8 Cf. Greene (2000), p Wahrenburg and Nethen (2000) compare Value-at-rsk estmates of CredtMetrcs and CredtRsk + n a two-state settng (default / no default) for the German constructon ndustry. Default correlatons are estmated from equty correlatons for CredtMetrcs, whle they are nferred from default volatltes for CredtRsk +. The 5%-value-at-rsk estmates dffer by a factor up to 3 dependng on the number of oblgors. Cf. also Crouhy, Gala and Mark (2000) and Koyluoglu, Banga, and Garsde (1999). 12

13 Table 3: Model specfcatons Case 1 Model A (taken to be the true model) A = vz + 1 v 2 ε Model B (representng the H 0 -hypothess) A = wz + 1 w 2 ε Case 2 A = v Z + 1 v ε, 2 v =v for 5000, v =v for > A = w Z + 1 w ε, Case 3 A = vz1 + 1 v 2 ε A = 3 3 Σ wk Z k + 1 Σ k = 2 k= 2 w 2 k ε w 1 = w I 5000, w 2 = w I >5000 default correlatons, the use of dfferent tme perods or return frequences could produce conflctng parameter estmates. When conductng such mutual comparsons, t s useful but not necessary to start wth an a-pror assumpton on whch of two rval models s the better one. An obvous crteron for ths decson s the observed lkelhood,.e. one would test the null hypothess that the model wth the smaller lkelhood s correct. For such a test, t s not necessary that the two alternatve models are nested n each other. An mportant characterstc of many credt rsk portfolo models s that the models parameters are not estmated usng a hstory of the varable whch s to be predcted. Credt rsk models predct the dstrbuton of losses, whle the parameters are estmated, for example, based on equty returns (n the case of CredtMetrcs), or equty prces and volatltes (n the case of the KMV model). Thus, we beleve that evaluators of credt rsk models often face a stuaton n whch they have to choose between alternatve a-pror specfcatons. In addton, the models under analyss may exhbt structural dfferences whch complcate the standard testng procedure. For these reasons, we focus on the second way of constructng a test, and examne the frst only n the course of the senstvty analyses n secton 5. If we do not maxmze the lkelhood functon n the frst place, the test statstc, even though t s a rato of lkelhoods, does not belong to the class of lkelhood rato tests used n standard statstcs. In consequence, the lkelhood rato statstc wll not be 13

14 dstrbuted as a ch-squared varable. 10 However, we can smulate the dstrbuton of the test statstc under the null hypothess and then work wth ths statstc as we do wth standard ones. In partcular, we can freely choose a sgnfcance level, and assess the power of the test. In the base case settng, the constructon of the test nvolves two steps (cf. Fgure 2): 1. Set up the null hypothess: parameters = parameter set of model B 2. Calculate the lkelhood rato statstc for 10,000 scenaros of 10-year hstores under H 0 (= Model B), and compute the (1-α)-quantle of ths dstrbuton. In an applcaton of the test, the null hypothess s rejected when the test statstc exceeds ths quantle. Thus, the type I error, whch s the probablty of rejectng the null hypothess when t s true, equals α. The power of the test can be assessed as follows: 3. Calculate the lkelhood rato statstc (5) for 10,000 scenaros of 10-year hstores under model A and compute the frequency that the test statstc under model A exceeds the (1-α)-quantle smulated n step 2. Ths s the power of the test f model A s correct. The type II error, whch ndcates the lkelhood of not rejectng the null hypothess when t s false, s equal to one mnus the power. Fgure 2: Illustraton of the lkelhood rato test statstc 7% 6% 5% 4% 3% λ smulated under H 0 (=model B) λ smulated under model A power of test gven α 2% 1% α quantle 0% Cf. Greene (2000), p

15 4.2 Test specfcatons and ther power In ths secton, we wll suggest varants of the lkelhood rato test ntroduced above, and determne ther power for selected specfcatons of factor models (see Table 3). In each case, we wll take alternatve A to be the correct model, and test the hypothess that model B s correct. We start by comparng two one-factor models wth dfferent senstvtes towards the common rsk factor (Case 1). We contnue by allowng two dfferent senstvtes towards the common rsk factor (Case 2). Here, cross-sectonal dfferences between the two rval models provde addtonal nformaton. Fnally, a one-factor model s compared wth a two-factor model, so that there are three dfferent rsk factors (Case 3). Agan, there s nformaton n the cross-secton. 11 Throughout, we do not test alternatve assumptons on uncondtonal default rates, but only assumptons on correlaton parameters. The applcablty of our results s not affected by ths restrcton. In practce, there wll most probably be more than one combnaton of default rates and correlaton assumptons whch results n a default hstory wth an equally hgh probablty, especally n large nhomogeneous portfolos wth many rsk factors. Case 1: The base case We start by assumng that the observed default hstory s best explaned by a onefactor asset value model wth an uncondtonal default rate of 1% and a constant mutual asset correlaton of 5% (model A). We test dfferent null hypotheses by changng the correlaton parameter (model B). The test statstc s the lkelhood rato gven n (5). Fgure 3 shows the smulated power of our test statstc n the base case,.e. the test s based on a 10-year hstory of a portfolo wth 10,000 borrowers. We choose a 10% (one-sded) sgnfcance level for the test. If the false model posts a zero default correlaton, the null hypothess s rejected n 100% of all cases. For models whch are close to the correct 5%, the power s lower. However, t s larger than 50% f the assumed correlaton s below 3% or above 9%. 11 For descrptve statstcs of smulated expected loss dstrbutons see appendx. 15

16 Fgure 3: Power of test statstc n Case 1 (sze of test=10%) 100% 75% 50% 25% 0% 0% 3% 5% 8% 10% 13% 15% Correlaton assumpton n model B (true = 5% = model A) It s nstructve to compare our test statstc s power to the power of an analogous test n a market rsk setup. Consder the stuaton n whch we compare two market rsk models (model A and model B) whose expected loss dstrbutons are normal wth zero mean and 90%-quantles (10% Value at Rsk) equal to those of the credt rsk models used n constructng Fgure 3. Then we perform a test that Model B correctly predcts the rsk of the portfolo gven that the portfolo returns are generated by model A. The most powerful test wll be based on the sample portfolo varance, and wll use the fact that the sample varance dvded by the true varance follows a chsquared dstrbuton. (In the test we assume that the mean s known.) Fgure 4 shows the result of ths comparson, usng a one-sded test and varyng the number of observatons avalable for the test. As can be seen, our test statstc s power s smlar to a market rsk setup n whch a test s based on 60 observatons. Whle the power wth 250 observatons s far better, t s consderably lower wth 10 observatons - even though our test also uses only 10 observatons Ths can be explaned by notng that the normal dstrbuton s the dstrbuton wth maxmum entropy. For a gven number of observatons, t s thus easer to dscrmnate between dfferent credt models as more dstrbutonal assumptons are mposed. 16

17 Fgure 4: Power of dscrmnatng credt portfolo models n Case 1 and comparable normally dstrbuted portfolos (sze of test = 10%) 100% Credt portfolo Market portfolo wth NOB= % 50% 25% 0% 0% 3% 5% 8% 10% 13% 15% Correlaton assumpton n model B (true = 5% = model A) Case 2: Explotng uncondtonal cross-sectonal dfferences So far, we used only the aggregate annual defaults to construct a test. Ths wll not be effcent f there s addtonal nformaton n the cross-secton of the data. Consder two models whch exhbt an dentcal dstrbuton of overall defaults, yet dffer n the predcton about the number of defaults n subsets of the portfolo. The lkelhood crteron would not be able to dscrmnate between the two models because the lkelhoods based on the total portfolo defaults would be the same. As an llustraton we analyze a portfolo whose defaults are drven by a one-factor asset value model wth an uncondtonal default rate of 1%. One half of the oblgors belongs to a sector wth an ntra-sector correlaton of 2%, the other half to a sector wth an ntra-sector correlaton of 9%. (The nter-sector correlaton s 4.2%). If we take ths setup to be model A and compare t wth an alternatve one-factor model B whch assumes a common factor senstvty, the power curve, whch s shown n Fgure 5, looks very smlar to the one n whch model A s a one-factor model wth 5% correlaton (cf. Fgure 3). Ths s due to the fact that the aggregate expected loss dstrbutons of these two A-models are nearly dentcal n ther frst and second moments and do not dffer much n hgher moments, even though the sector portfolo dstrbutons dffer. As a consequence, f the null hypothess posts a unform 17

18 Fgure 5: Power of test statstc n Case 2 usng the lkelhood rato for aggregate defaults (sze of test = 10%) 100% 75% 50% 25% 0% 0% 3% 5% 8% 10% 13% 15% Common correlaton assumpton n model B (true = 2% for <5000 and 9% else = model A) correlaton of 5% the power s only 16%, lttle more than the sze of the test. One possble way of explotng the cross-sectonal nformaton s to utlze the dea of Lopez and Sadenberg and apply the lkelhood-crteron (5) to randomly drawn portfolo subsets. Ths would not make effcent use of the nformaton, though. The man dsadvantage of drawng the subportfolos randomly s that we hardly ever get extreme subportfolo compostons. Consder the present settng,.e. Case 2 n whch a large portfolo conssts of two sectors, each of whch represents half of the portfolo. If we draw a large number of reasonably large subportfolos (say, wth 2,000 borrowers each), the probablty that we obtan at least one subportfolo whch conssts only of companes of one sector s extremely low. 13 If the null hypothess s a common correlaton of 5%, t s these extreme portfolo compostons whch have the greatest nformatonal value for our purpose. The more evenly mxed a subportfolo s, the more smlar are the two models under comparson. Even f we obtan some extreme portfolo compostons through resamplng, ther nformatonal value wll be lost by averagng across all subportfolos. As a consequence, randomly drawng subportfolos s unlkely to yeld a sgnfcant ncrease of power. A more effcent way of tacklng the problem s to dvde the portfolo nto the extreme 13 Consder a portfolo of 10,000 oblgors, one half of whch belongs to one sector, the other half to another. Drawng a subportfolo of 2,000 oblgors wthout replacement, the probablty that all oblgors belong to sngle sector s lower than By contrast, the probablty of an even mxture of sectors s 2%. 18

19 Fgure 6: Power of test statstc n Case 2 usng composte lkelhood ratos for aggregate defaults and two subportolos (sze of test = 10%) 100% 75% 50% 25% 0% 0% 2% 4% 6% 8% 10% 12% 14% Common correlaton assumpton n model B (true = 2% for <5000 and 9% else = model A) subportfolos and calculate the test statstc for each of them. Thus, f we have a twosector portfolo and assume that companes of these two sectors have dfferent senstvtes towards a common systematc factor, we form two subportfolos consstng of just one sector and proceed as though we were to test models on two dfferent portfolos. Ths produces an aggregaton problem. We propose to calculate the lkelhood rato test statstc for the entre portfolo ( λ Total extreme subportfolos ( λ Sub port optmal weght for each of these fgures: 1λTotal + a2λsub port(1) + a3 Sub port (2) ) as well as for the ), and then use numercal procedures to determne the λ = a λ (7) We optmze the weghts a by maxmzng the frequency that the lkelhood of model A s larger than the lkelhood of model B under 10,000 smulated hstores of model A. Afterwards we proceed as n the base case usng the composte lkelhood rato test statstc (7) nstead of the smple statstc (5). The hstores we use to assess power are dfferent from the ones used for optmzng the weghts. Applyng ths methodology to our example of a one-factor model A wth two ntrasector correlatons of 2% and 9%, we are able to enhance the power of the test. The power, depcted n Fgure 6, s close to 100% for every specfcaton of model B. The reason for ths substantal mprovement s that the two correlaton parameters of model A are suffcently dfferent from each other. When we set a model wth 1% correlaton aganst model A, for nstance, correlaton assumptons n the frst subportfolo are 1% and 2%, respectvely, yeldng a low power. Yet, n the second 19

20 Fgure 7: Standardzed optmal weghts for the composte test statstc n Case 2 (λ=a 1 λ Total + a 2 λ Sub-portfolo 1 + a 3 λ Sub-portfolo 2 ; sze = 10%) 100% 75% 50% 25% 0% 0% 2% 4% 6% 8% 10% 12% 14% Correlaton assumpton n model B (true = 2% for <5001 and 9% for >5000 = model A Subportfolo 1 (<5001) Subportfolo 2 (>5000) Total subportfolo we compare two models wth 1% and 9% correlaton, respectvely. Here, the power s large. After optmzng the weghts of the two subportfolo statstcs, we wll completely rely on the second subportfolo for the purpose of dscrmnatng between the two models. Fgure 7 shows the weghts of the three components of the composte test statstc whch yelded the result shown n Fgure 6. The optmal weghts are multpled by the standard devaton of the respectve components n order to facltate nterpretaton. The pattern of the optmal weghts s plausble. For example, when the correlaton assumpton of model B s equal to or less than 2%, the nformaton value of the subportfolo wth a model A correlaton of 9% s clearly domnant to the subportfolo wth a model A correlaton of 2%. For ths reason, the weght of the latter s equal to zero. The opposte holds when the model B correlaton s above 9%. In between, the weght of the frst subportfolo (model A correlaton = 2%) s ncreasng. The optmal weght of the total portfolo statstc always s close to zero, meanng that aggregate defaults do not contan useful nformaton n addton to the subportfolos. One mght expect the total statstc to receve a larger weght as the larger portfolo sze reduces nose n the observed defaults. In the chosen settng, however, ths effect s small. As wll be shown n secton 5, reducng the number of borrowers from 10,000 to 5,000 leads only to a mnor decrease of power. 20

21 Case 3: Explotng condtonal cross-sectonal dfferences Consder the case that model A s a one-factor model wth a constant senstvty v across all oblgors and that model B s a two-factor model wth constant senstvtes w 1 = w I 5000 and w 2 = w I >5000. Thus, model B does not have a common factor. If v equals w, factor senstvtes are dentcal for both models. As a result, the uncondtonal sector portfolo dstrbutons are also dentcal. If we calculated subportfolo test statstcs as before, we would compare two models wth a correlaton of w², and therefore could not extract any nformaton from the statstc. The pcture changes as soon as we condton the lkelhood crteron on the other sector s default nformaton. For model A, condtonng provdes worthwhle nformaton, because both sectors depend on a common rsk factor. For model B, condtonng does not change the predcted loss dstrbuton because the two sectors are not subject to a common factor. To explot ths dfference, we use condtonal probabltes to compute the lkelhood rato statstc. In equaton (4) we multpled the probabltes p A (x,t), the probablty of the number of defaults beng equal to x n year t under credt rsk model A for t=1,..., T. Now we use the condtonal probablty p A (x 1,t x 2 ) of observng x 1 defaults n sector 1 gven that x 2 borrowers have defaulted n sector 2. In the framework of ths paper, ths condtonal probablty s the bnomal densty of the number of defaults n sector 1 beng equal to x 1 n year t when the subportfolo sze equals y n both sectors, and the condtonal default probablty equals x 2 /y, That s, the condtonal default probablty s equal to the observed default rate n sector 2. The condtonal lkelhood of observng a default hstory h 1 (t) for subportfolo 1 under model A s: L C A = T t= 1 p A ( h1 ( t), t h2 ( t)), (8) C C C and the test statstc for a subportfolo s λ = 2log( L / L ) B A. Smlar to the prevous secton, we aggregate the total, uncondtonal lkelhood rato and the condtonal subportfolo ratos: λ = λ + λ λ (9) a C C 1 Total a2 a Sub port ( 1) 3 Sub port ( 2 ) + 21

22 Fgure 8: Power of test statstc n Case 3 usng composte lkelhood ratos for aggregate defaults and two subportolos (sze of test = 10%) 100% 75% 50% 25% 0% 0% 2% 4% 6% 8% 10% 12% 14% Intra-sector correlaton n model B (true common correlaton = 5% = model A) The weghts are chosen to maxmze the probablty that the lkelhood of model A s larger than the lkelhood of model B under the smulated hstory of model A. Fgure 8 shows that the use of condtonal lkelhoods can be very useful for dscrmnatng between models. We vary the factor senstvtes n model B, and hold the correlaton n model A constant at v²=5%. In each case, the power of the aggregate test statstc (9) s 100%. 4.3 Summary of the test procedure In general, we propose the followng procedure for comparng two credt portfolo rsk models, A and B. Examne the models to determne subportfolos where dfferences n the uncondtonal or condtonal loss dstrbuton are large. The test statstc aggregates lkelhood ratos of the two models, λ = 2log( L / ) B L A, computed separately for the total portfolo and the chosen subportfolos, and usng uncondtonal or condtonal lkelhoods: C C C λ = a λtotal + a2λ + a3λ amλ + am+ 1λ + amλ (10) Sub port ( 1) Sub port (2) Sub port ( m ) Sub port ( m+ 1) Sub port ( M ) The weghts are chosen to maxmze the probablty that the lkelhood of model A s larger than the lkelhood of model B under the smulated hstory of model A. Use the optmzed weghts to smulate the dstrbuton of λ under model B, whch s the null hypothess of the test. Choose a sgnfcance level α. For a gven data set, estmate λ. The hypothess s rejected f the emprcal λ exceeds the (1-α) quantle of the 22

23 Fgure 9: Power of test statstc n Case 1 for default hstores wth dfferent numbers of oblgors (sze = 10%) 10,000 oblgors (base case) 5,000 oblgors 1,000 oblgors 100% 75% 50% 25% 0% 0% 2% 4% 6% 8% 10% 12% 14% Correlaton assumpton n model B (true = 5% = model A) smulated dstrbuton. In the frst step, the descrpton of the test procedure s vague n the sense that there s no precse recommendaton on the nature and number of the chosen subportfolos. In prncple, one could choose all possble subportfolos or a random subset thereof, and optmze ther weghts n the aggregate test statstc. However, the assocated computng tme s lkely to be prohbtve. We do not regard ths as a serous lmtaton of the test procedure as the choce of subportfolos wll be evdent n many cases. In addton, choosng wrong subportfolos wll not decrease the power relatve to the benchmark test based solely on the aggregate portfolo dstrbuton. 5 Addtonal analyses In the prevous secton, we llustrated the power of lkelhood rato tests for varous credt portfolo rsk models whch dffered n ther correlaton assumptons. To put these results nto perspectve, we now vary the general settng of these tests, whch was held constant. Specfcally, we estmate the power of the test n case 1, where two one-factor models wth dfferent senstvtes are set aganst each other. The followng results should thus be compared to Fgure 3, whch shows the probablty of rejectng a model wth correlaton x% when the true model has 5% correlaton. We start by assessng the power when the portfolo under analyss contans only 1,000 or 5,000 borrowers nstead of 10,000. Fgure 9 shows that the power s lower when the portfolo sze s reduced, as should be expected. However, the loss of 23

24 Fgure 10: Power of the test statstc n case 1 varyng the sze of the test and the number of years n the default hstory 10 years, sze=10% (base case) 5 years, sze=10% 10 years, sze=5% 100% 75% 50% 25% 0% 0% 2% 4% 6% 8% 10% 12% 14% Correlaton assumpton n model B (true = 5% = model A) power s farly small when the number of borrowers s 5,000 nstead of 10,000. Wth 1,000 borrowers, the power s stll above 75% n some cases. The power also decreases when we lower the sze of the test, or reduce the number of years n the default hstory. Fgure 10 compares the power n the base case to the one when the chosen sze of the test s 5% nstead of 10%, or when the number of years n the observed default hstory s 5 nstead of 10. In general, reducng the number of years has a larger effect. Stll, even wth only 5 years of data the power appears to be satsfactory. The fnal two varatons assess the mportance of nformaton we presupposed n the constructon of the test statstc. We assumed that the predcton errors of the varous credt rsk models are ndependent across tme. Ths s a vald assumpton f there s no seral dependence n defaults, or credt rsk models effcently use avalable nformaton n the predcton of defaults. If these condtons are not met, the performance of the tests mght deterorate. We therefore modfy the base case by ntroducng autocorrelaton nto the tme seres of the systematc factor Z. In smulatng the default hstores, we use the followng autoregressve process for Z t : Z t = 0.5 Z t u t, u~ N(0,1), Z 1 ~ N(0,1) (11) The choce of 0.5 for the autocorrelaton coeffcent s based on the study of Belkn, 24

25 Fgure 11: Power of the test statstc n case 1 n the presence of autocorrelaton and absent knowledge of the true model (sze of test=10%) base case autocorrelated defaults ML estmate for model A 100% 75% 50% 25% 0% 0% 3% 6% 9% 12% 15% Correlaton assumpton n model B (true = 5% ) Suchower and Forest (1998a), who estmate such a process usng annual transton matrces and obtan a autocorrelaton coeffcent of We then test the power of the test as before, assumng that both models A and B do not use the tme seres nformaton to predct next year s factor realzaton. Another pece of nformaton we presupposed was that the rval model A s the true one. In a practcal settng, ths knowledge wll not be gven. We thus follow the frst way of settng up a lkelhood test whch we descrbed n secton 4.1. Instead of comparng model B to model A, the true model, we estmate the parameters of the model based on maxmum lkelhood, and test whether model B s a vald restrcton. Thus, we construct a standard lkelhood rato test. As before, we smulate ts dstrbuton because we cannot rely on the test beng dstrbuted as a ch-squared varable. Maxmzaton s done through a smple search procedure n whch we evaluate the lkelhood for each correlaton assumpton n the nterval [0%, 1%,..., 50%]. The power curves under these two varatons are shown n Fgure 11. If both the models under analyss and the test neglects seral correlaton n defaults, the power does not change sgnfcantly relatve to the base case. Ths suggests that autocorrelaton does not pose a major problem for the test proposed n ths paper. If the evaluator does not know the true model, the power goes down as well, but only modestly. To sum up, the power calculatons of secton 4 do not appear to depend crucally on the chosen settng. 25

26 6 Concludng remarks We have descrbed a procedure for evaluatng rval credt rsk models. In essence, we test whether one model s more lkely to have generated the data than another alternatve model. Monte Carlo smulatons show that the power of the tests s satsfactory. Wth ten years of annual data, for example, t s smlar to a market rsk settng where two rsk models are compared based on 60 observatons. A test should meet other crtera than a large power, for nstance ease of mplementaton and general applcablty. The use of the test nvolves smulatons, but the assocated computng tme s lkely to be smaller than when mplementng the proposal of Lopez and Sadenberg (2000). The smplest form of the test, whch s based only on aggregate defaults, provdes a benchmark whch s generally applcable. To explot addtonal nformaton contaned n the cross-secton of losses, we propose to aggregate the test statstcs of judcously chosen subportfolos. Thus, there s no general rule for the desgn of the test. We do not regard ths as a serous shortcomng as the choce of the most mportant subportfolos wll be straghtforward n many cases. Note, too, that the test procedures can easly be appled to models whch nclude mgraton and recovery rsk, and that the models need not be nested. Another possble crtcsm s that the test s based on the entre range of the dstrbuton, whereas rsk managers and regulators are manly concerned about the probablty of extreme events. There are two arguments aganst focusng on the rght tal of the dstrbuton when constructng a test. Frst, we observe only few of these rare events n the data, a problem even sophstcated smulaton procedures cannot overcome. Second, dfferences n the tals of two dstrbutons wll go along wth predctable dfferences n the rest of the dstrbuton. If default correlaton s ncreased, for example, the probablty of catastrophe losses rses, but so does the probablty of very small losses. 26

27 References Belkn, B., Suchower, S., Forest, L.R. Jr., 1998a. A one-parameter representaton of credt rsk and transton matrces. CredtMetrcs Montor, Thrd Quarter, Belkn, B., Suchower, S., Forest, L.R. Jr., 1998b. The effect of systematc credt rsk on loan portfolo value-at-rsk and loan prcng. CredtMetrcs Montor, Frst Quarter, Berkowtz, J., Evaluatng the forecasts of rsk models. Workng paper, Federal Reserve Board. Carey, M., Hrycay, M., Parameterzng credt rsk models wth ratng data. Journal of Bankng and Fnance 25, Crnkovc, C., Drachman, J., Qualty control. Rsk 9, No 9, Crouhy, M., Gala, D., Mark, R., A comparatve analyss of current credt rsk models. Journal of Bankng and Fnance 24, Fnger, C.C., Condtonal Approaches for CredtMetrcs Portfolo Dstrbutons. CredtMetrcs Montor, Frst Quarter, Fnger, C.C., Stcks and stones. Workng paper, The RskMetrcs Group, New York. Gordy, M., A comparatve anatomy of credt rsk models. Journal of Bankng and Fnance 24, Greene, W.H., Econometrc analyss. 4 th Internatonal Edton, Upper Saddle Rver, New Jersey. J.P. Morgan, CredtMetrcs Techncal document, New York. Koyluoglu, H.U., Banga, A., Garsde, T., Devl n the parameters. Workng Paper, Olver, Wyman & Company, New York Koyluoglu, H.U., Hckman, A., Reconclable dfferences. Rsk 11, No 10, Löffler, G., The effects of estmaton error on measures of portfolo credt rsk. Workng paper, Unversty of Frankfurt (Man). Lopez, J.A., Sadenberg, M.R., Evaluatng credt rsk models. Journal of Bankng and Fnance 24,