Analytic Models of the ROC Curve: Applications to Credit Rating Model Validation

Transcription

1 QUANTITATIVE FINANCE RESEARCH CENTRE QUANTITATIVE FINANCE RESEARCH CENTRE Research Paper 8 August 2006 Analtic Models of the ROC Curve: Applications to Credit Rating Model Validation Stephen Satchell and Wei Xia ISSN

2 Analtic Models of the ROC Curve: Applications to Credit Rating Model Validation Steve Satchell Facult of Economics and Politics Universit of Cambridge Wei Xia Birkbeck College Universit of London August,2006. Abstract: In this paper, the authors use the concept of the population ROC curve to build analtic models of ROC curves. Information about the population properties can be used to gain greater accurac of estimation relative to the non-parametric methods currentl in vogue. If used properl this is particularl helpful in some situations where the number of sick loans is rather small; a situation frequentl met in periods of benign macro-economic background. Kewords: Validation, Credit Analsis, Rating Models, ROC, Basel II Acknowledgement: Wei Xia would like to thank Birkbeck College for the generous funding support and Ron Smith for helpful comments. Introduction

3 The Internal Rating-Based (IRB) credit risk modelling approach is now allowed for qualified banks in their risk modelling and economical capital calculation, after the Basel Committee on Banking Supervision (BCBS) published the International Convergence of Capital Measurement and Capital Standards: A Revised Framework, more commonl known as Basel II, in June One of the important components of most IRB risk models is the rating sstem used for transforming and assigning the Probabilit of Default (PD) for each obligor in the credit portfolio. Banks and public rating agencies have developed a variet of rating methodologies in the last three decades. Therefore, questions arise as to which of these methodologies deliver acceptable discriminator power between the defaulting and non-defaulting obligor e ante, and which methodologies are to be preferred for different obligor sub-groups. It has become increasingl important for both the regulator and the banks to quantif and judge the qualit of rating sstems. This concern is reflected and stressed in the recent BCBS working paper No.4 (2005). This summarizes a number of statistical methodologies for the assessment of discriminator power which have been suggested in the literature. For eample, Cumulative Accurac Profile (CAP), Receiver Operating Characteristic (ROC), Baesian error rate, Conditional Information Entrop Ratio (CIER), Kendall s τ and Somers D, Brier score inter alia. Among those methodologies, the most popular ones are CAP and its summar inde, the Accurac Ratio (AR) as well as ROC and its summar inde; this inde is called the Area Under the ROC (AUROC). It is worth noting that, unlike some other measures that do not take sample size into account and are therefore substantiall affected b statistical errors, the CAP and the ROC measures eplicitl account for the size of the default sample and thus can be used for direct rating model comparison. A detailed eplanation of the CAP is presented in Sobehart, Keenan and Stein (2000), and Sobehart and Keenan (2004). ROC has been long used in medicine, pscholog and signal detection theor. There is a large bod of literature that analses the properties of the ROC curve. Bamber (975) shows the AUROC is related to the 2

4 Mann-Whitne Statistic and several different methods for constructing the confidence intervals are also discussed. An overview of possible applications of the ROC curves is given b Swets (988). Sobehart and Keenan (200) introduced the ROC concept to internal rating model validation. The focus on the calculation and interpretation of the ROC measure. Engelmann, Haden and Tasche (2003) showed that AR is a linear transformation of AUROC and the complement Sobehart and Keenan (200) with more statistical analsis of the ROC. However, previous works that we are familiar with, have been done using a finite sample of empirical data or simulated data. None of the above have analzed the analtic properties of the ROC Curve and the ROC measure under parametric assumptions concerning the distribution of the rating scores. In this paper, we further eplore the statistical properties of the ROC Curve and its summar indices, especiall under a number of rating score distribution assumptions. We focus on the analtical properties of ROC Curve onl since the CAP measure is just a linear transformation of the ROC measure. In section 2, in order to keep this paper self-contained, we first briefl introduce the credit rating model validation background and eplain the concepts and definitions of ROC and CAP. A general equation for the ROC Curve is derived. B assuming that there eists probabilit densit functions of the two variables that construct the ROC Curve, an unrestrictive assumption, we show that there is a link between the first derivative of the curve and the likelihood ratio of the two variables, a result derived b different methods in Bamber(975). In section 3, b further assuming certain statistical distributions for the credit rating scores, we derive analtic solutions for the ROC Curve and its summar indices respectivel. In particular, when the underling distributions are both Negative Eponential distributions, we have a closed form solution. 3

5 In section 4, we appl the results derived in section 3 to simulated data. Performance evaluation reports are presented. Section 5 concludes: We find that estimation results b our analtic approach are as good as and, in man cases, better than the non-parametric AUROC ones. Although the accurac of the approach in this paper is limited b the continuous rating score assumption and also affected b the accurac of estimation of the distribution parameters on rating score samples in some cases, it offers direct insight into more comple situations and a better tool in credit rating model selection procedure since the analtic solution can be used as object function. 2 Theoretical Implication and Applications: In this section, we first briefl review the credit rating sstem methodolog, in particular, the CAP and the ROC measures. The content presented in this part is ver similar to Engelmann, Haden and Tasche (2003) and BCBS working paper No.4 (2005). Then we introduce the Ordinar Dominance Graph (ODG) where ROC is a special case of ODG and some interesting theoretical implications of the ROC curve will be given. 2. The validation of credit rating sstem. The statistical analsis of rating models is based on the assumption that for a predefined time horizon there are two groups of a bank s obligor: obligors that will be in default, called defaulters, and obligors that will not be in default, called non-defaulters. It is not observable in advance whether an obligor is a defaulter or a non-defaulter in the net time horizon. Banks have loan books or credit portfolio, the have to assess an obligor s future status based on a set of his or her present observable characteristics. Rating sstems ma be regarded as classification tools to provide signals and indications of the obligor s possible future status. A rating score 4

6 is returned for each obligor based on a rating model, usuall an Epert Judgment Model. The main principal of rating sstems is that the better a grade, the smaller the proportion of defaulters and the greater the proportion of non-defaulters that are assigned this grade. Some eamples are the famous Altman s Z score, or some score from a Logit model or from other approach. Therefore, the qualit of a rating sstem is determined b its discriminator power between non-defaulting obligors and defaulters e ante for a specific time horizon, usuall a ear. The CAP measure and ROC provide statistical measures to assess the discriminator power of various rating models based on historical data. 2.2 Cumulative Accurac Profile and Accurac Ratio Consider an arbitrar rating model that produces a rating score. A high rating score is usuall an indicator of a low default probabilit. To obtain the CAP curve, all debtors are first ordered b their respective scores, from riskiest to safest, i.e. from the debtor with the lowest score to the debtor with the highest score. For a given fraction of the total number of debtors the CAP curve is constructed b calculating the percentage d() of the defaulters whose rating scores are equal to or lower than the maimum score of fraction. This is done for ranging from 0% to 00%. Figure illustrates CAP curves. 5

7 Figure. Cumulative Accurac Profile Fraction of defaulters a P a R Perfect Model Rating Model Random Model 0 Fraction of all obligors Apparentl a perfect rating model will assign the lowest scores to the defaulters. In this case the CAP is increasing linearl and than staing at one. For a random model without an discriminative power, the fraction of all debtors with the lowest rating scores will contain percent of all defaulters. Real rating sstem lies somewhere in between these two etremes. The qualit of a rating sstem is measured b the Accurac Ratio (AR). It is defined as the ratio of the area ar between the CAP of the rating model being validated and the CAP of the random model, and the area a P between the CAP of the perfect rating model and the CAP of the random model. a AR = a R P It is eas to see that for real rating models the AR range from zero to one and the rating model is the better if AR is closer to one. 2.3 Receiver Operating Characteristic and the Area Under the ROC curve The construction of a ROC curve is illustrated in Figure 2 which shows possible 6

8 distributions of rating scores for defaulting and non-defaulting debtors. For a perfect rating model the left distribution and the right distribution in Figure 2 would be separate. For real rating sstems, perfect discrimination in general is not possible. Both distributions will overlap as illustrated in Figure 2. Figure 2: Distribution of rating scores for defaulting and non-defaulting debtors Assume someone has to find out from the rating scores which debtors will survive during the net period and which debtors will default. One possibilit for the decision-maker would be to introduce a cut-off value C as in Figure 2, and to classif each debtor with a rating score lower than C as a potential defaulter and each debtor with a rating score higher than C as a non-defaulter. Then four decision results would be possible. If the rating score is below the cut-off value C and the debtor defaults subsequentl, the decision was correct. Otherwise the decision-maker wrongl classified a non-defaulter as a defaulter. If the rating score is above the cut-off value and the debtor does not default, the classification was correct. Otherwise a defaulter was incorrectl assigned to the non-defaulters group. Then one can define a hit rate HR(C) as: ( ) HR C ( ) H C = N where H(C) is the number of defaulters predicted correctl with the cut-off value C, D 7

9 and N D is the total number of defaulters in the sample. This means that the hit rate is the fraction of defaulters that was classified correctl for a given cut-off value C. The false alarm rate then FAR(C) is defined as: ( ) FAR C ( ) F C = N where F(C) is the number of false alarms, i.e. the number of non-defaulters that were classified incorrectl as defaulters b using the cut-off value C. The total number of non-defaulters in the sample is denoted b N. In Figure 2, HR(C) is the area to the left of the cut-off value C under the score distribution of the defaulters (coloured plus hatched area), while FAR(C) is the area to the left of C under the score distribution of the non-defaulters (chequered plus hatched area). ND ND The ROC curve is constructed as follows. For all possible cut-off values C that are contained in the range of the rating scores the quantities HR(C) and FAR(C) are computed. The ROC curve is a plot of HR(C) versus FAR(C). This is illustrated in Figure 3. A rating model s performance is the better the steeper the ROC curve is at the left end and the closer the ROC curve s position is to the point (0,). Similarl, the model is the better the larger the area under the ROC curve is. This area is called AUROC and is denoted b A. B means of a change of variable, it can be calculated as A = 0 HR( FAR) d( FAR) The area A is 0.5 for a random model without discriminative power and it is.0 for a perfect model. It is between 0.5 and.0 for an reasonable rating model in practice. 8

10 Figure 3: Receiver Operating Characteristic Curves. (Taken from Tasche. (2003)) It has been shown in Engelmann, Haden and Tasche (2003) that: ( A 0.5) a N R ND AR= = = 2( A 0.5) = 2A a 0.5N P ND 2.4 Some further statistical properties of ROC measures The ROC stems from the Ordinal Dominance Graph (ODG). Assume we have two sets of continuous random variables, X and Y. Let C be an arbitrar constant. Define: ( ) ( ) and ( ) ( ) = prob Y C = F C = prob X C = F C Y, where and lie between [0, ] and C lies in (, + ). Then the ODG is simpl a plot of against. See Figure 4. There are some straight forward properties of the ODG: ) The ODG curve is never decreasing as increases cannot decrease. 2) If Prob(Y C) = Prob(X C), then and are identicall distributed, = and the ODG curve is a 45 line. X 9

11 3) If X first order stochastic dominance (FSD) Y, then the ODG curve lies above the 45 line and vice versa. Proof: ( ) ( ) ( ) If X FSD Y F C F C, C X, lies above the 45 line Y If we regard as score signals of defaulters in the net predefined period and as those of the non-defaulters, then we epect an sensible rating sstem produces Prob(Y C) Prob(X C) for all C. Thus for all C and the ODG curve is above the 45 line. It is referred to as the ROC curve in the risk literature. In terms of section 2.3, is the HR(C) and is the FAR(C). B assuming there eists probabilit densit functions (PDF) of F X and F Y, i.e. that the are both absolutel continuous, the following lemma can be derived: Lemma : If = F ( C), C F ( ) X C = X, then = f C X ( ), where f ( ) X C is the PDF of X.. Proof: ( ) ( ) FX C FX C C C C = =. = fx ( C), = C f C X ( ) Lemma 2: fy C = Y, then = f C If F ( C) X ( ) ( ). We see from Lemma 2 that the slope of the ODG curve is just the likelihood ratio (LR) of Y and X evaluated at C. Proof: ( ) ( ) ( ) ( ) FY C FY C C C fy C = =. = fy ( C). = C f C X 0

12 Theorem: If If f ( C) ( ) Y f X C is increasing in C, then f ( C) ( ) Y f X C is decreasing in C, then is increasing the ODG curve is conve. is decreasing the ODG curve is concave. The later case is the one that we are interested in, as it is the ROC curve. See Figure 4. Figure 4: Ordinal Dominance Graph Concave Conve 0 We have assumed that X and Y have PDF s, thus the are continuous random variables. The AUROC can then be epressed as: A = Prob( ) = Prob ( = ) Prob( = ) Y X X C X C dc Since X and Y are scores from different obligor groups, the are independent. We have: Prob ( Y X X = C) = Prob( Y C). Since = F ( C) = Prob( Y C) and ( ) Y = f C C A = FY ( C) fx ( C) dc= Y X ( ) 0 A modelling eercise ma choose a rating model that maimizes the AUROC with X ( ) F F d ()

13 respect to the obligor group under review. But how would one estimate Prob( )? Bamber(975) and Engelmann, Haden and Tasche (2003) show that given a rating scores sample of the obligors assigned b a rating model, the AUROC could be estimated in an non-parametric wa using the Mann-Whitne U statistic. On the other hand, if we had a parametric distribution model for X and Y, then we could eplicitl derive a formula for the ROC curve and for the AUROC. In the net section, we will review some plausible distributions for X and Y and derive the close form solutions wherever possible. 3 Choices of Distributions In this section, we derive analtical formulae for the ROC b assuming that the rating scores produced b rating models follow some plausible distributions. The distributions we presented here are Weibull Distribution (including Eponential Distribution), Logistic Distribution, Normal Distribution and Mied models for X and Y respectivel. In the cases that we have eplicit closed forms for the ROC curve, we derive the closed form AUROC as well. The case of mied distributions for X and Y can be easil etended from the discussion followed. We use the smbol M for the location parameters (sometimes the mean, sometimes the minimum), λ for the scale parameter and α for the shape parameter in the distribution functions where appropriate. 3. Weibull Distribution We first present solutions under a Weibull Distribution assumption of rating scores. The Weibull Distribution is fleible and rich. A three parameter Weibull distribution cumulative probabilit function (CDF) is: 2

14 ( ) ( ) α z M λ F z P Z z = e, where z>m, α >0 and λ > 0 The inverse CDF of a three parameter Weibull Distribution is: ( ) ( ) = + λ ln F p M p α, where p [ 0,]. Assuming is the HR(C) and is the FAR(C), the three-parameter α C M Weibull Distribution ROC is derived as: λ ( ) λ ln( ) = FX C = e C = M + Y ( ) ( ) = F C = P Y C = e α C M λ α λ λ ROC : = ep ln( α ) + ( M M ) α (2) The above ROC formula is ver difficult to use to deduce an eplicit closed form formula for the AUROC, although a numerical solution eists in this case once all the parameters are estimated. However, the situation becomes much better if we impose a slightl stronger assumption that the shape parameters α of the F X and F Y are equal to one. We then have analtical closed form solutions in this case and the Weibull distribution degenerates to a truncated Eponential Distribution Famil if M is positive and an etended Eponential if M is negative. Assume: α = α =, we then rewrite the above equation as following: ROC : = λ ( ) M M e λ λ Let M M λ =, θ K e λ =, then λ M M ( θ ) K λ λ AUROC = K d = = e, (3) 0 + θ λ + λ We net discuss some of the properties of equation 3. 3

15 Propert. M (,0) M for an plausible rating sstem. The smaller is M M (or the larger is M M ), the closer is the AUROC to one. Recall that M is the location parameter, in this case the minimum. Therefore, the rating sstem will receive a higher AUROC if it can better discriminate the defaulter from the non-defaulter b the difference in their minimum values. λ It is also interesting to see that if M M 0, AUROC. As we λ + λ illustrated in earl section that AUROC of a plausible rating sstem is above 0.5 if it is not a random selection sstem. This implies that the value of the scale parameters that the rating scores being assigned have to be such that 0 < λ λin this case. Note that this condition is implied b both groups being eponential but also b both groups being truncated or etended eponentials with the same minima. Propert 2. AUROC is monotonicall increasing with respect to λ, but monotonicall decreasing with respect to λ. Proof: d K AUROC = M 2 M dλ λ + λ λ λ + λ λ λ + 2λ ( )( ) ( ) ( ) ( M M)( λ λ) λ( λ λ) ( λ λ)( M M λ) < + < Since plausible rating sstems, we epect M M, K 0 and λ > 0 d AUROC 0 dλ 4

16 3.2 Logistic Distribution A two parameter Logistic distribution CDF is: z M λ e F( z) P( Z z) = + e z M λ, where z and λ > 0. Here M is a mean parameter. The inverse CDF of a two parameter Logistic Distribution is: p ( ) = + λ ln p, F p M where p [ 0,]. Again assuming is the HR(C) and is the FAR(C), we have: C M = X ( ) = λ e = + λln C M λ F C C M + e and M M C M λ λ e e Y ( ) Y ( X ( )) C M M M λ λ λ = F C = F F = = + e + e Similar to the Weibull Distribution case, the AUROC with the above ROC specification can alwas be evaluated numericall. Moreover, b assuming λ = λ =, and in what follows assume that K does not equal, K = e, the above ROC equation can be simplified to M M e K = = M M + e + ( K ) The AUROC can be now derived analticall. ( ) ( K ) ( ) K AUROC = K d = d 0+ K K 0+ K λ λ λ M M 5

17 Let = + ( ) u K K K K lnk AUROC = du = lim AUROC = u K K K ( ) 2 K 3.3 Normal Distribution A two parameter Normal distribution CDF is: 2 M 2 λ ( ) ( ) z z F z P Z z e d M = =Φ λ 2π λ, where z and the Φ (). is the standard Normal probabilit distribution function.. The inverse CDF of a two parameter Logistic Distribution is: F ( p) M λ ( p) where p [ 0,]. For is the HR(C) and is the FAR(C), we have: C M =Φ = + Φ ( ) C M λ λ C M ( M ) M + λφ ( ) =Φ =Φ λ λ Therefore, AUROC ( M M ) + λφ ( ) = Φ 0 λ, which gives the ROC curve. d This function can be easil evaluated numericall to obtain the AUROC. = + Φ, Propert AUROC increases with M M and in particular if M M = 0, and λ = Proof: λ.,auroc = 0.5 ( M M ) + λφ ( ) λ ( ) AUROC = Φ d = Φ Φ d = d = λ 0 0 λ The above propert is also quite intuitive. If the means and variances of two normal distributed 6

18 populations equal each other,the distributions are equal and then overall there is no discriminator power of the models based on this rating mechanism. So the AUROC is Mied Models It is obvious that as long as we have parametric distribution families for the defaulter and non-defaulters, one can alwas calculate an AUROC for the two score samples from equation () in section 2 even with two different parametric distributions for the two populations respectivel. 4. Performance Evaluation on the AUROC Estimation with Simulated Data In this section we carr out a performance evaluation on AUROC estimations b the non-parametric Mann-Whitne Statistic and the analtic approach suggested in this paper respectivel with simulated data. We first assume known parametric distributions for the credit scores of defaulters and non-defaulters. B doing this we would know the value of the theoretical AUROC. After generating simulated sample data from the assumed distributions for defaulter score and non-defaulter s, we estimate the AUROC and its confidence interval (CI) b the above two approaches on the simulated samples. We repeat the simulation and estimation procedure a number of times. We then compare the accurac of the AUROC estimation and the CI of the two approaches. Finall, we change the parameter values of the assumed distribution and repeat the simulation. We repeat the above procedures to evaluate the performance of the two approaches subject to different theoretical AUROC inde values with different sample sizes of defaulters. We choose two-parameter Normal Distributions, one-parameter Eponential Distributions and Weibull Distributions with various shape and scale parameters. 7

19 4. Performance evaluations under Normal Distribution Assumption Normal Distributions are assumed as our parametric distributions of the credit scores of the defaulters and the credit scores of the non-defaulters in this performance evaluation. The theoretical value of AUROC for the normal score samples is evaluated numericall. The non-parametric estimate of AUROC is carried out using the ROC module in SPSS and we use the bootstrap to re-sample 000 replications to obtain the estimates of the analtic approach which also generates a two-sided 95% CI. The parameters of the parametric distribution are estimated for each replication and substituted back to the analtic AUROC formula. We then define error as the difference between model estimates based on a sample and the theoretical AUROC value, and compare the mean error and mean absolute error for the two approaches. The width of the confidence interval is also compared. We generate 50 normal samples from si different settings respectivel. Settings, 2 and 3, consisting of Group, target the AUROC at a low value, while settings 4, 5 and 6, Group 2, target the AUROC at a high value. Within each group the var in defaulter s sample size, which ranges at 20, 00 and 500. Credit rating models can be applied to at least three different tpes of groups: credit risk with Corporate, counter part default risk in Trading Books, and credit risk in credit card and other loan tpe Banking Books. The default sample of Corporate is usuall small, such as 50 in ten ears, especiall under a good economic ccle. Meanwhile, the number of defaults in a loan book or a credit card book in a commercial bank s banking book can be fairl large, possibl in ecess of several hundreds. The reason for selecting different defaulter sample sizes in the test is to assess for which tpe of problem the analtic approach outperforms. We define a performance statistic as follows: (Ratio to N) = Difference / ( Non-Parametric Estimate). Normal Setting -3: Normal Distributions Setting Setting 2 Setting 3 Sample Mean Standard # of # of # of 8

20 deviation Observation Observation Observation X Y Theoretical AUROC= Report on estimation error with 50 simulated normal samples & 000 replication bootstrap Setting Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 2 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 3 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Normal Setting 4-6: Normal Distributions Setting 4 Setting 5 Setting 6 Sample Mean Standard # of # of # of deviation Observation Observation Observation X Y Theoretical AUROC= Report on estimation error with 50 simulated normal samples & 000 replication bootstrap Setting 4 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % 9

21 Setting 5 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 6 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % In tables -6, all mean confidence interval widths show that the estimates of the analtic approach are better then the non-parametric estimates. As for the mean error and the mean absolute error, analtic estimates outperform the non-parametric estimates in tables, 2 and 4-6. (Ration to N) shows the percentage of the difference out of the non-parametric approach estimate. The larger the (Ration to N), the more the analtic approach outperform the non-parametric approach. 20

22 4.2Performance evaluations under Eponential Distribution Assumption Eponential Distributions are assumed as our parametric distribution of the credit scores of the defaulters and the credit scores of the non-defaulters in this performance evaluation. The theoretical value of AUROC for the Eponential score samples is evaluated analticall b the closed form formula (3) in section 3.. The performance evaluation setting is ver similar to the Normal Distribution one. There are 6 settings across different AUROC value and defaulter sample size as well. Eponential Setting -3: Normal Distributions Setting Setting 2 Setting 3 Scale Parameter # of Sample (Lamda) Observation # of Observation # of Observation X Y Theoretical AUROC= Report on estimation error with 50 simulated normal samples & 000 replication bootstrap Setting Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 2 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 3 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % 2

23 Eponential Setting 4-6: Normal Distributions Setting Setting 2 Setting 3 Scale Parameter # of Sample (Lamda) Observation # of Observation # of Observation X Y Theoretical AUROC= Report on estimation error with 50 simulated normal samples & 000 replication bootstrap Setting 4 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 5 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 6 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % In table -6, all the mean absolute error and the mean confidence interval widths show that the estimates of the analtic approach are better then the non-parametric estimates. (Ration to N) shows that the non-parametric approach estimates provide a significantl better confidence interval than the non-parametric estimates. 22

24 4.3Performance evaluations under Weibull Distribution Assumption Weibull Distributions with scale and shape parameters are assumed as our parametric distribution of the credit scores of the defaulters and the credit scores of the non-defaulters in this performance evaluation. The theoretical value of AUROC for the Weibull score samples is evaluated analticall b the closed form formula (2) in section 3. with the location parameters set to zero. The maimum likelihood estimations of sample distribution parameters are obtained b a numerical approimation. Since we have a shape parameter for Weibull Distribution, which ma shift the shape of the distribution significantl, we evaluate the performance of the two approaches under two cases: with the same shape parameter for defaulter and non-defaulter sample, and with different shape parameters. The theoretical value of AUROC for the normal score samples is evaluated numericall as well 2. The rest of the performance evaluation setting is ver similar to the Normal Distribution one. There are 6 settings across different AUROC values and defaulter sample sizes as well. 23

25 Weibull Setting -3: Normal Distributions Setting 4 Setting 5 Setting 6 Sample Shape # of # of # of Scale parameter Observation Observation Observation X Y Theoretical AUROC= Report on estimation error with 50 simulated normal samples & 000 replication bootstrap Setting Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 2 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 3 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % 24

26 Weibull Setting 4-6: Normal Distributions Setting 4 Setting 5 Setting 6 Sample Shape # of # of # of Scale parameter Observation Observation Observation X Y Theoretical AUROC= 0.75 Report on estimation error with 50 simulated normal samples & 000 replication bootstrap Setting 4 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 5 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % Setting 6 Approach Non-parametric Analtic Difference Ratio to N Mean Error Mean ABS Error % Mean CI Width % In tables -6, all mean confidence interval widths show that the estimates of the analtic approach are marginall better then the non-parametric estimates. As for the mean error and the mean absolute error, analtic estimates marginall outperform the non-parametric estimates in tables 2, 3, 5 and 6. Because we use numerical approimation for sample maimum likelihood estimates and the estimation error could be fairl large when we have a small sample, we observe this estimation error is passed through our analtic estimation for the AUROC inde, which made the mean 25

27 absolute errors estimated from the analtic approach are large than the non-parametric approach in setting and 3. This also reduces the gain of the analtic approach over the non-parametric approach in the Weibull Distribution case when comparing with the previous tests. Summar: Although the analtic approach gives no better estimates than non-parametric one when we use approimated maimum likelihood estimates for small samples, the performance evaluation shows that the analtic approach works at least as well as the non-parametric approach in above tests and provides better estimate in terms of mean absolute error estimates and confidence interval estimates in most cases. The above discussion has the following implications. If one can identif some appropriate parametric distributions to the scores of defaulter and non-defaulter, then one could estimate the AUROC and its confidence interval more accuratel b the analtic approach. On the other hand, if we can design the rating model so that the score sample is generated b some specific parametric distribution families, then we are able to find a better rating model b using the analtic AUROC as the objective function to maimize in the model selection process. Another interesting eperiment which has not been conducted in previous research is the effect of defaulter s sample size on AUROC. The above eperiments clearl show the level of estimation error in both methods with different sample sizes and the resulting error can be large if we onl have a small defaulter sample. In addition, although not ver clear but from the results in section 4. and 4.2, the analtic approach provides more gain over non-parametric approach when AUROC inde is in its high value region than low value region. The reason for this is not clear, we hope to investigate this in future research. 26

28 5 Conclusions This paper reviewed some of the prevailing credit rating model validation approaches and, in particular, studied the analtic properties of the ROC curve and its summar inde AUROC. We use the concept of the population ROC curve to build analtic models of ROC curves. It has been shown through simulation studies that greater accurac of estimation relative to the non-parametric methods can be achieved. We also show that there are some situations that the accurac gain of analtic ROC model ma decrease, which should be taken into account when appling the analtic models to practical applications. Moreover, with some particular distributions, where the closed form solution of AUROC is available, analtic AUROC can be directl used as an objective function to maimize during the rating model selection procedure. This means if we can transform the rating scores into those distributions, analtic AUROC could offer a powerful model selection tool. Finall, we also studied the performance of both non-parametric and analtic ROC models under different assumptions for the sample size of defaulters.. The magnitude of error size can be significant when we have a small sample on defaulters, which is a frequentl met situation in corporate credit risk stud and in periods of benign macro-economic background. 27

29 Appendi A: Non-parametric ROC curve for Normall Distributed Samples Non-parametric ROC curve under setting : Non-parametric ROC curve under setting 2: Non-parametric ROC curve under setting 3: Non-parametric ROC curve under setting 4: Non-parametric ROC curve under setting 5: Non-parametric ROC curve under setting 6: For eample, ROC curve Vi is plotted using the data of simulated sample number i generated under a specified setting. 28

30 Appendi A2: Non-parametric ROC curve for Eponentiall Distributed Samples Non-parametric ROC curve under setting : Non-parametric ROC curve under setting 2: Non-parametric ROC curve under setting 3: Non-parametric ROC curve under setting 4: Non-parametric ROC curve under setting 5: Non-parametric ROC curve under setting 6: For eample, ROC curve Vi is plotted using the data of simulated sample number i generated under a specified setting. 29

31 Appendi A3: Non-parametric ROC curve for Weibull Distributed Samples Non-parametric ROC curve under setting : Non-parametric ROC curve under setting 2: Non-parametric ROC curve under setting 3: Non-parametric ROC curve under setting 4: Non-parametric ROC curve under setting 5: Non-parametric ROC curve under setting 6: For eample, ROC curve Vi is plotted using the data of simulated sample number i generated under a specified setting. 30

32 References: Bamber, Donald (975), The Area above the Ordinal Dominance Graph and the Area below the Receiver Operating Characteristic Graph. Journal of Mathematical Pscholog 2, p Basel Committee on Banking Supervision (BCBS) (2004), International Convergence of Capital Measurement and Capital Standards: A Revised Framework. Bank for International Settlements, June. Basel Committee on Banking Supervision (BCBS) (2005) Working paper No.4 Blochwitz, Stefan, Stefan Hohl and Carsten Wehn (2003), Reconsidering Ratings. Unpublished Working Paper. Engelmann, Bernd, Eveln Haden, and Dirk Tasche, (2003), Testing Rating Accurac. Risk, Januar 2003, p Sobehart, Jorge R and Sean C Keenan (200), Measuring Default Accuratel, Risk Magazine, March 200, p Sobehart, Jorge R and Sean C Keenan (2004), Performance Evaluation for Credit Spread and Default Risk Models. In: Credit Risk: Models and Management. David Shimko (ed.). Second Edition. Risk Books: London, p Sobehart JR, Sean C Keenan and R Stein (2000), Validation methodologies for default risk models, Credit (4), p Swets JA, (988), Measuring the accurac of diagnostic sstems, Science 240, p The theoretical AUROC is approimated b partitions, while the bootstrap estimation is approimated b 0000 partitions. 2 The theoretical AUROC is approimated b partitions, while the bootstrap estimation is approimated b 0000 partitions. 3