National Bank of Romania. Occasional Papers

Transcription

1 Occasional Papers No. 7 October 2007

2

3 Early Warning System of CAAMPL Rating Downgrade Events Author: Bogdan Moinescu 1 1 The author is grateful to Cristian Bichi and Adrian Codirlaşu for the suggestions and comments made on the basis of a previous version of this research paper.

4 ISBN N o t e The opinions expressed in this paper are those of the author and do not necessarily represent the views of the National Bank of Romania, nor do they engage it in any way. Reproduction of the publication is forbidden. Data may be used only by indicating the source.

5 ABSTRACT This study aims to synthetically describe the opportunity of adding to the NBR s microprudential analysis framework a dynamic component, formalised through an early warning instrument of credit institutions whose performances are deteriorating as well as the manner in which the implementation of Basel II Accord requirements might facilitate the refinement of off-site analysis techniques. This project combines elements of heuristic analysis with elements of quantitative assessment, having as main component the development of an econometric model which quantifies the downgrade probability for the CAAMPL bank rating. The explanatory variables identified for the phenomenon of rating downgrade are the current rating, the market share on credit segment, the weight of nonperforming loans in total assets and the square of the deviation of the general risk rate from its natural level. Running this early warning system on the available data as for 31 December 2006 shows that the Romanian banking system will perform during 2007 at least as well as it did in 2006, except for three credit institutions, with an aggregated market share of 2 percent on total assets, for which it is likely that the CAAMPL rating will deteriorate from level 2 in December 2006 to level 3 by December JEL Classification: G21, G32, G33 Keywords: early warning system, downgrade probability, Basel II, off-site supervision, bank rating

6 Abbreviations AUROC Area Under ROC BCBS Basel Committee on Banking Supervision BdF Banque de France BdI Banca d Italia BIS Bank of International Settlements NBR National Bank of Romania CAAMPL The uniform bank rating system used by the NBR CCR Central Credit Register ECB European Central Bank EWS Early warning system FAR False Alarm Rate FDIC Federal Deposit Insurance Corporation FED Federal Reserve GMS Growth Monitoring System HR Hit Rate IMF International Monetary Fund ROC Receiver Operating Characteristics SCOR Statistical CAMEL off-site Rating SEER System for Estimating Examination Ratings

7 Contents INTRODUCTION...9 I. LITERATURE REVIEW II. METHODOLOGY Statistical model for estimating the rating downgrade probability Qualitative assessments related to the results provided by the statistical model III. DATA IV. EMPIRICAL ANALYSIS CONCLUSIONS REFERENCES ANNEXES

8

9 INTRODUCTION The role of the banking system as a key element in the saving-investment process makes its stability a priority on public authorities agenda. Among the main objectives of a central bank is that of preventing systemic risk by performing efficient banking supervision, which promotes stability and viability for the entire financial system. As such, central banks have developed methods and processes for ongoing bank supervision and assessment the prerequisites of preventing the occurrence of a large variety of banking crisis or of other unwelcomed surprises regarding the entities of the banking system. In particular, special attention is given to improving the quality of bank examination by developing adequate support systems which may assist supervisors in the early identification of potentially unfavourable trends in a bank s activity, which might lead to serious problems in the future (Bichi, Dumitru, Moinescu, 2003). Traditionally these trends include issues such as adverse changes in profitability, material deterioration of bank s market share and worsening of financial indicators as determined for each bank based on their previously registered values or by comparison with those determined for the peer group. Recent developments have focused on sophisticated systems which use econometric techniques to estimate the insolvency probability or the downgrade probability. Based on the information provided by these instruments, on-site inspections with specific targets are triggered or in case of general examinations, which are conducted on a regular basis, the priorities are ranked. Hence the early warning systems for bank performance deterioration allow both the improvement of the on-site supervision effectiveness and a better management of the limited resources available to the prudential control authority. Such efforts were encouraged by the work of the Bank of International Settlements (BIS) in order to develop an international convergence framework concerning the assessment and management of insolvency risk at the level of credit institutions. At this time, the new capital adequacy mechanism (Basel II) represents the most important reference in micro-prudential supervision. In the case of this new approach, the supervision activity is more and more directed towards the analysis of the institutions risk profile, the processes and instruments used by its managers to efficiently deal with specific risks. Rating and early warning systems, stress tests and inter-bank contagion tests are sophisticated techniques which allow the successful achievement of the objectives mentioned before. These instruments use relevant information about the features of a credit institution and its counterparts, based on which a synthetic measure of their performances and/or their vulnerabilities is provided. In this context, the purpose of this study is to describe the opportunity of adding to the NBR s microprudential analysis framework a dynamic component, formalised through an early warning instrument of credit institutions whose performances are deteriorating as well as the manner in which the implementation of Basel II Accord requirements might facilitate the refinement of off-site analysis techniques. This project combines elements of heuristic analysis with elements of quantitative analysis, having as main component the development of 9

10 an econometric model which quantifies the downgrade probability for the CAAMPL bank rating. The paper comprises four sections, ending with the main conclusions and future analysis issues. The first section comprises the literature review focusing on methodological issues and explanatory variables used in the development of early warning systems. The second section describes the methodology used in the downgrade CAAMPL bank rating prediction system, starting with technical elements regarding the estimation of the statistical model used to quantify the rating downgrade probability. Moreover, the issue concerning the need to integrate expert judgment within the process of explaining signals provided by statistical models was tackled. This completes the analysis framework with a qualitative component, which is to be considered a key element in overcoming statistical limits of the estimated scoring function. The third section presents the data, describing in detail both the economic rationale and the statistical evidence which have led to the selection of the explanatory variables. The fourth section describes the main empirical issues concerning the development and testing of the statistical model used in quantifying the downgrade probability of the CAAMPL bank rating. 10

11 I. LITERATURE REVIEW The advantages of using rating systems are recognised by supervision authorities worldwide, as proven by their extensive use in preventing the bank contagion phenomenon 2. By separating well-performing credit institutions from those in distress, the bank rating allows a better allocation of the limited resources available to the supervision authority in order to prevent the spillover of individual imbalances to the entire system. However, bank rating methodologies used by supervision authorities are mostly represented by expert systems which provide assessments only for the period under review, without providing insight to future developments. As such, ex-post results must be supplemented by information provided by prediction instruments, the use of which gives more time to the supervision authority to adopt the necessary measures. The role of the micro-prudential early warning system (EWS) is that of providing ex ante signals on potential financial distress of credit institutions, based on their current financial statements. EWSs used by central banks or by supervision authorities combine qualitative and quantitative analysis in different ratios. Since the 90s, quantitative methods gained ground and at present they are the main methodological component of EWSs. Among those, the most popular systems in the field of empirical research are oriented towards estimating bankruptcy or rating downgrade probability (Jagtiani, Kolari, Lemieux, Shin, 2003), using logit/probit regressions or duration models. Distinguishing between the insolvency or rating downgrade events and those of adequate capitalisation or stable / upgrading rating is often made by using a single threshold, established to either maximise the performance of the model or to lead to a desired level between the non-identified events and the false alarms. On the other hand, the recent requirements regarding the rating scale granularity made in the Basel II Accord (BCBS 2004 para. 389) suggest that the use of a single threshold is no longer state of the art in the field, the solution being the development of a rating scale with a sufficiently large number of risk classes. In the United States, both the Federal Reserve (FED) and the Federal Deposit Insurance Corporation (FDIC) are using statistical models in the off-site assessment process. The SEER 2 Bank rating systems are used mainly to prevent the phenomenon of indirect contagion, where this regards situations when, due to an incorrect perception over the existence of direct contagion effects, market operators react disproportionately in the case of solvent banks even if it was not the case to act as such. In this context, bank rating allows on the one hand the identification of credit institutions with low financial performances as well as the probability of increase in their level, and on the other hand the improvement of communication between the supervision authority and market operators when a credit institution goes into default. The quantification of the probability of occurrence and the severity of a potentially direct inter-bank contagion phenomenon, respectively the contagion arising exclusively from inter-bank exposures, theory and empirical research have made the inter-bank contagion test the most adequate instrument for this purpose (Upper and Worms 2004, Furfine 2003). 11

12 Banking Model (System for Estimating Examination Ratings) used by FED estimates the bankruptcy or severe undercapitalisation probability through a probit regression. The SCOR Model (Statistical CAMELS Off-site Rating) applied by FDIC is included in the same category of early warning systems of credit institutions with deteriorating financial position. This instrument was estimated using a logit regression with a discriminatory power evaluated through the area under the ROC curve of 79 percent (Gilbert, Meyer, Vaughan, 2002). The set of explanatory variables comprises only financial indicators, such as the weight of nonperforming loans, of own funds or of the general reserve for credit risk in total assets. The threshold probability was set at 35 percent, but the downgrading signal is validated only after the microprudential analysis is performed by the expert supervising that bank (Collier, Forbush, Nuxoll, Keefe 2003). Apart from SCOR, FDIC uses another model which combines the quantitative assessment with the qualitative one, i.e. Growth Monitoring System (GMS). GMS aims to detect the banks with very large growth in terms of loans and assets against their peer-group. The SAABA System applied by the French Banking Commission uses historical data to estimate a three year expected loss from the loan portfolio. The diagnostic analysis and the warning mechanism are based on the level of the solvency rate and the shareholders quality. Banks are divided into four categories: (a) banks with a solvency rate lower than 8 percent and a weak shareholders quality; (b) banks with a solvency rate lower than 8 percent, but with strong financial support from the shareholders; (c) banks with a solvency rate higher than 8 percent, but a weak shareholders quality; (d) banks with a solvency rate higher than 8 percent and an adequate shareholders quality. The early warning system aims to identify the banks whose features are described under (a) and (c). In this respect, the level of own funds, of solvency rate and the shareholders quality are analysed in both static and dynamic terms. As regards the prediction of the change in the own funds, the future level of gross revenues and the expected loss are estimated. The SAABA system quantifies the expected loss using the Basel II approach. In this respect, the probabilities of default estimated through a specific BdF methodology are used, which possess a discriminatory power evaluated through the area under the ROC curve of 85 percent; the loss given default parameter is considered at 45 percent (as in Basel II), while the exposure at default value is obtained out of the credit register database. All these pieces of information are integrated in a large data warehouse, with 25 sources, out of which five are the most important. Banca d Italia (BdI) has estimated a survival function of Italian credit institutions using the Cox Proportional Hazards model. This instrument quantifies the probability of a severe distress occurring at the level of Italian credit institutions over a time horizon of two years (Laviola, Marullo-Reedtz, Trapanese, 1999). The state of severe financial distress is established based on the events of legal failure, of takeover of a bank by a stronger one, and in 12

13 cases in which the bank rating system (PATROL 3 ) assigns the credit institution a rating of 4 or 5. The set of explanatory variables includes both prudential data and qualitative information, as the geographic region in which Italian 4 banks operate. The most significant prudential data are the indicators concerning credit quality, which possess a highly predictive power in the process of identifying distressed banks. It is important to mention that BdI experts are concerned about finding additional information on credit quality, on the basis of financial information registered by the Office of Banking Risks (Marullo-Reedtz, Trapanese, 2000). The model re-estimation occurs only when the climate in the banking system faces structural changes, and not at pre-established time periods. Economic literature consists in various approaches, both from the standpoint of techniques employed and of results obtained. Cole and Ghunter (1998) model banking insolvability by means of a logit regression. Their analysis show that data extracted from the off-site reports proved to have a better predictive power than the data collected during on-site inspections, the latter being more perishable, and having the tendency of significant alteration after only a few months. Gilbert, Meyer and Vaughan (1999) compare the capacity of univariate models to predict banking failures with that of multivariate models. The authors have found that, while univariate models feature a high volatility of their predictive power, multivariate models are much more robust and are able to provide long-lasting accurate information. On the other hand, Estrella, Park, and Peristiani (2000) argue that, in the case of the United States, a univariate model with only two variables, i.e. the capital adequacy ratio and its lag, behaves better than the multivariate or non-parametric models in predicting the events of banking failure. Having as a starting point the methodological approaches described above, we aim to develop a system for predicting the events of CAAMPL rating downgrade. 3 4 Capital adequacy (PATrimonio); profitability (Redditività); credit risk (Rischiosità); management (Organizzazione); liquidity (Liquidità). One can use a dummy variable to discriminate between banks that operate predominantly in northern Italy and banks that operate mainly in the south. 13

14 II. METHODOLOGY The architecture of the early warning system of CAAMPL bank rating downgrade events has two components: (2.1) a statistical model for quantifying the probability of rating downgrade; (2.2) qualitative appreciations performed by experts, based on additional information, besides the one obtained from the model Statistical model for estimating the rating downgrade probability In general, a rating downgrade prediction mechanism is based on a conditional probability model for decreasing bank performance over a specified time horizon. Early warning systems either developed by academia or created by supervision authorities use statistical models in one form or another. The interest in applying these approaches results both from their objective features and the possibility of automating the assessment process. Another important issue is the rating scale, which ensures a good segmentation of rating downgrade risk and, implicitly, a proper interpretation of signs provided by the statistical model. We present a brief description of the conceptual model and of the methodology employed in order to obtain the scoring function. Our analysis accomplishes the Basel II requirements concerning the statistical performance of internal models for credit risk. The objective consists in obtaining a sound, long-lasting accuracy, for both the procedure of classification of banks and a significant discrimination between probabilities of rating downgrade, and the granularity of the rating scale. In order to develop an early warning system for the CAAMPL bank rating downgrade events we have applied a logit approach which is able to predict the rating downgrade for a period of one year in advance. The endogenous variable is binary and it discriminates between rating downgrade events and rating upgrade or maintaining events in a time horizon of one year. Conventionally, we assign zero value to the dependent variable when the rating is improved or remains constant and the value of one when the rating is downgraded exactly over 12 months. The set of exogenous variables includes, exclusively, microprudential information, macroeconomic elements being left out in this version of the study. The selection of relevant variables has been realised according to the recommendations of staff involved in the supervisory activity and of the economic literature, and materialised in a series of indicators and criteria to fulfil. These were empirically tested, finally being employed only those indicators and criteria that have statistical significance. 14

15 The estimation methodology has the following model design: P( y i = 1/ x, 1 ) = 1+ e i β x, i β P ( y i = 1/ x i, β ) is the downgrade probability conditional upon banks features and macroeconomic environment We have notated with y i the categorical variable which indicates if a bank registered a weaker performance one year after that obtained at moment i and with y i * a latent variable explained by x ik, variables, k=(1,n), such as: y =β +β x +β x +,..., +β x +ε * yi = 1, yi > 0 y = < * i 0 1 i1 2 i2 n in i i * 0, yi 0 To estimate the model s coefficients we have chosen the maximum likelihood method MLE. The MLE s hypothesis states that each observation is extracted from Bernoulli s distribution. The probability of success is F ( `β ), so that we have: prob( Y x i ` ` 1 = y1, Y2 = y2,..., Yn = yn ) = (1 F( xiβ )) F( xiβ ) yi= 0 yi= 1 Accordingly, the likelihood function becomes: n ` yi ` [ F( xiβ )] [ 1 F( xi )] L( β / data) = β n ln L = i= 1 i= 1 ` ` { y F( x β ) + (1 y )[ 1 F( x β )]} i i i i 1 y i therefore To obtain the parameter values we have applied the quadratic hill climbing algorithm. It uses the matrix of secondary differentials of the log likelihood function to achieve convergence. From an economic standpoint, the signs of coefficients attributed to input variables show the influence of these variables on the binary values taken by the dependent variable. The marginal contribution of each variable to the downgrade risk is given by the first order derivation of the probability against the exogenous variable x i : P = f( β X) β i x i where f ( β X ) is the logistic probability density function assessed in terms of the average values of stochastic X vector. This process is equivalent to a factorial decomposition based on which it is possible to derive the main source of risk to which the performance of a certain credit institution is exposed. 15

16 Judging by the supervision activity, the estimated logit regression functional role is to provide the score (theoretical downgrade probability) based on which the banks are classified. In order to interpret the results, it is necessary, on the one hand, to set the alarm threshold (P* Chart 1), and on the other hand, the interval bounds that segment the rating scale. Chart 1 Banks classification based on the logit score Traditionally, when the logit score is above the threshold (P*), the model classifies the bank in the downgrade category. Otherwise, the model classifies the bank as non-downgrading institution. The overall statistical model performance is influenced by the value of chosen threshold, the hit rate being directly affected by this. The accuracy potential level of the statistical model is derived by optimising the threshold value based on the relative importance between the prediction errors. They fall into two categories: un-signalled downgrades (type 1 errors) and false alarms (type 2 errors). Type 1 error designates the situation in which the model classifies the bank in non-downgrading category when, in fact, it records a rating downgrade after one year. Type 2 error designates the situation in which the model classifies the bank in downgrading category when, in fact, it does not record a rating downgrade after one year. 16

17 From a methodological point of view, setting the relative importance between the two error rates depends on the scope in which the estimated logit function is used. Generally, equal importance is assigned to both error categories. From the supervision authority perspective, the un-signalled downgrades could prove costlier than false alarms, so that the relative importance coefficient becomes higher than one. An alternative method consists in choosing a desirable level for the type 1 error indicator, which will be the basis for computing the classification threshold. Adopting this approach is equivalent to assuming a certain level of unidentified rating downgrades. Of course, to supervisors, a smaller value of this parameter is desirable, but it implies a smaller value of the discrimination threshold too. Keeping in mind that a decrease of the threshold generates an increase of the number of false alarms, and that the sample comprising banks having unfaulty rating is much more numerous than the one comprising banks with faulty rating, the overall accuracy of the early warning system is, in this case, significantly lower than the potential maximum value (for the considered set of explanatory variables). We consider appropriate to use both approaches in this study: (a) equal importance for both error categories; (b) a value of 15% for type 1 error. On the other hand, the mechanical classification of credit institutions only according to the alarm threshold will limit the performance of the prediction system. To overcome this obstacle, it is recommended to build a rating scale with a sufficient number of risk classes, in order to ensure good risk dispersion, according to the value of the score obtained 5, in the case of a significant delimitation for the empirical probability of CAAMPL rating downgrade. Therefore, the model s user is able to judge more accurately the signal given by the scoring function, particular for the theoretical downgrade probability values which are close to the alarm threshold Qualitative assessments related to the results provided by the statistical model 6 The interpretation of signals provided by the model on the basis of additional qualitative information and on historical performance registered by each bank is a necessary condition in order to surpass the statistical boundaries of the scoring function. Qualitative information relates to issues of banking strategy, risk profile, the quality of the audit and internal control system, as well as to specific aspects concerning the income sources or the efficiency of operational expense management. 5 6 According to the requirements concerning the distinction and accurate, consequent and coherent measurement of risk, as shown in para. 389 and para. 390 of Basel II Accord (BCBS 2004). According to the requirement concerning the need that a credit institution should collect and register all relevant data, in order to develop its own processes for risk measurement and management, mentioned in para. 417 of Basel II Accord (BCBS 2004). 17

18 The analysis of the historical performance for each bank allows the distinct interpretation of signals provided by the statistical model. For this purpose, three statistical criteria were employed, as follows: a) the general rate of success, computed as a ratio between the number of correctly identified events and the total number of observations 7 ; b) the rate of success when the model signals a downgrade, computed as a ratio between the number of downgrade events correctly identified and the total number of alarms provided; c) the rate of success when the model does not signal a downgrade, computed as a ratio between the number of upgrade or constant rating events correctly identified and the number of observations in which the model has not signalled the occurrence of a potential downgrade. 7 The maximum number of observations per bank is 73 (months). 18

19 III. DATA The data sample used in order to build the CAAMPL rating downgrade prediction model is a panel of 31 credit institutions, Romanian legal persons. Mortgage banks and branches of foreign credit institutions were not included in this sample. Data used in estimation cover December 1999 December 2002 period, those for out of sample testing cover January 2003 January 2005 period, while the predictions were performed for December 2007, using data as of 31 December The initial set of candidate variables comprised around 30 elements, raw data and synthetic indicators covering the criteria traditionally used in bank micro-prudential analysis. The main data categories are: a) capital adequacy indicators; b) profitability indicators; c) liquidity indicators; d) asset quality indicators; e) market share indicators. An important criterion is the CAAMPL rating at the moment when we realize the prediction, which synthesizes all information in the indicators composing it. Keeping in mind that, in order to maintain a good general rating, a bank must keep all individual ratings (by component) at a high level 8, the lower (better) the present rating, the more probable the downgrade (Chart 2), and the expected sign of the attached coefficient is minus. This is sustained from a statistical point of view, too. Both the cases in which a bank with rating 4 registers a performance deterioration to rating 5 over a period of one year, and those of passing from rating 3 to 4 are rare 9. The main downgrades are from rating 2 to 3 (206 cases from a total of 292, i.e percent, but represent only percent of the number of observations with rating 2). On the other hand, it should be mentioned that, although the downgrades from rating 1 to rating 2 represent only percent of the total number of downgrade events, these occur every other situation in which, a year before, the rating obtained was maximum. Thus, after a year, half of the cases with rating 1 witnessed downgrades of CAAMPL rating to the immediately weaker rating (Table 1). 8 9 The composite rating 1 or 2 is granted to banks only if all components have ratings better than 3, which means that, in order to maintain the rating at level 1 or 2, a sustainable effort is necessary to keep the ratings of all the components. Only 4 of the 131 cases with rating 4 posted a downgrade to rating 5 over a time horizon of one year (3.05 percent of cases with rating 4) and only 22 of the 715 cases with rating 3 posted downgrades to rating 4 over a time horizon of one year (3.06 percent of cases with rating 3). 19

20 Table 1. Statistics concerning the events of rating downgrade according to the rating of one year ago: CAAMPL rating Number of ratings Number of rating downgrade events after one year Share of downgrade events according to the current rating Rating R1 R2 60 (57.14%) 20.55% Rating 2 1,122 R2 R3 206 (18.36%) 70.55% Rating R3 R4 22 (3.06%) 7.53% Rating R4 R5 4 (3.05%) 1.37% Rating Total 2, % As a consequence, if the actual performance is not satisfactory (rating 4), showing severe to critical deficiencies, the downgrade probability is much more reduced than, for instance, the case in which the actual performance is very high (rating 1). Moreover, the utility of a rating downgrade prediction model for a bank with weak actual performance is small, because the bank is already under the careful monitoring of the supervisory authority. Chart 2 Rating downgrade probability over a time horizon of one year according to actual performance Another important variable is the market share depending on the credit portfolio value of each credit institution. This reflects the reputation of a bank among its customers, the entities with a high share on the credit market registering fewer rating downgrade events than those with a reduced share, in the context of an improved macroeconomic climate. This argument was validated from a statistic standpoint, too, the arithmetic mean of the market share for the events signalling constant or improved CAAMPL rating being 3.99 percent, while the arithmetic mean of the market share for CAAMPL rating downgrade events is of only 1.58 percent. Thus, a high share on the credit market, ceteris paribus, reduces the actual rating downgrade probability, and the sign of the attached coefficient for this variable will be minus. On the other hand, an increase in the share of non-performing loans to total assets generates a potential rating worsening. 20

21 The more non-performing loans a bank has in its portfolio, the higher the likelihood of doubtful losses becoming certain and, therefore, the credit institution s financial performance deteriorating. In the same sense operates the square of the Chart 3 The rating downgrade probability general risk rate deviation from its natural value. over a time horizon of one year depending on On one hand, it is very probable that credit the general risk rate institutions with a level of the general risk rate much below the value of 50 percent could not obtain satisfactory incomes, and, on the other hand, credit institutions with a level of the general risk rate well above 50 percent can be assumed to face a precarious asset quality and hence high provision expenses. Thus, banks with significant deviations (positive or negative) of the general risk rate from the 50 percent level face a higher risk of CAAMPL rating downgrade than those with a level of the general risk rate close to 50 percent (Chart 3). Consequently, the expected sign for the attached coefficient of the variable representing the square of the general risk rate deviation is positive. 21

22 IV. EMPIRICAL ANALYSIS After the univariate tests, only four variables were retained for the multivariate estimations: (i) the bank rating for the period in which the analysis is done (RATING); (ii) the market share of loans to non-bank clients portfolio (MKS Loans); (iii) the weight of net past due and doubtful loans in total assets (NPL); (iv) the square of general risk rate deviation from its natural level (PARGR). Statistical analysis has shown that, until now, both capital adequacy and liquidity indicators did not provide any significant information in order to explain the rating downgrade phenomenon. At the same time, profitability indicators have a reduced capacity to identify CAAMPL rating downgrade events. Using the four criteria to estimate the polynomial logit function, the result obtained after the tests on the training sample resembles those obtained in other studies that modeled the rating downgrade probability: Table 2. CAAMPL rating downgrade model estimation results (the estimation sample: December December 2002) Variables Coefficients Standard Error z-statistic Probability RATING MKS Loans NPL PARGR Constant McFadden R Non-downgrade observations 880 Akaike info criterion Downgrade observations 194 Schwarz criterion Total observations The values of the statistical tests performed on the training sample emphasize that the estimated model is in line with the requirements of a good econometric performance. The coefficients are statistically 10 significant and their signs are in accordance with economic 10 Statistical relevance of the selected criteria is highlighted both by the substantial values of the z-statistic indicators associated to the coefficients of the multivariate function estimated, and by the low level of correlation between variables (Annex 1). Stability tests reveal that statistical significance is maintained for each of the four variables, even in the case of random re-sampling by subgroup. 22

23 theory. The downgrade probability 11 is negatively influenced by the current rating and by the credit market share, while the weight of non-performing loans in total assets and the square of the general risk rate deviation from the 0.5 value have a positive influence. The high accuracy of the downgrade rating model predictions is ensured by the scoring function performance, in terms of discriminatory power, stability and adequate calibration of its estimates. The discriminatory power has been tested both for observations included in the considered period (December December 2002) in order to estimate the scoring function, and for those of the Chart 4 ROC Curve subsequent period (January 2003 December 2005). We have used ROC (December 1999-December 2005) curve and the area under ROC curve indicator (AUROC). The results show high values of the AUROC indicator for the two periods analysed, i.e percent for the estimation period and percent for the subsequent period (out of time). This performance allows us to reach a level of percent when we test the whole sample (Chart 4), a value significantly above the 75 percent threshold, which is considered the benchmark. Moreover, the numerical results above are confirmed by the shape of the ROC curve. The concavity of the ROC curve emphasizes that the selected variables have a high discriminatory power, which ensures that the scoring function can provide a good ordering of banks, based on their downgrade probability. Hence, the model manages to concentrate the vast majority of downgrading cases in the riskiest classes, while the curvature of the ROC test goes near to the unit square margins; in fact, the concavity of the ROC curve is equivalent to highly informational content scores, being a decreasing function. If the model had not had significant discriminatory power, the scores associated with CAAMPL rating downgrade events would have been randomly spread across the chart, without any concentration, so that the ROC curvature would have resembled the main bisector. 11 The positive sign associated with a variable shows that an increase in that variable s value, ceteris paribus, determines an increase in rating downgrade probability, while a negative sign has an opposite influence. 23

24 Had the model been perfect, the scores for all rating downgrade events would have been worst than that of the least powerful non-downgrading bank. The assessment of the stability of this performance has been done by estimating the confidence interval of the area under the ROC curve indicator (AUROC), running the bootstrap procedure with 1,000 iterations. The results show that the model s ability to ex-ante discriminate between CAAMPL downgrade and nondowngrade events remains high for all those 1,000 random re-samplings, the percentiles of 97.5 and 2.5 being of 87.3 percent and 82.7 percent respectively (Chart 5). Chart 5 AUROC Bootstrap Forecasting CAAMPL rating downgrade events based on the estimated scoring function requires setting the value of the alarm threshold and, on the other hand, calibrating the rating scale, namely the CAAMPL rating downgrade risk categories. Conceptually speaking, when the theoretical probability of rating downgrade exceeds the value of the alarm threshold, the model signals that the credit institution will record a weaker rating in the future; otherwise, the model signals that the credit institution will have a rating at least as good as the current one. In this respect we used two thresholds, having in mind their relevance in the supervisory activity. When the threshold s value is set so that false alarms are as costly as non-identified downgrades, the success rate is percent, while if the threshold s value is set so as to obtain a level of 15 percent for type 1 error, the success rate decreases to percent (Table 3). In the former case, the decision threshold was set to 24.5 percent, while in the latter to percent. Table 3. Early warning system accuracy Success rate Prediction errors 24 P*=24.5% Type 1 Error = 15% Dep = 0 Dep = 1 Total Dep = 0 Dep = 1 Total 80.14% 80.00% 80.12% Success rate 71.03% 85.08% 72.94% 19.86% 20.00% 19.88% Prediction errors 28.97% 14.92% 27.06%

25 The reduction of the early warning system s overall accuracy in the latter case is an expected result, considering that the positive effect generated by the decreasing number of nonidentified downgrades was to be outperformed by the negative effect due to the increasing number of false alarms (Chart 6), on the background of the sample s structural disequilibrium 12. However, the significant increase of the model s performance in identifying the rating downgrade events is worth mentioning, on the background of diminished type 1 errors from 20 percent to percent. Thus, if we set the threshold at , only 15 percent of rating worsening events are un-signalled, given an acceptable false alarm rate. Consequently, the relative cost will be of two to one between type 1 and type 2 errors. Therefore, establishing an alarm threshold of percent is equivalent to attributing an importance twice as high to errors due to unidentified downgrades, instead of Chart 6 Prediction errors based on the threshold probability (C) false alarms errors. Although the general accuracy level of the scoring function is high 13 and comparable to those achieved by the early warning systems employed by other central banks 14, it is important to assess its performance from bank to bank. Banking activity heterogeneity could cause major discrepancies relative to the model s predictive power from a credit institution to another The ratio of rating downgrade events to upgrade or maintaining events is around one to seven, only percent of the total number of observations being cases of CAAMPL rating downgrades. Especially when we attach the same importance to non-identified downgrades and false alarms. For instance, the general rate of accuracy of the early warning system used by Banca d Italia was of 85 percent upon its implementation (according to Laviola, Marullo, Trapanese 1999). 25

26 On the other hand, understanding its statistical limits and adding qualitative information to its evaluations represent a fundamental premise in order to obtain an even higher performance level of the CAAMPL early warning system for rating downgrade events. Testing the manner in which the scoring function performs for each bank allows the distinct interpretation of signals provided by the model from bank to bank, based on its previous accuracy. Thus, for those banks having a good historical performance, i.e. a general rate of success of over 90 percent, the signal will be interpreted mainly 15 in the sense specified by the model, otherwise an additional qualitative analysis being necessary. By establishing the alarm threshold at 24.5 percent, we found that the model s general rate of success is extremely high, especially for banks of systemic importance. The simple application of the alarm threshold to the top eight credit institutions based on total asset value at end-2006 helps correctly identify over 90 percent of cases, except for bank B16, for which the model obtained a performance of percent; we have noticed the situation of banks B1 and B4, for which the success rate is of 100 percent (Annex 2). Moreover, by computing the average rate of success as a weighted value of the market share by assets, we obtain a level of percent, relative to percent, for the simple arithmetic mean. On the other hand, we noticed that the scoring function has an unsatisfactory performance for three of the banks with foreign capital (banks B28, B29 and B30) and for one of the banks with private domestic capital (bank B31). This result is determined by the great number of false alarms, keeping in mind that the rating downgrade events have been completely identified by the model. For these reasons, qualitative analysis will have an important role in interpreting the signals provided by the scoring function for each of the four credit institutions. The classification is similar if we use a threshold level of percent (Annex 2), except its lower side. In this alternative decision framework, banks for which the model performs worst are bank B21 (15.07 percent) and bank B27 (2.76 percent). For banks lying at the lower end of the classification, the sensitivity of the statistical model s performance to a change in the alarm threshold requires a more detailed statistical analysis of the informational content of the theoretical downgrade probability. When representing graphically, based on the score obtained, the distribution of the CAAMPL rating downgrade events comparatively with that of upgrading or maintaining events (Chart 7), we notice that 15 Depending on the error risk to which the signal provided by the model is exposed. For instance, if the model anticipates the rating downgrade of a bank after four quarters, the error risk is given by the probability that the signal is a false alarm. 26

27 the common area of the two distributions is quite large (70 percent) for a value of the theoretical downgrade probability between 0.1 and 0.29 (about 21 percent from observations). Chart 7 The distribution of rating downgrade probability This intuitive representation reveals that, although the model succeeds in precisely discriminating the two distributions, the exclusive interpretation of results obtained for values of the theoretical downgrade probability situated close to the score corresponding to the intersection of the two distributions on the basis of an alarm threshold is insufficient. To overcome this obstacle, we proceeded to calibrating the rating scale, by applying the methodological requirements of Basel II specific to the credit risk internal model approach. Having theoretical probabilities and the vector of rating downgrade events as a starting point, we have obtained six downgrade risk classes. The criteria employed were the homogeneity of events in the same class and the discrimination of empirical CAAMPL rating downgrade probability between the different risk categories. The empirical probabilities obtained are robust estimates in terms of CAAMPL rating downgrade risk, and allow a more flexible interpretation of alarms generated by applying the alarm threshold. The empirical values for the rating downgrade probability 16 attributed to each risk category 17 are illustrated in Table The empirical rating downgrade probability represents the share of non-paying debtors in total number of debtors of a risk class. According to the requirements laid down in para. 446 of the Basel II Accord (BCBS 2004), namely the fact that a rating system should include a rating scale for debtors, in order to exclusively reflect the measurement of probability of default. 27

28 Table 4. Segmentation on CAAMPL rating downgrade risk classes Score limits Empirical probability c1 [0; ) 1% c2 [0.0054; ) 2% c3 [0.0486; 0.102) 4% c4 [0.102; ) 13% c5 [0.2854; 0.46) 34% c6 [0.46; 1] 66% Source: NBR, own calculations Thus, there is a chance in eight that a bank with a score ranging between 0.1 and will record a weaker rating over a time horizon of one year, bearing in mind that only 13 percent of banks classified in class C4 recorded a CAAMPL rating downgrade. Class C4 represents a medium risk category, given that its empirical probability of 13 percent signals a level very close to the unconditional downgrade probability 18 existing for the whole data sample considered. At the same time, classes from C1 to C3 can be considered as implying a lower risk of rating downgrade, while classes C5 and C6 have an increased risk of downgrade. From the standpoint of the process of discrimination between downgrade events and upgrade or maintaining events, this result strengthens the finding that the simple use of an alarm threshold is insufficient in ensuring the accuracy of the early warning system for CAAMPL rating downgrade for all credit institutions, Romanian legal entities. On the other hand, it should be noticed that, although classes from C1 to C3 have a low risk of downgrade, banks classified in class C3 are twice riskier than those in class C2 and four times riskier than those in class C1. At the same time, although both classes C5 and C6 display a high risk of downgrade, we should mention that banks classified in C6 are twice riskier than those in class C5, the probability of a downgrade phenomenon emerging being of two events in three in case of C6, and of only one event in three in case of C5. 18 Unconditional probability refers to the mean of the dependent variable for the whole sample. In our case, the dependent variable took the value 0 in 1,862 cases and 1 in 292 cases, i.e. a weight of percent in the former case and of percent in the latter. 28

29 Besides the precise discrimination of risk based on the empirical downgrade probability, the quality of the rating scale is also emphasized by debtor distribution across the six risk classes; its granularity allows avoiding a situation of concentration of notes into a certain class 19 (Chart 8). The most concentrated level is in class C3 (22.2 percent), which is under the 35 percent ceiling (the benchmark for the cases of unjustified concentration). Chart 8 Banks distribution on CAAMPL rating downgrade risk classes The above-mentioned arguments lead us to the conclusion that there is a set of indicators which makes possible the modelling of the C1 C2 C3 C4 C5 C6 CAAMPL rating downgrade phenomenon. Even the exclusive use of microprudential data allows us to develop a good predictive scoring function. The results are encouraging also because the estimation sample included all credit institutions, Romanian legal persons, irrespective of their business strategy or lending activity. 25% 20% 15% 10% 5% 0% Distribution of banks on risk classes - left scale Empirical downgrade probability - right scale 70% 60% 50% 40% 30% 20% 10% 0% 19 According to the requirements under para. 403 and para. 406 of the Basel II Accord (BCBS 2004), namely the fact that credit institutions holding portfolios highly concentrated on a particular market segment and lying in a certain interval of default risk should have a sufficient number of rating classes in this particular interval in order to avoid an excessive concentration of debtors into a particular rating class. 29

30 CONCLUSIONS The present study synthetically describes the mechanism of an early warning system of CAAMPL bank rating downgrade events, with a prediction horizon of one year, and the results of its estimations as of 31 December Its main component is represented by a statistical model for measuring the CAAMPL rating downgrade probability, being estimated and tested exclusively on microprudential data covering the period December 1999 December The specific methodology employed for estimation, tests and implementation gathers issues both from economic best practice and supervisory requirements laid down in the Basel II Accord, concerning the topic of credit risk internal models, having as purpose the achievement of a long-lasting, highly-performing process of identification of the rating downgrade events for Romanian credit institutions. The high accuracy of the statistical model is ensured by the rating function s performance, in terms of discriminatory power, stability and adequate calibration of its estimates. The value of the area under ROC indicator is high and robust, the stability tests indicating small fluctuations around the level of 85 percent, which represents the value for the entire sample of observations. Also, the check up of the capacity to ex-ante distinguish between CAAMPL rating downgrade events and upgrade or maintaining events showed a higher performance for the testing sample (January December 2005) than for the estimation sample (December December 2002). The model performs very well especially for credit institutions with a large share in the Romanian banking system, the weighted mean of the general accuracy rate being of over 90 percent, instead of 80 percent for the arithmetic mean, assuming the calibration of the alarm threshold so that the percentage of false alarms equal that of unidentified downgrades. Notable are the percentages of performance for BCR, BRD and Raiffeisen Bank, of 100 percent for the first and third bank and percent for the second one. On the other hand, for three of the foreign capital banks and for one bank with private domestic capital the scoring function registered an unsatisfactory performance. This result is determined by the great number of false alarms, keeping in mind that the rating downgrade events had been completely identified by the model. For these reasons, the ex-ante identification of downgrade rating events is not made exclusively according to the alarm threshold, the probability of success of the signal provided by the model being evaluated also by means of a rating scale and of qualitative analysis. The rating scale includes six risk classes, which ensure a good segmentation of the downgrade probability based on the score obtained. The criteria employed for constructing this scale were the homogeneity of events in the same class and the discrimination of empirical CAAMPL 30