Kalman flerng as a performance monorng echnque for a propensy scorecard Kaarzyna Bjak * Unversy of Souhampon, Souhampon, UK, and Buro Informacj Kredyowej S.A., Warsaw, Poland Absrac Propensy scorecards allow forecasng, whch bank cusomers would lke o be graned new creds n he near fuure, hrough assessng her wllngness o apply for new loans. Kalman flerng can help o monor scorecard performance. Daa from successve monhs are used o updae he baselne model. The updaed scorecard s he oupu of he Kalman fler. There s no assumpon concernng he scorng model specfcaon and no specfc esmaon mehod s presupposed. Thus, he esmaor covarance s derved from he boosrap. The focus s on a relaonshp beween he score and he naural logarhm of he odds for ha score, whch s used o deermne a cusomer s propensy level. The propensy levels correspondng o he baselne and updaed scores are compared. Tha comparson allows for monorng wheher he scorecard s sll up-o-dae n erms of assgnng he odds. The presened echnque s llusraed wh an example of a propensy scorecard developed on he bass of cred bureau daa. Keywords Propensy scorecard, scorecard monorng, Kalman flerng, boosrap. Inroducon Propensy scorng Accordng o Thomas e al (2002, p 1), cred scorng s he se of decson models and her underlyng echnques ha ad lenders n he granng of consumer cred. Nowadays mos banks use scorng o predc he cred rsk of her acual or poenal cusomers. Scorng * I am graeful o an anonymous referee for suggesons and commens ha helped mprove hs paper. I would also lke o say hank you o Sophe N Ja for proof-readng he draf. 1
models are also developed by cred bureaus o help banks assess cred rsk on he bass of daa comng from he bankng secor as a whole. The mos common form of such models s a scorecard. Mays (2004, p 63) defnes he scorecard as a formula for assgnng pons o applcan characerscs n order o derve a numerc value ha reflecs how lkely a borrower s, relave o oher ndvduals, o experence a gven even or perform a gven acon. The characerscs (varables) can have several dscree arbues o whch he scorecard assgns pons (arbue scores). A cusomer s score s calculaed as a sum of he arbue scores. Mos cred scorng models and echnques can be adaped for oher bank acves such as collecon (see Mays, 2004, p 7), fraud deecon or markeng (see Thomas e al, 2002, p 4). In order o selec cusomers for markeng campagns (especally drec-mal ones), some banks use propensy scorecards ha allow for he forecasng of whch of her cusomers wll soon be neresed n new creds. Such models faclae he predcon of cusomers wllngness o apply for new loans (cred propensy) n he same way ha he cred scorecards make possble o predc cred rsk. Whle usually he hgher he cred score, he lower he rsk (and he beer he cusomer), s assumed here ha he hgher he propensy score, he lower he cusomer s wllngness o apply for new loans. However, n pracce, propensy scorecards are somemes scaled so ha he hgher he score, he hgher he wllngness (and he more aracve he cusomer). In cred scorng, cusomers are dvded no goods (credworhy) and bads (uncredworhy). Smlarly, n propensy scorng hey can be dvded no he wllng and he unwllng o apply for new loans. In hs research he wllng cusomers are defned as hose who appled for new loans n a four-monh oucome perod beween he observaon pon and he oucome pon. The observaon pon s a dae on whch daa on a cusomer s behavour are colleced, and he oucome pon refers o he dae on whch her saus s deermned. Because he cred bureau daa, ha peran o he whole bankng secor, are used, a cusomer s saus s deermned regardless of whch bank hey appled o for a loan. A rao of goods o bads s referred o as he odds n cred scorng. Smlarly, here he odds are defned as a rao of he unwllng o he wllng among cusomers havng a gven score or a score comng from a gven range. In parcular, he odds can be calculaed as a rao of he unwllng o he wllng n he whole sample. Irrespecve of wheher he odds are compued for a score, a score range or a sample, hey can be reaed as a measure of cred propensy. 2
Scorecard monorng Once a scorecard has been mplemened, s monorng (usually called valdaon) has o be performed regularly. Accordng o Thomas e al (2002, p 17), monorng a scorecard s a se of acves nvolved n examnng he curren bach of applcaons and new accouns and assessng how close hey are o some benchmark, whch s usually deermned on he bass of he developmen sample. A dsncon s made beween monorng and rackng; he laer conssng of comparng expeced and observed performance of cohors of accouns over me. However, oher researchers consder rackng repors as a specfc ype of monorng repors. A complee se of scorecard monorng repors s suggesed and descrbed n deal n Mays (2004, chaper 13). Those repors can be dvded no fron-end and back-end ones. Fron-end repors do no requre nformaon abou defauls. There are repors on populaon sably and approval rae, characersc analyss, overrde rae and overrde reasons. On he conrary, back-end repors are based on nformaon abou defauls. There are good/bad separaon and early performance score repors. The good/bad separaon repors allow for he evaluaon of how well he scorecard separaes goods from bads (usng e.g. he Kolmogorov-Smrnov (KS) sasc). They also enable he observaon of changes n a relaonshp beween he score and he odds for ha score. Those shfs - as well as changes n dscrmnaory power - can be denfed by comparng curren resuls wh prevous ones (e.g. one year ago) and wh hose based on he developmen sample. In order o produce a good/bad separaon repor, one has o collec daa coverng an oucome perod of he same lengh as assumed n he scorecard developmen process (whch s usually a leas welve monhs n cred scorng). Snce hs akes some me, he early performance score repors can be useful. Those repors conss of bad raes n esablshed score ranges. There are bad raes n dfferen cohors of accouns afer, for example, hree and sx monhs of bookng. For even earler evaluaon of he scorecard effecveness, he defaul can be replaced wh 30+ or 60+ days-pas-due n hose analyses (see Mays, 2004). Anoher se of monorng repors s proposed and descrbed n Anderson (2007, chaper 25). There are he followng repor ypes: porfolo analyss, performance monorng, drf repor, decson process monorng and ohers (overrde analyss ec.). Porfolo analyses nclude delnquency dsrbuons and ranson marces, whle drf repors cover populaon sably checks and possble score shfs. Performance monorng consss of examnng 3
dscrmnaory power, accuracy (calbraon) and sably of he scorecard. There are scorecard performance repors, vnage analyses and score msalgnmen repors. Scorecard performance repors are smlar o he good/bad separaon repors, whle vnage analyses are smlar o he early performance score repors. The score msalgnmen repors allow for he denfcaon of problems a he characersc level: pons assgned o arbues of one or more characerscs mgh have sopped reflecng cred rsk relaed o hose arbues (see Anderson, 2007). Sandard scorecard monorng repors are also descrbed n Sddq (2006, chaper 9); he usual mehods of scorecard monorng are presened n Lucas (2004) as well as n Van Gesel and Baesens (2009, p 269-272), and some useful advce on he opc s provded n Schffman (2001). Moreover, dealed nformaon on measurng dfferen aspecs of scorecard qualy (ncludng a wde selecon of dscrmnaory power measures) can be found n Thomas (2009, chaper 2). The above-menoned repors are desgned for cred scorng models. However, mos of hem can be used o monor scorecards appled n oher areas. Obvously, he defaul has hen o be replaced wh he modelled phenomenon, and he cusomer s saus has o be redefned accordngly. In parcular, performance repors can be prepared for a propensy scorecard. The man drawback of he commonly used approach o such repors les n usng - besdes he developmen sample - only he curren monorng sample whch s colleced for one seleced momen and hus may be aypcal (e.g. because here was a perod n whch some cred producs have been offered a unusually aracve condons). Whaker e al (2007) presen a new scorecard performance monorng echnque ha s free from he above-menoned dsadvanage. The echnque s derved from he Kalman flerng. There s an assumpon ha he model parameers change consanly and he successve monorng samples provde her measuremens. Those measuremens are used o updae he baselne model and he updaed scorecard s he oupu of he Kalman fler. The echnque s demonsraed for a logsc regresson model esmaed usng he maxmum lkelhood mehod, and llusraed wh an example of a dynamc morgage scorecard (see Whaker e al, 2007). In hs paper he same echnque s used bu a more general approach s presened and appled. There s no assumpon concernng he scorng model specfcaon and no specfc esmaon 4
mehod s presupposed. Unlke n Whaker e al (2007), rackng all arbue scores s no of neres here. The focus s on a relaonshp beween he score and he naural logarhm of he odds for ha score. Tha relaonshp s used o deermne he propensy level of a cusomer havng a gven score. The log odds esmae, whch represens he propensy level (provded ha he baselne scorecard s sll up-o-dae), s compared wh he esmae calculaed usng he relaonshp beween he updaed score and he log odds. Tha comparson allows for conrollng wheher he scorecard s n fac up-o-dae n erms of assgnng he odds. As an example a propensy scorecard s used, developed and sysemacally updaed on he bass of cred bureau daa. The presened model s a sample one, developed only for he purposes of hs research, and he analysed propensy scores dffer from hose offered by he Polsh cred bureau, Buro Informacj Kredyowej S.A. (BIK). Mehodology Kalman fler The Kalman fler s a common mehod for esmang he sae of a nosy process (see Kalman, 1960). I enables he esmaon, when he exac sae canno be observed and here are only some measuremens (observaons) whch conan a nose. The mehod allows flerng he measuremens n order o remove ha nose (see Wells, 1996, chaper 4). I s assumed ha he curren sae of a process depends sochascally on he prevous sae. Ths relaonshp s descrbed by he sae equaon (also known as he ranson equaon). I s also assumed ha he measuremen depends sochascally on he sae a he same momen. Tha relaonshp s descrbed by he observaon equaon (also known as he measuremen equaon). Thus, he Kalman fler s used o esmae he sae of a process governed by he sae equaon, when a lnk beween he measuremen and he sae s expressed by he observaon equaon. The above-menoned equaons creae he sae space model (see Harvey, 1990, chaper 3). Accordng o Welch and Bshop (2006), he Kalman fler s a se of mahemacal equaons ha provdes an effcen compuaonal (recursve) means o esmae he sae of a process. There are wo groups of equaons, whch allow he Kalman fler esmaes o be calculaed: 5
me updae ones and measuremen updae ones. The resuls of he me updae equaons consue a pror esmaes, whch are hen used, ogeher wh measuremens, n he measuremen updae equaons o oban a poseror esmaes (see Welch and Bshop, 2006). The a poseror esmae of he sae of a process s an oupu of he Kalman fler. In parcular, parameers of a sascal model can be reaed as he sae of a process (snce hey are lkely o change over me). I seems reasonable o assume ha here s a sochasc dependence beween hem now and n he pas. Obvously, s no possble o calculae he exac values of such parameers. However, hey can be esmaed usng an approprae esmaon mehod (e.g. maxmum lkelhood, ML) and her esmaes can be hough of as he measuremen. Thus, here mus be a relaonshp beween he measuremen and he model parameers. The measuremen s assumed o conan a nose (e.g. as a resul of sample selecon). Therefore, has o be flered n order o deermne he acual esmaes of he model parameers. In hs case, he a pror esmae of he curren sae s he prevous oupu of he Kalman fler. Then, he a pror esmaes are updaed no he a poseror ones usng he measuremen (e.g. MLEs). The a poseror esmaes of he model parameers are reaed as he acual ones and consue he Kalman fler oupu. Hence, he (acual) curren esmaes depend boh on he prevous ones and on he measuremen. As suggesed n Whaker e al (2007), n hs research he scorecard parameers are he sae of a process. Ther esmaon, whch s based on a monorng sample, provdes he measuremen. The acual esmaes of he scorecard parameers are obaned usng he Kalman fler. The oupu of he Kalman fler s referred o as he updaed scorecard. The sarng model, whch s esmaed on he bass of he ranng daase, s called he baselne scorecard. Baselne scorecard The baselne scorecard s developed usng a random sample S 0. The sample S 0 s randomly dvded no ranng and es daases ha nclude, for example, 60% and 40% of cusomers, respecvely. In boh he daases he same odds are ensured. All varables, whch descrbe a cusomer s behavour, are bnned and hen some of hem are seleced no he scorecard. The bnned varables are used n he form of dummes. The model parameers are esmaed on he 6
bass of he ranng daase and he scorecard dscrmnaory power s confrmed on he bass of he es daase. As a resul here are he baselne model parameer esmaes are used as nal a pror esmaes n he Kalman fler. b ˆ 0 whch hen Sae equaon The sae of a process s consued by he scorecard parameers, whle he sae equaon descrbes he relaonshp beween he curren sae and he prevous one. I s assumed ha hs relaonshp akes he form of a mul-dmensonal random walk: q, 1 where var( ) Q for all. Accordng o Whaker e al (2007), s assumed ha he q covarance marx Q does no depend on me and ha he ndvdual model parameers vary ndependenly. As a resul of he laer assumpon, off-dagonal enres of he marx Q equal zero. Because all varables, whch are used n he model, are n he form of dummes, dagonal enres of he marx Q are equal. In consequence, he marx Q s a dagonal one: Q I, where s referred o as a sgnal o nose rao (see Whaker e al, 2007). In order o allow he model parameers o vary n me only slghly, le = 0.00001. Observaon equaon The monorng samples S come from successve monhs ( = 1, 2, ). On he bass of each sample S he model parameer esmaes m ˆ are found. Whle he parameers deermne he sae of a process, hose esmaes are used as a measuremen. The relaonshp beween hem s descrbed by he observaon equaon, whch s supposed o have he followng form: ˆ r, m 7
where var(r ) = R for all. Because here s neher assumpon on he scorng model specfcaon nor assumpon on he esmaon mehod, he esmaor feaures are unknown. However, seems safe o assume ha he esmaor s unbased and follows an asympoc mulvarae normal dsrbuon: ˆ m ~ N(, R ). Conrary o Whaker e al (2007), here s no reason o presuppose any specfc form of he esmaor covarance marx R. In parcular would be unjusfed o assume, lke Whaker e al (2007), who use he maxmum lkelhood mehod, ha he marx R s an nverse of he Fsher nformaon marx. Such an assumpon would be unjusfed because s no known wheher he esmaor s he mos effcen one. Therefore, an esmae of he marx R s derved from he paramerc boosrap. In order o perform he boosrap, a new sample B s chosen from he orgnal one S, usng proporonal samplng wh replacemen (.e. wh repeon allowed). As a resul he new sample s equal n sze o he orgnal one and he odds are he same. On he bass of he sample B he model parameers are esmaed and hen he obaned esmaes are colleced. The samplng and he esmaon are repeaed, for example, 100 mes. The colleced parameer esmaes are used o compue he covarance marx whch consues he boosrap esmae of he marx R. Updaed scorecard The updaed scorecard s he oupu of he Kalman fler. Is parameer esmaes ˆ are found usng he Kalman fler on he n-dmensonal sae space, where n s a number of he model parameers. Those esmaes are he acual ones, whle he esmaes on he bass of a monorng sample, are reaed as only a nosy measuremen. m ˆ, whch are obaned In order o calculae he Kalman fler esmaes, he me and measuremen updae equaons are used (see Welch and Bshop, 2006). Frsly, usng he me updae equaons, he a pror esmaes ˆ are deermned and he a pror error covarance marx P s compued: 8
ˆ ˆ, P 1 P 1 Q. In hs case he curren a pror esmaes are equal o he prevous a poseror esmaes. Secondly, he Kalman gan K s calculaed accordng o he followng formula: 1 K P ( P R ). Fnally, usng he measuremen updae equaons, he a poseror esmaes he a poseror error covarance marx P s compued: ˆ are found and ˆ ˆ ( ˆ m ˆ K ), P ( I K ) P. The parameer esmaes ˆ of he updaed scorecard are deermned on he bass of he a pror esmaes ˆ, he Kalman gan K and he esmaes m ˆ from he monorng sample. As far as nal values are concerned, s assumed ha he a pror esmaes 1 ˆ are equal o he parameer esmaes b ˆ 0 of he baselne model: ˆ ˆ ˆ. 1 0 b 0 Accordng o Whaker e al (2007), he nal error covarance marx P 0 should be such ha he baselne model parameer esmaes have a relavely weak nfluence on he esmaes ˆ 1. Therefore, s supposed ha P 0 = 10000I. As a consequence, he esmaes ˆ are affeced more by he esmaes m ˆ from he monorng sample han by he parameer esmaes of he baselne model. The updaed scorecard s deermned on he bass of boh he curren monorng sample and he prevous ones bu depends more on he former han on he laer. 9
Lnear model I s common pracce ha once a scorecard has been developed, an addonal lnear model s esmaed, n order o fnd a relaonshp beween he score and he naural logarhm of he odds for ha score (see Mays, 2004, p 71). Tha lnear relaonshp s ofen used o scale he scorecard,.e. o change he model parameers so ha here s a requred dependency beween he score and he odds or he probably of he modelled phenomenon (see Sddq, 2005, p 113). I s especally useful when an nsuon (a bank or a cred bureau) has several scorecards and wans hem o be conssen n erms of scale. In order o esmae an addonal lnear model, he whole score range s dvded no m equallengh ranges. The model s developed on he bass of he daa on he md-pons and he log odds of hose ranges (see Mays, 2004, p 71). Some cusomers wh he lowes and hghes scores can be reaed as oulers and hus excluded from he model esmaon. In hs research, addonal lnear models are bul for boh he baselne and updaed scorecards. For he baselne scorecard, he followng model s assumed: ln( oˆ ) ˆ ˆ s, b b a0 b0 b where b s s a score comng from ha scorecard. The baselne lnear model s esmaed on he bass of he sample S 0. The above relaonshp s used o deermne he cusomer s cred propensy accordng o he baselne scorecard. Usng he pon esmaon, one could predc ha a cusomer, whose baselne score equals b s, s wllng o apply for new loans a he level correspondng o he odds ô. Usng he nerval esmaon, one could oban he 90% l u confdence nerval l, l ) ln( oˆ ) S,ln( oˆ ) S ( of he log odds, such ha: P ln( ˆ ) S ln( o ) ln( oˆ ) S 0. 9 o, where s he approprae value of he Suden s dsrbuon wh m 2 degrees of freedom and S s he ex ane forecas error (see Greene, 2000, p 307). 10
As far as he updaed scorecard s concerned, he lnear model, whch s esmaed on he bass of he monorng sample S, akes he followng form: ln( oˆ ) aˆ bˆ s, where s s a score ha comes from he menoned scorecard. The above relaonshp enables he cusomer s cred propensy o be deermned accordng o he updaed scorecard. Usng he pon esmaon, could be predced ha a cusomer, whose updaed score s equal o s, s wllng o apply for new loans a he level correspondng o he odds ô. Performance monorng Each cusomer belongng o he monorng sample s scored usng boh he baselne scorecard and he updaed one. Boh scores are calculaed on he bass of he cusomer s daa for he same momen. Then he log odds are esmaed usng he baselne lnear model and he lnear model for he updaed scorecard, respecvely. Those odds are reaed as measures of he cusomer s cred propensy accordng o he baselne and updaed scorecards. Provded ha he baselne scorecard s sll up-o-dae, he odds should no dffer oo much from each oher. In parcular, he log odds esmae, whch s obaned on he bass of he updaed score, should n prncple le whn he 90% confdence nerval deermned usng he baselne score of he cusomer for he same momen. If he esmae does no f whn he nerval, he baselne and updaed scorecards dffer consderably n her assessmen of he cusomer s cred propensy level. If here are numerous cases lke ha, one can conclude ha he baselne scorecard s no up-o-dae n erms of assgnng he odds and probably of applyng for a new loan. Therefore, he percenage of cusomers, for whom he above-menoned condon s no fulflled, s analysed for each monorng sample. Smulaneously, he scorecard performance measures, he Gn coeffcen and he KS sasc, are racked n order o verfy he dscrmnaory power of he baselne scorecard over successve monhs. In propensy scorng he Gn coeffcen s a measure of ably o rank cusomers accordng o her cred propensy whle he KS sasc measures ably o separae he wllng from he unwllng. However, even f he rankng and separaon sascs reman unchanged, he relaonshp beween he score and he log odds can change 11
consderably (see Mays, 2004, p 116). I can mean ha he cred propensy level s sysemacally under- or overesmaed. Therefore, he propensy scorecard monorng should nclude an analyss of he menoned relaonshp. Such an analyss usually consss of comparng he acual (emprcal) odds wh her esmaes obaned usng he baselne lnear model. The baselne scorecard performance s assessed usng one monorng sample each me. Thus, a sngle unypcal sample can lead o a negave monorng resul and redevelopmen of he model. However, n hs paper ha approach s replaced wh rackng he percenage of cusomers whose updaed odds do no le whn he 90% confdence nervals deermned usng her baselne scores. Thus, he baselne scorecard performance s assessed usng no only he curren monorng sample, bu hrough he Kalman fler all prevous ones as well. Emprcal resuls Daa The presened example s based on he cred bureau daa conssng of welve samples: a baselne sample and eleven monorng ones. Each sample has a dfferen observaon pon. Those observaon pons are derved from welve successve monhs. The oucome perod always equals four monhs here. In each sample here are cusomer s characerscs (varables) as of he observaon pon and a cusomer s saus (wllng or unwllng) as of he oucome pon. The baselne scorecard s developed usng a random sample conssng of 6309 cusomers (ncludng 1229 wllng ones) whose daa are colleced n he BIK daabase. The observaon pon s he 1s of Sepember 2005 and he oucome pon s he 1s of January 2006 (four monhs laer). The sample s also used o develop he baselne lnear model. The monorng samples, whch are used o calculae esmaes servng as measuremens n he Kalman fler, come from eleven successve monhs ( = 1, 2,, 11). Each monorng sample consss of over sx housand cusomers randomly seleced from he daabase. In he consecuve samples here are he followng observaon pons: he 1s of Ocober 2005, he 1s of November 2005,, he 1s of Augus 2006. Because a four-monh oucome perod s assumed, he respecve oucome pons are: he 1s of February 2006, he 1s of March 2006,, he 1s of 12
December 2006. Boh n he baselne sample and n he monorng ones he odds are smlar and equal ca 4. Model parameers The developed model has 29 parameers (nne characerscs n he form of bnned varables). Snce hs s a propensy scorecard based on cred bureau daa, he varables descrbe he cusomer s cred hsory and cred acvy (especally whn he las year). There are such characerscs as: number of cred nqures whn he las 12 monhs (0, 1, 2 or 3 and more), number of loans graned whn he las 12 monhs (0, 1 or 2 and more), number of pas loans (0, 1, 2-3 or 4 and more), me snce las cred nqury (below 6 monhs or 6 monhs and above, or no nqures), and number of dfferen producs appled for whn he las 12 monhs (1 or 2 and more, or no nqures). The model parameers are esmaed usng commercal sofware dedcaed o scorecard developmen. The esmaon mehod s no menoned n he sofware documenaon and hus he esmaor feaures reman unknown. Scorecard monorng The baselne scorecard s monored usng he Kalman-fler-based echnque descrbed n hs paper. In he begnnng (for = 0) here s no updaed model. However, for = 1 he baselne scorecard s reaed as f were an updaed one from he precedng momen (n order o deermne he nal a pror esmaes of he Kalman fler). Therefore, n he begnnng he updaed model can be assumed o be he same as he baselne one, and he racked percenage of cusomers s equal o zero. The nex updaed scorecard s he frs oupu (a poseror esmaes) of he Kalman fler. The model parameer esmaes, whch are obaned on he bass of he frs monorng sample, consue he measuremen used o produce ha oupu. The parameer esmaes of he updaed scorecard serve as he a pror esmaes, whch are hen ransformed no he a poseror ones usng esmaes from he second monorng sample (he nex measuremen). The a poseror esmaes consue he second updaed scorecard. They serve hen as he a pror esmaes used (ogeher wh he new measuremen) o esmae parameers of he hrd updaed scorecard, and so on. As a resul, here s a sequence of updaed scorecards. As an example, measuremens and updaed arbue scores of he seleced characersc (number of pas loans) are presened n Fgure 1. 13
For each updaed scorecard an addonal lnear model s bul (based on he monorng sample). The baselne scorecard s observed over a year. For each monh cusomers from he monorng sample are scored usng ha model. Then he Gn coeffcen and he KS sasc of he baselne scorecard are compued. The updaed scorecard and lnear model are used o calculae he updaed odds for each cusomer. I s checked wheher hose odds f whn he 90% confdence nerval deermned usng he cusomer s baselne score and lnear model. The percenage of cusomers, whose updaed odds do no f whn he nervals, s compued. Scorecard performance All he racked measures are presened n Table 1 and llusraed n Fgures 2, 3 and 4. In he begnnng (for = 0) he Gn coeffcen and he KS sasc of he baselne scorecard equal ca 0.43 and 0.33, respecvely. However, n he monorng perod hey are slghly lower. Alhough hey reman relavely sable over me, here s some evdence ha he baselne scorecard has deeroraed. The percenage of cusomers, whose updaed odds do no le whn he 90% confdence nervals deermned usng her baselne scores, ncreases generally over he successve monhs. The observed endency s clear: he cred propensy level s eher under- or overesmaed for he ncreasng percenage of cusomers. Afer egh monhs of he model monorng, for = 8, he racked percenage exceeds 20%, whch means ha more han one n fve cusomers has an updaed odds lyng beyond he nerval. I seems ha one could expec hs o ncrease furher and, as a consequence, furher degradaon of he baselne scorecard n he subsequen monhs could be also expeced. The obaned resuls could be nerpreed n he followng way. Snce he dscrmnaory power measures are reasonably sable, he scorecard reans s ably o separae he wllng from he unwllng as well as o rank cusomers accordng o her wllngness o apply for new loans. However, here s an ncrease n he percenage of cusomers whose updaed odds do no f whn he deermned nervals. Thus, he successve updaed scorecards dffer more and more from he baselne one n her assessmen of he cusomer s cred propensy level. Ths could be nerpreed ha n he consecuve monhs he model becomes less and less upo-dae n erms of assgnng he odds and probably of applyng for a new loan. 14
Conclusons The presened example demonsraes ha a scorecard may become less up-o-dae, alhough he commonly used performance measures such as he Gn coeffcen or he KS sasc do no change consderably. I s up o he decson makers as o wha he maxmum value of he analysed percenage ha can sll be acceped s (probably 10% would be a good dea n he case ha he 90% confdence nervals are used). Once such a value has been exceeded, he model has o be redeveloped (or a compleely new model should be bul). Usng a degraded scorecard may resul n wrong busness decsons and hus s no recommended, especally n he case of a cu-off deermned on he bass of a relaonshp beween he score and he odds for ha score. Kalman flerng can help deec such scorecard falures n he monorng process. One of he man advanages of ha echnque seems o le n usng no only he curren monorng sample bu hrough he Kalman fler all prevous ones as well. In effec, possble local dsurbance should have a lmed nfluence on he monorng resuls and hus on he decsons based on hem (poor monorng resuls can ndcae ha a new model should be bul). Anoher advanage of he approach, whch s presened n hs paper, les n he lack of assumpons concernng boh model specfcaon and esmaon mehod (ofen he case n praccal applcaons based on commercal sofware). Thus he demonsraed echnque seems useful as a monorng ool for dfferen scorecards, ncludng, bu no lmed o, propensy ones. A monorng resul, whch ndcaes ha he scorecard does no assgn odds correcly, s of grea mporance for he scorecard user. However, s somemes more mporan o know wheher he odds are sysemacally under- or overesmaed. In cred scorng overesmaed odds mean underesmaed cred rsk. In such a suaon, score-based cred decsons may resul n an unexpeced decrease n he bank porfolo qualy. In he reverse suaon an excessve number of applcans are rejeced, whch reduces he bank prof. In propensy scorng, he underesmaed odds seem o be a more serous problem han he overesmaed ones, because among cusomers seleced for a markeng campagn here are less wllng ones han expeced. Thus, he response rae may be lower han assumed, whch has a negave nfluence on he campagn effcency. Therefore, furher modfcaons of he presened echnque could more specfcally dsngush beween under- and overesmaon of he odds. 15
Monh Percenage of cusomers Gn coeffcen KS sasc Sep-05 0.0% 0.431 0.327 Oc-05 7.2% 0.359 0.263 Nov-05 3.9% 0.363 0.289 Dec-05 11.8% 0.350 0.275 Jan-06 4.7% 0.406 0.309 Feb-06 8.3% 0.370 0.278 Mar-06 14.3% 0.349 0.260 Apr-06 9.5% 0.355 0.265 May-06 22.1% 0.370 0.276 Jun-06 13.3% 0.375 0.276 Jul-06 16.5% 0.376 0.282 Aug-06 18.4% 0.406 0.322 Table 1. The monorng resuls 16
arbue scores 0.50 0.25 0.00-0.25-0.50-0.75-1.00 Sep-05 Oc-05 Nov-05 Dec-05 Jan-06 Feb-06 Mar-06 Apr-06 May-06 Jun-06 Jul-06 Aug-06 monh 0 0 (updaed) 1 1 (updaed) 2-3 2-3 (updaed) 4+ 4+ (updaed) Fgure 1. Arbue scores of he seleced characersc (number of pas loans = 0, 1, 2-3 or 4 and more): esmaed on he bass of successve samples and updaed usng he Kalman fler 17
% cusomers 25% 20% 15% 10% 5% 0% Sep-05 Oc-05 Nov-05 Dec-05 Jan-06 Feb-06 Mar-06 Apr-06 May-06 Jun-06 Jul-06 Aug-06 monh Fgure 2. The percenage of cusomers whose updaed odds do no le whn he nervals 18
Gn coeffcen 0.5 0.4 0.3 0.2 0.1 0.0 Sep-05 Oc-05 Nov-05 Dec-05 Jan-06 Feb-06 Mar-06 Apr-06 May-06 Jun-06 Jul-06 Aug-06 monh Fgure 3. The Gn coeffcen of he baselne scorecard 19
KS sasc 0.4 0.3 0.2 0.1 0.0 Sep-05 Oc-05 Nov-05 Dec-05 Jan-06 Feb-06 Mar-06 Apr-06 May-06 Jun-06 Jul-06 Aug-06 monh Fgure 4. The KS sasc of he baselne scorecard 20
References Anderson R (2007). The Cred Scorng Toolk. Oxford Unversy Press: New York. Greene WH (2000). Economerc Analyss. Prence Hall: Upper Saddle Rver. Harvey AC (1990). Forecasng, srucural me seres models and he Kalman fler. Cambrdge Unversy Press: Cambrdge. Kalman RE (1960). A New Approach o Lnear Flerng and Predcon Problems. Transacons of he ASME Journal of Basc Engneerng 82 (Seres D): 35-45. Lucas A (2004). Updang scorecards: removng he mysque. In: Thomas LC, Edelman DB and Crook JN (eds). Readngs n Cred Scorng: Foundaons, Developmens, and Ams. Oxford Unversy Press: New York, pp 93-109. Mays E (2004). Cred Scorng for Rsk Managers. The Handbook for Lenders. Thomson Souh-Wesern: Mason, Oho. Schffman R (2001). Evaluang and Monorng Your Model. In: Mays E (ed). Handbook of Cred Scorng. Glenlake Publshng Company, Ld.: Chcago, pp 285-300. Sddq N (2005). Cred Rsk Scorecards: Developng and Implemenng Inellgen Cred Scorng. Wley: New York. Thomas LC, Edelman DB and Crook JN (2002). Cred Scorng and Is Applcaons. SIAM: Phladelpha. Thomas LC (2009). Consumer Cred Models: Prcng, Prof, and Porfolos. Oxford Unversy Press: New York. Van Gesel T and Baesens B (2009). Cred Rsk Managemen. Basc conceps: fnancal rsk componens, rang analyss, models, economc and regulaory capal. Oxford Unversy Press: New York. Welch G and Bshop G (2006). An Inroducon o he Kalman Fler. Tech. Repor TR 95-041, Deparmen of Compuer Scence a he Unversy of Norh Carolna a Chapel Hll. hp://www.cs.unc.edu/~welch/meda/pdf/kalman_nro.pdf, accessed 2 June 2008. Wells C (1996). The Kalman Fler n Fnance. Kluwer Academc Publshers: Dordrech. Whaker J, Whehead C and Somers M (2007). A dynamc scorecard for monorng baselne performance wh applcaon o rackng a morgage porfolo. J Opl Res Soc 58: 911-921. 21
Fgure 1. Arbue scores of he seleced characersc (number of pas loans = 0, 1, 2-3 or 4 and more): esmaed on he bass of successve samples and updaed usng he Kalman fler Fgure 2. The percenage of cusomers whose updaed odds do no le whn he nervals Fgure 3. The Gn coeffcen of he baselne scorecard Fgure 4. The KS sasc of he baselne scorecard Table 1. The monorng resuls 22