Information Theoretic Evaluation of Change Prediction Models for Large-Scale Software

Transcription

1 Informaion Theoreic Evaluaion of Change Predicion Models for Large-Scale Sofware Mina Askari School of Compuer Science Universiy of Waerloo Waerloo, Canada Ric Hol School of Compuer Science Universiy of Waerloo Waerloo, Canada ABSTRACT In his paper, we analyze he daa exraced from several open source sofware reposiories. We observe ha he change daa follows a Zipf disribuion. Based on he exraced daa, we hen develop hree probabilisic models o predic which files will have changes or bugs. The firs model is Maximum Likelihood Esimaion (MLE, which simply couns he number of evens, i.e., changes or bugs, ha happen o each file and normalizes he couns o compue a probabiliy disribuion. The second model is Reflexive Exponenial Decay (RED in which we posulae ha he predicive rae of modificaion in a file is incremened by any modificaion o ha file and decays exponenially. The hird model is called RED-Co-Change. Wih each modificaion o a given file, he RED-Co-Change model no only incremens is predicive rae, bu also incremens he rae for oher files ha are relaed o he given file hrough previous co-changes. We hen presen an informaion-heoreic approach o evaluae he performance of differen predicion models. In his approach, he closeness of model disribuion o he acual unknown probabiliy disribuion of he sysem is measured using cross enropy. We evaluae our predicion models empirically using he proposed informaion-heoreic approach for six large open source sysems. Based on his evaluaion, we observe ha of our hree predicion models, he RED-Co-Change model predics he disribuion ha is closes o he acual disribuion for all he sudied sysems. Caegories and Subjec Descripors D.2.7 [Sofware Engineering]: Disribuion, Mainenance, and Enhancemen- Version conrol D.2.8 [Sofware Engineering]: Merics- Performance measures, Process merics General Terms Performance, Reliabiliy, Theory Keywords Predicion Models, Evaluaion approach, Informaion Theory Permission o make digial or hard copies of all or par of his work for personal or classroom use is graned wihou fee provided ha copies are no made or disribued for profi or commercial advanage and ha copies bear his noice and he full ciaion on he firs page. To copy oherwise, or republish, o pos on servers or o redisribue o liss, requires prior specific permission and/or a fee. MSR 6, May 22 23, 26, Shanghai, China. Copyrigh 26 ACM X/6/5...$ INTRODUCTION Sofware sysems are coninuously being changed o adap o mee he needs of heir users or o correc he fauls appearing in sysems during developmen or afer deploymen. There has been exensive research on new processes and approaches for developing sofware sysems o minimize hese new modificaions. The idea is ha during sofware developmen by following some specific principles, he probabiliy of cerain kinds of modificaions can be decreased. Despie his progress, new changes and bugs are ineviable during sofware developmen. However, if sofware developers were able o forecas he occurrence of changes and bugs hen hey could miigae heir impac. Therefore, developing accurae echniques o predic he fuure behavior of changes and bugs can be valuable for sofware developmen and mainenance. The idea for predicing which files/subsysems are mos suscepible o having a faul in he near fuure is a well-known idea. There exis several predicion models [5][6][8][9][][14] and more are emerging. However, many of hese faul predicion models have no been evaluaed in pracice and some of hem are no applicable o large-scale sofware sysems. The majoriy of faul predicion models are applicable o deployed sysems only. The general approach for evaluaing hese models is o run he sysem and collec he observed informaion during is execuion and hen compare i wih he resuls prediced by he models [18]. The problem is ha oo ofen hese models are no general and hence, hey are no applicable o differen sofware sysems. In many cases, because he models measure differen merics, he resuls are no comparable [18]. There are many quesions wih respec o he validiy of he underlying assumpions, accuracy, and applicabiliy of sofware predicion models. In his paper, our goal is o conribue oward more general and realisic assessmen and predicion of sofware modificaions based on heoreical and empirical sudies. We are ineresed in mehods and models ha have wo properies. Firs, hey use daa colleced during developmen process and second, heir disance from he acual bu unknown disribuion of he colleced daa can be measured. Our goal is wofold: firs, o develop predicion models driven by sofware reposiories and second, evaluae and compare differen models using a mahemaical approach. We use hisorical records, from source conrol reposiories of large sofware sysems, o develop predicion models and o esimae how much informaion is capured by he models. Figure 1 illusraes he problem we are rying o solve. Suppose we have a lis of all he evens ha have so far occurred on differen files of a sofware sysem during developmen process. 126

2 These evens are file changes o fix bugs, or o add new feaures or change exising feaures. We have exraced hese evens from he hisory of he sofware. For example f 21 shows ha one modificaion has happened on file 21 a he specific ime (see Figure 1. We ask his quesion: To wha degree are hese changes unpredicable? More paricularly, wha will be he uncerainy of he nex sample, if all pas samples are known? In some cases, i may be impossible o say anyhing abou he nex sample regardless of how many pas samples are already known. In oher cases, he process may be much less uncerain abou he nex sample when given he hisory of he changes. Having exraced hisorical daa, we wan o deermine how much informaion his daa provides us abou he fuure. Our resuls indicae ha CVS log daa conains informaion abou he pas ha can help o predic he fuure. The quesions include: How much informaion is buried in he CVS logs and how can we capure his informaion? How good are he predicion models ha use his informaion o predic he fuure? Figure 1. Does pas predic fuure? Afer inroducing relaed work in Secion 1.1, he res of he paper is organized as follows. Secion 2 presens he echniques and approaches we used o perform differen experimens in order o analyze he daa exraced from several open source sofware reposiories. In Secion 3, we presen our hree predicion models and describe he seps involved in developing he models. In Secion 4, we presen an informaion heoreic approach for evaluaion predicion models. In Secion 5 presens he resuls of using wo approaches for evaluaing our proposed predicion models: he Top Ten Lis approach proposed by Hassan e al. [8] and our informaion heory based approach. Finally, Secion 6 concludes he paper and discusses possible fuure works. 1.1 Relaed Work Many researchers [2][5][6][8][9][16][17] in sofware developmen area have realized he value of hisorical daa and have used hem in heir research ranging from sofware design o sofware undersanding, sofware mainenance, developmen process and many more areas. There is considerable research on developing ools o recover such hisorical daa. Hassan e al. [7] developed C-REX ool, an evoluionary code exracor, which recovers informaion from source conrol reposiories. Zimmermann e al. [19] used version reposiories o deermine co-change clusers. They applied daa mining o version hisories in order o guide programmers hrough relaed changes. For deecing anoher kind of cochanges Gall e al. [5] used sofware reposiories. They uncovered he dependencies and inerrelaions beween classes and modules (logical dependencies which can be used by developers in mainenance phase of a sysem. Graves e al. [6] showed ha here is a relaion beween he number of changes a subsysem has wih he fuure fauls in ha subsysem. Hassan e al. [8] presened various heurisics using hisoric version conrol daa o creae he Top Ten Lis. Top Ten Lis highlighs o managers he en mos suscepible subsysems o have a faul. They also developed echniques o measure he performance of hese heurisics. Mockus e al. [12] sudied a large legacy sysem o es he hypohesis ha hisoric version conrol daa can be used o deermine he purpose of sofware changes and o undersand and predic he sae of a sofware projec [13]. Khoshgofaar e al. [9][] used process hisory o predic sofware reliabiliy and o show ha he number of prior modificaions o a file is a good predicor of is fuure fauls. Eick e al. [4] presened visualizaion echniques o explore change daa o help engineers undersand and manage he sofware change process. Osrand, e al. [14] suggesed a model o predic he number of fauls for a large indusrial invenory sysem based on he hisory of he previous releases. Our approach akes guidance from his previous work, bu is noably differen by suggesing new predicion models and by using an informaion heoreic approach o measure he effeciveness of such models. 2. CHARACTERISTICS OF THE DATA The predicion models and he evaluaion mehods presened in his paper are based on change hisory daa. Change daa is he informaion generaed during developmen process and can be obained hrough mining he reposiories of he sofware. We began by analyzing he exraced daa o undersand is saisical properies. In paricular, we observed ha hisory daa has Zipf disribuion [2]. 2.1 Sudied Sysems To perform our sudy we used several CVS logs of open source sofware sysems. Table 1 summarizes he deails of he sofware sysems we sudied. The oldes sysem is over en years old and he younges sysem is five years old. We ried o choose he applicaions from differen domains and differen sizes. We were looking for any kind of change and bug which happens o differen files of a sysem. The process of acquiring such specific daa is very challenging, since CVS logs are mainly designed as record keeping reposiories and commis aren aomic and large amoun of daa sored in hese reposiories complicaes he daa exracing process. For analyzing daa and creaing predicion models and comparing hem based on he daa, our main concern was o perform our sudies on he daa of several CVS logs sofware sysems in a sandard forma ha is easier o process and no developing ools ha auomaically recover daa from hese reposiories. So we obained and used he daa which were exraced from hese CVS logs by ools developed by Hassan e al. [7]. This le us concenrae on analyzing he exraced daa insead of spending ime developing ools o recover he daa. 127

3 Table 1. Number of evens available for differen sysems Applicaion Name Duraion (Monh Toal Evens Bug or Changes Toal Files OpenBSD FreeBSD KDE Koffice NeBSD Posgres Zipf s Law We sared by couning he number of modificaions which happened for each file during he developmen process. Based on he hisory of he developmen, if we coun how ofen each file is modified, and hen lis he files in order of he frequency of occurrence, we can explore he relaionship beween he frequency of a file and is posiion in he lis, known as is rank. Figure 2 illusraes he number of modificaions for each file for differen sysems we sudied. Differen sysems have differen number of files. Therefore, o compare all he sudied sysems in a single plo, we used he percenage of files and percenage of aciviies for each file in Figures 2 o 4. As i can be seen from he figure 2, here are few files wih high frequency of changes bu many files wih very low number of changes. I also can be seen in he figure, hese frequencies follow a similar paern in all sudied sysems. This behavior indicaes ha he change daa follows he general form he Pareo (or 8-2 law [15] and Zipf's law [2]. Percenage a ha Frequency Posgres KDE OpenBSD FreeBSD Koffice NeBSD Percenage of he Files Figure 2. Change daa follows Zipf s law. poins are close o a single sraigh line hereby confirming ha he daa approximaes Zipf s law [2]. 3. CHANGE PREDICTION MODELS The predicion of fuure modificaions in a large sofware sysem is an imporan par in sofware evoluion. Since mos predicion models in pas sudies have been consruced and used for individual sysems, i has no been pracically invesigaed wheher a predicion model based on one sysem can also predic fauls and changes accuraely in oher sysems. Our expecaion was ha if we could build a model applicable o differen range of sysems based on he informaion which is generaed during developmen process, e.g. CVS logs, i would be useful for sofware developers. In his Secion we will show several predicion models which can use he CVS logs o predic he fuure bugs and changes in any arbirary sysem. These models are generally in form of probabiliy models. Cumulaive Frequency Percenage of he Files Posgres KDE OpenBSD FreeBSD Koffice NeBSD Figure 3. Cumulaive frequency disribuions. Log-Cumulaive Frequency Posgres KDE OpenBSD FreeBSD Koffice NeBSD The Pareo law, in is generalized form, saes ha 8% of he objecives - or more generally he effecs - are achieved wih 2% of he means. In order o show ha here is 8-2 law in our daa, we ploed he cumulaive disribuions of he file frequencies in Figure 3. I can be seen from he figure ha almos 2% of he files in he sysems have (almos 8% of aciviies during developmen. To show ha Zipf s law holds for he daa, we ploed he log-log scale of he cumulaive frequency disribuions; see Figure 4. I can be seen ha he 1 1 Log-Percenage of he Files Figure 4. Log-log scale of cumulaive disribuions. Afer exracing he changes and bugs ha occurred in he various files of a sysem during developmen, we creaed a sequence of evens showing file changes o fix bugs or o add feaures. Having his sequence of evens our goal is o predic fuure comparable evens. There are many files in he sysems 128

4 we sudied, ranging from o 2 files. We waned o consruc a probabilisic model of his process, in oher words, o define a probabilisic model ha characerizes he resul of he nex elemen in he sequence. We assume ha we know ha he possible value space, i.e., he Domain D (i.e., sample space for even e (considered as a random variable. In our work, D is he se of files in he sysem. We denoe he elemens of his Domain as f 1, f 2... f m. Our goal is o define a good probabiliy model o give he probabiliy ha he i h (i.e., nex elemen in he sequence will have a paricular value (will be a paricular file; in oher words for finding he probabiliy disribuion of random variable e i, wha we need o do is o decide on he form of he underlying model of he sequence of evens. Ideally his would be a condiional probabiliy funcion of form P(e i e 1, e 2,...,e i-1. Our work is complicaed by he fac ha, in general, a new probabiliy funcion is needed for each e i. Based on his approach, we will now presen hree probabilisic models. 3.1 Mos Likely Esimaion (MLE Model Our firs model, maximum likelihood esimae (MLE, simply uses he couns from he sequence o esimae he disribuion. PMLE(e fi Coun(fi / N ( 1 In (1, fi D, N is he size of sequence, and Coun(fi is he number of occurrences of fi in he sequence. The proporion of imes a cerain even f i occurs is called he relaive frequency of he even. In he MLE model, we compue (predic he relaive frequency of each new even based on he preceding sequence. Empirically for our daa we observed if one performs a large number of rials, he relaive frequency (for each file ends o sabilize around some number. In our experimens, insead of definiion (1, we compued our MLE probabiliy disribuions [1] using his formula: In (2, fi D, N is he size of sequence, Coun(fi is he number of occurrences of fi and d is he size of domain D. We use his equaion because equaion (1 has wo compuaional problems. The firs problem is ha i implicily assigns a zero probabiliy o elemens of domain ha have no been observed in he sequence. This means i will assign a zero probabiliy o any sequence conaining a previously unseen elemen. The second problem is ha i does no disinguish beween differen levels of cerainy based on he amoun of evidence we have seen. One soluion is o assign a small probabiliy o each possible observaion a he sar. We do his by adding a small number (we use l o he coun of each oucome o ge he esimaion formula. This echnique, using value 1, is called Laplace esimaion [1]. If we never see a oken of a ype f in a corpus of size N and domain size d, he probabiliy esimae of a oken of f occurring will be 1/(N+d. For he second problem, using Laplace formula, our prior knowledge ha here is D differen ypes of evens makes our esimae say close o he uniform disribuion [1]. 3.2 Reflexive Exponenial Decay (RED Model Our second model relies on he idea ha when a change is observed in a file, i is likely ha more changes will be observed in ha file, bu ha his effec decreases (decays wih ime. We are given a sequence of evens called e 1, e 2 e n, occurring respecively a monoonically increasing imes 1, 2 n. We assume ha evens probabilisically predic evens, e.g., bug fixes predic bug fixes. By analogy, yeserday s weaher is a good predicor of oday s weaher. We posulae ha he predicive rae of bugs induced by any even decays exponenially. We call his model reflexive because each even in urn predics more evens. More generally, we call i he reflexive exponenial decay (RED model. A paricular even occurring a ime i on he file f j, implies (predics a fuure frequency rae R (j for ha file a fuure ime. Our model defines R (j as follows: R (j where k(-i k (-i /h I ( / ( 32 -ln( 2 /h and In formula (3, h is he half life (measured ypically in monhs and I is he impac of an even (measured ypically in evens per monh. This means ha if in he sequence of evens, e i happens a ime i and e i is a modificaion of file f j, for all ime > i, he prediced incremenal frequency for file will be R j (. A larger half life h means ha he effecs of a change las longer. Figure 5 shows R (j for differen half lives and wih impac of I = 1 and i =. Based on R (j for each even on file f j, we define he RED model as he summaion of he effecs all (hisorical evens happening o each file. We now formalize he RED frequency model. Suppose ha he PMLE(E fi ( Coun(fi / (N d ( 21 sequence of evens, e, e 1, e 2,.., e l has happened on file f j up ime. Then RED predics ha he fuure frequency of changes o his file will be: R (j for all k(- l k(- i k(-21 k(-l... ( 4 Figure 6 shows how he effec of each even is added o he previous ones for a specific file. In he figure, specific file f j has been observed o change a imes, 5 and 15; he individual exponenially decaying predicive effecs of hese hree evens are shown as he hree lower curves. The cumulaive effec of hese firs wo of hese (from imes and 5 is shown as anoher curve. Then he effec of all hree of hese is shown by ye anoher (he highes curve. RED Disribuion Model We will now conver our RED model so ha i predics probabiliy disribuion raher han frequency. Given a sequence of evens: e, e 1, e 2,.., e m, having R (j for all files j = 1..n, we can define he disribuion of RED a ime as follows: RED (e R (i 1 f im for 5 m ( R (j j 1..n 129

5 Prediced Frequency Time (monh 1 monh 2 monh 3 monh 6 monh Figure 5. Exponenial decay for differen half lives. 3.3 RED Co-Change (REDCC Model Our hird model is an enhanced version of RED. When each even occurs we updae he probabiliy no only for he changed file bu also for he co-changed files. There are several differen approaches for concluding ha (or defining ha he files change (co-change ogeher during sofware developmen. Developers commonly modify files ogeher (co-change hem o inroduce new feaures or fix bugs. Developers should ensure ha when one file is changed, oher relaed files in he sofware sysem are updaed o be consisen wih he modificaions. We use a definiion of co-change ha is inspired by he lieraure [8]. If file f 1 and f 2 changed ogeher (on he same day in previous change ses, hen hey are candidaes o be considered as co-changed files. We will define ha co-change files are hose ses of files which have changed on he same day in he pas a leas 3 imes wihin he preceding 7 days. We now define he RED Co-Change (REDCC model. We assume ha a ime, he sequence of evens, e, e 1, e 2,.., e m has happened on file f i or on he co-change files of f i up o his ime. k(- k(- k(-m REDCC(j... ( 6 Reflexive Exponenial Decay Effec of change #1 Effec of change #2 Effec of change #3 Effec of changes #1 & #2 Effec of changes #1, #2 & # Time Figure 6. Reflexive exponenial decay for a file. Using REDCC (j frequency model, we conver i o he probabiliy model REDCC (e m+1 =f i in he same way we convered he RED frequency model o a probabiliy model. 4. EVALUATION OF PREDICTION MODELS In his secion we presen an informaion heoreic approach o quanify he goodness or finess of a guessed probabiliy g (g is a predicion model compared o he acual probabiliy p. Our approach uses enropy conceps o evaluae predicion models. Our goal is o compare hese predicive models (disribuions o see how good hey are. By good we mean how close hey are o he rue disribuions of he evens. I also could mean ha how well hey predic he occurrence of he nex even. The approach we ake is well known in Naural Language Processing (NLP area, where a sequence of words in language is called corpus, bu o our knowledge has no been used in he field of Mining Sofware Reposiories. NLP uses informaion heory o find he disance beween predicion models and acual disribuion of corpus [11]. 4.1 Enropy and Cross Enropy Before inroducing our informaion heoreic approach, we will review some relaed conceps. Informaion heory echniques define he amoun of informaion in a message. The heory measures he amoun of uncerainy/enropy in a disribuion. Shannon enropy [11], given probabiliy p(x, is defined as: H(p = - p(xlog p(x Larger values of H(p imply ha more bis are needed for coding messages. There is a relaed concep called cross enropy which allows us o compare wo probabiliy funcions. (Cross enropy is closely relaed o Kullback-Leibler divergence [11]. The cross enropy beween wo probabiliy disribuions measures he overall difference beween he wo disribuions p and m and is defined as: H(p,m = - p(xlog m(x Where p(x is he rue disribuion and m(x he model disribuion. The cross enropy is minimal when p and m are idenical, in which case i reduces o simply H(p. The closer he cross enropy is o enropy proper, he beer m is an approximaion of p. If we have wo models m 1 and m 2, if H(p, m 1 < H(p, m 2 hen m 1 is a closer approximaion o disribuion o p. This approach seems o require ha we know p, he acual disribuion of daa, which unforunaely we do no know. One of he cenral problems we face in using probabiliy models is obaining he acual disribuion p(x of daa. The rue disribuions are no known, ye we wan o esimae predicive models and validae hem using he exising daa. Here here is a paradox: if we had p(x in advance, we wouldn need o make any model for esimaing p(x. 4.2 Corpus Cross Enropy We solve his problem wih using corpus cross enropy (CCE. Given a sequence c wih of lengh N consising of evens e 1 e N, he corpus cross enropy of a probabiliy funcion m is defined as follows: H c (m = -(1/N log m(e i 13

6 Hi Raio Koffice-RED Koffice-MLE Koffice-RED+CoChange Hi Raio OpenBSD-RED OpenBSD-MLE OpenBSD-REDCC Lis size (Percenage of oal files Lis size (Percenage of oal files Figure 7. Evaluaion of 3 models based on Hi Raio of Top Ten Lis, wih varying size of lis. I is sraighforward o prove ha corpus cross enropy H c (m approaches cross enropy H(p,m as N approaches infiniy, given ha p is he rue disribuion of corpus c and given ha p is saionary. We can compue H c (m, as an approximaion o H(p,m, even hough we do no know disribuion p. As is done in NPL lieraure [11], we assume ha given wo models m 1 and m 2 we can compare H c (m 1 and H c (m 2 o deermine which of m 1 and m 2 is he beer model, even hough we do no know he rue disribuion p, given ha p is reasonably saionary. Tha is, when H c (m 1 < H c (m 2 we conclude ha m 1 is a closer disribuion o he rue disribuion and hence is a beer model. 5. EMPIRICAL STUDIES In his secion we evaluae our hree proposed predicion models (MLE, RED and REDCC empirically, using wo approaches, for six large open source sysems. Table 1 summarizes he deails of he sofware sysems we sudied. Due o space limiaion we will only shown he resuls for wo sysems (Koffice and NeBSD. The oher sysems had similar behavior. 5.1 Top Ten Lis evaluaion For evaluaing he qualiy of our hree models, firs we use he Top Ten Lis [8] approach. This approach evaluaes which model predics more accuraely. In his approach, he model predics a lis of he files (more generally, a lis of n files ha are mos likely o be changed nex. A new lis is generaed for each new even. Given a prediced disribuion m for he nex even, we creae he corresponding Top Ten Lis for ha upcoming even by picking he en (or n files wih he highes probabiliy according o m. Wih he occurrence of each even, here is a change o a file, call i file f i. We record wheher file f i is in he even s Top Ten Lis. We define he Hi Raio as he fracion of evens in which file f i was observed o be in is Top Ten Lis. Models wih higher Hi Raios are considered o be beer models. We applied he Top Ten Lis approach o evaluae our hree proposed models, for all he sudied sysem; see Figure 7 for he resuls for wo of hese sysems: Koffice and OpenBSD. (Resuls for he oher sudied sysems are comparable. As can be seen, for boh sysems, REDCC and RED have very similar resuls, wih REDCC being slighly superior. By conras, MLE s resuls are considerably worse. In oher words, he Top Ten Lis approach evaluaes REDCC is slighly beer han RED, and boh of hese considerable beer han MLE. As can be seen in Figure 7, as he size of lis increases, we have a higher hi raio. Ineresingly, using REDCC or RED model, when we use 2 percen of oal files in he sysem, he hi raio is almos 8 percen. 5.2 Informaion heoreic evaluaion We also applied he informaion heoreic approach o compare our hree predicion models. Due o space limiaions, we only presen he resul for one of he sudied sysem, Posgres. The resuls for he oher sysems are similar. Using hisorical Posgres daa, we developed insances of our hree models: MLE, RED and REDCC. To develop he MLE model, we used he firs evens and kep i fixed for he remaining corpus. Figure 8 shows he corpus cross enropy of our hree predicive models when applied o Posgres. As i can be seen in he figure, REDCC has he lowes corpus cross enropy which means is disribuion is he closes o he acual disribuion of he daa. The nex closes (see middle curve in Figure 9 is RED, and he wors (op curve is MLE. Noe ha his ordering is he same ha we observed when our evaluaions were based on he Top Ten Lis. As can be seen in Figure 8, as he size of corpus increases he MLE disribuion ges farher from he real disribuion of daa bu for wo oher models, RED and REDCC, he opposie is rue. This suggess ha he RED and REDCC models benefi by updaing heir disribuions based on he evens in he corpus as ime passes. Corpus cross enropy REDCC (boom curve RED (middle curve MLE (op curve Size of corpus Figure 8. Evaluaion of 3 models using corpus cross enropy on Posgres. 131

7 6. CONCLUSION We developed hree models (MLE, RED and REDCC for predicing fuure modificaion of files based on available change hisories of sofware. We proposed a rigorous approach for evaluaing such predicive models. This approach has been used in Naural Language Processing, bu no in Mining Sofware Reposiories, as far as we know. This is an informaion heoreic approach in ha he closeness of a predicive model disribuion o an acual bu unknown probabiliy disribuion of he sysem is measured using cross enropy. We evaluaed our proposed predicion models empirically using wo approaches for six large open source sysems. Firs we used he Top Ten Lis [8] approach o see which model predics more accuraely. Using his approach we showed ha he REDCC model works bes of our hree models. Then using our informaion heoreic evaluaion approach, we observe ha he REDCC model again has he disribuion ha is closes o he acual disribuion for all he sudied sysems. An advanage of our informaion heoreic approach over he Top Ten Lis approach is ha using our approach we know quaniaively, as measured by cross enropy, how much beer or worse is he predicion model compared o ideal resul. Our hope is ha our approach can be used o help beer predic fuure changes and bugs, based on he hisory of sofware. Our approach also can be used by researchers who have developed new predicion models o evaluae hem using a informaion heoreic approach. 7. ACKNOWLEDGMENT The auhors would like o hank Ahmed Hassan. This paper would no have been possible wihou his generous help and his daa. We also would like o hank he referees for heir exremely helpful suggesions. 8. REFERENCES [1] Allen, J. F. Using Enropy for Evaluaing and Comparing Probabiliy Disribuions, available a: hp:// [2] Basili, V. R., and Perricone, B. Sofware errors and complexiy: An empirical invesigaion. Communicaions of he ACM, 27(1:42 52, [3] Eick, S. G., Graves, T. L., Karr, A. F., Marron, J.S., and Mockus, A. Does Code Decay? Assessing he Evidence from Change Managemen Daa. IEEE Trans. on Sofware Engineering, 27(1:1 12, 21. [4] Eick, S.G., Graves, T.L., Karr, A.F., Mockus, A., Schuser, P. Visualizing Sofware Changes, IEEE Trans. on Sofware Engineering, vol. 28, no. 4, pp , April, 22. [5] Gall, H., Hajek, K., and Jazayeri, M. Deecion of logical coupling based on produc release hisory. In Proceedings of he 14h Inernaional Conference on Sofware Mainenance, Behesda, Washingon D.C., November [6] Graves, T. L., Karr, A. F., Marron, J. S. and Siy, H. P. Predicing faul incidence using sofware change hisory. IEEE Trans. on Sofware Engineering, 26(7: , 2. [7] Hassan, A. E., Mining Sofware Reposiories o Assis Developers and Suppor Managers. PhD Thesis, Universiy of Waerloo, Onario, Canada, 24 [8] Hassan, A. E. and Hol, R. C., The Top Ten Lis: Dynamic Faul Predicion, Proceedings of ICSM 25: Inernaional Conference on Sofware Mainenance, Budapes, Hungary, Sep 25-3, 25. [9] Khoshgofaar, T. M., Allen, E. B., Halsead, R., Trio, G. P. and Flass, R. M. Using Process Hisory o Predic Sofware Qualiy. Compuer, 31(4, [] Khoshgofaar, T. M., Allen, E. B., Jones, W. D., and Hudepohl, J. P. Daa Mining for Predicors of Sofware Qualiy. Inernaional Journal of Sofware Engineering and Knowledge Engineering, 9(5, [11] Manning, C. and Schüze, H. Foundaions of Saisical Naural Language Processing, MIT Press. Cambridge, MA: May [12] Mockus, A. and Voa, L. G. Idenifying reasons for sofware change using hisoric daabases. In Inernaional Conference on Sofware Mainenance, pages 12-13, San Jose, California, Ocober [13] Mockus, A., Weiss, D. M., and Zhang, Ping. Undersanding and predicing effor in sofware projecs. In 23 Inernaional Conference on Sofware Engineering, pages , Porland, Oregon, May ACM Press. [14] Osrand, T. J., Weyuker, E. J., Bell, R. M. Predicing he Locaion and Number of Fauls in Large Sofware Sysems. IEEE Trans. Sofware Eng. 31(4: (25 [15] Pareo Law: hp:// [16] Perry, D. E. and Evangelis, W. M. An Empirical Sudy of Sofware Inerface Fauls An Updae. In Proceedings of he 2h Annual Hawaii Inernaional Conference on Sysems Sciences, pages , Hawaii, USA, January [17] Perry, D. E. and Seig, C.S. Sofware Fauls in Evolving a Large, Real-Time Sysem: a Case Sudy. In Proceedings of he 4h European Sofware Engineering Conference, Garmisch, Germany, Sepember [18] Reliabiliy Analysis Cener, Inroducion o Sofware Reliabiliy: A sae of he Ar Review. Reliabiliy Analysis Cener (RAC, hp://rome.iiri.com/rac/ [19] Zimmermann, T., Weissgerber, P., Diehl, S., Zeller, A. Mining Version Hisories o Guide Sofware Changes, IEEE Trans. on Sofware Engineering, vol. 31, no. 6, pp , June, 25. [2] Zipf, G. K. Human Behavior and he Principle of Leas Effor.Addison-Wesley,