Predicting Software Development Project Outcomes *

Transcription

1 Predctng Software Development Project Outcomes * Rosna Weber, Mchael Waller, June Verner, Wllam Evanco College of Informaton Scence & Technology, Drexel Unversty 3141 Chestnut Street Phladelpha, PA {rosna.weber,mwaller}@drexel.edu; {june.verner,wllam.evanco}@cs.drexel.edu Abstract. Case-based reasonng s a flexble methodology to manage software development related tasks. However, when the reasoner s task s predcton, there are a number of dfferent CBR technques that could be chosen to address the characterstcs of a dataset. We examne several of these technques to assess ther accuracy n predctng software development project outcomes (.e., whether the project s a success or falure) and dentfy crtcal success factors wthn our data. We collected the data from software developers who answered a questonnare targetng a software development project they had recently worked on. The questonnare addresses both techncal and manageral features of software development projects. The results of these evaluatons are compared wth results from logstc regresson analyss, whch serves as a comparatve baselne. The research n ths paper can gude desgn decsons n future CBR mplementatons to predct the outcome of projects descrbed wth manageral factors. 1 Introducton and Background Software development project falure can be very costly. Rsk factors that can determne project falure tend to become evdent only n the later stages of software development lfe cycle; often too late to steer a project safely back on course. As a result, software project managers can be aded greatly by tools that can dentfy lkely success or falure at an early stage, as well as those factors that may contrbute to development problems. Our goal s to develop a tool that can make a good predcton of a project s outcome early n the development lfe cycle and ndcate success or rsk factors. Ths tool has to be flexble enough to accommodate manageral features and be able to manage a varety of knowledge tasks (e.g., capture, reuse) comprsng a knowledge management (KM) system. We chose case-based reasonng (CBR) for ths tool because CBR s a methodology [1] that can support the flexble automaton of the entre process. Addtonally, CBR s approprate for software development related tasks because the methodology resembles human judgment [2]. CBR also offers advantages to support KM efforts [4, 5, 6, 7]. CBR s characterzed as a lazy learner that predcts the outcome of new * D. Brdge and K. Ashley (eds.) Case-Based Reasonng Research and Development. LNAI 2689, , Berln Hedelberg:Sprnger-Verlag.

2 cases by usng a k-nearest neghbor classfer, thus presentng some potental benefts to predct project success (e.g., a relatvely small tranng cost and effort conjugated wth an explanaton for the classfcaton [3]). There are a number of dfferent technques that can be used to mplement CBR; the current vew s that certan desgn choces can bas the system s qualty [8, 9]. Therefore, we need to test dfferent technques and select the one that performs best wth the problem data. Our data was collected from 122 software developers who responded to a questonnare about a software project they had recently worked on. The questons addressed both techncal and manageral features of the chosen software development projects. Past work usng CBR for predcton of software development projects has focused on the techncal, quantfable aspects of software development, predctng, for example, development effort [2, 8, 9] rather than addressng qualtatve manageral factors. In [9], Watson et al. compared three CBR technques and found that the weghted Eucldean dstance was the most accurate method for predctng development effort for web hypermeda software. In [2] Fnne et al. compared CBR technques wth lnear regresson analyss. They predcted effort represented by a contnuous dependent varable, whch s better suted for lnear regresson models. Kadoda et al. [8] also predcted software development effort wth CBR and compared ts performance wth stepwse regresson. They concluded that there s strong assocaton between features of the dataset (e.g., tranng set sze, nature of the cost functon) and the success of a technque, and that the best technque can be determned on the bass of each dataset [8]. The use of CBR to help manage software engneerng projects s commonly assocated wth the experence factory (EF) [10, 11] - a framework that structures the reuse of experence and products obtaned durng the software development lfe cycle. For example, Althoff et al. [7] have adopted the EF model to create an experence base to reuse experences and products obtaned n the development of CBR systems. Our approach dffers because our core goal s to predct success and provde advce to software project managers. Our strategy s not ntended to explctly record methods employed n one development and enable detaled reuse, but smply to gather general experence n software development projects n order to dentfy success factors. In our approach, t s the user s responsblty to search for mtgants when a gven factor seems to suggest potental falure. Our approach s more superfcal and desgned for easer mplementaton and acquston. When usng CBR to manage or predct success of manageral projects, sometmes the number of features descrbng these projects may outnumber the number of cases. The way Can et al. [11] dealt wth ths problem was by ncorporatng doman knowledge to choose the features to explan success or falure. They combned the noton of explanaton-based learnng (EBL), whch uses doman knowledge to generalze a concept from a tranng example [13]. Ther parameterzed combnaton of CBR plus EBL s detaled n Subsecton 3.3. We tested dfferent technques to assess how well they performed wth our data. After testng unweghted measures we used a hll-clmbng feedback method to learn weghts and prevent mperfect features from partcpatng equally n predctng

3 project outcome. We then ntroduced and nvestgated a weghted verson of the CBR plus EBL approach. As we dentfy the best smlarty measure to provde accurate outcome predctons, we wll ntroduce a method to use ths technque to dentfy success factors wth CBR. Because falure factors may have multple mtgants, managers wll have the nformaton necessary to avod project ptfalls. We also used a non-cbr technque, logstc regresson (LR) [14], to predct the outcomes for the same dataset. LR s generally consdered the most accurate statstcal technque for modelng dchotomous dependent varables, partcularly n datasets wth fewer than 100 examples, but t s also a method that tends to be expensve [14]. Cleary et al., explan that LR s a hghly desrable statstcal model and should be used for model fttng and hypothess testng [14]. Hence, we use LR as a pont of comparson for testng varous CBR technques. Ths paper s organzed as follows. Secton 2 dscusses the questonnare used for data collecton. Secton 3 descrbes the predcton technques we used. In Secton 4 we evaluate the technques used and descrbe our results. Secton 5 extends the use of the most accurate CBR technque to dentfy software project success factors. In Concludng Remarks, we present our conclusons and future work. 2 Collectng Software Development Project Data The collecton method s crucal n obtanng relable data. Once data s collected and understood by a KM system t can be managed to systematcally beneft other projects. Verner [15, 16] s a software engneer who collected the data we used n ths research to nvestgate software project crtcal success factors. The data was collected for the sole purpose of analyzng software development projects and dentfyng the factors that determne project success or falure. Table 1 Examples of questons n the questonnare ID q2_1 q2_4 q3_3 q3_5 q3_7 q4_2 q4_8 q4_11 Queston What was the level of nvolvement of the customers/users? Were the customers/users nvolved n makng schedule estmates? Was the scope of the project well defned? Dd the customers/users make adequate tme avalable for requrements gatherng? Dd the requrements result n well-defned software delverables? Was the delvery date decson made wth approprate requrements nformaton? Dd the project have adequate staff to meet the schedule? Dd the schedule take nto account staff leave, tranng, etc.? Removng all varables that descrbe features unknown early n a project resulted n a total of 23 varables. The questonnare ncorporates both objectve and subjectve human judgments about these projects. Table 1 provdes some example questons,

4 whch wll be further dscussed n ths paper, referenced by ther ID. The format of these answers are yes/no and multple-choce. Ths questonnare addresses the areas of management support, customer/user nteracton, requrements, estmaton and schedulng, n relaton to project outcome. The outcome secton orgnally ncluded questons that allowed for conflctng answers. The questonnare asked for project success or falure from the perspectve of the organzaton and from the perspectve of the developer answerng the queston. We merged these varables nto a sngle varable and removed all projects where outcome varables were n conflct. After elmnatng the conflctng answers, the ntal group of 122 project records was reduced to descrbng successful projects and 21 descrbng faled projects. 3 Technques to Predct Project Outcomes Our goal s to desgn a system to systematcally acqure software development project data and understand t wthn a KM framework. A case-based reasoner predcts the outcome of new projects n ths system. In ths secton, we dscuss the CBR system we mplemented to evaluate dfferent smlarty measures to predct project s outcome, descrbe CBR technques and LR. 3.1 CBR Implementaton Our CBR mplementaton uses the data gathered wth the questonnare dscussed n Secton 2, from whch we created a case base wth 88 cases descrbed by 23 features and a bnary varable for project outcome. Ths mplementaton entals the use of four subtasks of the retreve CBR process, namely, dentfy features, ntal match, search, and select [17]. These are standard mplementatons except for the select subtask. In order to accommodate stuatons n whch multple cases have the same smlarty but dfferent outcomes, we broke all tes by selectng the class of the case wth the next hghest smlarty. We treat all features as symbolc; f they have the same value the smlarty equals one, otherwse t equals zero; there are no ntermedary degrees for smlarty. Currently, our mplementaton does not dentfy factors that contrbute to success or falure. The results dscussed later n ths paper wll lay the foundaton for the development of a method to provde dentfcaton of these factors along wth an outcome predcton. Suggestng countermeasures based upon these factors s an addtonal step to ncorporate.

5 3.2 Unweghted k-nn We frst mplemented CBR wth a tradtonal unweghted k-nn classfer to serve as a baselne to compare wth predctons developed wth other technques. Ths measure smply consders the number of smlar features between a canddate case and a target case. 3.3 Explanaton-Based Learnng Explanaton-based learnng (EBL) uses doman-specfc knowledge to generalze a concept [13]. Can et al. [11] used the EBL noton to explore feature relevance -n a combned approach wth CBR - to classfy a dchotomous dependent varable because the features n ther dataset outnumbered the cases. We nvestgate ther CBR+EBL approach because we are desgnng a tool wth an evolvng collecton process, and thus havng more features than cases s a future possblty. The work descrbed n [11] has successfully classfed foregn trade negotaton cases (50 cases, 76 features) by applyng the CBR+EBL approach. The approach entals a parameterzed smlarty measure that ncorporates both elements of tradtonal CBR (explorng the smlarty of features between projects) and elements of EBL (explorng features relevance supported by doman knowledge). The parameterzed Equaton (1) of CBR+EBL ntroduced n [11] s gven by smlarty measure S: S = n = 1 α * sm ( case, cue ) + β n + β = 1 n = 1 n relevance ( case relevance ( case ) ) * sm ( case, cue ). (1) To mplement the CBR+EBL approach, we need to acqure doman-specfc knowledge and choose approprate values for α and β. Knowledge acquston for EBL. Ths knowledge elctaton process s amed at determnng a relevance factor 1 or 0 for each possble queston answered n the questonnare. These factors represent, respectvely, whether or not the answers nfluence the project s outcome. Thus, a factor 1 s gven when the answer for a queston s such that t supports the outcome of that specfc project. For example, software engneerng knowledge mandates that, to be successful, projects should have a schedule (consequently the lack of a schedule could explan falure). Therefore, there are two possble answers for the queston askng whether the project had a schedule: yes or no. The fnal factor 1 or 0 can only be obtaned when we also consder the gven project s outcome. Thus, n a project wth outcome of success, f the answer to the queston about the exstence of the schedule s also yes, then the relevance factor s 1; because, based on doman knowledge, we can state that ths answer supports the outcome of success. The relevance factors vary as shown n Table 2.

6 We ntervewed two software engneers for consstency and represented the knowledge wth smple rules. For example, Fgure 1 shows a multple-choce queston from the customer/user secton of the questonnare, and ts assocated rule obtaned through knowledge acquston from experts. Table 2. Relevance factors for queston about exstence of schedule Does the project have a schedule? Project outcome s Relevance Factor s Yes Success 1 Yes Falure 0 No Success 0 No Falure 1 By mplementng rules for all features where an assocaton occurs based on domanspecfc knowledge, we were able to determne the relevance factor of a partcular feature for a case. Some features can be left wthout rules because no assocaton was evdent. For example, the Yes/No queston q1_7, Dd senor management mpact the project n any other way? does not drectly assess the type of mpact and thus was the only feature n our dataset left wthout a rule. Ths queston exemplfes how we use EBL to help evolve the collecton method; by confrmng the mportance of a gven queston through ts mpact on project outcomes (queston q1_7 wll be removed or reworded n the next verson of the questonnare). Queston: 2.1 What was the level of nvolvement of the customers/users? 1. none 2. lttle 3.some 4. reasonable level 5. hgh nvolvement Rule: IF ((outcome s success) AND (answer s ether (4) OR (5))) OR IF ((outcome s falure) AND (answer s ether (1) OR (2))) THEN relevance factor equals 1, otherwse relevance factor equals 0. Fg. 1. Example queston and ts assocated rule Experts were able to provde assocatons for nearly all questons because the entre questonnare was conceved and desgned wth the sole purpose of analyzng software development projects. Consderable research has been publshed n ths area [e.g., 16, 18]. An aspect that we dd not mplement concerns combnatons of features. It s possble, and very lkely, that experts usng doman knowledge could fnd assocatons between two or more features. One assocaton could explan a gven outcome, whle another may neutralze the effect of ndvdual features. We dd not extend our nvestgaton to address ths, although we wll consder ths aspect n future research.

7 Determnng α and β. Equaton (1) expresses smlarty S as the weghted dstance between two cases that ncorporates β as a weght measurement for the relevance factor. In the EBL component, features are consdered relevant when ther values support the outcome of each case, and thus they are assgned a relevance factor. Therefore, dfferent cases have dfferent sets of relevant features. The CBR component of the equaton, represented by α, explores the smlarty of features between cases. The fnal step n applyng the parameterzed equaton from [11] s to defne values for α and β. The authors [11] who conceved the equaton do not explctly recommend values for α and β; they used α = 1 and β = 15. They stated that the equaton s not very senstve to β and that β values greater than one wll produce smlar results. The baselne for these parameters occurs when α = 0, n whch case only the EBL component s evaluated and dfferent values for β do not mpact the accuracy. When β =0, however, the equaton evaluates smple feature countng (unweghted k-nn). We also evaluated the senstvty of varatons of these two parameters wth respect to our data. Intally we set α = 1, and then 5, 10 and 20 and vared β n search of the greatest accuracy. We found the maxmum accuracy to occur n multple pars of ponts for α and β. To account for the senstvty of these parameters n dfferent datasets, to ensure consstency of the results, and gven that the authors n [11] clam that values above one for β do not mpact the results, we chose to use four dfferent pars for α and β: (1,1), (5,7), (10, 11.5), and (20, 32). Though we perform all the tests usng all the pars, we wll present only the results obtaned wth the frst par (1,1). Gven the bas mposed by applyng the same effect to all features (some may be rrelevant) [19] on the case-based component of the formula, we next nvestgate varable feature weghts. Our goal s to extend the CBR+EBL approach (Equaton (1)) by addng a representaton of relatve feature relevance to that awarded by doman knowledge. Hence, once we have the feature weghts for all the varables, we mplement the combned verson of the parameterzed equaton: S = n = 1 w * sm ( case, cue n w ) + β + β = 1 n = 1 = 1 n relevance relevance ( case ( case ) * sm ( case ), cue ) ; (2) where we use ndvdual weghts nstead of α. To ensure the consstency of the results, we compute S (Equaton (2)) for β = 1 and = 15. The results are dscussed n Secton Feature Weght Learnng The framework for feature weghtng methods descrbed n [19] suggests the use of ncremental hll-clmbng methods when the dataset contans nteractng features. We

8 selected gradent descent (GD), whch s a hll-clmber that uses feedback from the smlarty measure when examnng each case. GD can be mplemented and modfed by adjustng ts geometrc parameters. We used startng step sze 0.5, endng step 0.02, step sze update 0.9, and the number of cases tested was 10. These parameters resulted most effectve n our prelmnary tests. Havng obtaned weghts to account for the relatve mportance of each feature, we used these weghts to predct project outcomes usng the weghted k-nn and weghted CBR+EBL. 3.5 Logstc Regresson LR s generally consdered the most accurate and theoretcally approprate statstcal technque for modelng dchotomous dependent varables, partcularly n datasets wth fewer than 100 examples, though t s also a method that tends to be expensve [14]. Cleary et al., explan that LR s a hghly desrable statstcal model and should be used for model fttng and hypothess testng [14]. It s approprate for our data because outcome s a dchotomous varable. LR produces a formula that predcts the probablty of an occurrence as a functon of the ndependent varables. LR overcomes the problem wth lnear models producng values for the probablty outsde the range of (0,1) desred for a dchotomous dependent varable [19]. Unlke ordnary least squares (OLS) regresson, LR does not assume lnearty of relatonshp between the ndependent varables and the dependent, does not requre normally dstrbuted varables, and n general has less strngent requrements wth respect to the data. 4 Evaluaton Frst we want our evaluaton to determne whch of the CBR technques performs best across three accuracy metrcs when predctng project outcomes wth our dataset. Second we want to compare the performance of the CBR technques to LR for each metrc. Thrd, we want to determne whether the weghted verson of CBR+EBL (S ) s more accurate than ts unweghted verson. 4.1 Methodology We represent the performance of these technques by usng three metrcs. Accuracy represents the number of correct predctons n relaton to the total predctons. True postves represent the number of correct predctons of projects wth outcomes of success. True negatves gve the number of correct predctons of projects wth outcomes of falure. In ths paper, values of accuracy, true postves and true negatves are expressed as percentages.

9 The orgn of the data s explaned n Secton 2. We used the data generated by the questonnare and also prepared for LR. It has 23 symbolc features descrbng 88 tranng examples; 67 successes and 21 falures. For the evaluaton we used stratfed samplng by randomly choosng sx pars of tranng sets (test sets were the complements), wth 44 cases each, mantanng the overall proporton of postve and negatve examples across all sets. Tranng sets 1, 3 and 5 have 33 postve and 11 negatve examples; tranng sets 2, 4 and 6 have 34 postve and 10 negatve examples. We wll present our results n terms of the average and standard devaton across these sx test sets. 1 LR and the weghted forms of CBR requre the use of tranng sets. Tranng sets were used to ether generate equatons for LR or learn feature weghts. Once completed, the tranng parameters were then tested on the testng sets,.e. on data not ncluded n tranng. 4.2 Results A summary of our results s presented n Table 3. These results were obtaned by applyng unweghted k-nn, (unweghted) CBR+EBL, weghted k-nn, weghted CBR+EBL wth β=1, and logstc regresson (LR). The results are gven wth the average and standard devatons across the sx test sets. Table 3. Average and standard devaton for accuracy, true postves, and true negatves Technque Accuracy True Postves True Negatves Ave. St.Dev Ave. St.Dev Ave. St.Dev Unweghted k-nn CBR+EBL Weghted k-nn Weghted CBR+EBL LR On the frst hypothess evaluated, we found CBR+EBL to be the most accurate among the CBR technques for all three metrcs. It performed just as well as the baselne (unweghted k-nn) n predctng successful projects, and was superor n the other two metrcs, though one should note that the standard devaton resulted from the CBR+EBL performance s hgher than the baselne. The results also confrm the conclusons presented n [11] that CBR+EBL outperforms unweghted k-nn. These results support the concluson to adopt CBR+EBL, partcularly because we can expect to have case bases n whch the number of features s greater than the number of cases, whch seems to be no obstacle for CBR+EBL. 1 Though we beleve LOOCV s preferable, applyng LR would requre that we developed 88 sets of equatons, therefore we reled on stratfed samplng.

10 Wth respect to the second hypothess evaluated, CBR+EBL slghtly outperformed LR n the average accuracy, but LR presented a hgher standard devaton. In the average of true postves, LR was superor. LR predcted successful projects well, but was not able to predct faled cases as accurately (less than 30% for true negatves). Ths was lkely due to the sparsty of data among the group of faled software projects (.e., 21 out of 88). Incluson of addtonal faled project cases would very lkely mprove these results. Gven that the data for successful projects s suffcently dense, the metrc true postves n Table 3 emphaszes the loss n accuracy caused by usng a combnaton of feature weghts and the EBL measure (weghted CBR+EBL). Ths s the only technque not able to predct at least 80% of successful projects. Even the unweghted k-nn performs well, easly fndng smlar cases among postve examples. LR s the most accurate n ths metrc. The relatve lack of negatve examples (or faled projects) makes the accuracy of CBR+EBL stand out. Ths s probably the reason why CBR+EBL tends to be more accurate n general. Even wth fewer negatve examples, t provdes better predctons. Our results suggest that there s an advantage n usng CBR especally CBR+EBLwth ths type of data, whch may often be sparse n real world problems. Therefore, we recommend CBR+EBL to predct the (bnary) outcome of software development projects. For the thrd hypothess evaluated, we wanted to compare the performance of the weghted verson of CBR+EBL (S ) wth respect to the unweghted S. S provded the lowest accuracy wth the hghest standard devaton. It actually performed more poorly n predctng successful projects from dense data than t dd predctng faled cases from sparse data. These results suggest nvestgatng further why the weghted verson of CBR+EBL dd not perform well n accuracy and true postves but performed much better (wth respect to other technques) n true negatves. The varaton n performance of LR suggests an assocaton wth the number of negatve examples; hence we want to nvestgate f a smlar assocaton could be made. Possble causes for performance of S. Table 3 shows that the weghted verson of CBR+EBL outperforms all technques that do not use doman knowledge for the metrc true negatves. Ths may suggest that the combnaton of both the CBR and EBL components would be approprate to learn from sparse data (and predct faled projects). When we compare the performance of the unweghted versons alone, CBR+EBL performs (a lttle) better than k-nn for accuracy, exactly the same as k- NN for true postves, and (much) better than k-nn for true negatves. Ths comparson suggests that when we add the EBL component, the performance mproves wth respect to true negatves. Addtonally, f we analyze the weghted versons, the weghted CBR+EBL performs worse than the weghted k-nn. It performs just as poorly for true postves and better only for true negatves. These facts, combned wth the hgh standard devatons, found for the weghted CBR+EBL measure, nstgate further examnaton. Gven ths prelmnary analyss, our

11 hypothess s that when we combne feature weghts wth the EBL component, t overestmates the relatve mportance of some features. Ths s detrmental to predctve accuracy wth dense data, but when appled to sparse data, the method seems to work farly well. In future work, we wll perform a second experment to evaluate ths hypothess; n ths paper, we smply examne further our results. In order to fully gauge the effect of combnng feature weghts wth EBL, we examne the weghts and the EBL component n dfferent test sets. The weghts were determned by gradent descent (see Secton 3.4), whle the EBL component s provded by the assgnment of relevance factors (see Secton 3.3). Table 4 ranks the sx varables that receved the largest number of relevance factors overall. The fnal row shows the performance of the weghted CBR+EBL for true postves for these sets. The two columns n Table 4 dedcated to the two test sets show each varable s overall rankng wthn each test set based upon ther feature weghts (the hgher the number, the smaller the weght). For example, varable q3_7 2 was fourth among the varables overall n terms of the number of relevance factors assgned, ranked fourth n test set 2 and 16 th n test set 6. Table 4. Relevance factors and weghts n test sets 2 and 6 Rank Varable ID Test Set 2 Test Set 6 1 q3_ q3_ q4_ q3_ q4_ q3_ True Postves The analyss of Table 4 shows that fve of the sx varables were heavly weghted n test set 2 (rankngs 1, 3, 4, 5, 7) and lghtly weghted n test set 6 (rankngs 10, 16, 17, 19). When comparng these rankngs to the performance for true postves, t s clear that where the majorty of these varables were heavly weghted, weghted CBR+EBL performed the most poorly and where these varables were most lghtly weghted, CBR+EBL performed the most accurately. Ths supports our nterpretaton that predcton accuracy decreases when the same features receve hgh weghts and are assgned relevance factors n the majorty of the cases (overestmatng the relatve mportance of some features). We wll evaluate ths hypothess n future work, because f the combnaton of the two technques ncreases predcton accuracy when the dataset s sparsely populated, ths measure can be used n these cases. 2 See meanng of feature n Table 1.

12 5 Assocatng Outcomes to Project Management Ths paper s fnal challenge s to make use of the most accurate technque to perform an addtonal task. We would lke to determne whch factors have the strongest assocatons wth partcular project outcomes. These could then be hghlghted as crtcal rsk factors n projects headng for falure, allowng a project manager to dentfy key strengths and weaknesses early enough to establsh correctve measures when needed. Gven the sutablty of LR, we agan use t as a benchmark. The LR process ncludes the dentfcaton of the varables that most strongly predct the dependent varable; thus, predctor varables are a byproduct of LR. Based on LR, the varables that have the most nfluence on project outcome are q3_3, q3_5, q4_2, and q4_8. We use the parameterzed equaton S (CBR+EBL) to suggest predctor varables by predctng the outcome of each of the four questonnare sectons separately. We compare these predcton results wth the predcton generated usng the entre dataset. Isolated problem areas and features that predct outcome nearly as well as the entre dataset are assumed to be those most responsble for project outcome. Table 5. Average accuracy (n %) for the four sectons across the sx test sets Problem Area Ave. Std dev. Management Support Customer/User Requrements Estmaton/Schedule Table 5 shows the average accuracy n each of the four sectons for the sx test sets. Gven the smlarty of average accuracy of the three problem areas customer/user, requrements, and estmaton-schedule, our strategy s to further examne these three areas for potentally useful predctors. We wll exclude the management support secton because of ts lower accuracy when compared to the other sectons. In order to further nvestgate the three problem areas, we assess the frequences of the relevance factors n each test set. Our assumpton s that features that have scored a hgher number of relevance factors are those most responsble for the project outcome. We note that these tests were performed on test sets wth 44 cases, so that a feature that has, for example, been assgned a relevance factor 29 tmes, has nfluenced 65.9% of the cases. Table 6 summarzes the averages across the sx test sets for the four most relevant varables n each of the three sectons. These varables are the ones wth the best potental to be predctors. Among these varables, q3_3 and q3_5 n the requrements secton, and q4_8 and q4_2 n estmaton-schedule are also the varables dentfed by LR. Table 6. Average assgnments of relevance factors Customer-user Requrements Estmaton and Schedule Varable ID Ave Varable ID Ave Varable ID Ave

13 q2_ q3_ q4_ q2_ q3_ q4_1lst 22.5 q2_ q3_ q4_ q2_ q3_ q4_ We now examne the varable wth the hghest relevance factor frequences (q3_5) to determne whether t has been valued because t s relevant to successful or faled projects. For ths last analyss, we do not want to use averages, so we select test sets fve and sx because they are the sets that present relevance frequences that are the most smlar to the averages for the requrements secton. In test set 5, feature q3_5 supported successful outcomes n 87.9% of the cases, that s, 29 out of 33. It supported falure outcomes n 45.5% of the cases, 5 out of 11. In test set 6, q3_5 supported success n only 26 of 34 cases, whch represented 76.5% of the cases. The number of faled projects supported by q3_5 n ths set was 7, representng 70% of the cases (7 out of 10). Lookng exclusvely at these two test sets, q3_5 supported success n over 80% of the cases; and supported falure n almost 60% of the cases. It would be premature to state whether ths feature can be consdered as a crtcal success factor n successful or faled projects. Further study s necessary to dentfy a method to valdate such condtons for a varable. Because ths varable deals wth the tme dedcated for requrements gatherng by the customer/users, t s easy to accept t conceptually as a crtcal factor for both success and falure n ths nstance. These tests are not conclusve but show promsng results for further research nto automatcally determnng success and rsk factors. Further research wll delve more deeply nto methods for dentfyng crtcal factors for success and falure; ths wll help project managers to better understand falure and how to ncrease chances of success by takng acton early n software project development. 6 Concludng Remarks Ths paper extends our understandng of the amenablty of usng CBR to support KM tasks for managng knowledge from projects descrbed by manageral factors. We conclude that the parameterzed CBR+EBL smlarty measure [11] s the most accurate technque for ths applcaton. The second most accurate CBR technque, weghted k-nn, performed nearly as accurately as CBR+EBL overall, but t dd not perform as well for predctng project falures because t appears to be less tolerant of sparse data sets. When the dataset was suffcently dense, CBR+EBL performed at or above the level of all other methods and t stood out when the data was sparse. For ths reason, the CBR+EBL measure s the best choce for ths type of predcton. Wth respect to the CBR technques we used, one ntrgung result was that the weghted k-nn performed smlarly to CBR+EBL (S) n some nstances but combnng the two methods (S ) generally decreased predcton accuracy. We beleve that ths s because the two methods provde smlar relatve levels of mportance to

14 the same data and a combnaton of these methods ultmately overstates the relatve strength of these varables. In general, ths may lead to less accurate predcton results. As dscussed n Subsecton 4.2, however, when we analyzed predcton for sparse data sets, t s possble that the combnaton of these methods wll allow for a strong assocaton between key factors and outcomes. It s also mportant to note, however, that even wth sparse data, the combnaton of these methods dd not outperform the CBR+EBL method alone. Probng ths queston more deeply wll be a focus of future research. The accuracy provded by logstc regresson suggests ts use as a benchmark for the predcton accuracy of a case base, when the dependent varable s dchotomous. These technques cannot be consdered as compettve alternatves for supportng a KM framework because LR poses hgher engneerng requrements, as t requres statstcal expertse and a complex process of analyss. These tasks may be more easly performed usng a CBR technque, whch has less strngent engneerng requrements. The prmary cost beneft of CBR s manfested through automaton knowledge acquston and system reuse may be fully automated so that staff members may use the tool wthout needng n-depth knowledge about the technques used by the tool. We have shown how to detect potental predctor varables for project success. Our work dentfes q3_3, a well defned project scope; q3_5, customers/users makng adequate tme for requrements gatherng; q4_2, a delvery decson (schedule) made wth adequate requrements nformaton; and q4_8, assgnment of adequate staff to meet the project schedule, as the success factors n our dataset. It s noteworthy that CBR has found q3_5 to be the most mportant factor for both project success and falure. As project managers well know, estmaton of a reasonable schedule s mpossble wthout good requrements; good requrements and a well-defned project scope go hand-n-hand; and good requrements are essental for assgnng adequate staff to a project schedule. There are a few areas of future work that we wsh to explore: evaluate the causes of the performance of S, valdate predctor varable dentfcaton, nvestgate the use of predctor varables to derve mtgants for project management, and analyze combnatons of features to assgn relevance factors n EBL. In the CBR mplementaton, we wll ncorporate the step that ndcates the success or falure factors. Acknowledgements. Rosna Weber s research s supported n part by the Natonal Insttute for Systems Test and Productvty at USF under the USA Space and Naval Warfare Systems Command grant no. N C-3244, for L0, References 1. Watson, I.: CBR s a methodology not a technology. Knowledge Based Systems, 12 (5-6) (1999)

15 2. Fnne, G.R., Wttg, G.E., Desharnas, J.M.: A Comparson of Software Effort Estmaton Technques: Usng Functon Ponts wth Neural Networks, Case-Based Reasonng and Regresson Models. Journal of Systems Software 39 (1997) Aha, D. W.: Feature weghtng for lazy learnng algorthms. In H. Lu & H. Motoda (eds.): Feature Extracton, Constructon and Selecton: A Data Mnng Perspectve. Norwell, MA, Kluwer (1998) Aha, D.W., Becerra-Fernandez, I., Maurer, F., & Muñoz-Avla, H.: (eds.) Explorng Synerges of Knowledge Management and Case-Based Reasonng: Papers from the AAAI Workshop (Techncal Report WS-99-10). Menlo Park, CA, AAAI Press (1999) 5. Watson, I.: Knowledge Management and Case-Based Reasonng: a perfect match? In Proc. of the Fourteenth Annual Conference of the Internatonal Florda Artfcal Intellgence Research Socety Menlo Park, CA, AAAI Press (2001) Weber, R., Aha, D.W., Becerra-Fernandez, I.: Intellgent lessons learned systems. Internatonal Journal of Expert Systems Research & Applcatons 20 1 (2001) Althoff, K.D., Nck, M., Tautz, C.: CBR-PEB: An Applcaton Implementng Reuse Concepts of the Experence Factory for the Transfer of CBR System Know-How. In Proc. of 7th German Workshop on Case-Based Reasonng, Würzburg (1999) Kadoda, G, Cartwrght, M., Shepperd, M.: Issues on the Effectve use of CBR Technology for Software Project Predcton. In Aha, D., Watson, I., (eds.): Case-Based Reasonng Research and Development, LNAI, 2080, Sprnger (2001) Watson, I., Mendes, E., Mosley, N., Counsell, S.: Usng CBR to Estmate Development Effort for Web Hypermeda Applcatons. In Proc. of the Ffteenth Annual Conference of the Internatonal Florda Artfcal Intellgence Research Socety. Menlo Park, CA, AAAI Press (2002) Basl, V.R., Caldera, G., Rombach, H.D.: Experence Factory. In J. J. Marcnak, (ed), Encyclopeda of Software Engneerng 1 (1994) Jedltschka, A., Althoff, K.-D., Decker, B., Hartkopf, S., Nck, M.: Corporate Informaton Network (COIN): The Fraunhofer IESE Experence Factory, IESE-Report No /E. Verson 1.0, May (2001) 12. Can, T., Pazzan, M. J., Slversten, G. Usng doman knowledge to nfluence smlarty judgment. In Proc. of the Case-Based Reasonng Workshop. Washngton, DC, Morgan Kaufmann (1991) Mtchell, T., Keller, R., Kedar-Cabell, S.: Explanaton-based generalzaton: A Unfyng Vew. Machne learnng 2 (1986) Cleary, P.D., Angel, R.: The Analyss of Relatonshps Involvng Dchotomous Dependent Varables, Journal of Health and Socal Behavor 25 (1984) Verner, J. M, Overmyer, S. P. and McCan, K. W.: In the 25 Years Snce the Mythcal Man-Month What Have we Learned About Project Management? Informaton and Software Technology 41 (1999) Procaccno, J.D., Verner, J.M., Overmyer, S. P., Darter, M.: Case Study: Factors for Early Predcton of Software Development Success, Informaton and Software Technology 44 (2001) Aamodt, A., Plaza, E.: Case-Based Reasonng: Foundatonal Issues, Methodologcal Varatons, and System Approaches. Artfcal Intellgence Communcatons 7(1) (1994) Reel, J.S.: Crtcal Success Factors n Software Projects, IEEE Software 16(3) (1999) Wettschereck, D., Aha, D.W.: Weghtng features. Veloso, M., Aamodt, A. (eds). Case- Based Reasonng Research and Development, LNAI 1010, Sprnger-Verlag (1995)

16 20. Whtehead, J.: Wllngness to Pay for Bass Fshng Trps n the Carolnas. In An Introducton to Logstc Regresson, Wrtng up results. (1998) Avalable onlne (last vsted 03/31/2003):