Analysis of Empirical Software Effort Estimation Models

Transcription

1 (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, Analyss of Emprcal Software Effort Estmaton Models Saleem Basha Department of Computer Scence Pondcherry Unversty Puducherry, Inda Dhavachelvan P Department of Computer Scence Pondcherry Unversty Puducherry, Inda [email protected] Abstract Relable effort estmaton remans an ongong challenge to software engneers. Accurate effort estmaton s the state of art of software engneerng, effort estmaton of software s the prelmnary phase between the clent and the busness enterprse. he relatonshp between the clent and the busness enterprse begns wth the estmaton of the software. he credblty of the clent to the busness enterprse ncreases wth the accurate estmaton. Effort estmaton often requres generalzng from a small number of hstorcal projects. Generalzaton from such lmted experence s an nherently under constraned problem. Accurate estmaton s a complex process because t can be vsualzed as software effort predcton, as the term ndcates predcton never becomes an actual. hs work follows the bascs of the emprcal software effort estmaton models. he goal of ths paper s to study the emprcal software effort estmaton. he prmary concluson s that no sngle technque s best for all stuatons, and that a careful comparson of the results of several approaches s most lkely to produce realstc estmates. Keywords-Software Estmaton Models, Conte s Crtera, Wlcoxon Sgned-Rank est. I. INRODUCION Software effort estmaton s one of the most crtcal and complex, but an nevtable actvty n the software development processes. Over the last three decades, a growng trend has been observed n usng varety of software effort estmaton models n dversfed software development processes. Along wth ths tremendous growth, t s also realzed the essentalty of all these models n estmatng the software development costs and preparng the schedules more quckly and easly n the antcpated envronments. Although a great amount of research tme, and money have been devoted to mprovng accuracy of the varous estmaton models, due to the nherent uncertanty n software development projects as lke complex and dynamc nteracton factors, ntrnsc software complexty, pressure on standardzaton and lack of software data, t s unrealstc to expect very accurate effort estmaton of software development processes [1]. hough there s no proof on software cost estmaton models to perform consstently accurate wthn 25% of the actual cost and 75% of the tme [30], stll the avalable cost estmaton models extendng ther support for ntended actvtes to the possble extents. he accuracy of the ndvdual models decdes ther applcablty n the projected envronments, whereas the accuracy can be defned based on understandng the calbraton of the software data. Snce the precson and relablty of the effort estmaton s very mportant for the compettveness of software companes, the enterprses and researchers have put ther maxmum effort to develop the accurate models to estmate effort near to accurate levels. here are many estmaton models have been proposed and can be categorzed based on ther basc formulaton schemes; estmaton by expert [5], analogy based estmaton schemes [6], algorthmc methods ncludng emprcal methods [7], rule nducton methods [8], artfcal neural network based approaches [9] [17] [18], Bayesan network approaches [19], decson tree based methods [21] and fuzzy logc based estmaton schemes [10] [20]. Among these dversfed models, emprcal estmaton models are found to be possbly accurate compared to other estmaton schemes and COCOMO, SLIM, SEER-SEM and FP analyss schemes are popular n practce n the emprcal category [24] [25]. In case of emprcal estmaton models, the estmaton parameters are commonly derved from emprcal data that are usually collected from varous sources of hstorcal or passed projects. Accurate effort and cost estmaton of software applcatons contnues to be a crtcal ssue for software project managers [23]. here are many ntroductons, modfcatons and updates on emprcal estmaton models. A common modfcaton among most of the models s to ncrease the number of nput parameters and to assgn approprate values to them. hough some models have been nundated wth more number of nputs and output features and thereby the complexty of the estmaton schemes s ncreased, but also the accuracy of these models has shown wth lttle mprovement. Although they are dversfed, they are not generalzed well for all types of envronments [13]. Hence there s no slver bullet estmaton scheme for dfferent envronments and the avalable models are envronment specfc. II. COCOMO ESIMAION MODEL A. COCOMO 81 COCOMO 81 (Constructve Cost Model) s an emprcal estmaton scheme proposed n 1981 [29] as a model for estmatng effort, cost, and schedule for software projects. It 68

2 (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, was derved from the large data sets from 63 software projects varables, whch s hghly dependent on development the rangng n sze from 2,000 to 100,000 lnes of code, and uncertanty at the nput level of the COCOMO yelds programmng languages rangng from assembly to PL/I. hese uncertanty at the output, whch leads to gross estmaton error data were analyzed to dscover a set of formulae that were the n the effort estmaton [33]. Irrespectve of these drawbacks, best ft to the observatons. hese formulae lnk the sze of the COCOMO II models are stll nfluencng n the effort system and Effort Multplers (EM) to fnd the effort to develop estmaton actvtes due to ther better accuracy compared to a software system. In COCOMO 81, effort s expressed as other estmaton schemes. Person Months (PM) and t can be calculated as III. SEER-SEM ESIMAION MODEL where, 15 b PM a * Sze * EM 1 a and b are the doman constants n the model. It contans 15 effort multplers. hs estmaton scheme accounts the experence and data of the past projects, whch s extremely complex to understand and apply the same. Cost drves have a ratng level that expresses the mpact of the drver on development effort, PM. hese ratng can range from Extra Low to Extra Hgh. For the purpose of quanttatve analyss, each ratng level of each cost drver has a weght assocated wth t. he weght s called Effort Multpler. he average EM assgned to a cost drver s 1.0 and the ratng level assocated wth that weght s called Nomnal. B. COCOMO II In 1997, an enhanced scheme for estmatng the effort for software development actvtes, whch s called as COCOMO II. In COCOMO II, the effort requrement can be calculated as SEER (System Evaluaton and Estmaton of Resources) s a propretary model owned by Galorath Assocates, Inc. In 1988, Galorath Incorporated began work on the ntal verson of SEER-SEM whch resulted n an ntal soluton of 22,000 lnes of code. SEER (SEER-SEM) s an algorthmc project management software applcaton desgned specfcally to estmate, plan and montor the effort and resources requred for any type of software development and/or mantenance project. SEER, whch comes from the noun, referrng to one havng the ablty to foresee the future, reles on parametrc algorthms, knowledge bases, smulaton-based probablty, and hstorcal precedents to allow project managers, engneers, and cost analysts to accurately estmate a project's cost schedule, rsk and effort before the project s started. Galorath chose Wndows due to the ablty to provde a more graphcal user envronment, allowng more robust management tradeoffs and understandng of what drves software projects.[4] hs model s based upon the ntal work of Dr. Randall Jensen. he mathematcal equatons used n SEER are not avalable to the publc, but the wrtngs of Dr. Jensen make the basc equatons avalable for revew. he basc equaton, Dr. Jensen calls t the "software equaton" s: where 17 E PM a * Sze * EM E 1 B 0.01 * SF 5 j 1 COCOMO II s assocated wth 31 factors; LOC measure as the estmaton varable, 17 cost drves, 5 scale factors, 3 adaptaton percentage of modfcaton, 3 adaptaton cost drves and requrements & volatlty. Cost drves are used to capture characterstcs of the software development that affect the effort to complete the project. COCOMO II used 31 parameters to predct effort and tme [11] [12] and ths larger number of parameters resulted n havng strong co-lnearty and hghly varable predcton accuracy. Besdes these mertorous clams, COCOMO II estmaton schemes are havng some dsadvantages. he underlyng concepts and deas are not publcly defned and the model has been provded as a black box to the users [26]. hs model uses LOC (Lnes of Code) as one of the estmaton varables, whereas Fenton et. al [27] explored the shortfalls of the LOC measure as an estmaton varable. he COCOMO also uses FP (Functon Pont) as one of the estmaton j where, 0.5 Se Cte ( Ktd ) S s the effectve lnes of code, ct s the effectve developer technology constant, k s the total lfe cycle cost n man-years, and td s the development tme n years. hs equaton relates the effectve sze of the system and the technology beng appled by the developer to the mplementaton of the system. he technology factor s used to calbrate the model to a partcular envronment. hs factor consders two aspects of the producton technology -- techncal and envronmental. he techncal aspects nclude those dealng wth the basc development capablty: Organzaton capabltes, experence of the developers, development practces and tools etc. he envronmental aspects address the specfc software target envronment: CPU tme constrants, system relablty, real-tme operaton, etc. he SEER-SEM developers have taken the approach to nclude over 30 nput parameters, ncludng the ablty to run Monte Carlo smulaton to compensate for rsk [2]. Development modes covered nclude object orented, reuse, COS, spral, waterfall, prototype and ncremental 69

3 development. Languages covered are 3rd and 4th generaton languages (C++, FORRAN, COBOL, Ada, etc.), as well as applcaton generators. It allows staff capablty, requred desgn and process standards, and levels of acceptable development rsk to be nput as constrants [15]. Fgure 1 s adapted from a Galorath llustraton and shows gross categores of model nputs and outputs, but each of these represents dozens of specfc nput and output possbltes and parameters. Fgure 1. SEER-SEM I/O Parameters Features of the model nclude the followng: Allows probablty level of estmates, staffng and schedule constrants to be nput as ndependent varables. Facltates extensve senstvty and trade-off analyses on model nput parameters. Organzes project elements nto work breakdown structures for convenent plannng and control. Dsplays project cost drvers. Allows the nteractve schedulng of project elements on Gantt charts. Bulds estmates upon a szable knowledge base of exstng projects. Model specfcatons nclude these: Sze Personnel Envronment Complexty Constrants Input Parameters SEER-SEM Effort Cost Schedule Rsk Mantenance Relablty Output Parameters Parameters: sze, personnel, complexty, envronment and constrants - each wth many ndvdual parameters; knowledge base categores for platform & applcaton, development & acquston method, applcable standards, plus a user customzable knowledge base. Predctons: effort, schedule, staffng, defects and cost estmates; estmates can be schedule or effort drven; constrants can be specfed on schedule and staffng. Rsk Analyss: senstvty analyss avalable on all least/lkely/most values of output parameters; probablty settngs for ndvdual WBS elements adjustable, allowng for sortng of estmates by degree of WBS element crtcalty. Szng Methods: functon ponts, both IFPUG sanctoned plus an augmented set; lnes of code, both new and exstng. (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, Outputs and Interfaces: many capablty metrcs, plus hundreds of reports and charts; trade-off analyses wth sde-bysde comparson of alternatves; ntegraton wth other Wndows applcatons plus user customzable nterfaces. Asde from SEER-SEM, Galorath, Inc. offers a sute of many tools addressng hardware as well as software concerns. One of partcular nterest to software estmators mght be SEER-SEM, a tool desgned to perform szng of software projects. he study done by hbodeau n 1981 and a study done by II Research Insttute (IIRI) n 1989 states that they calbrated SEER-SEM model usng three databases. he sgnfcance of ths study s as follows: 1. Results greatly mproved wth calbraton, n fact, as hgh as a factor of fve. 2. Models consstently obtaned better results when used wth certan types of applcatons. he IIRI study was sgnfcant because t analyzed the results of seven cost models (PRICE-S, two varants of COCOMO, System-3, SPQR/20, SASE, SoftCost-Ada) to eght Ada specfc programs. Ada was specfcally desgned for and s the prncpal language used n mltary applcatons, and more specfcally, weapons system software. Weapons system software s dfferent then the normal corporate type of software, commonly known as Management Informaton System (MIS) software. he major dfferences between weapons system and MIS software are that weapons system software s real tme and uses a hgh proporton of complex mathematcal codng. Up to 1997, DOD mandated Ada as the requred language to be used unless a waver was approved. Lloyd Mosemann stated: he results of ths study, lke other studes, showed estmatng accuracy mproved wth calbraton. he best results were acheved by SEER-SEM model were accurate wthn 30 percent, 62 percent of the tme. IV. SLIM ESIAMION MODEL SLIM Software Lfe-Cycle Model was developed by Larry Putnam [3]. SLIM hres the probablstc prncple called Raylegh dstrbuton between personnel level and tme. SLIM s bascally applcable for large projects exceedng 70,000 lnes of code. [4]. Percentage of otal Effort =0 D td dy dt me Fgure 2. he Raylegh Model 2 Kate 2 2 at 70

4 It makes use of Raylegh curve referred from [14] as shown n fgure 2 for effort predcton. hs curve represents manpower measured n person per tme as a functon of tme. It s usually expressed n personyear/ year (PY/YR). It can be expressed as where, dy dt 2 at 2 Kate dy/dt s the manpower utlzaton per unt tme, t s the elapsed tme, a s the parameter that affects the shape of the curve and K s the area under the curve. here are two mportant terms assocated wth ths curve: 1) Manpower Buld up gven by D 0 =K/t d 3 2) Productvty = Lnes of Code/ Cumulatve Manpower.e. P=S/E and S= CK 1/3 t d 4/3,where C s the technology factor whch reflects the effects of varous factors on productvty such as hardware constrants, program complexty, programmng envronment and personal experence. he SLIM model uses two equatons: the software the manpower equaton and software productvty level equaton he SLIM model uses Raylegh dstrbuton to estmate to estmate project schedule and defect rate. wo key attrbutes used n SLIM method are productvty Index (PI) and Manpower Buldup Index (MBI). he PI s measure of process effcency (cost-effectveness of assets), and the MBI determnes the effects on total project effort that result from varatons n the development schedule [A Probablstc Model]. Inputs Requred: o use the SLIM method, t s necessary to estmate system sze, to determne the technology factor, and approprate values of the manpower acceleraton. echnology factor and manpower acceleraton can be calculated usng smlar past projects. System sze n terms of KDSI s to be subjectvely estmated. hs s a dsadvantage, because of the dffculty of estmatng KDSI at the begnnng of a project and the dependence of the measure on the programmng language. Completeness of Estmate: he SLIM model provdes estmates for effort, duraton, and staffng nformaton for the total lfe cycle and the development part of the lfe cycle. COCOMO I provdes equatons to estmate effort, duraton, and handles the effect of re-usng code from prevously developed software. COCOMO II provdes cost, effort, and schedule estmaton, dependng on the model used (.e., dependng on the degree of product understandng and marketplace of the project). It handles the effect of reuse, reengneerng, and mantenance adjustng the used sze measures usng parameters such as percentage of code modfcaton, or percentage of desgn modfcaton Assumptons: SLIM assumes the Raylegh curve dstrbuton of staff loadng. he underlyng Raylegh curve assumpton does not hold for small and medum szed projects. Cost estmaton s only expected to take place at the start of the 2 (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, desgn and codng, because requrement and specfcaton engneerng s not ncluded n the model. Complexty: he SLIM model s complexty s relatvely low. For COCOMO the complexty ncreases wth the level of detal of the model. For COCOMO I the ncreasng levels of detal and complexty are the three model types: basc, ntermedate, and detaled. For COCOMO II the level of complexty ncreases accordng to the followng order: Applcaton Composton, Early Desgn, Post Archtecture. Automaton of Model Development: he Putnam method s supported by a tool called SLIM (Software Lfe-Cycle Management). he tool ncorporates an estmaton of the requred parameter technology factor from the descrpton of the project. SLIM determnes the mnmum tme to develop a gven software system. Several commercal tools exst to use COCOMO models. Applcaton Coverage: SLIM ams at nvestgatng relatonshps among staffng levels, schedule, and effort. he SLIM tool provdes facltes to nvestgate trade-offs among cost drvers and the effects of uncertanty n the sze estmate. Generalzablty: he SLIM model s clamed to be generally vald for large systems. COCOMO I was developed wthn a tradtonal development process, and was a pror not sutable for ncremental development. Dfferent development modes are dstngushed (organc, semdetached, embedded). COCOMO II s adapted to feed the needs of new development practces such as development processes talored to COS, or reusable software avalablty. No emprcal results are currently avalable regardng the nvestgaton these capabltes. Comprehensveness: Putnam s method does not consder phase or actvty work breakdown. he SLIM tool provdes nformaton n terms of the effort per major actvty per month throughout development. In addton, the tool provdes error estmates and feasblty analyses. As the model does not consder the requrement phase, estmaton before desgn or codng s not possble. Both COCOMO I and II are extremely comprehensve. hey provde detaled actvty dstrbutons of effort and schedule. hey also nclude estmates for mantenance effort, and an adjustment for code re-use. COCOMO II provdes prototypng effort when usng the Applcaton Composton model. he Archtectural Desgn model nvolves estmaton of the actual development and mantenance phase. he granularty s about the same as for COCOMO I. V. REVIC ESIMAION MODEL REVIC (REVsed verson of Intermedate COCOMO) s a drect descendent of COCOMO. Ourada [16] was one of the frst to analyze valdaton, usng a large Ar Force database for calbraton of the REVIC model. here are several key dfferences between REVIC and the 1981 verson of COCOMO, however. 71

5 REVIC adds an Ada development mode to the three orgnal COCOMO modes; Organc, Sem-detached, and Embedded. REVIC ncludes Systems Engneerng as a startng phase as opposed to Prelmnary Desgn for COCOMO. REVIC ncludes Development, est, and Evaluaton as the endng phase, as opposed to COCOMO endng wth Integraton and est. he REVIC basc coeffcents and exponents were derved from the analyss of a database of completed DoD projects. On the average, the estmates obtaned wth REVIC wll be greater than the comparable estmates obtaned wth COCOMO. REVIC uses PER (Program Evaluaton and Revew echnque) statstcal technques to determne the lnes-of-code nput value, Low, hgh, and most probable estmates for each program component are used to calculate the effectve lnes-of-code and the standard devaton. he effectve lnes-of-code and standard devaton are then used n the estmaton equatons rather than the lnear sum of the lne-of-code estmates. REVIC ncludes more cost multplers than COCOMO. Requrements volatlty, securty, management reserve, and an Ada mode are added. (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, under study. hese ssues nclude: level of system defnton, system tmng and crtcalty, documentaton, etc. A complexty multpler s then derved and used to alter the prelmnary budget and schedule estmates from er II. he software system effort estmaton s then calculated. er IV and V are not necessary for an effort estmaton. er IV addresses the nscope mantenance assocated wth the project. he output of er IV s the monthly man-loadng for the mantenance lfe-cycle. er V provdes the user wth a capablty to perform rsk analyss on the szng, schedule and budget data. he actual mathematcal expressons used n SASE are publshed n the User's Gude, but the Gude s very unclear as to what they mean and how to use them VII. COSMODL ESIMAION MODEL COSMODL (Cost MODeL) s a COCOMO based estmaton model developed by the NASA Johnson Space Center. he program delvered on computer dsk for COSMODL ncludes several versons of the orgnal COCOMO and a NASA developed estmaton model KISS (Keep It Smple, Stupd). he KISS model wll not be evaluated here, but t s very smple to understand and easy to use; however, the calbraton envronment s unknown. he COSMODL model ncludes the basc COCOMO equatons and modes, along wth some modfcatons to nclude an Ada mode and other cost multplers. VI. SASE ESIMAIN MODEL SASE (Software Archtecture, Szng and Estmatng ool) s a forward channg, rule-based expert system usng a herarchcally structured knowledge database of normalzed parameters to provde derved software szng values. hese values can be presented n many formats to nclude functonalty, optmal development schedule, and man-loadng charts. SASE was developed by Martn Maretta Denver Aerospace Corp. on contract to the Naval Center for Cost Analyss. o use SASE, the user must frst perform a software decomposton of the system and defne the functonaltes assocated wth the gven software system [22]. SASE uses a tered approach for system decomposton; er 1 addresses software developmental and envronmental ssues. hese ssues nclude che class of the software to be developed, programmng language, developmental, schedule, securty, etc. er 1 output values represent prelmnary budget and schedule multplers. er II specfes the functonal aspects of the software system, specfcally the total lnes-of-code (LOC). he total LOC estmate s then translated nto a prelmnary budget estmate and prelmnary schedule estmate. he prelmnary budget and schedule estmates are derved by applyng the multplers from er I to the total LOC estmate. er III develops the software complexty ssues of the system he COSMODL as delvered ncludes several calbratons based upon dfferent data sets. he user can choose one of these calbratons or enter user specfed values. he model also ncludes a capablty to perform a self-calbraton. he user enters the necessary nformaton and the model wll "reverse" calculate and derve the coeffcent and exponent or a coeffcent only for the nput envronment data. he model uses the COCOMO cost multplers and does not nclude more as does REVIC. hs model ncludes all the phases of a software lfe cycle. PER technques are used to estmate the nput lnes-of-code n both the development and mantenance calculatons VIII. SUDY OF EMPIRICAL MODELS Emprcal estmaton models were studed for the past couple of decades, out of these studes many came wth the result of accuracy and performance. able I summares the bref study of the most relevant emprcal models. Studes are lsted n chronologcal order. For each study, estmaton methods are ranked accordng to ther performance. A 1 ndcates the best model, 2 the second best, and so on. 72

6 (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, ABLE I. SUMMARY OF EMPIRICAL ESIMAION SUDY Sl No. Author Regresson COCOMO Analogy SLIM CAR ANN A. Impact of Cost Drvers Emprcal Software estmaton models manly stands over the cost drvers and scale factors. hese model reveals the problem of nstablty due to values of the cost drvers and scale factors, thus affects the senstvty of the effort. Also, most of the model depends on the sze of the project, a change n the sze leads to the proportonate change n the effort. Mscalculatons of the cost drves have even more vvd change n the result too. For example, a msjudgment n personnel capablty n COCOMO or REVIC from very hgh to very low wll result n 300% ncrease n effort. Smlarly n SEER- SEM changng securty requrements from low to hgh wll result n 400% ncrease n effort. In PRICE-S, 20% change n effort wll occur due to small change n the value of the Productvty factor. All models have one or more nputs for whch small changes wll result n large changes n effort and, perhaps, schedule. he nput data problem s further compounded n that some nputs are dffcult to obtan, especally early n a program. he sze must be estmated early n a program usng one or more szng models. hese models usually have not been valdated for a wde range of projects. Some senstve nputs, such as analyst and programmer capablty, are subjectve and often dffcult to determne. Studes lke one performed by Brent L. Barber, Investgatve Search of Qualty Hstorcal Software Support Cost Data and Software Support Cost-Related Data, show that personnel parameter data are dffcult to collect. Fgure 3, extended from the SEER-SEM User s Manual shows Stepwse ANOVA OSR Expert Judgment 1 Lucana Q, Yeong-Seok Seo, Janfeng Wen, Petrôno L. Braga, Jngzhou L, Irs Fabana de Barcelos ronto, Chao-Jung Hsu, Krstan M Furulund, Blge Başkeleş, Da Deng, Smon, m Menzes, Bente Anda, Cuauhtémoc López Martín, Parag C, Randy K. Smth, Myrtvet, Stensrud, Walkerden, Jeffery, Ktchenham, Fnne et al., Shepperd, Schofeld, Jorgensen, Srnvasan, Fscher, Bso, Malaboccha, Subramanan, Breslawsk Mukhopadhyay, Kerke Mukhopadhyay et al., Brand et al Vcnanza et al., Fgure 3. Relatve Cost Drver Impact[32] Other Methods 73

7 (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, the relatve mpact on cost/effort of the dfferent nput 100 parameters for that model. Even "objectve" nputs lke MMRE predcted actual / actual (6) Securty Requrements n SEER-SEM may be dffcult to confrm early n a program, and later changes may result n substantally dfferent cost and schedule estmates. Some senstve nputs such as the PRICE-S Productvty Factor and the SLIM PI should be calbrated from past data. If data are not avalable, or f consstent values of these parameters cannot be calbrated, the model s usefulness may be questonable. IX. ANALYSIS OF SUDY N where, N = total number of estmates RMS (Root Mean Square): Now, calculate the Root Mean Square (model s ablty to accurately forecast the ndvdual actual effort) for each data set. hs step s a precedent to the next step only. Agan, satsfactory results are ndcated by a value of 25 percent or less[30]. RMS 1 / n * predcted actual 2 (7) A. Accuracy Estmatos he feld of cost estmaton suffers a lack of clarty about the nterpretaton of a cost estmate. Jørgensen reports dfferent applcaton of the term cost estmate beng the most lkely cost, the planned cost, the budget, the prce, or, somethng else. Consequently there s also dsagreement about how to measure the accuracy of estmates. Varous ndcators for accuracy relatve and absolute have been ntroduced throughout the cost estmaton lterature such as mean squared error (MSE), absolute resduals (AR) or balanced resdual error (BRE). Our lterature revew ndcated that the most commonly used by far are the mean magntude relatve error or MMRE, and predcton wthn x or PRED(x). Of these two, the MMRE s the most wdely used, yet both are based on the same basc value of magntude relatve error (MRE) whch s defned as he frst step wll be to apply Conte s crtera to determne the accuracy of the calbrated and uncalbrated model. hs wll be acheved usng the followng equatons. Conte s Crtera: he performance of model generatng contnuous output can be assesses n many ways ncludng PRED(30), MMRE, correlaton etc., PRED(30) s a measure calculated from the relatve error, or RE, whch s the relatve sze of the dfference between the actual and estmated value. One way to vew these measures s to say that tranng data contans records wth varables 1,2,3, N and performance measures and addtonal new varables N+1, N+2,. MRE(Magntude of Relatve Error): Frst, calculate the Magntude of Relatve Error (degree of estmatng error n an ndvdual estmate) for each data pont. hs step s a precedent to the next step and s also used to calculate PRED(n). Satsfactory results are ndcated by a value of 25 percent or less [30]. MRE predcted actual / actual MMRE(mean magntude of the relatve error): he mean magntude of the relatve error, or MMRE, s the average percentage of the absolute values of the relatve errors over an entre data set. RRMS(Relatve Root Mean Square): Lastly, calculate the Relatve Root Mean Square (model s ablty to accurately forecast the average actual effort) for each data set. Accordng to Conte, the RRMS should have a value of 25 percent or less[30]. RRMS RMS /( actual/ ) PRED(n): A model should also be wthn 25 percent accuracy, 75 percent of the tme [30]. o fnd ths accuracy rate PRED(n), dvde the total number of ponts wthn a data set that have an MRE = 0.25 or less (represented by k) by the total number of data ponts wthn the data set (represented by n). he equaton then s: PRED(n) = k/n where n equals 0.25 [30]. In general, PRED(n) reports the average percentage of estmates that were wthn n percent of the actual values. Gven N datasets, then PRED(n)= 100 N N Σ =1 For example, PRED(30) = 50% means that half the estmates are wthn 30 percent of the actual. Wlcoxon Sgned-Rank est. 1 f MRE <=n/100 0 otherwse he next step wll be to test the estmates for bas. he Wlcoxon sgned-rank test s a smple, nonparametrc test that determnes level of bas. A nonparametrc test may be thought of as a dstrbuton-free test;.e. no assumptons about the dstrbuton are made. he best results that can be acheved by the model estmates s to show no dfference between the number of estmates that over estmated versus those that under estmated. he Wlcoxon sgned-rank test s accomplshed usng the followng steps [31], 1. Dvde each valdated subset nto two groups based on whether the estmated effort was greater (+) or less (-) than the actual effort. 2. Sum the absolute value of the dfferences for the + and -groups. he closer the sums of these values for each group are to each other, the lower the bas. (8) (9) 74

8 3. Any sgnfcant dfference ndcates a bas to over or under estmate. Another performance measure of a model predctng numerc values s the correlaton between predcted and actual values. Correlaton ranges from +1 to -1 and a correlaton of +1 means that there s a perfect postve lnear relatonshp between varables. And can be calculates as follows he correlaton coeffcent for COCOMO II s and the correlaton coeffcent for proposed model s (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, cost models could be shown to be accurate. he software support estmaton problem s further convoluted by lack of qualty software support cost data for model development, calbraton, and valdaton. Even f models can be shown to be accurate, another effect must be consdered. able III summarzes the parameters used and actvtes covered by the models dscussed. Overall, model based technques are good for budgetng, tradeoff analyss, plannng and control, and nvestment analyss. As they are calbrated to past experence, ther prmary dffculty s wth unprecedented stuatons. P= Σ Predcted, S p= Σ (Predcted p) 2, -1 a= Σ Actual, All these performance measures (correlaton, MMRE, and PRED) address subtly dfferent ssues. Overall, PRED measures how well an effort model performs, whle MMRE measures poor performance. A sngle large mstake can skew the MMREs and not effect the PREDs. Sheppard and Schofeld comment that MMRE s farly conservatve wth a bas aganst overestmate whle PRED(30) wll dentfy those predcton systems that are generally accurate but occasonally wldly naccurate[28] B. Model Accuracy here s no proof on software cost estmaton models to perform consstently accurate wthn 25% of the cost and 75% of the tme[30]. In general model fals to produce accurate result wth perfect nput data. he above studes have compared emprcal estmaton models wth known nput data and actual cost and schedule nformaton, and have not found the accuracy to be scntllatng. Most model were accurate wthn 30% of the actual cost and 57% of the tme. he Ourada study showed even worse results for SEER-SEM, SASE, and REVIC for the 28 mltary ground programs n an early edton of the Space and Mssles Center database. A 1981 study by Robert hbodeau enttled An Evaluaton of Software Cost Estmatng Models showed that calbraton could mprove model accuracy by up to 400%. However, the average accuracy was stll only 30% for an early verson of SLIM and 25% for an early verson of PRICE-S. PRICE-S and System-3 are wthn 30%, 62% of the tme. An Ar Force study performed by Ferens n 1983 and publshed n the ISPA Journal of Parametrcs, concluded that no software support S a= S Σ 1 (Predcted p) (Actual a), pa= -1 Corr= S pa / S p * S a Σ (Actual a) 2, -1 (10) ABLE II. ANALYSIS OF EMPIRICAL ESIMAION MODELS Study Model Applcaton ype Valdated Accuracy MMRE / RRMS Pred MRE COCOMO Flght Software Karen SEER Lum, 2002 COCOMO Ground Software SEER Knd:mn 31-60(0.3) Lang:ftn 44-42(0.3) Knd:max 38-52(0.3) All 40-60(0.3) Mode:org 32-62(0.3) Lang:mol 36-56(0.3) Project:Y 22-78(0.3) Karen Lum, 2006 COSEEKMO Msson Plannng 36-50(0.3) Avocsmontorng 38-53(0.3) Mode:sd 33-62(0.3) Project:X 42-42(0.3) Fg:g 32-65(0.3) Center: (0.3) All 48-43(0.3) Mode:e 64-42(0.3) Cemter: (0.3) REVIC (0.25) Gerald L SASE (0.25) Ourada, Aero Space SEER (0.25) 1992 Cost Model (0.25) SLIM COCOMO ABC Software FP Chrs F Estmac Kemour, SLIM COCOMO Busness:App FP Estmac Jeremah Machne D Deng, Learnng 2009 Random (30) De ran- Cao, 2007 Cosmc B (0.25) In able II, Summary of the analyss of the study, the result of a collaboratve effort of the authors, whch ncludes author name, cost model name, applcaton type, valdated accuracy (MMRE, RRMS, Pred) s the percentage of estmates that fall wthn the specfed predcton level of 25 or 30 percent. In ths Chrs F Kemour valdated SLIM and obtaned the result of hghest MRE of 772, COCOMO obtaned the result of hghest 75

9 (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, ABLE III. ACIVIIES COVERED/FACORS EXPLICILY CONSIDERED BY VARIOUS MODELS[33] Group Factor SLIM CheckPont Prce-s Estmacs SEER-SEM Select COCOM Estmator O II Source Instructon Yes Yes Yes No Yes No Yes Sze Attrbutes Functon Ponts Yes Yes Yes Yes Yes No Yes OO-Related Metrcs Yes Yes Yes! Yes Yes Yes ype/doman Yes Yes Yes Yes Yes Yes No Complexty Yes Yes Yes Yes Yes Yes Yes Program Attrbutes Language Yes Yes Yes! Yes Yes Yes Reuse Yes Yes Yes! Yes Yes Yes Requred Relablty!! Yes Yes Yes No Yes Computer Resource Constrants Yes Yes Yes Yes Yes No Yes Attrbutes Platform Volatlty!!!! Yes No Yes Personnel Capablty Yes Yes Yes Yes Yes Yes Yes Personnel Personnel Contnuty!!!!! No Yes Attrbutes Personnel Experence Yes Yes Yes Yes Yes No Yes ools and echnques Yes Yes Yes Yes Yes Yes Yes Breakage Yes Yes Yes! Yes Yes Yes Schedule Constrants Yes Yes Yes Yes Yes Yes Yes Project Attrbutes Process Maturty Yes Yes!! Yes No Yes eam Coheson! Yes Yes! Yes Yes Yes Securty Issues!!!! Yes No No Mult Ste Development! Yes Yes Yes Yes No Yes Incepton Yes Yes Yes Yes Yes Yes Yes Actvty Covered Elaboraton Yes Yes Yes Yes Yes Yes Yes Constructon Yes Yes Yes Yes Yes Yes Yes ranston and Mantenance Yes Yes Yes No Yes No Yes [3] B Boehm, C Abts, and S Chulan. "Software Development Cost Estmaton Approaches A Survey, echncal Report USC-CSE ", Unversty of Southern Calforna Center for Software Engneerng, USA, (2000). [4] S. chulan, B. Boehm, and B. Steece, Bayesan Analyss of Empercal Software Engneerng Cost Models, IEEE rans. Software Eng., vol.25, no. 4, pp , [5] Jorgen M, Sjoberg D.I.K, he Impact of Customer Expectaton on Software Development Effort Estmates Internatonal Journal of Project Management, Elsever, pp , 2004 MRE rate of Karen Lum has valdated SEER- SEM and found to be the valdaton of COSEEKMO n Mode:e has an enormous MRE of 64 evaluated by Karen Lum. Ourada study showed REVIC has MMRE of and SASE even worse result of X. CONCLUSION Based upon the background readngs, ths paper states that the exstng models were hghly credble; however, ths survey found ths not to be so based upon the research performed. All the models could not predct the actual aganst ether the calbraton data or valdaton data to any level of accuracy or consstency. Surprsngly, SEER and machne learnng technques were relable good at predctng the effort. But however they are not accurate because all the model les n the term predcton, predcton never comes true s proved n ths estmaton models. In all the models, the two key factors that nfluenced the estmate were project sze ether n terms of LOC or FP and the capabltes of the development team personnel. hs paper s not convnced that no model s so senstve to the abltes of the development team can be appled across the board to any software development effort. Fnally ths paper concludes that the no model s best for all stuatons and envronment. REFERENCES [1] Satyananda, An Improved Fuzzy Approach for COCOMO s Effort Estmaton Usng Gaussan Membershp Functon Journal of Software, vol 4, pp , 2009 [2] R. Jensen, An mproved macrolevel software development resource estmaton model. In 5th ISPA Conference, pp 88 92, 1983 [6] Chu NH, Huang SJ, he Adjusted Analogy-Based Software Effort Estmaton Based on Smlarty Dstances, Journal of Systems and Software, Volume 80, Issue 4, pp , 2007 [7] Kaczmarek J, Kucharsk M, Sze and Effort Estmaton for Applcatons Wrtten n Java, Journal of Informaton and Software echnology, Volume 46, Issue 9, pp , 2004 [8] Jeffery R, Ruhe M,Weczorek I, Usng Publc Doman Metrcs to Estmate Software Development Effort, In Proceedngs of the 7th Internatonal Symposum on Software Metrcs, IEEE Computer Socety, Washngton, DC, pp 16 27, 2001 [9] Heat A, Comparson of Artfcal Neural Network and Regresson Models for Estmatng Software Development Effort, Journal of Informaton and Software echnology, Volume 44, Issue 15, pp , 2002 [10] Huang SJ, Ln CY, Chu NH, Fuzzy Decson ree Approach for Embeddng Rsk Assessment Informaton nto Software Cost Estmaton Model, Journal of Informaton Scence and Engneerng, Volume 22, Number 2, pp , 2006 [11] B.W. Boehm, Software Engneerng Economcs, Prentce Hall, [12] B.W. Boehm, E. Horowtz, R. Madachy, D. Refer, B. K. Clark, B. Steece, A. W. Brown, S. Chulan, and C. Abts, Software Cost Estmaton wth COCOMO II, Prentce Hall, [13] Vu Nguyen, Bert Steece, Barry Boehm A Constraned Regresson echnque for COCOMO Calbraton ESEM 08, ACM, pp ,

10 (IJCSIS) Internatonal Journal of Computer Scence and Informaton Securty, [14] K.K.Aggarwal, Yogesh Sngh, A.Kaur, O.P.Sangwan "A Neural Net Based Approach to est Oracle" ACM Software Engneerng Notes Vol. 29, No. 4, pp 1-6, [15] R. W Jensen, A Comparson of the Jensen and COCOMO Schedule and Cost Estmaton Models, Proceedngs of the Internatonal Socety of Parametrc Analysts, pp , [16] Oruada, Gerald L, Software Cost Estmaton Models: A Callbraton, Evaluaton, and Compartson, Ar fore nsttute of echnology. [17] K. Srnvasan and D. Fsher, "Machne learnng approaches to estmatng software development effort," IEEE ransactons on Software Engneerng, vol. 21, pp , [18] A. R. Venkatachalam, "Software Cost Estmaton Usng Artfcal Neural Networks," Presented at 1993 Internatonal Jont Conference on Neural Networks, Nagoya, Japan, [19] G. H. Subramanan, P. C. Pendharkar, and M. Wallace, "An Emprcal Study of the Effect of Complexty, Platform, and Program ype on Software Development Effort of Busness Applcatons," Emprcal Software Engneerng, vol. 11, pp , [20] S. Kumar, B. A. Krshna, and P. S. Satsang, "Fuzzy systems and neural networks n software engneerng project management," Journal of Appled Intellgence, vol. 4, pp , [21] R. W. Selby and A. A. Porter, "Learnng from examples: generaton and evaluaton of decson trees for software resource analyss," IEEE ransactons on Software Engneerng, vol. 14, pp , [22] Denver, CO, Martn-Maretta, Ratlff, Robert W., SASE 3.0 user Gude, 1993 [23] K. Maxwell, L. Van Wassenhove, and S. Dutta, "Performance Evaluaton of General and Company Specfc Models n Software Development Effort Estmaton," Management Scence, vol. 45, pp , 1999 [24] M. van Genuchten and H. Koolen, "On the Use of Software Cost Models," Informaton & Management, vol. 21, pp , [25]. K. Abdel-Hamd, "Adaptng, Correctng, and Perfectng softwareestmates: Amantenance metaphor " n Computer, vol. 26, pp , 1993 [26] F. J. Heemstra, "Software cost estmaton," Informaton and Software echnology, vol. 34, pp , 1992 [27] N. Fenton, "Software Measurement: A necessary Scentfc Bass," IEEE ransactons on Software Engneerng, vol. 20, pp , [28] M.Sheppered and C. Schofed, Estmatng Software Project Effort Usng Analoges, IEEE rans. Software Eng. Vol. 23, pp , 1997 [29] Barry Boehm. Software engneerng economcs. Englewood Clffs, NJ:Prentce-Hall, ISBN [30] S.D. Conte, H.E. Dunsmore, V.Y.Shen, Software Engneerng Metrcs and Models, Benjamn-Cummngs Publshng Co., Inc., 1986 [31] Mendenhall, W., Wackerly, D.D., Schaeffer, R.L, Mathematcal statstcs wth applcatons, 4th edton, PWS-KEN Publshng Company, Boston (1990),752 pp,isbn [32] [33] Barry Boehma, Chrs Abts a and Sunta Chulan, Software development cost estmaton approaches -A survey Annals of Software Engneerng, pp , 2000 AUHORS PROFILE Saleem Basha s a Ph.D research scholar n the Department of Computer Scence, Pondcherry Unversty. He has obtaned B.E n the feld of Electrcal and Electroncs Engneerng, Bangalore Unversty, Bangalore, Inda and M.E n the feld of Computer Scence and Engneerng, Anna Unversty, Chenna, Inda. He s currently workng n the area of SDLC specfc effort estmaton models and web servce modellng systems. Dr. Dhavachelvan Ponnurangam s workng as Assocate Professor, Department of Computer Scence, Pondcherry Unversty, Inda. He has obtaned hs M.E. and Ph.D. n the feld of Computer Scence and Engneerng n Anna Unversty, Chenna, Inda. He s havng more than a decade of experence as an academcan and hs research areas nclude Software Engneerng and Standards, web servce computng and technologes. He has publshed around 75 research papers n Natonal and Internatonal Journals and Conferences. He s collaboratng and coordnatng wth the research groups workng towards to develop the standards for Attrbutes Specfc SDLC Models & Web Servces computng and technologes. 77