IMPROVED SIZE AND EFFORT ESTIMATION MODELS FOR SOFTWARE MAINTENANCE. Vu Nguyen

Transcription

1 IMPROVED SIZE AND EFFORT ESTIMATION MODELS FOR SOFTWARE MAINTENANCE by Vu Nguyen A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) December 2010 Copyright 2010 Vu Nguyen

2 DEDICATION Kính tặng ba mẹ, Nguyễn Đình Hoàng, Nguyễn Thị Thiên You survived and struggled through one of the darkest chapters in the history of Vietnam to raise and love your children. ii

3 ACKNOWLEDGEMENTS I am deeply indebted to my advisor, Dr. Barry Boehm, for his encouragement, support, and constructive advice. I have learned not only from his deep knowledge and experience in software engineering but also from his remarkable personality. I would also like to thank A Winsor Brown who brought me to work on the CodeCount project and make the tool more useful for the software development community. I am grateful to Dr. Bert Steece, whom I regard as my unofficial advisor, for his statistical insights that shape my understanding of statistical analysis applied to this dissertation work. My thanks also go to other members of my qualifying exam and defense committees, Dr. Nenad Medvidović, Dr. Ellis Horowitz, and Dr. Rick Selby. Their criticism, encouragement, and suggestion effectively helped to shape my work. I am thankful to my friends and faculty for their constructive feedback on my work, including Dan Port, Tim Menzies, Jo Ann Lane, Brad Clark, Don Reifer, Supannika Koolmanojwong, Pongtip Aroonvatanaporn, and Qi Li. Other colleagues Julie Sanchez of the USC Center for Systems and Software Engineering, Marilyn A Sperka, Ryan E Pfeiffer, and Michael Lee of the Aerospace Corporation, and Lori Vaughan of Northrop Grumman also assisted me in various capacities. This work was made possible with the support for data collection from the Affiliates of Center for Systems and Software Engineering and two organizations in Vietnam and Thailand. Especially, Phuong Ngo, Ngoc Do, Phong Nguyen, Long Truong, iii

4 Ha Ta, Hoai Tang, Tuan Vo, and Phongphan Danphitsanuphan have provided me tremendous help in collecting historical data from the organizations in Vietnam and Thailand. I awe much to the Fulbright Program for financial support during my Master s program at USC, giving me an opportunity to fulfill my dream of studying abroad and doing research with remarkable researchers in the software engineering research community. My cultural and educational experiences in the United States made possible by the program are priceless. iv

5 TABLE OF CONTENTS Dedication... ii Acknowledgements...iii List of Tables... vii List of Figures... ix Abbreviations... x Abstract... xii Chapter 1. Introduction The Problem A Solution Research Hypotheses Definitions... 6 Chapter 2. Related Work Software Sizing Code-based Sizing Metrics Functional Size Measurement (FSM) Major Cost Estimation Models SLIM SEER-SEM PRICE-S KnowledgePlan (Checkpoint) COCOMO Maintenance Cost Estimation Models Phase-Level Models Release-Level Models Task-Level Models Summary of Maintenance Estimation Models Chapter 3. The Research Approach The Modeling Methodology The Calibration Techniques Ordinary Least Squares Regression The Bayesian Analysis A Constrained Multiple Regression Technique v

6 3.3 Evaluation Strategies Model Accuracy Measures Cross-Validation Chapter 4. the COCOMO II Model for Software Maintenance Software Maintenance Sizing Methods The COCOMO II Reuse and Maintenance Models A Unified Reuse and Maintenance Model COCOMO II Effort Model for Software Maintenance Chapter 5. Research Results The Controlled Experiment Results Description of the Experiment Experiment Results Limitations of the Experiment Delphi Survey Results Industry Sample Data Model Calibrations and Validation The Bayesian Calibrated Model The Constrained Regression Calibrated Models Reduced Parameter Models Local Calibration Summary Chapter 6. Contributions and Future Work Contributions Future Work Bibliography Appendix A. UFNM and AA Rating Scale Appendix B. Delphi Survey Form Appendix C. Data Collection Forms Appendix D. The COCOMO II.2000 Parameters Used in the Experiment Appendix E. Histograms for the Cost Drivers Appendix F. Correlation Matrix for Effort, Size, and Cost Drivers vi

7 LIST OF TABLES Table 2-1. COCOMO Sizing Models Table 2-2. COCOMO II Calibrations Table 2-3. Maintenance Cost Estimation Models Table 4-1. Maintenance Model s Initial Cost Drivers Table 4-2. Ratings of Personnel Experience Factors (APEX, PLEX, LTEX) Table 4-3. Ratings of RELY Table 5-1. Summary of results obtained from fitting the models Table 5-2. Differences in Productivity Ranges Table 5-3. Rating Values for Cost Drivers from Delphi Survey Table 5-4. RELY Rating Values Estimated by Experts Table 5-5. Maintenance Core Data Attributes Table 5-6. Summary Statistics of 86 Data Points Table 5-7. Differences in Productivity Ranges between Bayesian Calibrated Model and COCOMO II Table 5-8. Rating Values for Cost Drivers from Bayesian Approach Table 5-9. Estimation Accuracies Generated by the Bayesian Approach Table Estimation Accuracies of COCOMO II.2000 on the Data Set Table Retained Cost Drivers of the Constrained Models Table Estimation Accuracies of Constrained Approaches Table Estimation Accuracies of Constrained Approaches using LOOC Crossvalidation vii

8 Table Correlation Matrix for Highly Correlated Cost Drivers Table Estimation Accuracies of Reduced Calibrated Models Table Stratification by Organization Table Stratification by Program Table Stratification by Organization on 45 Releases viii

9 LIST OF FIGURES Figure 3-1. The Modeling Process Figure 3-2. A Posteriori Bayesian Update in the Presence of Noisy Data RUSE Figure 3-3. Boxplot of mean of PRED(0.3) on the COCOMO II.2000 data set Figure 3-4. Boxplot of mean of PRED(0.3) on the COCOMO 81 data set Figure 4-1. Types of Code Figure 4-2. Nonlinear Reuse Effects Figure 4-3. AAM Curves Reflecting Nonlinear Effects Figure 5-1. Effort Distribution Figure 5-2. Maintenance Project Collection Range Figure 5-3. Distribution of Equivalent SLOC Figure 5-4. Correlation between PM and EKSLOC Figure 5-5. Correlation between log(pm) and log(eksloc) Figure 5-6. Adjusted Productivity for the 86 Releases Figure 5-7. Adjusted Productivity Histogram for the 86 Releases Figure 5-8. Productivity Ranges Calibrated by the Bayesian Approach Figure 5-9. Productivity Ranges Generated by CMRE Figure Distribution of TIME Figure Distribution of STOR ix

10 ABBREVIATIONS COCOMO COCOMO II CMMI EM PM OLS MSE MAE CMSE CMAE CMRE MMRE MRE PRED ICM PR SF Constructive Cost Model Constructive Cost Model version II Capability Maturity Model Integration Effort Multiplier Person Month Ordinary Least Squares Mean Square Error Mean Absolute Error Constrained Minimum Sum of Square Errors Constrained Minimum Sum of Absolute Errors Constrained Minimum Sum of Relative Errors Mean of Magnitude of Relative Errors Magnitude of Relative Errors Prediction level Incremental Commitment Model Productivity Range Scale Factor MODEL PARAMETERS Size Parameters AA Assessment and Assimilation AAF Adaptation Adjustment Factor AAM Adaptation Adjustment Multiplier AKSLOC Kilo Source Lines of Code of the Adapted Modules CM Code Modified DM Design Modified EKSLOC Equivalent Kilo Source Lines of Code x

11 ESLOC IM KSLOC RKSLOC SLOC SU UNFM Cost Drivers ACAP APEX CPLX DATA DOCU FLEX LTEX PCAP PCON PERS PLEX PMAT PREC PREX PVOL RELY RESL SITE STOR TEAM TIME TOOL Equivalent Source Lines of Code Integration Modified Kilo Source Lines of Code Kilo Source Lines of Code of the Reused Modules Source Lines of Code Software Understanding Programmer Unfamiliarity Analyst Capability Applications Experience Product Complexity Database Size Documentation Match to Life-Cycle Needs Development Flexibility Language and Tool Experience Programmer Capability Personnel Continuity Personnel Capability Platform Experience Equivalent Process Maturity Level Precedentedness of Application Personnel Experience Platform Volatility Required Software Reliability Risk Resolution Multisite Development Main Storage Constraint Team Cohesion Execution Time Constraint Use of Software Tools xi

12 ABSTRACT Accurately estimating the cost of software projects is one of the most desired capabilities in software development organizations. Accurate cost estimates not only help the customer make successful investments but also assist the software project manager in coming up with appropriate plans for the project and making reasonable decisions during the project execution. Although there have been reports that software maintenance accounts for the majority of the software total cost, the software estimation research has focused considerably on new development and much less on maintenance. In this dissertation, an extension to the well-known model for software estimation, COCOMO II, is introduced for better determining the size of maintained software and improving the effort estimation accuracy of software maintenance. While COCOMO II emphasizes the cost estimation of software development, the extension captures various characteristics of software maintenance through a number of enhancements to the COCOMO II size and effort estimation models to support the cost estimation of software maintenance. Expert input and an industry data set of eighty completed software maintenance projects from three software organizations were used to build the model. A number of models were derived through various calibration approaches, and these models were then evaluated using the industry data set. The full model, which was derived through the Bayesian analysis, yields effort estimates within 30% of the actuals 51% of the time, outperforming the original COCOMO II model when it was used to estimate these xii

13 projects by 34%. Further performance improvement was obtained when calibrating the full model to each individual program, generating effort estimates within 30% of the actuals 80% of the time. xiii

14 Chapter 1. INTRODUCTION Software maintenance is an important activity in software engineering. Over the decades, software maintenance costs have been continually reported to account for a large majority of software costs [Zelkowitz 1979, Boehm 1981, McKee 1984, Boehm 1988, Erlikh 2000]. This fact is not surprising. On the one hand, software environments and requirements are constantly changing, which lead to new software system upgrades to keep pace with the changes. On the other hand, the economic benefits of software reuse have encouraged the software industry to reuse and enhance the existing systems rather than to build new ones [Boehm 1981, 1999]. Thus, it is crucial for project managers to estimate and manage the software maintenance costs effectively. Software cost estimation plays an important role in software engineering practice, often determining the success or failure of contract negotiation and project execution. Cost estimation s deliverables, such as effort, schedule, and staff requirements are valuable pieces of information for project formation and execution. They are used as key inputs for project bidding and proposal, budget and staff allocation, project planning, progress monitoring and control, etc. Unreasonable and unreliable estimates are a major cause of project failure, which is evidenced by a CompTIA survey of 1,000 IT respondents in 2007, finding that two of the three most-cited causes of IT-project failure are concerned with unrealistic resource estimation [Rosencrance 2007]. Recognizing the importance of software estimation, the software engineering community has put tremendous effort into developing models in order to help estimators 1

15 generate accurate cost estimates for software projects. In the last three decades, many software estimation models and methods have been proposed and used, such as COCOMO, SLIM, SEER-SEM, and Price-S. 1.1 THE PROBLEM COCOMO is the most popular non-proprietary software estimation model in literature as well as in industry. The model was built using historical data and assumptions of software (ab initio) development projects. With a few exceptions, the model s properties (e.g., forms, cost factors and constants) are supposed to be applicable to estimating the cost of software maintenance. However, inherent differences exist between software development and maintenance. For example, software maintenance depends on quality and complexity of the existing architecture, design, source code, and supporting documentation. The problem is that these differences make the model s properties less relevant in the software maintenance context, resulting in low estimation accuracies achieved. Unfortunately, there is a lack of empirical studies that evaluate and extend COCOMO or other models, in order to better estimate the cost of software maintenance. 1.2 A SOLUTION Instead of using the COCOMO model that was designed for new development, what if we build an extension that takes into account characteristics of software maintenance. Thus, my thesis is 2

16 Improved COCOMO models that allow estimators to better determine the equivalent size of maintained software and estimate maintenance effort can improve the accuracy of effort estimation of software maintenance. The goal of this study is to investigate such models. I will test a number of hypotheses testing the accuracy of alternative models that predict the effort of software maintenance projects, the explanatory power of such cost drivers as execution time and memory constraints, and potential effects on quality attributes as better determinants of the effort involved in deleting code. Using the industry dataset that I have collected, I will also evaluate the differences in the effects of the COCOMO cost drivers on the project's effort between new development and maintenance projects. Finally, COCOMO models for software maintenance will be introduced and validated using industry data sets. 1.3 RESEARCH HYPOTHESES The main research question that this study attempts to address, reflecting the proposed solution, is as follows Are there extended COCOMO models that allow estimators to better determine the equivalent size of maintained software and estimate maintenance effort that can improve the accuracy of effort estimation of software maintenance? To address this question, I implement the modeling process as described in Section 3.2. The process involves testing a number of hypotheses. The inclusion of hypothesis tests in the process helps frame the discussion and analysis of the results 3

17 obtained during the modeling process. This section summarizes the hypotheses to be tested in this dissertation. It is clear from prior work discussed in Chapter 2 that there is no common maintenance size metric used as compared to the size for new development. In software (ab initio) development, a certain level of consistency is achieved; either SLOC or Function Point is widely used. In software maintenance, on the other hand, some maintenance effort estimation models use the sum of SLOC added, modified, and deleted, others do not include the SLOC deleted. Due to this inconsistency, evidence on whether the SLOC deleted is a significant predictor of effort is desired. Thus, the following hypothesis is investigated. Hypothesis 1: The measure of the SLOC deleted from the modified modules is not a significant size metric for estimating the effort of software maintenance projects. This hypothesis was tested through a controlled experiment of student programmers performing maintenance tasks. The SLOC deleted metric was used as a predictor of effort in linear regression models, and the significance of the coefficient estimate of this predictor was tested using the typical 0.05 level of significance. Software maintenance is different from new software development in many ways. The software maintenance team works on the system of which the legacy code and documentation, thus it is constrained by the existing system s requirements, architecture, design, etc. This characteristic leads to differences in project activities to be performed, system complexity, personnel necessary skill set, etc. As a result, we expect that the impact of each cost driver on the effort of software maintenance would be significantly 4

18 different in comparison with the COCOMO II.2000 model. We, therefore, investigate the following hypothesis. Hypothesis 2: The productivity ranges of the cost drivers in the COCOMO II model for maintenance are different from those of the cost drivers in the COCOMO II.2000 model. The productivity range specifies the maximum impact of a cost driver on effort. In other words, it indicates the percentage of effort increase or decrease if the rating of a cost driver increases the lowest to the highest level. This hypothesis was tested by performing simple comparisons on the productivity ranges of the cost drivers between the two models. Finally, the estimation model must be validated and compared with other estimation approaches regarding its estimation performance. As the model for maintenance proposed in this study is an extension of the COCOMO II model, we will analyze its performance with that of the COCOMO II model and other two unsophisticated but commonly used approaches including the simple linear regression and the productivity index. Hypothesis 3: The COCOMO II model for maintenance outperforms the COCOMO II.2000 model when estimating the effort of software maintenance projects. Hypothesis 4: The COCOMO II model for maintenance outperforms the simple linear regression model and the productivity index estimation method. 5

19 Hypotheses 3 and 4 were tested by comparing the estimation performance of various models using the estimation accuracy metrics MMRE and PRED. 1.4 DEFINITIONS This section provides definitions of common terms and phrases referred to throughout this dissertation. Software maintenance or maintenance is used to refer to the work of modifying, enhancing, and providing cost-effective support to the existing software. Software maintenance in this definition has a broader meaning than the IEEE definition given in [IEEE 1999] as it includes minor and major functional enhancements and error corrections. As opposed to software maintenance, new (ab initio) development refers to the work of developing and delivering the new software product. COCOMO II is used to refer to a set of estimation models that were developed and released in [Boehm 2000b] as a major extension of COCOMO to distinguish itself from the version originally created and published in [Boehm 1981]. COCOMO II.2000 refers to the COCOMO II model whose constants and cost driver ratings were released in [Boehm 2000b]. COCOMO II model for maintenance is used to indicate the sizing method and effort estimation model for software maintenance. 6

20 Chapter 2. RELATED WORK Software cost estimation has attracted tremendous attention from the software engineering research community. A number of studies have been published to address cost estimation related problems, such as software sizing, software productivity factors, cost estimation models for software development and maintenance. This chapter presents a review of software sizing techniques in Section 2.1, major cost estimation models in Section 2.2, and maintenance cost estimation models in Section SOFTWARE SIZING Size is one of the most important attributes of a software product. It is a key indicator of software cost and time; it is also a base unit to derive other metrics for software project measurements, such as productivity and defect density. This section describes the most popular sizing metrics and techniques that have been proposed and applied in practice. These techniques can be categorized into code-based sizing metrics and functional size measurements CODE-BASED SIZING METRICS Code-based sizing metrics measure the size or complexity of software using the programmed source code. Because a significant amount of effort is devoted to programming, it is believed that an appropriate measure correctly quantifying the code 7

21 can be a perceivable indicator of software cost. Halstead s software length equation based on program s operands and operators, McCabe s Cyclometic Complexity, number of modules, source lines of code (SLOC) among others have been proposed and used as code-based sizing metrics. Of these, SLOC is the most popular. It is used as a primary input by most major cost estimation models, such as SLIM, SEER-SEM, PRICE-S, COCOMO, and KnowledgePlan (see Section 2.2). Many different definitions of SLOC exist. SLOC can be the number of physical lines, the number of physical lines excluding comments and blank lines, or the number of statements commonly called logical SLOC, etc. To help provide a consistent SLOC measurement, the SEI published a counting framework that consists of SLOC definitions, counting rules, and checklists [Park 1992]. Boehm et al. adapted this framework for use in the COCOMO models [Boehm 2000b]. USC s Center for Systems and Software Engineering (USC CSSE) has published further detailed counting rules for most of the major languages along with the CodeCount tool 1. In COCOMO, the number of statements or logical SLOC, is the standard SLOC input. Logical SLOC is less sensitive to formats and programming styles, but it is dependent on the programming languages used in the source code [Nguyen 2007]. For software maintenance, the count of added/new, modified, unmodified, adapted, reused, and deleted SLOC can be used. Software estimation models usually aggregate these measures in a certain way to derive a single metric commonly called effective SLOC or equivalent SLOC [Boehm 2000b, SEER-SEM, Jorgensen 1995, Basili 1 8

22 1996, De Lucia 2005]. Surprisingly, there is a lack of consensus on how to measure SLOC for maintenance work. For example, COCOMO, SLIM, and SEER-SEM exclude the deleted SLOC metric while KnowledgePlan and PRICE-S include this metric in their size measures. Several maintenance effort estimation models proposed in the literature use the size metric as the sum of SLOC added, modified, and deleted [Jorgensen 1995, Basili 1996]. SLOC has been widely accepted for several reasons. It has been shown to be highly correlated with the software cost; thus, they are relevant inputs for software estimation models [Boehm 1981, 2000b]. In addition, code-based metrics can be easily and precisely counted using software tools, eliminating inconsistencies in SLOC counts, given that the same counting rules are applied. However, the source code is not available in early project stages, which means that it is difficult to accurately measure SLOC until the source code is available. Another limitation is that SLOC is dependent on the programmer s skills and programming styles. For example, an experienced programmer may write fewer lines of code than an inexperienced one for the same purpose, resulting in a problem called productivity paradox [Jones 2008]. A third limitation is a lack of a consistent standard for measurements of SLOC. As aforementioned, SLOC could mean different things, physical lines of code, physical lines of code excluding comments and blanks, or logical SLOC. There is also no consistent definition of logical SLOC [Nguyen 2007]. The lack of consistency in measuring SLOC can cause low estimation accuracy as described in [Jeffery 2000]. A forth limitation is that SLOC is technology and language 9

23 dependent. Thus, it is difficult to compare productivity gains of projects of varying technologies FUNCTIONAL SIZE MEASUREMENT (FSM) Having faced these limitations, Albrecht developed and published the Function Point Analysis (FPA) method as an alternative to code-based sizing methods [Albrecht 1979]. Albrecht and Gaffney later extended and published the method in [Albrecht and Gaffney 1983]. The International Function Point Users Group (IFPUG), a non-profit organization, was later established to maintain and promote the practice. IFPUG has extended and published several versions of the FPA Counting Practices Manual to standardize the application of FPA [IFPUG 2004, 1999]. Other significant extensions to the FPA method have been introduced and widely applied in practice, such as Mark II FPA [Symons 1988] and COSMIC-FFP [Abran 1998] FUNCTION POINTS ANALYSIS (FPA) FPA takes into account both static and dynamic aspects of the system. The static aspect is represented by data stored or accessed by the system, and the dynamic aspect reflects transactions performed to access and manipulate the data. FPA defines two data function types (Logical Internal File, External Interface File) and three transaction function types (External Input, External Output, Query). Function point counts are the sum of the scores that are assigned to each of the data and the transactional functions. The score of each of the data and transaction functions is determined by its type and its 10

24 complexity. Function counts are then adjusted by a system complexity factor called Value Adjustment Factor (VAF) to obtain adjusted function point counts. The IFPUG s Function Point Counting Practices Manual (CPM) provides guidelines, rules, and examples for counting function points. The manual specifies three different types of function point counts for the Development project, the Enhancement project, and the Application count. The Development project type refers to the count of functions delivered to the user in the first installation of the new software. The Enhancement project type refers to function point counts for modifications made to the preexisting software. And the Application function point count measures the functions provided to the user by a software product and is referred to as the baseline function point count. It can be considered the actual function point count of the development project developing and delivering the system. Thus, the application function point count can be determined by applying the same process given for the development project if the user requirements can be obtained from the working software. In FPA, the enhancement project involves the changes that result in functions added, modified or deleted from the existing system. The procedure and complexity scales are the same as those of the development, except that it takes into account changes in complexity of the modified functions and the overall system characteristics. The process involves identifying added, modified, deleted functions and determining value adjustment factors of the system before and after changes. 11

25 The Effective Function Point count (EFP) of the enhancement project is computed using the formula EFP = (ADD + CHGA) * VAFA + DEL * VAFB (Eq. 2-1) Where, ADD is the unadjusted function point count of added functions. CHGA is the unadjusted function point count of functions that are obtained by modifying the preexisting ones. It is important to note that CHGA counts any function that is modified, regardless of how much the function is changed. DEL is the unadjusted function point count of deleted functions. VAFA is the Value Adjustment Factor of the application after the enhancement project is completed. VAFB is the Value Adjustment Factor of the preexisting application MARK II FUNCTION POINT ANALYSIS In 1988, Symons [1988] proposed Mark II Function Point Analysis (MkII FPA) as an extension to Albrecht s FPA. MkII FPA was later extended and published by the United Kingdom Software Metrics Association (UKSMA) in the MkII FPA Counting Practices Manual [UKSMA 1998]. The MkII FPA method is a certified ISO as an international standard FSM method [ISO 2002]. 12

26 MkII FPA measures the functionality of a software system by viewing the software system as consisting of logical transactions. Each logical transaction is a finestgrained unit of a self-consistent process that is recognizable by the user. The logical transaction consists of three constituent parts: input, processing, and output components. The input and output components contain data element types that are sent across the application boundary and processed by the processing component. The processing component handles the input and output by referencing data entity types. Conceptually, MkII FPA s definitions of data entity types and data element types are similar to those of ILFs/EIFs and DETs in Albrecht s FPA. The size of the input and output components is the count of data element types, and the size of the processing component is the count of data entity types. The MkII function point count, which is referred to as MFI, is determined by computing a weighted sum of the sizes of the input, processing, and output components of all logical transactions. That is, MFI = W i N i + W e N e + W o N o (Eq. 2-2) Where, W i, W e, W o are the weights of input data element types, data entity types, and output data element types, respectively; and N i, N e, N o are the counts of input data element types, data entity types, and output data element types, respectively. The weights W i, W e, W o are calibrated using historical data by relating MFI to the effort required to develop the respective functions. In the CPM version [UKSMA 1998], the industry average weights are as W i = 0.58, W e = 1.66, and W o =

27 For sizing the software maintenance work (referred to as changes in CPM 1.3.1), the MkII function point count includes the size of the logical transactions that are added, changed, or deleted. This involves counting all individual input/output data element types and data entity types that are added, modified, or deleted. The formula (Eq. 2-2) can be used to calculate the MkII function point count of maintenance work, where N i, N e, N o are treated as the counts of input data element types, data entity types, and output data element types that are added, modified, or deleted COSMIC FFP Full Function Point, which is often referred to as FFP 1.0, is a FSM method proposed by St-Pierre et al. [St-Pierre 1997]. FFP was designed as an extension to IFPUG FPA to better measure the functional size of real-time software. The method was later extended and renamed to COSMIC-FFP by the Common Software Measurement International Consortium (COSMIC). Several extensions have been published, and the latest version is COSMIC-FFP 2.2 [COSMIC 2003]. COSMIC-FFP has been certificated as an ISO international standard [ISO 2003]. COSMIC-FFP views the functionality of a software application as consisting of functional processes, each having sub-processes. There are two types of sub-processes, data movement type and data manipulation type. COSMIC-FFP does not handle the data manipulation type separately, but it assumes some association between the two types. It defines four types of data movement sub-processes, including Entry, Exit, Read, and Write. An Entry is a movement of the data attributes contained in one data group from 14

28 the outside to the inside of the application boundary; an Exit moves a data group in the opposite direction to that of the entry; and a Read or a Write refers to a movement of a data group from or to storage. The COSMIC-FFP measurement method determines the functional size by measuring the data movement sub-processes, each moving exactly one data group. That is, the COSMIC-FFP function point count, which is measured in cfsu (COSMIC Functional Size Unit), is computed by summing all Entries, Exits, Reads, and Writes. The complexity of the measurement procedure involves identifying application layers, application boundaries, and data groups that each sub-process handles. Detailed rules and guidelines for this process are given in the COSMIC-FFP measurement manual [COSMIC 2003]. In COSMIC-FFP, the functional size of the software maintenance work is the simple sum of all Entry, Exit, Read, and Write data movement sub-processes whose functional processes are affected by the change. By this calculation, COSMIC-FFP assumes that three different types of change to the functionality (add, modify, and delete) have the same level of complexity (e.g., the size of adding a function is counted the same as that of modifying or deleting the function). That is, the COSMIC-FFP count for the maintenance work (Sizecfsu) is computed as Size cfsu = Size added + Size modified + Size deleted (Eq. 2-3) Where, Size added, Size modified, and Size deleted are the COSMIC-FFP counts measured in csfu of the added, modified, and deleted data movements, respectively. 15

29 The above-discussed functional size measurement methods measure the size of software maintenance consistently in the sense that added, modified, and deleted functions are all included in the count, and they are assigned the same weight. Thus, the number of function point counts assigned for a function is a constant regardless of whether the function is added, modified, or deleted. This calculation also implies that the effort required to add a function is expected to be the same as the effort to delete or modify the same function. 2.2 MAJOR COST ESTIMATION MODELS Many estimation models have been proposed and applied over the years. Instead of describing them all, this section provides a brief review of major estimation models that have been developed, continued to be applied, and marketed by respective developers. These models include SLIM, SEER-SEM, PRICE-S, KnowledgePlan, and COCOMO. There are several reasons for this selection. First, they represent the core set of models that was developed in the early 1980 s and 1990 s. Second, they are still being investigated and used widely in practice and literature. Their long history of extensions and adoptions is proof of their robustness and usefulness. Third, these models perform estimation for a broad range of software development and maintenance activities, covering a number of phases of software lifecycles such as requirements, architecture, implementation, testing, and maintenance. 16

30 2.2.1 SLIM SLIM is one of the most popular cost estimation models that has been in the market for decades. The model was originally developed in the late 1970s by Larry Putnam of Quantitative Software Measurement 2, and its mathematical formulas and analysis were published in [Putnam and Myers 1992]. As the model is proprietary, the subsequent upgrades of the model structures and mathematical formulas are not available in the public domain. Generally, the SLIM model assumes that the staffing profile follows a form of Rayleigh probability distribution of project staff buildup over time. The shapes and sizes of the Rayleigh curve reflect the project size, manpower buildup index, and other productivity parameters. The Rayleigh staffing level at time t is presented as 2 t K 2 2td ( t) = te 2 td p (Eq. 2-4) Where K is the total lifecycle effort and t d the schedule to the peak of the staffing K curve. The quality D = 2 t d is considered staffing complexity of the project. The total lifecycle effort is calculated using the project size S, the technology factor C, and t d, and is defined as 3 S 1 K = x 3 (Eq. 2-5) C 4 t d

31 Boehm et al. note that some assumptions of staffing profile following the Rayleigh distribution do not always hold in practice [Boehm 2000a]. For example, some development practices such as maintenance and incremental development may employ a constant level of staff. In subsequent adjustments, SLIM handles this limitation by allowing the staffing profile to be adjusted by a staffing curve parameter. There are multiple project lifecycle phases defined by the model, such as feasibility study, functional design, main build (development), and maintenance. Each lifecycle phase may have a different staffing curve parameter. In the main build phase, for example, the staffing curve parameter can specify the curve as Medium Front Load (for staff peaking at 40% of the phase), Medium Rear Load (peaking at 80% of the phase), or Rear Load (peaking at the end of the phase). In the maintenance phase, this parameter specifies a flat staffing curve. SLIM views software maintenance as a software lifecycle phase following the main build or development phase. The maintenance phase may have major enhancements, minor enhancements, and baseline support including emergency fixes, help desk support, infrastructure upgrades, operational support, small research projects, etc. The maintenance phase can be estimated independently with the other phases. The model uses the effective SLOC as a unit of project size. Function points and user-defined metrics such as the number of modules, screens, etc. can be used, but they have to be converted to effective SLOC using a gear factor. SLIM counts new code and modified code, but it excludes deleted code. Clearly, the model assumes that new code and modified code have the same influence on the maintenance effort. 18

32 2.2.2 SEER-SEM SEER-SEM is a commercial and proprietary model developed and marketed by Galorath, Inc 3. This model is an extension of the Jensen model [Jensen 1983] from which model structures and parameters are extended while sharing the same core formulas. Like SLIM, the model uses the Rayleigh probability distribution of staffing profile versus time to determine development effort. The size of the project and other parameters can change the Rayleigh curve which then gives estimates for effort, time, and peak staff accordingly. In SEER-SEM, the traditional Rayleigh staffing level at time t is calculated as 2 t K 2 2td ( t) = te 2 td p (Eq. 2-6) Where K is the total life cycle effort and t d the time to the peak of the staffing curve; these terms are calculated as K = D 0.4 S x 3 e C te 1.2 (Eq. 2-7) = S e t d D C (Eq. 2-8) te Where D is the staffing complexity, S e the effective size, and C te the effective technology. Although sharing the same meaning of D with SLIM, SEER-SEM defines D

33 K as 3, which is slightly different from that of SLIM. The derivative of the Rayleigh curve t d p(t) at t = 0 is defined as staffing rate, generally measuring the number of people added to the project per year. The effective technology C te is an aggregated factor determined by using a set of technology and environment parameters. The effective size S e is the size of the project that can be measured in source lines of code (SLOC), function points, or other units. In SEER-SEM, the software maintenance cost covers necessary maintenance activities performed to ensure the operation of the software after its delivery. These software maintenance activities involve correcting faults, changing the software to new operating environments, fine-tuning and perfecting the software, and minor enhancements. They are usually triggered by change requests and software faults that are found during the operation of the software. SEER-SEM uses a number of maintenancespecific parameters to estimate the maintenance cost for a given period of time, in addition to the development cost of the software. The maintenance cost is allocated into corrective, adaptive, perfective maintenance, and minor enhancement cost categories. Major enhancements and re-engineering are not included in the software maintenance cost, but are treated separately as a new software development project. SEER-SEM handles major enhancements and new development similarly except that their differences are reflected in the way that the effective size of the software is determined. 20

34 The effective size, S e, of the software is calculated as S e = New + [P x (0.4 A B C)] (Eq. 2-9) Where, New and P are the new size and the pre-existing software size, respectively. The size can be measured in either function points or lines of code. Discarded code is not included in calculating the effective size of the pre-existing software (P). A, B, and C are the respective percentages of code redesign, code reimplementation, and code retest required to adapt the pre-existing software. The element [P x (0.4 A B C)] in formula (Eq. 2-9) is the equivalent size of rework or reuse of the pre-existing software. The parameters A, B, and C are subjective, and their values range from 0 to 100%. The SEER-SEM user manual provides formulas and guidelines to help determine these parameters [Galorath 2002]. It does not give details, however, on how to reevaluate these parameters when the completed software is available (it is important to collect the actual size data because the actual size data is used to recalibrate the model and to measure the actual productivity, defect density, etc). The parameters A, B, and C maximum limit of 100% might potentially result in underestimation of the rework because it does not account for possible code expansion, full retest and integration of the pre-existing code. 21

35 2.2.3 PRICE-S PRICE-S was originally developed by Frank Freiman of RCA for estimating acquisition and development of hardware systems in the late 1960 s and then released as a first commercial software cost estimation model in After multiple upgrades, extensions, and changes of ownership, the current model, True S, is now implemented as a component in the TruePlanning tool marketed by PRICE Systems 4. As True S is built on the same core methodology as its predecessor, this review uses the name PRICE-S to maintain its originality. PRICE-S is an activity based estimation model that estimates the effort required for each activity. In PRICE-S, an activity represents the work that people, equipment, technologies, or facilities perform to produce a product or deliver a service [PRICE-S 2009]. As described in [IPSA 2008], the effort E of each activity is modeled in the form of Where, E = S x PB x PA (Eq. 2-10) S is the software size that can be measured in source lines of code, function points, predictive object points (POPs) or use case conversion points (UCCPs). POPs is an object oriented (OO) metric introduced by PRICE Systems to measure the size of OO projects [Minkiewicz 1998], and UCCPs is a metric used to quantify the size of use cases

36 BP is the ideal baseline productivity of an industry or an application domain. PA is the productivity adjustment factor that accounts for the overall effects of cost drivers on the productivity of the project. PRICE-S views the software maintenance phase as an activity that focuses on fixing latent defects, deployment, and changing the software release to improve performance, efficiency or portability. Thus, the maintenance cost is determined by these activities other than as a function of how much the software cost is modified. PRICE-S assumes that the maintenance cost only includes changes required to improve performance, efficiency, and portability or to correct defects. Other changes not included in the maintenance cost (i.e., functional enhancements) are estimated the same way as the development project. The model uses a number of maintenance-specific parameters to calculate the cost of software maintenance. The costs associated with functional enhancements and adaptations for preexisting software involve specifying the amount of new, adapted, reused, changed, and deleted code. Other size measures are also used such as the Percentage of Design Adapted, Percentage of Code Adapted, and Percentage of Test Adapted, which are similar to those of SEER-SEM. But unlike SEER-SEM, PRICE-S does not use the effective size aggregated from different types of code in its model calculations. Rather, it uses the different size components separately in various calculations to determine the effort. 23

37 2.2.4 KNOWLEDGEPLAN (CHECKPOINT) KnowledgePlan is a commercial software estimation tool developed and first released by Software Productivity Research (SPR 5 ) in KnowledgePlan is an extension of several previous tools, Checkpoint and SPQR/20, which were originally based on Capers Jones works [Jones 1997]. According to the KnowledgePlan user s guide, the model relies on knowledge bases of thousands of completed projects to provide estimation and scheduling capabilities [KnowledgePlan 2005]. KnowledgePlan estimates effort at project, phase, and task levels of granularity. In addition to the effort, the model main outputs include resources, duration, defects, schedules and dependencies. Estimating and scheduling project tasks are enabled through the knowledge bases containing hundreds of standard task categories to represent typical activities in software projects. These standard task categories cover planning, management, analysis, design, implementation, testing, documentation, installation, training, and maintenance. Each of these task categories is associated with predefined inclusion rules and algorithms that are used to suggest relevant tasks and determine the size of deliverables and productivity for the given size of the project being estimated. The KnowledgePlan tool supports different project types such as new development, enhancement, reengineering, reverse engineering, and maintenance and support of a legacy system. KnowledgePlan uses eight sizing categories called code types, including New, Reused, Leveraged, Prototype, Base, Changed, Deleted, and System Base. Differences in project types may be reflected in the distribution of size

38 estimates of these categories. For example, a new development project typically has a high proportion of New code while enhancement may have a high value of Base code COCOMO The Constructive Cost Model (COCOMO), a well-known cost and schedule estimation model, was originally published in the text Software Engineering Economics [Boehm 1981]. This original model is often referred to as COCOMO 81. The model was defined based on the analysis of 63 completed projects from different domains during the 1970s and the early 1980s. To address the issues emerging from advancements and changes in technologies and development processes, the USC Center for Systems and Software Engineering has developed and published COCOMO II. The model was initially released in [Boehm 1995] and then published in the definitive book [Boehm 2000b]. Among the main upgrades are the introduction of new functional forms that use scale factors, new cost drivers, and a new set of parameters values. COCOMO II comprises three sub-models, Applications Composition, Early Design, and Post-Architecture. The Applications Composition model is used to compute the effort and schedule to develop the system that is integrated from reusable components and other reusable assets using integrated development tools for design, construction, integration, and test. The Applications Composition model has a different estimation form from the other models. It uses a size input measured in terms of Application Points or Object Points [Kauffman and Kumar 1993, Banker 1994] and a productivity rate to calculate effort. The Early Design model is used in the early stages of the project when 25

39 the project information is not detailed enough for a fine-grained estimate. When the detailed information is available (i.e., the high level design is complete, development environment is determined), the Post-Architecture model is used alternatively. The Early Design and Post-Architecture models use source lines of code as the basic size unit and follow the same arithmetic form. The general form of the effort formulas of the COCOMO 81, Early Design, and Post-Architecture models can be written as Where, p B PM = A* Size * EM (Eq. 2-11) i i PM is effort estimate in person months. A is a multiplicative constant, which can be calibrated using historical data. Size is an estimated size of the software, measured in KSLOC. B is an exponential constant (COCOMO I) or scale factors (COCOMO II). EM specifies effort multipliers, which represent the multiplicative component of the equation. In COCOMO 81, the B term is an exponential constant, which is usually greater than 1.0, indicating diseconomies of scale. In COCOMO II, B is defined as a function of scale factors, in form of B = β + β i= 1 SF i where β 0 and β 1 are constants, and SF i is one of the five scale factors. The COCOMO 81 model identifies 15 effort multipliers. 26

40 COCOMO II uses 7 in the Early Design model and 17 in the Post-Architecture model. The effort multipliers are the cost drivers that have multiplicative impact on the effort of the project while the scale factors have exponential impact. The Early Design and Post- Architecture models have the same set of scale factors while the cost drivers in the Early Design model were derived from those of the Post-Architecture model by combining drivers that were found to be highly correlated. The COCOMO II Post-Architecture model s rating values of the model cost drivers were calibrated using the Bayesian technique on a database of 161 project data points [Chulani 1999a]. Table 2-1. COCOMO Sizing Models Development Type Source Code Types Sizing Model New System New New/Added New SLOC Pre-existing Adapted, Reused Reuse Major enhancements Pre-existing Adapted, Reused Reuse Maintenance (repairs and minor updates) Pre-existing Added, Modified Maintenance COCOMO II provides different models to determine the source code size of the project, dependent on the origin of the code (new, pre-existing) and types of change (addition, modification, reuse, automatic translation). Table 2-1 shows the sizing models for various development types. In COCOMO, software maintenance involves repairs and minor updates that do not change its primary functions [Boehm 1981]. Major enhancements, which include changes that are not considered as maintenance, are 27