Mean-Variance Combination (MVC): A New Method for Evaluating Effort Estimation Models

Transcription

1 Mean-Variance Combination (MVC): A New Method for Evaluating Effort Estimation Models Lang Xie (ISCAS), Ye Yang (ISCAS),Da Yang(ISCAS),Qing Wang(ISCAS),Mingshu Li(ISCAS) April. 27, 2011

2 Agenda 1 2 Introduction to cost estimation research at ISCAS MVC method for evaluating cost estimation models 3 Ongoing work

3 Cost Estimation Research Framework ISCAS Perspective Local Government Cost estimation for contract pricing Literature review COCOMO family models COCOMO-U Budgeting under uncertainty Defects prediction Simulation Software Quality Industry Cost estimation tool Software Cost Estimation Basic Research Coping with the cone of uncertainty Software Process USC JSP WikiWinWin SoftPM COCOMO Coping with the uncertainty Combining estimations Estimation based on Use- Case Cost estimation & process management integration Cost drivers auto-rating Software Measurement SoftPM 3 3

4 Uncertainty ranges of cost estimations present a decreasing trend as the software development lifecycle proceeds 4x 2x 1.5x Early Design (13 parameters) 1.25x Relative Size Range x 0.8x 0.67x Post-Architecture (23 parameters) 0.5x Applications Composition (3 parameters) 0.25x Concept of Operation Rqts. Spec. Product Design Spec. Detail Design Spec. Accepted Software Feasibility Plans and Rqts. Product Design Detail Design Devel. and Test

5 Input COCOMO-U The COCOMO-U takes the probability distributions of the estimated project size and other 22 cost factors as input. Output the probability distribution of software development effort 5

6 The InCoME Process Cost Drivers Analysis & Data Collection Build Cost Models Yes Evaluate Cost Models Require Further Improvement? No Risk Assessment Cost Estimation Decision Support 6

7 Estimation Process based on COGOMO ---constructive government contract pricing model Government history projects Industry history projects Human capital in China Industry benchmark Project size Customized model input Calibrated parameters Establishment of government knowledge base Effort Estimation Effort distribution Wage-rate in China Estimated effort Cost Analysis Total cost 7

8 Data:7 versions of Qone Localize of COCOMO Result: A: 1.32 B: 0.94 Qone: a commercial software process management tool, released by a Chinese software enterprise

9 Data: Cost Estimation based on Use Cases 7 versions of Qone Estimation Model Effort = A * (UCadjusted) B UCadjusted = newuc + Wmod * moduc + Wreu * reuuc + Wdel * deluc QONE case UCadjusted = newuc * moduc * adouc version adduc moduc reuuc adjuste d UC effort v v v v v v v A B R 2 P-value The method provides guidance for organizations to conduct the maintenance effort estimation based on use cases. It apply use cases as the size metric. The added, modified, reused and deleted types of use cases are identified to be included in the use case metric for estimating the effort of software maintenance.

10 Propheta-a cost estimation tool Three cost estimation methods: Analogy estimation based ondatabase from CSBSG and ISBSG COCOMO Integrated estimation for software product with multiple modules CSBSG: The China Benchmarking Standards Group ISBSG: The International Software Benchmarking Standards Group

11 Agenda 1 2 Introduction to cost estimation research at ISCAS MVC method for evaluating cost estimation models 3 Ongoing work

12 MVC method Background && Motivation MVC(mean-variance combination) method Experiment result

13 Background(1/3) A wealth of estimation methods existed Evaluation method is important Indicate the problem of estimation models. Drive the improvement of estimation models. Statistic view of model s character Bias and variance

14 Background(2/3) Bias and Variance Ideal Model y Model 2 Structure: horizontal line Data: whole data set Model 1 Structure: y = a*x + b Data: part of data x

15 Background(3/3) The true bias can not be caught The distance between the observed value and estimated value contains bias and variance Accuracy indicators: MMRE gives the bias information while stdmre gives the variation information and part of bias

16 Motivation Indicators: based on RE or MRE MMRE, stdmre, PRED(N), MdMRE, etc. Evaluation: Cross Validation(CV) Average value of indicators above Interval of indicators above Traditional mean value of indicators in CV are challenged Do not combine the bias and variance together The comparing result varies

17 MVC method: the whole process History data Model structure Resampling Train and test MVC Process Generate indicators Split Ratio, Re-sampling times

18 MVC Method: Re-sampling process Re-sampling process Input: data, model structure Output: pairs of (MMRE, stdmre) Fix the ratio of test set and sampling times N Randomly split whole data set N times to get N pairs of (train set, test set) Train and test current model structure N times using the N pairs Calculate N pairs of (MMRE, stdmre)

19 MVC method: why Re-sampling The history data is limited, small size Independent and identity distribution may not be satisfied Re-sampling is like to simulate the situations: train set VS test set History data Vs the new data C(n,m), the number of possible combination is large

20 MVC Method: Generate Indicator paradigm Scatter-plot Convex_hull AUC_L (AUC Lower) AUC_M (AUC Middle) AUC_U (AUC Upper) ACU: Area Under Curve

21 MVC Method: Generate Indicator Algorithm Input : N pairs of (MMRE, stdmre) Output: three types of area Get the scatter plot of MMRE and stdmre Get the convex hull, and split the convex hull as up part and lower part Extend the two part of convex hull to three types of area

22 AUC_U Convex hull AUC_M AUC_L

23 MMRE std_mre Result: Performance of traditional indicators CV times CV times

24 Results: the scatter plot of MMRE and stdmre

25 Results: MVC s indicators on two models COCOMO81 dataset. NASA93 dataset model Auc_U Auc_M Auc_L Auc_U Auc_M Auc_L COCOMO Analogy

26 Results: variance of indicators divided by mean value

27 Discussion Benefit: The reason of more stable: distribution replace point The combination

28 Agenda 1 2 Introduction to cost estimation research at ISCAS MVC method for evaluating cost estimation models 3 Ongoing work

29 Improve MVC: MMRE Ongoing work Splitting of four areas Left-up High bias and low variance Mean_threshold Left-bottom low bias and low variance stdmre right-up High bias and high variance Right-bottom Low bias and high variance Std_threshold How to determine the threshold for dividing four regions?

30 Ongoing work Deal with Cross-Company data Definition and measurement of Local Bias (Ye, etc ESEM) Build new models to deal with Organization ID Measure uncertainty more accurate Reduce the bias under the indicating of MVC method and express the variance more accurately

31 Thank you!