Estimating Project Outcomes

Estimating Project Outcomes June Verner 1, Barbara Kitchenham 1 and Narciso Cerpa 2 1 National ICT Australia Ltd. 2 Facultad de Ingeniería, Universidad de Talca, Chile {June.Verner, Barbara.Kitchenham}@nicta.com.au, Ncerpa@utalca.cl Phone 61 2 8374 5513; Fax 61 2 8274 5520 Abstract: In spite of many years of research, many software projects still fail. As the goal of this paper is to contribute to ongoing research into the identification of factors that affect project success and failure, i.e. project outcome, we undertook a correlation study of project variables and project outcome. Our data consisted of three data sets. We developed two logistic models for the combined data selected by stepwise regression: one based on three factors and the other based on four of the raw variables. The models were compared for predictive accuracy. Twelve variables significantly associated with project success and failure, in all data sets, were identified. We found that logistic models did not predict failure well, unless the cut-off probability for classifying a project as successful corresponded to the proportion of successful projects in the data set. Only a limited number of the 12 key variables predicted project success and failure. A much larger set of variables appear not to impact project outcome. Keywords: project management, project estimation, logistic models, project success and failure 1. BACKGROUND We have been developing software since the 1960s but still have not learned enough to ensure that our IT development projects are successful. Charette suggests that Billions of dollars are wasted each year on failed software projects and that We have a dismal history of projects that have gone awry [1]. Most organizations try to hide their failures which incur not only monetary loss, but also lost opportunity. Charette provides a long list of high profile failed projects from around the world in his Hall of Shame [1]. Many software project success and failure factors have been described in the literature [1]-[21]. For example: organizational structure, communication with customer/users; personality conflicts; user requirements and requirements specification, scheduling and project budget, customer satisfaction; product quality; leadership; upper management support; software development methodologies; business processes and resources; and the project management process and tracking tools [1]-[21]. Some project failures are predictable and avoidable though it is not always possible to identify what success and failure factors are important early enough to take evasive action. Our research is focussed on developing practical, actionable tools that managers can apply in the very early stages of projects through an understanding of why software development projects succeed or fail [6]. This paper describes an empirical study, using data that we collected by surveying US, and Australia developers. The survey was conducted in order to investigate factors that software developers believe contribute to project success and failure (i.e. project outcome). Our contribution is to provide project managers with insight into: 1) which practices software developers believe have the greatest impact on project outcome, and hence 2) where to focus attention when resources are constrained. The research questions addressed in this paper are: 1) Which factors are most likely to impact project success and failure? 2) If we use logistic regression to predict failure is it better to use the raw variables or to reduce the variable set with factor analysis? 3) If the rate of failure and success are very different in the data sets, will changing the standard logistic regression cut-off value of 0.5 improve predictive capability. 1/8

In previous articles we have used some subsets of our data to address issues such as cost and effort estimation [15][19], US project management practices [16][18], predicting good requirements [17][20], and applying case-based reasoning techniques to projects considered successful by both developers and management [23][24]. The added value provided by this paper is that for the first time all the data from our three data sources are analysed together. This allows us to identify factors that are important across three different data sets and those that appear to be unimportant in all sources. In the next section we describe our survey, the respondents, and the data we collected including some survey demographics. Subsequent sections describe the methodology used and the data analysis. Finally, we provide some discussion and conclusions. 2. DATA In this section we discuss the survey construction, survey administration, survey responses and some project demographics. 2.1 Survey Construction We had discussions with over 100 software developers in the US and Australia regarding project success and failure (outcome). During these discussions the meaning of project success and project failure and factors developers thought important for project outcomes were explored. Based on our discussions and the software engineering literature, we developed a questionnaire to investigate software development practices developers believe lead to successful projects. The survey, which comprised 88 questions, was organized into a number of sections covering the development process: 1) sponsor/senior management, 2) customer/user, 3) requirements, 4) estimation and scheduling, 5) development process, 6) the project manager and project management, 7) development team. We also asked questionnaire respondents if they considered the project they referenced when answering was a success or failure. While we did not define a successful or failed project in the questionnaire, we felt that the discussions we had had, meant that our respondents had a good understanding of what was meant by successful and failed projects. Sections of the questionnaire have been reported before [11]-[13], however we have not previously considered the data in its entirety and compared different sections of the data for their comparative importance in predicting project outcome. 2.2 Survey Administration Our questionnaire was distributed to practitioners from a large US financial institution (Group 1) who each responded by answering it twice, once for a recent project they considered successful and once for a recent project they considered a failure. The questionnaire was later distributed to a second group of practitioners from various companies in North Eastern U.S.A. (Group 2), and a third group of practitioners from Australia (Group 3) who each answered the questionnaire once. Each of these groups of respondents had earlier participated in discussions as described earlier. Our sample is not random but rather a convenience sample. 2.3 Response rate and project demographics In all we received completed questionnaires from 102 respondents, reporting on 164 distinct projects. The majority of our respondents were developers involved with software for use within their own organizations. Projects were classified as a success or not according to the participants view of the project. The responses to the first set of 42 questionnaires (Group 1), described 42 separate projects, 21 regarded as successful and 21 unsuccessful. The second set of responses (Group 2) included descriptions of 80 unique 2/8

projects with a 71% of success rate. While the third set of responses included 42 mainly in house projects with a 78% success rate. Overall 62% of projects were regarded as successful and 38% unsuccessful, 87% were development projects (55% successful), and 13% were large maintenance/enhancement projects in terms of effort (66% successful). The percentage of projects by number of full-time IT employees is 1-4 = 44%; 5-9 = 19%; 10-19 = 21%; >20=16%. 3. DATA ANALYSIS Our data analysis methodology consisted of the following steps: 1. Reduce variable set by a. Remove variables with large numbers of missing values. b. Correlate all remaining variables with project outcome, within each data set and overall. Spearman correlation is used as the variables are categorical. c. Select variables significantly correlated with project outcome at the 90% level or above for each of the three data sets and overall. 2. Use principal components factor analysis on the reduced set of variables to develop a new set of variables (principal components) suitable for predicting project outcome with a minimum of 0.5 as the variable loading for a factor. 3. Use the new variables to develop prediction equations using logistic regression (Factor-based model). 4. Change the cut-off point (from the standard 0.5) to match the overall success rate in the data set when using logistic regression. This will allow us to investigate if improved failure prediction results by changing the cut-off point to reflect the successes and failures in the data. 5. Use the original set of variables, correlated with project outcome, to develop prediction equations using logistic regression (raw variable model). 6. Using the original set of variables change the cut-off point (from the standard 0.5) to match the overall success rate in the data set when using logistic regression. This will allow us to investigate if improved failure prediction for the raw variable model results by changing cut-off point to reflect the successes and failures in the data. 4. RESULTS Our results are presented in the same order in which we describe the methodology we used. 1. We obtained a Spearman correlation between each variable and project ooutcome for each group. Table 1 identifies all variables for which significant correlations (p<0.1) were found in every dataset. Table 2 identifies the variables that were not significantly correlated with project outcome in any of the groups 2. We applied exploratory principal components factor analysis (principal component analysis with Varimax rotation) to the eight variables in Table 1, noted in Sections 2-5 of that table. There was a relatively large number of missing values for the variables in Section 6 and 7 (between 25% and 28%) and hence these were omitted from the factor analysis. No other variables had more than 5% missing values. The first three factors together explained 75% of the variance. Each factor appeared to measure a distinct concept (see Table 3): PM capability, realistic plans and adding staff late to the project. 3. Using the whole data set, excluding missing values we used the factors in a stepwise logistic regression (136 data points in all). Logistic regression was used because our dependent variable outcome is categorical. All three factors were included in the model. Accuracy statistics for the logistic regression are shown in Table 4. The first column provides the results if the standard cut-off probability for classifying a project as successful is 0.5. This cut-off point leads to poor predictive accuracy for failures but good predictive ability for successes 4. However, if we use a cut-off probability corresponding to the actual proportion of successful projects (0.66=90/136), we improve the accuracy prediction of failures, have a more balanced result, while decreasing the accuracy prediction for successes; i.e., changing the cut-off probability in a logistic regression to reflect the proportion of successes in the data improves the prediction of failed projects. 3/8

5. Performing a forward stepwise logistic regression using the original eight variables, four variables are included in the model. In the order they were introduced into the model they are: Good requirements; How good were estimates; Staff added late to achieve aggressive schedule; and Level of confidence of customers in PM and team. All the variables, except Staff added late which is associated with Factor 3, correspond to variables associated with Factor 2. No Factor 1 variables are included in the model. 6. The predictive accuracy of the model is shown in Table 5. Again the predictive accuracy for failures is poor when we use the standard cut-off probability of 0.5 and is improved when the cut-off probability is set to the proportion of successes (i.e. 0.66=90/136). Overall there is little to choose between two models as the predictive accuracy is similar for both cut-off points Table 1. Correlations with significant project outcome for all data sets and overall (** corresponds to p<0.01, * corresponds to p<0.05) Section ID Variable Overall Data set 1 Data set 2 Data set 3 Sponsor 1 Nil Nil Nil Nil Customers and Users 2 Level of confidence the customer has in project manager (PM) and team members 0.418** 0.495** 0.289* 0.367* The customers had realistic expectations 0.378** 0.370* 0.271* 0.393* Requirements 3 The project had good requirements overall 0.498** 0.406** 0.465** 0.447** Estimation and Scheduling 4 How good were the estimates 0.474** 0.548** 0.367** 0.335* Staff were added late to meet an aggressive schedule -0.356** -0.278-0.428** -0.364* Project manager 5 The project manager communicated well with staff 0.399* 0.335** 0.330** 0.415** How good was the project manager 0.382** 0.350* 0.360* 0.366* Development process Development team The project manager related well to software development staff 0.350** 0.444** 0.273* 0.358* 6 Adequate time was allowed for each of the 0.407** 0.331 0.409** 0.487** phases 7 How well did team members work together 0.557** 0.460** 0.625** 0.582** How high was the motivation of team members 0.404** 0.450** 0.422** 0.331* How good was the working environment? 0.366** 0.314 0.393** 0.367** 4/8

Table 2. Variables Not Significantly Correlated with Project Outcome in Any Data Set Section Variable Sponsor project manager given full authority Customer and users Requirements Estimation and Scheduling Project Manager Development process Development team the project began with a committed champion the commitment lasted right through the project the sponsor was involved with decisions other stakeholders committed and involved senior management impacted the project customers/users involved in schedule estimates customers had realistic expectations problems were caused by large by the number of customers/users involved were you affected by large numbers of users requirements were gathered using a particular method the requirements were complete and accurate did the scope change during the project the project had a schedule there was a project manager was project manager changed during project was everyone in project treated equally or did project manager plan favorites project manager's background years of experience of the PM was project manager experienced in application area was project manager able to pitch and help if needed How frequently were meetings held on the average The project was over organized Other projects negatively impacted this project What was the effect of the schedule on the development process project manger motivated team project itself motivated team other motivation for team senior team leader motivated team technology used motivated team other team member motivated team rewards at end of project motivated team Aggressive schedule affected motivation Number of consultants The consultants reported to the project manager The project manager had the authority to choose team members Did all the key people stay throughout the project? Table 3. Three Factors Based on Principle Components Analysis with Varimax Rotation Variables Factor 1 Factor 2 Factor 3 Level of confidence of customers in PM & team 0.191 0.747-0.160 Customers had realistic expectations 0.215 0.732 0.120 Good requirements overall 0.251 0.776-0.195 How good were the estimates 0.473 0.636-0.141 Staff were added late to achieve aggressive schedule -0.148-0.116 0.958 The PM communicated well with staff 0.877 0.301-0.021 How good was the project manager 0.892 0.259-0.075 Project manager related well to staff 0.793 0.252-0.232 Percentage variation explained 51.7 11.9 11.3 Factor description Project manager capability Realistic project plans Adding staff to meet schedules 5/8

Table 4 Logistic Regression Accuracy for Factor Model Cut-off 0.5 0.66 Correct failures 63 69.6 Correct successes 88.9 80 Overall 80.1 76.5 Table 5 Logistic Regression Accuracy for Raw Variable Model Cut-off 0.5 0.66 Correct failures 58.7 78.4 Correct successes 91.1 78.9 Overall 80.1 78.7 5. DISCUSSION, CONCLUSIONS AND FURTHER WORK Although our results are based on a convenience sample, our results contribute to assessing the generalisability of project outcome factors, since the data is drawn from three different sources. Our analysis has identified a small set of 12 variables that are significantly correlated with outcome in all three groups of data (see Table 1). We believe that these represent a key set of variables for project outcome and provide insight into which practices have the greatest impact on project success or failure. We have also identified a large number of variables that are not associated with project outcome in any of the three groups of data. We believe these may be omitted from any general model of project success and failure factors. We were able to derive three factors capable of representing 75% of the variance from eight of the key variables. However, we did not find that a model based on factors was much more accurate than a step-wise model based on 4 of the raw variables. These results are based on goodness of fit statistics not predictive accuracy. They need to be confirmed using an independent hold-out sample. When resources are constrained our results show that a project manager should focus attention on Ensuing good requirements; Ensuring good estimates (this is a consequence of having good requirements) ; Not adding staff added late to achieve an aggressive schedule (this is a consequence of good requirements and good estimates); and Improving the level of confidence of customers in the PM and team. We found that using the default cut-off probability of 0.5 for classifying a project as successful led to poor predictive accuracy for failures. Prediction accuracy was more balanced for successes and failures when the cutoff probability was altered to reflect the actual proportion of successful projects. This implies that logistic models built in one environment will not work well in another unless the proportion of successful projects in the new environment is known in advance. Our logistic analysis omitted four of the key variables because they exhibited a relatively large number of missing values. The variables are: Adequate time was allowed for each of the phases How well did the team members work together? How high was the motivation of the team members? Working environment Future data collection activities need to ensure that these variables are collected, so we can assess the impact they have on models for predicting project success. In addition, we will investigate the best approaches for dealing with missing values. For example, analysis methods such as Bayesian networks are better able to deal with missing values than standard logistic regression. We also need to consider improving our questionnaire by the addition explicit failure factors if we wish to develop better predictive models. 6. Acknowledgements This research was done while Dr Cerpa was visiting NICTA. NICTA is funded through the Australian Government's Backing Australia's Ability initiative, in part through the Australian Research Council. Funds were also provided by the University of New South Wales. 6/8

7. REFERENCES [1] Charettte, R. Why software fails, IEEE Spectrum, Sept 2005 pp.42-49. [2] Glass, Robert L, 1999. Evolving a New Theory of Project Success, Communications of the ACM, Volume 42, Number 11, November. [3] Hagerty, Nicole, 2000. Understanding The Link Between IT Project Manager Skills and Project Success. Research In Progress, Proceedings of SIGCPR Conference, Evanston, IL pp. 192-195. [4] Kanter, R. (1988). Three tiers for innovation research. Communication Research, 15, 509-523. [5] Linberg, Kurt R., 1999. Software Developer Perceptions About Software Project Failure: A Case Study, Journal of Systems and Software, Volume 49. [6] Mc Cabe, B., (2006). No Shorcuts to Project Success, The Australian, 20 th of June. [7] Pereira, J., Cerpa, N., and Rivas, M., 2004. Risk factors in software development projects: Analysis of the Chilean software industry. First Experimental Software Engineering Latin American Workshop, Brasilia, 18 th October 2004. [8] Pinto, Jeffrey K. and Mandel, S. J., 1990. The Causes of Project Failure, IEEE Transactions on Engineering Management, Volume 34, Number 7, November. [9] Pinto, Jeffrey K. and D.P. Slevin, 1988. Project Success: Definitions and Measurement Techniques. Project Management Journal. Volume 19, Number 1, pp. 67-72. [10] Pressman, Roger, 1996. Software Engineering: A Practitioners Approach, McGraw Hill. [11] Procaccino, J. D., and Verner, J. M., 2002. Software Practitioner s Perception of Project Success: A Pilot Study. International Journal of Computers, the Internet & Management, Volume 10, Number 1, January- April 2002. (01-02) (A). [12] Procaccino, J. D., Verner, J. M. and S. Overmyer, P., 2001. Case Study: Factors for Early Prediction of Software Development Success,Information and Software Technology. [13] Procaccino, J.D.; Verner, J.M.; Shelfer, K.M:, and Gefen, D., 2004. What do software practitioners really think about project success: an exploratory study. Journal of Systems and Software, (78:2), pp. 194-203, 2005. [14] Verner, J., Cox, K., Bleistein, S., Cerpa, N., 2005. Requirements engineering and software project success: An industrial survey in Australia and the US. Australian Journal of Information Systems, vol. 13, pp. 225-238. [15] Verner, J. M., Evanco, W. M., and Cerpa, N., [2007] State of the practice: An exploratory analysis of schedule estimation and software project success prediction Information and Software Technology Volume 49, Issue 2, February 2007, Pages 181-193. [16] Verner, J., and Evanco W., [2005], In house Software Development: What Software Project Management Practices Lead to Success? IEEE Software Jan/Feb, Vol. 22, No 1, pp. 86-93. [17] Verner, J.M., Cox, K. and Bleistein S., [2006] Predicting good requirements for in-house development projects, ACM International Symposium on Empirical Software Engineering, Rio de Janeiro, Brazil, September pp 154-163. [18] Verner. June M. and Cerpa, N., (2005), Australian Software Development: What Software Project Management Practices Lead to Success? [2005] IEEE Proceedings of ASWEC, Ed Paul Strooper, Brisbane, Australia, March 29-31 pp 70-77, ISBN 0-7695-2257-2. [19] Verner, J. and Evanco, W., [2003] The Effect of Cost and Effort Estimation on Project Success, Proceedings of ICSSEA 2003, Dec 2-4, Paris, Volume 4 pp 8.1-10. [20] June M Verner, Cox, Karl, Bleistein Steven and Cerpa Narciso, [2004] Requirements Engineering and Software Project Success: An Industrial Survey in Australia and the US, Australian Workshop on Requirements Engineering, AWRE, Adelaide, Australia, 6-7 December 2004, pp.10.1-10.10 ISBN 1920927204, Best paper award. [21] Walsh, K., and Schneider, H., 2002. The role of motivation and risk behaviour in software development success. Information Research, 7(3), April. [22] Wateridge, John, 1995. IT Projects: A Basis For Success, International Journal of Project Management, Volume 13, Number 3, June. [23] Weber, R., Evanco, W., Waller, M., and Verner, J., (2004), Identifying Critical Factors in Case-Based Prediction.. In Valerie Barr and Zdravko Markov (eds.) Proceedings of the Seventeenth Annual 7/8

Conference of the International Florida Artificial Intelligence Research Society, 207-212. Menlo Park, CA: AAAI Press. [24] Weber, R., Waller, M., Verner J. M., and Evanco, W., [2003] Predicting Software Development Project Outcomes Proceedings of the Fifth International Conference on Case-Based Reasoning (ICCBR), (eds) D Bridge and K Ashley, Springer-Verlag, June, pp. 595-609. [25] Wohlin, C., von Mayrhauser, A., Host M., and Regnell, B., 2000, Subjective Evaluation As A Tool For Learning From Software Project Success, Information and Software Technology, Volume 42. pp. 983-992. [26] Wohlin C. and von Mayrhauser A., 2000, Assessing Project Success using Subjective Evaluation factors, Journal of Software Quality, August. 8/8