2012 ITEA Annual Symposium David G. Smith Sep 2012

2012 ITEA Annual Symposium David G. Smith Sep 2012

The Flight Plan Why; the Research Problem and questions Basis and Background; literature review Clarification; unique terminology How; the methodology and limitations What; the findings what did they say? Why; implications and future research Finally; Conclusion and recommendations

Overview of the Research Question There is a lack of quantitative evidence either supporting or refuting the claim that ToC PM improves project cost, schedule, performance, and overall effectiveness Theory of Constraints Project Management was brought to the Air Force Flight Test Center at considerable expense, and has never been comparatively studied.

Overview of the Research Question Questions were designed to address project: Cost Schedule Performance Overall Effectiveness Four research (and associated null) hypotheses were crafted to address each question

Definitions of Uncommon Terms 412th Test Wing Air Force Flight Test Center Concerto Critical chain project management (CCPM) Critical path project management (CPPM) Developmental test and evaluation Program evaluation and review technique (PERT)

Project Management Overview Projects fail at alarming rates Costing time, money and resources Failing to deliver the desired end state Volatile projects have the highest failure rate Flight test projects are highly volatile Prior AFFTC project management lacked structure CCPM introduced in 2001 Never quantitatively studied Gap in the literature

The Triad-pick any two? Cost Schedule Overall Effectiveness Performance

Program Evaluation and Review Technique (PERT) Critical Path Method (CPM) Critical Chain Project Management (CCPM)

Review of the Literature Lack of comparative studies noted No peer-reviewed studies of volatile projects Only anecdotal evidence of success at the AFFTC None-the-less, ToC implemented in 2001 Literature strongly supports ToC PM Reduced cost, ABC automotive-3 more units weekly Increased timely performance-late pharmaceutical projects reduced by 50% Delivering project content-777 from late and over budget to on-time delivery as designed Improving overall effectiveness-quicker project execution w/ fewer resources

ToC PM 5 Step Process: Identify the system constraint Exploit the system constraint Subordinate everything else to the system constraint. Elevate the system constraint Lather, rinse, repeat (Goldratt & Cox, 2004)

Why test military aircraft? Required by regulation Enhances safety of flight Fiscally smart Testing minimizes risk ID s problems early Provides fix before production Multiple examples: Bradley B-1 Bomber F-35 Air Refueling lights

Description of the Methodology Retrospective quantitative causal-comparative design Self-report descriptive research (2 groups) N=100, n=62 (line managers) N=10, n=10 (senior managers) Likert-type scale ANOVA (t-test, Mann-Whitney U-test, modified Bonferroni correction) Factor analysis Bayesian analysis Comparison of actual projects Researchers role and setting

Bayesian approach Different approach from frequentist analyses: Frequentist: Analysis based on a model: Look at p (getting the data we observed null hypothesis value of the parameter) No prior information Estimate an unknown constant (parameter) Hypothesis test, p-value, confidence intervals type I/II errors Bayesian: Use hierarchical models Estimate a random variable, get a density function Incorporate prior information Clean interpretation: no hypothesis tests, no p-values, no type I, type II errors, conclusions driven by posterior density

Limitations of the Research Lack of randomization, manipulation, and in-equality of groups Larger sample size? Ethics and perception? No comparison of identical projects executed differently Lack of other comparative studies

Discussion of Findings 62 line manager responses (Edwards AFB population of 100) 36 experiment (uses ToC) 26 control 10 senior responses (entire population) 5 experiment 5 control ANOVA (t-test and Mann-Whitney U-test) Modified Bonferroni correction Factor analysis Bayesian (WinBUGS estimated w/ Markov-Chain Monte Carlo processes)

Data 4.5 4 5 4.5 3.5 4 3 3.5 2.5 2 1.5 1 Using ToC PM Not Using ToC PM 3 2.5 2 1.5 1 Using TOC Not Using TOC 0.5 0.5 0 Q-1 Q-2 Q-3 Q-4 Q-5 Q-6 Q-7 Q-8 Q-9 Q-10 0 Q-1 Q-2 Q-3 Q-4 Q-5 Q-6 E F G H I J K 25 20 15 10 5 0 Favors ToC Favors Non-ToC Tie Critical Critical Path Chain Days Saved Savings Duration Duration Project 1 123 162 39 24% Project 2 57 72 15 21% Project 3 63 66 3 10%

Significance Cost Line managers support rejection (r(62) = 3.521, p =.0008) Bayesian statistical analysis supported rejection of the null hypothesis (p <.05) Senior managers did not support rejection (U = 13.5, p =.841) Schedule Line managers not significant (r(62) = 1.71, p =.0919) Bayesian statistical analysis supported rejection of the null hypothesis (p <.05) Moderately significant for senior managers (U = 23, p =.032)

Significance Performance Line managers not significant: r(62) = 0.796, p =.429 Bayesian statistical analysis supported rejection of the null hypothesis (p <.05) Senior managers did not support rejection (U = 21, p =.095)

Significance Overall Effectiveness (pooled data) Line managers significant (r(615) = 2.194, p =.029) Bayesian statistical analysis supported rejection of the null hypothesis (p <.05) Significant for senior managers (1-6: U = 698.5, p <.001; E-K U = 601, p =.014) Compared projects demonstrate efficiencies in use of ToC PM

Implications Literature and this study support ToC PM as improving project cost Mixed study results addressing schedule; in contrast to the literature which strongly supports ToC PM Limited study support for performance; again, in contrast to the literature Literature and this study support ToC PM as improving project overall effectiveness Power: high value for sensitivity and specificity Study addresses a clear literature void

Recommendations Future research Schedule & performance not consistent with literature Bayesian analysis original contribution to the lierature Continuous vs likert scale Address age/gender/rank etc Empirical study with larger group Continue the use of ToC PM at the 412 th Test Wing Expand ToC PM across the AFFTC Consider ToC PM for any highly volatile project management scenario

Recommendations, continued Complete one or more comparative studies: Bayesian analysis of results, using current study as prior information Use Edwards as well as other AF bases Consider inclusion in the PM BOK Applications in private industry?

Concluding Remarks Questions?