Risk Analysis Methodology Overview - Monte Carlo Simulation

Transcription

1 Risk Analysis Methodology Overview - Monte Carlo Simulation We may all agree that risk analysis is a necessary, vital part of any valid/defensible cost estimate. We may not agree as much on the best approach to take to quantify risk in an estimate. All estimates contain risk. In the words of a wise cost estimator I know, That s why they re called estimates, and not exactimates! We must quantify and manage levels of risk. Why? One vital part of a successful program is the ability to build a budget based on reliable cost projections. Reliability increases when we can analyze inherent risk, quantify a range of possible outcomes, and predict cost impacts of those outcomes. This article will be the first of several discussions I present to simplify and demystify risk analysis. The first topic, presented here, is a broad overview of one risk analysis approach: Monte Carlo simulation. This approach is used in various software products such as Crystal Ball The Monte Carlo approach was developed in the 1940s by Stanislaw Ulam and Nicholas Metropolis. The name Monte Carlo refers to games of chance popular at the time in Monte Carlo, Monaco. Monte Carlo utilizes simulation, drawing random numbers from input probability distributions to result in an output distribution. To demonstrate the Monte Carlo process, consider the following example. We have a high level cost estimating relationship to determine the unit production cost of a military helicopter. The equation is based on three performance characteristics: maximum airspeed, service ceiling, and maximum payload. The equation is shown below: Cost = * Service Ceiling * Max Payload * Max Airspeed Hopefully, we have some requirements analysis documentation that states broad system performance parameters, or targets. We will use these targets as our most likely or point estimate. Service Ceiling: 18,000 feet

2 Maximum Payload: 3,900 pounds Maximum Airspeed: 145 miles per hour Using our CER, the unit production cost should be approximately $8.56 Million. On a side note, the analysis required to develop this CER and resulting value becomes a trivial exercise when we leverage publicly available data and the powerful data analysis features in TrueFindings more to come in the near future on this new capability from PRICE Systems! NOTE: This article is not an endorsement of Crystal Ball or Monte Carlo risk analysis. It is a demonstration and broad overview of the process involved in Monte Carlo simulation, utilizing the Crystal Ball software. However, PRICE Systems developers have created a companion application allowing users to build an estimate in TruePlanning and export the inputs into a Crystal Ball interface to perform risk analysis. Visit pricesystems.com to request a download of this and several other companion applications. Back to our example. Through our interviews with engineers and other subject matter experts, we ve determined a range around these three inputs, using minimum, likeliest, and maximum values in a Crystal Ball risk input. We then run a Monte Carlo simulation consisting of 1,000 trials.

3 Service Ceiling: 17,000 feet to 19,000 feet Maximum Payload: 3,500 pounds to 4,300 pounds Maximum Airspeed: 135 mph to 160 mph Resulting cost distribution output For those of us not familiar with Crystal Ball, or Monte Carlo, we may be asking ourselves What just happened? How did we go from system requirements, to determination of input ranges, to a riskbased cost distribution? Now, we will walk step by step through the process in a Monte Carlo simulation. Thankfully, I ll only walk through three iterations and leave the remaining 997 trials for Crystal Ball to deal with. Remember that we have our basic model inputs: service ceiling, speed, and payload. We then assigned a distribution around each of these inputs. As you can see from the screen shots above, I used a triangular distribution. In Crystal Ball, there are over 20 different distribution types you can choose from. Depending on the type of variable and the ranges around the variable, the user should choose an appropriate distribution. An example of this is service ceiling. We know that there are military requirements to fly within certain elevations based on mission profiling, threat envelopes, etc. We know we cannot fly below zero feet, or up to infinity. The fact that we have firm minimum and maximum altitudes within our requirements document points us towards a triangular distribution. Another example might be with software sizing. We know we cannot write less than zero lines of code. Also, albeit a cynical perspective, code growth can result in massive overruns in a program. An appropriate type of distribution in this example may be a logarithmic distribution. Key properties of a

4 logarithmic distribution are that values below 0 are impossible, and the maximum can statistically go to infinity. We d hope our code size does not reach to infinity, but there is a very small probability that the code size grows well past our initial estimate. The logarithmic distribution allows us to account for that. Once we have our input distributions defined and set up in Crystal Ball, we are ready to perform our risk analysis. We ll talk through exactly what happens in Crystal Ball from a very simplistic view in order to explain how Monte Carlo simulation works. When we start the simulation, the software will randomly select a value from each input distribution. In our example, assume the following values were selected: Service Ceiling: 17,600 Maximum Payload: 4,100 Maximum Airspeed: 152 The software then enters those values into the cost estimating relationship, resulting in a unit production cost of $9.37Million. On the next iteration, the software selects new random values from each input distribution: Service Ceiling: 18,400 Maximum Payload: 3,700 Maximum Airspeed: 139 The resulting unit production cost is $8.12 Million. On the third iteration, the software again selects new random variables from each input distribution: Service Ceiling: 17,150 Maximum Payload: 3,525 Maximum Airspeed: 146 The resulting unit production cost is $6.21 Million. Throughout this process, and along with the other 997 iterations, the software is saving the unit production costs from each iteration. Once we collect all of these results and plot the data, we have an output distribution. The shape of the output distribution is tied to several factors. One of these factors is the type and range of underlying input distributions. Another factor is correlation. Stay tuned for my upcoming blog on correlation.

5 The output distribution will have its own set of statistical characteristics. It will have a mean, median, mode, standard deviation, skewness, kurtosis, etc. Now that we have an output distribution, using the characteristics of the distribution, we can answer some important questions. Examples: How confident are we in our original point estimate? Recall that our point estimate, based on our original most likely inputs was $8.56 Million. Just as we have done hundreds of times in Statistics 101, we can quantify how much of the bell curve is above or below that value. If our point estimate is at the 41 st percentile, we can say we have a confidence level of 41% in our original point estimate. In other words, there is a 41% probability that we will either meet or come below our original point estimate. On the flip side, we can say we have a 59% probability of over running the original cost estimate. What if we want to be cautious and include enough money in the budget to be at a 70% probability of executing at or below our budget? Using the statistical characteristics of our distribution, we can quickly calculate the unit production cost that lies at the 70% mark: $9.76 Million Of course, we can also calculate the amount of risk dollars in the estimate by comparing the base case with the 70% percentile, resulting in $1.20 Million per unit. Monte Carlo simulation is a powerful risk analysis process. Each commercially available tool, such as Crystal Ball has its own set of features, limitations, and user learning curve. However,

6 through this example presented here, hopefully I ve provided a better understanding (or maybe just a refresher) on some of the basic steps in a Monte Carlo simulation. To recap, a Monte Carlo simulation uses random drawings out of input distributions to calculate a formula, such as a cost estimating relationship. The result of each iteration is saved and then compiled into an output distribution. We can then use the statistical characteristics of the output distribution to make decisions regarding the confidence of our estimate and the likelihood of successful program execution. Monte Carlo simulation is one small piece of the larger risk analysis puzzle. Commercially available software programs perform the vast majority of work involved in these complex calculations. It is important that we remind ourselve from time to time of the activities happening behind the software. Equally important is our need to communicate what is happening to key decision makers. This builds confidence in our estimating process and results in higher program level support. Please look for other upcoming articles on Method of Moments risk analysis and correlation.