Modeling, Simulation & Data Mining: Answering Tough Cost, Date & Staff Forecasts Questions

Transcription

1 Modeling, Simulation & Data Mining: Answering Tough Cost, Date & Staff Forecasts Questions Troy Magennis (Focused Objective) Larry Maccherone (Rally)

2 Pain Point My Boss Needs A Date

3 Getting Quantitative Evidence

4 Assessing & Communicating Risk / Uncertainty

5 My Mission Arm my teams (and yours) with the tools and techniques to solve these problems

6

7 2 Minutes About Larry Larry is a Pisces who enjoys skiing, reading and wine (red, or white in outdoor setting) We have a lot in common over to Larry!

8 Metrics & Measurement

9 Why measure? Feedback Diagnostics Forecasting Lever

10 When to NOT take a shot Good players? Monta Ellis 9 th highest scorer (8 th last season) Carmelo Anthony (Melo) 8 th highest scorer (3 rd last season)

11 The ODIM Framework better Measurement better Insight better Decisions better Outcomes

12

13 What is normal? Cumulative -> 0.1% 2.3% 15.9% 50.0% 84.1% 97.7% 99.9% Roughly -> 85% 98%

14 Are you normal?

15 You will be wrong by 3x-10x when assuming Normal distribution 2.5x-5x when assuming Poisson distribution 7x-20x if you use Shewhart s method Heavy tail phenomena are not incomprehensible but they cannot be understood with traditional statistical tools. Using the wrong tools is incomprehensible. ~ Roger Cooke and Daan Nieboer

16 Bad application of control chart Control is an illusion, you infantile egomaniac. Nobody knows what's gonna happen next: not on a freeway, not in an airplane, not inside our own bodies and certainly not on a racetrack with 40 other infantile egomaniacs. ~Days of Thunder

17 Time in Process (TIP) Chart A good alternative to control chart

18 Collection Perceived cost is high Little need for explicit collection activities Use a 1-question NPS survey for customer and employee satisfaction Plenty to learn in passive data from ALM and other tools How you use the tools will drive your use of metrics from them

19 Summary of how to make good metric choices Start with outcomes and use ODIM to make metrics choices. Make sure your metrics are balanced so you don t over-emphasize one at the cost of others. Be careful in your analysis. The TIP chart is a good Data visualization is like photography. Impact is a function of perspective, illumination, and focus. ~Larry Maccherone alternative to control chart. Troy s approach is excellent for forecasting. We ve shown that there are many out there that are not so good. Consider collection costs. Get maximal value out of passively gathered data.

20 Flaw of Averages, Risk & Monte Carlo Sim

21

22 A model is a tool used to mimic a real world process A tool for low-cost experimentation

23 Monte Carlo Simulation? Performing a simulation of a model multiple times using random input conditions and recording the frequency of each result occurrence

24 Scrum Backlog This Iteration Deployed Run Sim Total Iterations

25 Kanban Backlog Design Develop Test days 1 5 days 1 2 days Deployed Run Time Total

26 Frequency of Result Result versus Frequency (50 runs) 25 More Often Result Values For example, Days Less Often

27 Frequency of Result Result versus Frequency (250 runs) 25 More Often Result Values For example, Days Less Often

28 Frequency of Result Result versus Frequency (1000+ runs) 25 More Often Result Values For example, Days Less Often

29 Key Point There is NO single forecast result There will always be many possible results, some more likely

30 Likelihood 50% Possible Outcomes 50% Possible Outcomes Time to Complete Backlog When pressed for a single number, we often give the average.

31 Likelihood 95% Outcomes 5% Time to Complete Backlog Monte Carlo Simulation Yields More Information 95% Common.

32 Key Point Average is NEVER an option WARNING: Regression lines are most often average

33 But, I.T. gets worse

34 Likelihood Promised New Average Time to Delivery 50% Possible Outcomes Planned Backlog Perf. Issues Vendor Delay 1 2 3

35 Key Point Risks play a BIG role in forecasts Yes, more than backlog.

36 Velocity is NOT Linear nor is defect rate, scope-creep, story expertise requirements, team skill, etc.

37 Likelihood (0-100%) Date for likelihood

38 Key Point Forecasts should be presented with the right amount of uncertainty

39 Demo: Forecasting PAIN POINT My Boss Needs a Date

40 In this demo Basic Scrum and Kanban Modeling How to build a simple model SimML Modeling Language Visual checking of models Forecasting Date and Cost The Law of Large Numbers

41 Demo: Finding What Matters Most Cost of Defects & Staff Analysis

42 Sensitivity Report Actively Manage Ignore for the moment

43 Staff Skill Impact Report Explore what staff changes have the greatest impact

44 Key Point Modeling helps find what matters Fewer estimates required

45 In this demo Finding what matters most Manual experiments Sensitivity Testing Finding the next best 3 staff skill hires Minimizing and simplifying estimation Grouping backlog Range Estimates Deleting un-important model elements

46 Demo: Finding the Cost / Benefit of Outsourcing

47 Outsourcing Cost & Benefits Outsourcing often controversial Often fails when pursued for cost savings alone Doesn t always reduce local employment An important tool to remain competitive I.Q. has no geographic boundaries Many models Entire project Augmentation of local team

48 Build Date & Cost Matrix 1 x Estimates 1 x Staff Best Case 1.5 x Estimates 1.5 x Staff Midpoint 2 x Estimates 2 x Staff Worst Case Benefit = (Baseline Dev Cost New Dev Cost) - Cost of Delay + Local Staff Cost Savings

49 NOT LINEAR & NOT YOUR PROJECT $150,000 $100,000 $50,000 $- $(50,000) x Multiplier 1.5x Multiplier 2x Multiplier $(100,000) $(150,000)

50 In this demo Model the impact of various outsourcing models

51 New Project Rules of Thumb Cost of Delay plays a significant role High cost of delay project poor candidates Increase staffing some compensation Knowledge transfer and ramp-up time critical Complex products poor candidates Captive teams better choices for these projects NEVER as simple as direct lower costs!

52 Assessing and Communicating Risk

53 Speaking Risk To Executives Buy them a copy of Flaw of Averages Show them you are tracking & managing risk Do We are 95% certain of hitting date x With 1 week of analysis, that may drop to date y We identified risk x, y & z that we will track weekly Don t Give them a date without likelihood February 29 th 2013 Give them a date without risk factors considered To do the backlog of features, February 29 th, 2013

54 **Major risk events have the predominate role in deciding where deliver actually occurs ** We spend all our time estimating here Plan Performance Issues External Vendor Delay 1 2 3

55 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

56 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

57 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

58 Risk likelihood changes constantly 95 th Confidence Interval 1 2 3

59 Key Points There is no single release date forecast Never use Average as a quoted forecast Risk factors play a major role (not just backlog) Data has shape: beware of Non-Normal data Measurement Insight Decisions Outcomes : Work Backwards! Communicate Risk early with executive peers

60 Read these books Call to action Download the software FocusedObjective.com

61 Please Submit an Eval Form! We want to learn too!

62

63 BEST PRACTICES

64 Sensitivity Test Model (a little) The Model Creation Cycle Monte- Carlo Test Visually Test

65 Make Informed Decision(s) Baseline Compare Results The Experiment Cycle Make Single Change

66 Best Practice 1 Start simple and add ONE input condition at a time. Visually / Monte-carlo test each input to verify it works

67 Best Practice 2 Find the likelihood of major events and estimate delay E.g. vendor dependencies, performance/memory issues, third party component failures.

68 Best Practice 3 Only obtain and add detailed estimates and opinion to a model if Sensitivity Analysis says that input is material

69 Best Practice 4 Use a uniform random input distribution UNTIL sensitivity analysis says that input is influencing the output

70 Best Practice 5 Educate your managers about risk. They will still want a single date for planning, but let them decide 75 th or 95 th confidence level (average is NEVER an option)

71 SIMULATION EXAMPLES Return to main presentation

72 certain unlikely Forecasts Return to main presentation

73 certain unlikely Forecasts Return to main presentation 50% Possible Outcomes 50% Possible Outcomes

74 Return to main presentation Sensitivity Report Actively Manage Ignore for the moment

75 Return to main presentation Staff Skill Impact Report Explore what staff changes have the greatest impact

76 Return to main presentation

77 Focused Objective Risk Tools for Software Dev Scrum/Agile Simulation Kanban/Lean Simulation Forecasting Staff, Date & Cost Automated Sensitivity Analysis Data Reverse Engineering Consulting / Training Book

78 We Use & Recommend: EasyFit MathWave.com Invaluable for Analyzing data Fitting Distributions Generating Random Numbers Determining Percentiles

79 Free Images: MorgueFile.com Calendar: Calculator: Dice: