Measuring for Results: Metrics and Myths Peter Hundermark Certified Scrum Coach and Trainer ScrumSense 1
Project Success Rates Succeeded Challenged Failed 44% Late Over budget Missing features On time On budget Required features 32% 24% Source: Standish Group CHAOS Summary 2009 Cancelled Never used 2
Agile is Value-Driven Predictive Adaptive Constraints Requirements Cost Schedule Plan Driven Value Driven Estimates Cost Schedule Features 3 Cost
What is Valued? 64% rarely or never used 19% 16% Never Rarely Sometimes Often Always 45% 13% 7% 20% frequently used Source: Standish Group (2002) 4
Value of Time Incremental delivery Traditional single delivery Real Value Delivered / Earned Delivery 2 Delivery 2 Delivery 3 Delivery 4 Delivery 1-5 Delivery 5 Value gap Time 5
knowledge workers measure themselves 6
Tell me how you will measure me and I ll tell you how I will behave. 7 Eli Goldratt The Goal (2004)
On the Folly... We hope for Long-term growth Commitment to quality Teamwork and collaboration Innovative thinking and risk taking Employee involvement and empowerment Candour: surfacing bad news early But we often reward Quarterly earnings Shipping with defects Individual effort Proven methods and not making mistakes Tight control of operations and resources Reporting good news, whether it s true or not Agreeing with the boss, whether or not (s)he s right! Adapted from Steven Kerr On the folly of rewarding A, while hoping for B, Academy of Management Executive (1995 vol 9 no 1) 8
Metrics and Myths Level of performance Measurement indicators True performance Time Source: Robert Austin, Measuring and Managing Performance in Organisations (1996) 9
Good Agile Metrics 10
Good Agile Metrics Reinforce desired behaviour 10
Good Agile Metrics Reinforce desired behaviour Measure results not output 10
Good Agile Metrics Reinforce desired behaviour Measure results not output Measure trends 10
Good Agile Metrics Reinforce desired behaviour Measure results not output Measure trends Vital few 10
Good Agile Metrics Reinforce desired behaviour Measure results not output Measure trends Vital few 10 Easy to collect
Good Agile Metrics Reinforce desired behaviour Reveal their context Measure results not output Measure trends Vital few Easy to collect 10
Good Agile Metrics Reinforce desired behaviour Reveal their context Measure results not output Fuel meaningful conversation Measure trends Vital few Easy to collect 10
Good Agile Metrics Reinforce desired behaviour Reveal their context Measure results not output Fuel meaningful conversation Measure trends Amplify learning Vital few Easy to collect 10
Good Agile Metrics Reinforce desired behaviour Reveal their context Measure results not output Fuel meaningful conversation Measure trends Amplify learning Vital few May measure value or process Easy to collect 10
Good Agile Metrics Reinforce desired behaviour Reveal their context Measure results not output Fuel meaningful conversation Measure trends Amplify learning Vital few May measure value or process Easy to collect Good enough quality 10
Leading and Lagging 11
Leading and Lagging Leading indicator Signals future events Changes before the event E.g. amber traffic light 11
Leading and Lagging Leading indicator Signals future events Changes before the event E.g. amber traffic light Lagging indicator Reports past events Changes as an outcome E.g. unemployment 11
Measurement Dimensions Value (to Customer) Predictability (Schedule) Collaboration (Process) Quality (Product) 12 Source: Pete Behrens, Measuring Agility - Top 5 Metrics and Myths (2009)
Customer Surveys Many opportunities over time Baseline and measure quarterly Qualitative and quantitative Responsiveness On-time delivery Value Quality Support Source: Pete Behrens, Measuring Agility - Top 5 Metrics and Myths (2009) Lagging Value Metric 13
Value Customer Surveys Lagging Ask: How likely are you to recommend this product or service to a friend? Scale 0-10 15 9-10! Promoters 7-8!Passive 0-6!Detractors 5 8 10 12 Net promoters score (promoters - detractors) sample size 1 1 1 2 3 2 0 1 2 3 4 5 6 7 8 9 10 Adapted from Sanjiv Augustine and Roland Cuellar, Agile Metrics for Senior Managers & Executives (2009) 14
Value Customer Surveys Lagging Ask: How likely are you to recommend this product or service to a friend? Net promoters score: (27-14) / 57 = 23% 27 Scale 0-10 18 15 9-10! Promoters 7-8!Passive 0-6!Detractors 14 5 8 10 12 Net promoters score (promoters - detractors) sample size 1 1 1 2 3 2 0 1 2 3 4 5 6 7 8 9 10 Adapted from Sanjiv Augustine and Roland Cuellar, Agile Metrics for Senior Managers & Executives (2009) 14
Predict Velocity Leading 40 30 Velocity 20 10 0 1 2 3 4 5 6 7 8 Sprints 15
Myth Myth Higher velocity is always a good thing Truth Risk of incurring technical debt Do Add technical debt to product backlog Monitor and reduce over time Cumulative Cost Time Optimal Cost Actual Cost 16
Velocity measures productivity or value Truth Story points are relative Cannot compare velocity Teams are different Size measures complexity Do Use the metric as designed Myth Source: Pete Behrens, Measuring Agility - Top 5 Metrics and Myths (2009) 17
100% Committed vs. actual drives estimation accuracy 40 Myth Truth Teams learn to be safe Leads to lower productivity Better Velocity 30 20 10 What features did we deliver? Why did we miss one? What is the impact? 0 1 2 3 4 5 Sprints! Learning Committed Actual 18
Sprints fail Myth Truth Less probable events! more information Maximum information at 50% probability Knowledge from low-probability events Source: Donald Reinertsen Managing the Design Factory: A Product Developer s Toolkit (1997) 19
Rate of Features Delivered Product or Release Burnup / Burndown Chart Story Points 300 250 200 150 100 50 Leading 0 0 1 2 3 4 5 6 7 8 Sprints Predict Metric Story points delivered Story points remaining 20
Leading Predict Quality Metric Source: Ron Jeffries, http://xprogramming.com/xpmag/jatrtsmetric (2004) Running Tested Features / Running Automated Tests 21
Technical Debt What is technical debt? Why does it occur? Leading Quality Metric Add stories to backlog Team is responsible! 22
Work-in-process Story ToDo WIP Done Leading Collab. Metric Stories in-process Aim for " 2 No silos or blockages! 23
Story Cycle Time Story ToDo WIP Done Leading Collab. 30 Metric Story Points 20 10 Average time to complete Aim for around 3 days No lagging tests! 0 1 2 3 4 5 6 7 8 9 10 Days 24
Basic Agile Metrics Value Predicability Customer surveys Velocity Burnup / Burndown Story cycle time RTF / running automated tests Collaboration Work-in-process Technical debt Product Quality 25
Extending the Metrics Value Predicability Customer surveys Velocity Burnup / Burndown Story cycle time RTF / running automated tests Collaboration Work-in-process Technical debt Product Quality 26
Extending the Metrics Value Predicability Customer surveys Velocity Burnup / Burndown Story cycle time RTF / running automated tests Collaboration Work-in-process Technical debt Product Quality 26
Extending the Metrics Value Predicability Customer surveys Velocity Burnup / Burndown Team surveys Story cycle time RTF / running automated tests Work-in-process Technical debt Collaboration Product Quality 26
Extending the Metrics Value Customer surveys Velocity Predicability Cost per sprint / point Burnup / Burndown Team surveys Story cycle time RTF / running automated tests Work-in-process Technical debt Collaboration Product Quality 26
Extending the Metrics Value Predicability Customer surveys Real value delivered Velocity Cost per sprint / point Burnup / Burndown Team surveys Story cycle time RTF / running automated tests Work-in-process Technical debt Collaboration Product Quality 26
Extending the Metrics Value Predicability Customer surveys Real value delivered Velocity Cost per sprint / point Team surveys Story cycle time ROI / NPV Burnup / Burndown RTF / running automated tests Work-in-process Technical debt Collaboration Product Quality 26
Puzzles? Metrics and Myths 27
Peter Hundermark Copyright in these slides is owned by Peter Hundermark and Scrum Sense CC. Some content and images may the copyright of others. Further information on Scrum training and coaching is available from: peter@scrumsense.com www.scrumsense.com ScrumSense 28