Measures to get the best performance from your software suppliers

Transcription

1 Measures to get the best performance from your software suppliers Charles Symons Co-Founder & Past President, COSMIC ISMA 9 Conference, 27 th March, ITMPI005

2 Aims of this talk* What is publicly known about software industry performance? Why do we get this performance? How might customers use software contracts and metrics to get better performance? * For more see Software industry performance: what you measure is what you get. IEEE Software, November/December

3 First, what is the software industry? Very difficult to define a few % of world GDP I have no data from organizations that do not usually publish their performance, e.g. Major hardware and packaged software suppliers (MS, Oracle, Google, SAP, IBM, Apple, etc) Much real-time software (defence & transportation systems, appliances, etc.) and for games I include data from publicly-available sources: Business application and real-time software Typical average performance, not exemplars 3

4 Performance is a matter of balance Delivery to budget & time: Actual vs. Estimated Cost Actual vs. Estimated Duration Project productivity Size / Effort Project speed Size / Duration Product quality Post-delivery defects / Size 4

5 Performance varies by parameter - and is not commonly measured Delivery to budget & time: Project productivity Continues to be terrible No sign of improvement 4 Project speed Product quality Usually OK In some domains amazingly good Increasing 4.! 5

6 The cost of project over-runs is a public scandal Study No. of Cost Over-runs/ Country Projects Write-offs UK Public Sect B 9B (31%) Mostly US $ 146B $ 45B (27%) Annual cost of failures and over-runs: US market 1 ~100 Billion US$ European market 2 ~100 Billion N.B. The cost of failures is usually mainly hardware The cause of failures is usually problems with software 6

7 Meanwhile: most software suppliers make nice profits The world-class software suppliers profit margins on the UK contracts 3 : 7 26 % It s very rare for a major software supplier to go bankrupt But in-house suppliers who are believed to under-perform tend to get outsourced to the major suppliers! 7

8 UK Government frustration has led to a poor ICT Strategy 6 If possible avoid projects > 100M? Break the oligopoly of large suppliers. More work for small & medium enterprises? Adopt Agile methods Improve public sector ability to manage ICT projects / Use measurements & benchmarks to improve performance X But no action! 8

9 The UK Government uses measurements to drive performance improvement in Crime Transport Health Education Why not software system delivery? Charles Symons

10 Agenda What is publicly known about software industry performance? Why do we get this performance? How might customers use software contracts and metrics to get better performance? 10

11 Some of the main reasons are obvious (High) Quality No choice (for mission-critical & customerfacing systems) Supplier aim: to get paid (Poor) Delivery to time & budget Changing requirements Poor estimating & project management... in spite of ~30 years of process improvement! 11

12 Traditional software contracting is flaky, especially early in a project Customer produces outline requirements & asks for estimates Supplier measures requirements using a simple sizing method Supplier estimates cost & time from past experience and adds contingency Supplier is selected based on total cost and other qualities 12

13 ... and goes from bad to worse Customer commits and then has to add & change requirements Supplier now has excuse to try to re-negotiate cost & time estimates (or) De-scope Supplier focuses on meeting finallyagreed requirements to ensure payment 13

14 ... so the final result? Delivery to budget & time: Over-run, run, as usual, for which the customer pays Project productivity Customer has no idea, (and learns nothing) Project speed Product quality Customer has no idea OK. (Defects are (and learns nothing) corrected during Warranty) 14

15 Conclusion: the traditional contracting process is seriously flawed Customers reasonably want an early estimate of total cost, but an accurate estimate is impossible from outline requirements Estimating tools are only as good as their input Agile approaches can help, but do not meet all the customer s needs, especially for large projects 15

16 Agenda What is publicly known about software industry performance? Why do we get this performance? How might customers use software contracts and metrics to get better performance? 16

17 Imagine buying vegetables in this market $2 $5 $8 Customer: I want some onions for a soup Vegetable seller 1: Onions are $5 today Vegetable seller 2: Onions are $8 today Vegetable seller 3: Onions are $4 today... 17

18 Key elements for software customers to re-balance risk Contract on a price-per-unit size basis using: appropriate software sizing method(s) open project estimating methods that the customer can understand independent scope management 18

19 The key difference is to contract with an agreed unit price Customer produces outline requirements & asks for estimates Supplier measures requirements using appropriate sizing method Supplier: estimates total size (including contingency) x fixed unit price = Total price, and estimated duration Supplier is selected based on unit price and other qualities 19

20 As requirements change, the unit price stays fixed* Customer commits to the unit price and then adds/changes requirements* Agile activities can start here Total cost varies with size**. Duration may need to be re-estimated Final total price = Final size** x unit price * The unit price may increase for very late changes ** Sizes may be measured by an independent scope manager 20

21 The process helps balance the customer -supplier relationship Customer Takes risk on total size of requirements & changes Supplier Takes risk on unit price, quality & delivery Shared Understanding of sizes & estimates Knowledge of productivity & speed 21

22 And the benefits of Scope Management processes? Finnish experience 7 Project Mgmt. /FP FP/ Approach Month Traditional (107) >500 <50 Scope mgmt. (6) <250 >100 and improved estimating accuracy Similar results reported from Australia, Poland 22

23 Renault: Scope Management processes = major benefits 8 Software embedded in automotive Electronic Control Units Requirements stored in modelling tools Software sizes measured automatically Past data used to predict software development & enhancement costs and memory requirement Renault invites bids from software suppliers knowing what to expect 23

24 Conclusion: Customer/Supplier relationships must be re-balanced Suppliers deliver amazing software products and enormous benefits, but do not deliver reliably, or demonstrate good value for money Customers must use advanced processes, & measurement methods to get a better deal 24

25 The approach is powerful but not easy, and needs patience Requirements for success A good understanding of software size measurement, its uses, and of estimating Good quality measurements of size, effort, and project characteristics A lot of data to understand what drives performance in your environment Strong commitment from senior management There are no silver bullets, but the potential ROI is huge 25

26 Postscript: What does business want from new IT/computer science graduates? 9 Top of the list of missing skills: Project planning and management Metrics, estimating, statistical analysis Presentations 26

27 Thank you for your attention Questions? 27

28 References 1. Standish CHAOS Report, 2009, 2. McManus, J. and Wood-Harper, T., A Study in Project Failure, June Whitfield, D., Cost over-runs, delays and terminations: 105 outsourced public sector ICT projects, European Services Strategy Unit, Research Report No. 3, December The SIP (Software Industry Performance) Report and ISBSG benchmark data, 5. Why your IT project may be riskier than you think Bent Flyvbjerg, Alexander Budzier, Harvard Business Review, September Government ICT Strategy, Cabinet Office, March 2011, 7. Validation of the Northern Scope concept for managing the sourcing and development of software, Forselius, P., Kakola, T., ICSE (See also Southern SCOPE: avoiding software budget blowouts, the Government of the State of Victoria, e-government Resource Centre, ) 8. Practical experimentations with the COSMIC method in automotive embedded software field, Stern, S., International Workshop on Software Measurement, November (UK) Science & Engineering Council 2013 Information on the COSMIC method and its uses is available for free download from 28