Driving software development through measurement Results from PwC s Software Measurement Study

Transcription

1 Driving software development through measurement Results from PwC s Software Measurement Study Technology Institute November 2014 At a glance Software has evolved to be central to most high tech products. Leading companies recognize that measurement of the R&D engine that produces that software is more critical than ever.

2 Introduction The performance of software development teams is an area of high interest for Technology executives especially with the increasing impact of software on the usability and functionality of high-tech products. Unlike hardware development, software is less amenable to the hard, quantitative metrics that enable side-by-side performance comparisons. The soft nature of software development makes concepts like content, time to market (TTM), and quality more ambiguous and, as a result, harder to measure. PwC s Software Measurement Study (SMS) was designed to analyze the current state of measurement and practices in software development environments. Thirty-one companies participated in the study, representing a range of industries, software development methodologies, and software domains. The study compared measurement practices to industry leading practices to provide an assessment of software measurement. PwC s Software Measurement Practices Maturity Model, shown in Figure 1, served as the framework for software measurement leading practices. Figure 1: 1: PwC s Software measurement practices practices maturity maturity model model Stage 4 Stage 3 Stage 1 Metrics are used in limited occasions and/or by few departments Financial or code-based productivity metrics are rarely used Limited tracking of time to market (TTM) and planned vs. actual effort Defect and test effectiveness measurement occurs rarely Testing staff allocation is high Stage 2 Metrics are primarily backward looking and used solely for planning Productivity is measured and benchmarked within departments or for projects in the organization Tracking of TTM and/or planned vs. actual occurs within a department or for specific projects Departments or projects measure defects and test effectiveness or employ automated tests Metrics are reviewed and published across the organization Quantitative productivity metrics are measured and benchmarked cross the organization TTM and plan vs. actual measurements are used in planning projects Organization-wide measurement of defects and test effectiveness. Staff allocation to testing is a priority Productivity metrics are used to drive improvements/set targets TTM and plan vs actual measurements drive continuous improvement behavior Optimal number of maintenance releases and prioritized staff allocation guides consistent software quality Metrics are tied to manager and developer incentives Ensure Metrics Usage Enable Process Improvements Drive Organizational Behavior

3 Study focus The study focused on the four main areas of software measurement shown in Figure 2: productivity, schedule, quality, and use of metrics. Key findings are included in the following section. Figure 2: 2: Software measurement measurement study focus study areas focus areas Financial and code-based measures and how they are used to assess productivity The use of schedule measurements, how time to market is measured and the freezing of requirements How software quality, defects, and test effectiveness are measured Productivity Schedule Quality Use of metric How do companies use metrics to drive behaviors and overall success Driving Software Development Through Measurement 1

4 Productivity Companies use both financial and code-based metrics to gauge software team productivity. In the area of financial metrics, most survey participants, as shown in Figure 3, favored R&D as a percentage of revenue. However, there are other financial metrics such as percentage of revenue from new product/services that should provide a better indication of software team productivity. In the area of code-based metrics, most participants favored easy, but vague metrics such as features and requirements delivered over more quantitative metrics like Function or Story Points (as shown in Figure 4). Furthermore, participants performed very little benchmarking internal or external for their code-based metrics. As a result, the value of tracking these metrics is low (a considerable percentage of companies (14%) said they do not use code-based metrics at all). Figure 3: Financial productivity metrics Percentage of survey participants selecting the metric R&D as a % of revenue New products/services revenue None product/service revenue Per employee product/service Revenue per developer 14% 11% 11% 32% 68% R&D as a % of revenue is the primary financial metric used to measure development productivity Figure 4: Code-based productivity metrics Figure 4: Code-based productivity metrics Percentage of survey participants selecting the metric Features Delivered Requirement Delivered Lines of Code None Story Points Delivered Function Points Other 15% 1 6% 26% 39% 48% Story points delivered and function points are pertinent (but time consuming) metrics to measure, however, less than 15% of companies do it 2 PwC

5 Schedule Measuring development schedules and TTM ensures more predictable releases and provides insight into development challenges. Despite these benefits, one-third of the companies surveyed, as shown in Figure 5, do not measure TTM. The remaining two-thirds measure the start of TTM as the point where the concept is defined and the end as the point where the release criteria have been met. While most participants measure TTM, half do not measure the predictability of their schedule forecasts. Similarly, half do not measure the predictability of the effort required to achieve those schedules. The lack of focus on tracking schedules and effort commitments is likely the cause of much of the unpredictability surrounding software development. Figure 5: Software TTM measurement practices Figure 5: Software TTM measurement practices Development process step percentage of survey participants that responded Breakdown of 3 of survey participants not measuring TTM Initial idea Concept defined 17% Design specification complete 7% Functional specification complete 14% Release to manufacturing 6% The most common measurement of TTM is from concept definition to the point at which the release criteria System test completion 6% have been met Release criteria completion Initial customer Release active customer use Post-implementation warranty complete 14% 25% 6% 14% 6% 14% Web IT Embedded DoD Application (Single User) Application (Distributed) All data warehouse Driving Software Development Through Measurement 3

6 Quality Software quality is an area where the opportunity for using metrics is large, but companies often focus on too few or too many metrics. Striking a balance on the right number and mix of metrics is essential. Two quality-related metrics in the study that were of particular interest focused on the allocation of staff. Figure 6 shows the percent of staff allocated to fixing defects. Most participants were in the range of 0 20%, but a significant number (~25%) reported that more than 40% of staff members were allocated to fixing defects. These staff members were not only working on non-revenue producing tasks, but were also fixing defects at the most costly point possible once it was in the field. The second metric related to staff allocation was the ratio of feature releases to defect fixing releases. Figure 7 shows the range of ratios provided by the survey participants. Companies with a ratio above five likely have quality problems both in their code bases and in their release criteria and are fixing those quality issues in the least cost-effective manner possible. Figure 6: Percentage of SW staff allocated to Figure bug 6: fixing Percentage of SW staff allocated to bug fixing Percentage of survey participants that responded Figure 7: 7: Number of maintenance of maintenance releases releases per year per feature year per release feature release Percentage of survey participants that responded 0 20% 56% 21 40% 20% 41 60% 61 80% % 12% 8% 4% % 8% 4% 4% 39% 50% Staff allocated to bug fixing are not working on new features and so are not contributing to creating new product revenue opportunities for the company PwC

7 Summary The results of the Software Measurement Study show that software development is still a practice where a culture of metrics has not been implemented at most companies. Many participants are not using metrics to effectively drive the performance of their software teams. Figure 8 shows that only about half of those surveyed link manager compensation to measurable performance targets, and roughly a quarter link developer compensation to similar targets. A structured approach to designing and implementing a balanced set of software performance metrics may provide software development organizations with significant benefits. Figure 8: 8: Use of of metrics Metrics are reviewed at a standard cadence Explicit metrics targets are set and visible to the organization Managers have financial benefits tied to hitting metric targets Developers have financial benefits tied to hitting metric targets None of the above Other 28% 50% 52% 60% Driving Software Development Through Measurement 5

8 PwC can help A balanced set of software performance metrics provide software development organizations with significant benefits. Is your company prepared? For a deeper software measurement discussion, please contact one of our practice leaders: Ari Shinozaki Technology Industry Principal ari.shinozaki@us.pwc.com Walter Sun Technology Industry Principal walter.c.sun@us.pwc.com Let's talk Please reach out to any of our technology leaders to discuss this or other challenges. We re here to help. Tom Archer US Technology Industry Leader thomas.archer@us.pwc.com Kayvan Shahabi US Technology Advisory Leader kayvan.shahabi@us.pwc.com Cory Starr US Technology Assurance Leader cory.j.starr@us.pwc.com Diane Baylor US Technology Tax Leader diane.baylor@us.pwc.com Acknowledgements The following PwC professionals contributed their experience and knowledge to produce this paper. Bill Noel Technology Industry Director bill.noel@us.pwc.com About PwC's Technology Institute The Technology Institute is PwC s global research network that studies the business of technology and the technology of business with the purpose of creating thought leadership that offers both fact-based analysis and experience-based perspectives. Technology Institute insights and viewpoints originate from active collaboration between our professionals across the globe and their first-hand experiences working in and with the technology industry. For more information please contact Tom Archer, US Technology Industry Leader PricewaterhouseCoopers LLP, a Delaware limited liability partnership. All rights reserved. PwC refers to the US member firm, and may sometimes refer to the PwC network. Each member firm is a separate legal entity. Please see for further details. MW MZ This content is for general information purposes only, and should not be used as a substitute for consultation with professional advisors. PwC US helps organizations and individuals create the value they re looking for. We re a member of the PwC network of firms in 158 countries with more than 180,000 people. We re committed to delivering quality in assurance, tax and advisory services. Tell us what matters to you and find out more by visiting us at 6 PwC