Software Quality - Getting Right Metrics, Getting Metrics Right

Transcription

1 Software Quaity - Getting Right Metrics, Getting Metrics Right How to set the right performance metrics and then benchmark it for continuous improvement? Whie metrics are important means to quantify performance of a process, we often define metrics to compy with process standards, overooking actua aignment with business factors. We may put substantia effort and discipine in getting metrics depoyed, but we might not be using the ones that reay matter. How can we define metrics that remain anchored to business needs? In other words, how can we transate business requirements to the right set of metrics? Once the right metrics are defined, how can we monitor and benchmark it to contro performance and make breakthrough improvements? This white paper discusses best practices around metrics, from its very definition to using it for continuous improvement.

2 About the Authors Dr. Kanhaiya Jethani Kanhaiya is a senior Process and Quaity consutant at Tata Consutancy Services. He has extensive experience in the fieds of software project management, software process and quaity, CMM and CMMI. He has ed and executed process and quaity consuting engagements for various IT organizations in USA, Spain, Poand, Taiwan, Singapore, Japan and India. He is a Certified Software Quaity Anayst (CSQA), Project Management Professiona (PMP) and Candidate SCAMPI Lead Appraiser. A Doctorate in Chemica Engineering from Okahoma State University, USA, he was aso a Post-Doctora Research Feow at Okahoma State University, USA and University Institute of Chemica Technoogy (UICT), Mumbai. He is a member of PMI. Vanishri Veoo Vanishri is a senior Process and Quaity consutant at Tata Consutancy Services. She is a Candidate Lead Assessor for Peope CMM, a quaified auditor for ISO 9001:2000, a trained assessor for CMM and SCAMPISM and an Externa Assessor for Tata Business Exceence Mode (TBEM). She has ed and executed process and quaity consuting engagements for various IT organizations in USA, UK, Germany, The Netherands, Spain, Taiwan and India. Vanishri has worked in the financing of renewabe energy, e-business and financia services industry before moving to the Information Technoogy Industry. Vanishri is a graduate in Biomedica Engineering from Mumbai University, India and has done her Masters in Management Studies (M.M.S.) in Finance from the Sydenham Institute of Management Studies, Research & Entrepreneurship Education, Mumbai University, India. Pragnya Misra Pragnya Misra is a senior Process and Quaity consutant at Tata Consutancy Services. She has extensive experience in the fieds of appication deveopment and maintenance, software project management, software process and quaity, CMM and CMMI. She has ed and executed process definition and depoyment engagements for various customers across the word. She is a Certified Software Quaity Anayst (CSQA).She is an SEI authorized assessor for CMM and CMMI and aso ISO-9000 Interna Auditor. She is a Masters in Business Administration (Finance) and a Bacheor of Engineering (Instrumentation and Contro) from Gujarat University, India. 1

3 Tabe of Content 1. Introduction 3 2. Ask the Right Question 3 3. GQM Keeps Us Anchored to Changing 5 Business Imperatives 4. Using SPC to Contro Process Performance 5 5. Continuous Improvement Needs 7 Breakthroughs where it Matters 6. Concusion 9 2

4 Introduction Pursuit of process frameworks and quaity standards ike CMMI or ITIL has brought more discipine and maturity to software deveopment and IT services. The fip side is that we go overboard with a pethora of metrics, which often are dictated by our preoccupation for compiance to such frameworks. With itte aignment to the business-needs, many of these metrics can turn rituaistic and time consuming, et aone the questionabe sanity of the data. For exampe, whie we may remain compacent with ow defect eves; we may ignore a high defect concentration in a critica modue or the severity of defects. Many IT organizations have a metrics handbook that suggests the set of metrics to be foowed. If such a handbook prescribes a set of mandatory metrics, we woud ike such metrics to be rather sma in number. A arger set of metrics can remain optiona, eaving it to the discretion of the project. Now the question is - how can one decide what woud be the right metric. It is the question of finding the right metric first before getting the metric right; the irony that comes out of Joseph Juran s adage do right things before doing things right. Ask the Right Question Consider this: a modue in software was found to have abnormay high defects. The metric that was cosey monitored was the defect density - defects per unit functionaity (for exampe defects per function point). This modue was escaated for re-engineering. On inspection of the design, it was found to be good in terms of maintainabiity and defect prevention. Questioning the rationae for re-engineering, the defects were revisited. It was found that surprisingy, the defects were more in number but ow in severity, being concentrated mosty on output formats. Unfortunatey, the severity of the defects was not factored in the defect density metric. Metrics wi remain meaningess ceremonies uness they are aigned to business goas. In this exampe, the business goa was to reduce injection of defects critica to the quaity of the product. Whie we inherit or improvise the metrics that woud support the monitoring of the project, it is essentia to be anchored to the business needs. In fact, the business goa shoud transate to the right set of metrics required to monitor and contro quaity. A three step process, Goa-Question-Metric (GQM), woud be very handy here: It is a way to identify the right metrics by asking questions around the business goa. GQM heps transate the business needs from conceptua eve to the operationa eve by asking appropriate questions, thereby identifying the measures that can provide objective insight into the achievement of the business goas. For exampe, if our goa is to improve estimation, what questions woud we have in mind? We woud most probaby ask how good the current estimation method is. Effort overrun woud then be a good metric to indicate the accuracy of our estimation method. 3

5 Conceptua Leve Estabish measurement goas aigned with business needs i.e. what we want to achieve Operationa Leve Generate a set of questions that can be used to determine achievement of goas Quantitative Leve Estabish the set of data, objective or subjective, that wi answer each question in a quantitative way Figure 1: Transating business needs at conceptua to operationa and quantitative parameters Goa Approach Question Metric Improve Estimation - Estimating effort based on requirements - Panning based on estimation How good is effort estimation? Effort Sippage Minimize Defects - Detecting defects as eary as possibe - Minimize acceptance testing defects What are current defect eves? Defect Density Reduce Costs - Identifying non-- vaue added activities - Detecting defects eary - Reducing rework What is the rework Effort? Rework Effort 4

6 GQM Keeps Us Anchored to Changing Business Imperatives A arge project cacuated biabe vaue in terms of the project effort expended unti the time of costing. However, the vaue in terms of project deiverabes and miestones met were found to be esser than the effort spent. This was primariy due to schedue sippages at certain phases in the project. Due to the enormity of the project, the actua vaue deivered during the project remained confusing. In this case, the business need was to deiver the miestones as per schedue. The question that comes to mind is what miestones have been deivered and what was the corresponding estimated effort? This suggested tracking the deivered vaue in terms of Earned Vaue. Earned vaue is the vaue of the deiverabes made in terms of estimated effort, irrespective of the sippages. This gave the project a better understanding of how much of true vaue is being deivered by the project at a given point in time, vis- a- vis the tota effort spent. Now, when we have seected the right metric for monitoring project or process performance, how do we ensure that the performance suggested by the metric is within acceptabe imits? Or, how do we know that the project wi be abe to deiver the deiverabe as per pan? Statistica Process Contro (SPC) is a technique that is usefu in this. It defines the operationa imits for acceptabe variation in the process data. Using SPC to Contro Process Performance Statistica Process Contro primariy foows four steps. These are: Measuring the process Reducing variation to acceptabe imits Monitoring and controing Improving the process to achieve target vaue Whie many toos are used in process improvement, with each being appicabe to specific contexts, we wi discuss the use of Contro Charts to contro process variation. Contro chart is potting of the process parameter over time to identify causes of any abnorma variation. Any parameter woud have some variation that woud be confined to an acceptabe or norma range. An instance of the high variation woud suggest an underying assignabe or specia cause, which woud then have to be eiminated. Normay, an instance of variation ying outside the range of u + 3s (u is the mean and s the standard deviation) is considered to be abnorma or outier. The imits u + 3s and u 3s are referred to as Upper Contro Limit (UCL) and Lower Contro Limit (LCL) respectivey. Processes having variation within the range LCL-UCL are considered stabe. 5

7 Abnorma variation due to assignabe sources Out of contro UCL 1000 Mean Norma variation due to chance Abnorma variation due to assignabe sources LCL Sampe number Figure 2 LSL USL Process Capabiity Band LCL UCL 3s 3s The reason we chose to discuss contro chart in this paper is that we find the technique usefu in gauging the capabiity of the process, than just the stabiity. Besides UCL and LCL, the process can set its own imits (target for process performance). This is referred to as Lower Specification Limit (LSL) and Upper Specification Limit (USL). A capabe process woud have a variation we within the LSL-USL range. The LCL-UCL of a capabe process is caed Process Capabiity Baseine (PCB). As the process matures to be more capabe, the USL-LSL range is shortened with a more tapered peak ( high Kurtosis ) at the mean. 6

8 The be-curve becomes: Taer Thinner with more process improvements Improvements Further Improvements Time X Time X Time X Capabiity Over Time Continuous Improvement Needs Breakthroughs where it Matters Continuous improvement essentiay invoves improving the performance of the process so that the specification imits are satisfied and the process becomes more capabe. Once the specification imits are satisfied, tighter specification imits or stretch targets are defined and another cyce of continuous improvement is initiated. We ca it a continuous improvement journey because the improvement actions are not warranted by exceptions or specia causes, but are more proactive in nature. It addresses the common causes, which are inherent in the nature of the process. In the context of contro charts, capabiity improvement in processes requires identifying the right breakthroughs, which woud shift the mean and reduce variation to make it centered on the mean. In simper words, this means reducing variation to the extent that it improves end quaity and shifts the mean to the desired specifications. Often, we reduce variation on a parameter even when the process is robust with the existing eve. For exampe, we may go overboard on improving the performance variation of a transaction when its highest variation is we accommodated in the actua transaction sequence. Rather, the business objective woud be to reduce the mean time taken to perform the series of reated transactions in the practica business scenario. Reduction of the mean woud require a breakthrough action or process change. This is because the od process may have aready hit its maximum performance state given the existing process definition. To improve from here, one woud need to introduce new interventions or activities within the process. For exampe, we may often find that, despite sufficient testing, there are a high number of defects during production. A process ateration such as peer review of source code can hep in detection of potentia defects. 7

9 As depicted in the foowing figure, breakthrough improvements wi ead to change in the mean performance of the process. Origina zone of Quaity contro Chronic waste New zone of Quaity contro Less chronic waste Breakthrough Improvement To summarize, quaity improvement starts with identifying the right metrics. We appy techniques such as contro charts to improve the stabiity of the process. Thereafter, we embark on a continuous improvement journey by increasing the capabiity of the process. A through, a fact-based approach is needed. The typica approach is iustrated as foows: Business Objectives Sub-process Seection Project-eve Change Process Change Pioting Measure Remove Specia Causes Remove Common Causes Org. Change Process/ Technoogy Change No No Statistica Anaysis (OPP) Process Stabe? Yes Process Capabe? Yes 8

10 Concusion A quantitativey managed process woud have metrics aigned to business needs. Such needs are better understood when we define the business goas. One can transate the goas to a set of representative metrics by asking the right questions, which is an important part of the exercise. The GQM approach heps in this transation. Once the right metrics are defined, SPC heps in monitoring the performance of the metrics and bringing the process under contro. We have discussed contro chart as one of the toos that is usefu here. As we move into the continuous improvement journey, we try to reduce variance where it matters and find breakthroughs/ process changes to shift the mean. This makes the process more capabe. At the end of the day, a a good process needs is the right metrics used to monitor it and good benchmarks set to contro its performance and continuousy improve its performance. 9

11 About Tata Consutancy Services (TCS) About TCS IT Process and Service Management Consuting Tata Consutancy Services is an IT services, business soutions and IT Process and Service Management Consuting partners with outsourcing organization that deivers rea resuts to goba organizations to enabe IT process improvement aigned to businesses, ensuring a eve of certainty no other firm can match. business needs. Leveraging TCS' experience in successfuy TCS offers a consuting-ed, integrated portfoio of IT and IT- adopting best-incass frameworks ike CMMI, ISO-9001, ITIL, enabed services deivered through its unique Goba Network P-CMM, BS15000, BS7799, and Six Sigma, the consuting group Deivery ModeTM recognized as the benchmark of exceence in provides services with a practitioner s approach to assessing, software deveopment. defining, improving and depoying IT processes. CMMI appraisa services and training A part of the Tata Group, India's argest industria congomerate, ITIL depoyment and assessment TCS has over 100,000 of the word's best trained IT consutants in 50 Agie deveopment for distributed environment countries. The company generated consoidated revenues of US Six Sigma-based improvements $5.7 biion for fisca year ended 31 March 2008 and is isted on the Quaity improvement initiatives Nationa Stock Exchange and Bombay Stock Exchange in India. For Digitized enabers which reduce 'Time to Depoy' and sustain more information, visit us at productivity for process goba.consuting@tcs.com A content / information present here is the excusive property of Tata Consutancy Services Limited (TCS). The content / information contained here is correct at the time of pubishing. No materia from here may be copied, modified, reproduced, repubished, upoaded, transmitted, posted or distributed in any form without prior written permission from TCS. Unauthorized use of the content / information appearing here may vioate copyright, trademark and other appicabe aws, and coud resut in crimina or civi penaties. Copyright 2008 Tata Consutancy Services Limited