Transactions on Information and Communications Technologies vol 11, 1995 WIT Press, ISSN

Transcription

1 Impact analysis of process change proposals* M. Host and C. Wohlin Department of Communication Systems, Lund University, PO Box 118, S Lund, Sweden Abstract Before software processes are changed in order to achieve improvements, an impact analysis can be performed to predict the impact of the proposed change on specific quality attributes, such as product reliability, time to market and development productivity. In this paper an impact analysis method is presented. It consists mainly of a prediction system, which based on a proper data collection can be used to derive the impact prior to making any changes. 1 Introduction Life is evolutionary and changes occur over time. Most persons strive to turn changes into improvements, for example when buying a new house, we make a very thorough impact analysis, which in turn is used to evaluate costs and benefits. The impacts when buying a house may be in terms of travel time to work and closeness to recreation areas. This type of evaluation must be made when developing software as well. The process of developing software can not be changed arbitrarily, it must be a conscious and well founded decision. Therefore impact analysis of software process change is needed as support in decision making. * This work is a part of the PERFECT project and it is sponsored by the Swedish National Board for Industrial Technical Development (NUTEK). PERFECT stands for Process Enhancement for Reduction of software defects and it denotes the ESPRIT project 9090 funded by the CEC in which the following organizations participate: CAP Gemini Innovation, Daimler Benz, LGI, Q-Labs, Robert Bosch, Siemens Norway, and University of Kaiserslautern. In PERFECT, the Department of Communication Systems at Lund University is a subcontractor of Q-Labs.

2 312 Software Quality Management The objective of the work presented subsequently is to provide a method to perform impact analysis of a software process change. The method supports impact analysis of: reliability (this must include both reliability* and the fault content in the software), cycle time (duration of a software development project) and productivity (mainly in terms of effort, which must be related to the product being developed in order to achieve productivity). The method is denoted RePlaCe, based on the three key attributes to analyse and also based on the wish to replace old techniques with new and better ones. Impact analysis provides the basis for cost/benefit analysis and hence also return-on-investment models. The objective is to provide management as well as software developers with a basis for decision making regarding a specific proposed process change. 2 Process Improvements The quality of the products developed by an organization depends on the quality of the development processes used by the organization. Therefore, to improve the quality of the products developed by an organization, the processes that are used to develop the products can be improved. Gained experiences in combination with public domain information can be used to improve the software processes, with respect to reliability, productivity, development time, predictability, etc. It is not advisable to become religious about a specific method or tool, instead a thorough impact analysis which forms the basis for cost/benefit analysis is needed and it must be combined with return-on-investment. No sensible manager would allow changes without some proof of expected success. The proof can not be based on published results as most figures presented in the software engineering literature is non-repeatable, which is quite different from most experiments etc. presented in other sciences. Therefore, a method which is able to predict the effect of a proposed change is a necessity in software engineering. This paper focuses on providing such a method. The impact analysis method is in an early version and further work is required, but it will already, in its current status, provide managers with support in their difficult task of planning an improvement programme. In an improvement programme, the best change proposals should be introduced into an organization. Improvements can be introduced in a number of different ways, such as directly if the result of the impact analysis points out a great certainty of actual improvement or in smaller steps if the impact analysis reveals a considerable uncertainty of actual improvement. Another way to learn more about actual impacts of a process change proposals is to run test projects. 1. We define reliability as "the probability of a device performing its purpose adequately for the period of time intended under the operating conditions encountered". The reliability is often expressed in terms of Mean Time Between Failures (MTBF). For a more informal definition of reliability, see for example [Sommerville92].

3 Software Quality Management PERFECT The PERFECT project is an ongoing ESPRIT project with the overall goal to assist industry in establishing measurement-based initiatives aimed at evolutionary improvement of software development processes relative to companyspecific quality goals. The project will result in both a methodology and a platform to support systematic process improvement. The PERFECT methodology provides an organization with systematic and continuous improvement of software processes, based on the Experience Factory and the Quality Improvement Paradigm (EF/QIP) [Basili93b] and three key technologies: Explicit modelling of processes, products and quality. Goal oriented measurement. Comprehensive reuse of experience. The QIP divides the improvement process into six steps, which can be iterated to continuously improve the software processes. The six steps can briefly be described as: 1. Characterize: Characterize the current processes to identify improvement needs. 2. Set goals: At this stage goals are set. It is important that the goals are realistic and measurable. 3. Choose process models: Based upon the characterization and the goals set appropriate process models are chosen. 4. Execute the processes: While a process is executed, data should be collected according to a measurement plan for real time feedback and corrective actions. 5. Analyse data: The collected data is interpreted and the process is evaluated according to the goals that were set. 6. Package experience: According to the experience gained by executing the process, models must be defined and refined for reuse in future iterations of the improvement cycle. The proposed impact analysis method, which is one technique of the PERFECT methodology, can serve as an important technique to determine if goals are realistic and if suggested process models will meet certain goals.

4 314 Software Quality Management 4 Impact analysis 4.1 Introduction As stated above, an impact analysis can be carried out to evaluate change proposals. This can either be done in a simple and basic way or in a more thorough and advanced way. A basic method can rely on expert statements of the overall impact, while an advanced method must rely on mathematical models, such as, in this case, models of reliability, productivity and cycle time, in combination with expert statements. To carry out an advanced impact analysis, the impact of the proposed change on the sub-processes of the process must be considered to derive an overall prediction of the impact. As afirststep, a basic impact analysis method can be carried out. The basic method naturally results in one of three different decisions: go, if it is obvious that the change proposal results in an improvement, no go, if it is obvious that the change proposal does not result in any improvement, or investigate further, if the overall impact of the change proposal should be further investigated. If the basic impact analysis results in the decision investigate further, then an advanced impact analysis must be carried out to find the overall impact of the change proposal. The latter outcome is unfortunately very common as it is hard to determine all implications of a specific process change. The advanced impact analysis results in one of two different decisions: go, if the change proposal turns out to result in an improvement, or no go, if the change proposal turns out not to result in any improvement. As is illustrated in figure 1, impact analysis is carried out when process models are chosen, i.e. in step 3 of the QIR To illustrate the position of the impact analysis, the third step is divided into two parts: propose change and decide upon models, where the impact analysis is carried out between these two activities. If the impact analysis results in a no go there are two different possible reasons for this: the goals may be unrealistic or the suggested process change will not meet the goals. If the impact analysis results in a go, then the next step is to introduce the change proposal as a part of an improvement programme. The accuracy of the predictions made in the impact analysis, can help to decide how to introduce the change. The change has to be systematically introduced and carefully planned. To learn more about the impact of the change in the current organization, test projects can be executed and the improvement can be introduced step by step. In the rest of this paper we will concentrate on the advanced impact analysis method. In section 4.2 we introduce RePlaCe, an advanced impact analysis method, in section 5 we describe how to collect data to this method and in

5 Software Quality Management 315 QIP Characterize Step 1 Set goals and Propose change Investigate Step 2 Step 3a I -a I no go no go go Basic IA Investigate further go _ Advanced IA Decide upon models, Execute process, Analyse data and Package experience Step 3b Step 4 Step 5 Step 6 Figure 1: Impact analysis can be carried out when models are chosen. section 6 we describe how to actually predict the overall impact of a proposed process change. 4.2 RePlaCe - an advanced impact analysis method The method consists of three steps that must be carried out to be able to decide whether or not to implement a process change proposal. The three steps are illustrated in figure 2 together with some more specific techniques which are discussed further below. First, data must be collected on the impact on quality attributes on sub-processes. This data can be collected as subjective measures, in for example interviews, or as objective measures as experience from former projects. The major reason for allowing for subjective measures on sub-processes is that it is often easier to find experts on different phases and methods in the process than trusting an overall judgement. Secondly, the impact on quality attributes for the overall process is determined with a prediction system, which consists of three different quality models: a reliability model, describing the relative improvement of fault content, which can be used to derive relative improvement of reliability. a productivity model, describing the relative improvement of the productivity.

6 316 Software Quality Management Presentation Figure 2: The advanced impact analysis method. a cycle time model, describing the relative improvement of the development time of the system. The models aim at increasing predictability of these key quality attributes. The models are also well in line with the objectives of most software organizations, which can be summarized by: reduce cost, reduce cycle time and move towards zero defects. Finally, the output of the prediction system must be presented to management, which can use this information as a basis to decide whether or not to implement the change proposal. 5 Data collection 5.1 Introduction This is thefirststep of the impact analysis method, where figures on the impact on the sub-processes are derived. These figures can be derived as subjective or objective measures of the impact of the proposed change. Impact analysis is needed in different contexts, i.e. in comparison with the experience base of the organization, for example: Internal experience is available, i.e. the process change means that experience can be reused from other projects within the organization. Public experience is available, i.e. no experience is available within the organization but experience of, for example, a new method has been published in the public domain. State-of-the-art technology, i.e. no experience is available. In this case some new techniques or methods are available but no organization has published any experience from using them. In thefirsttwo cases figures on local impact can be collected as a combination of subjective and objective measures, while in the third case figures on local impact must be collected as subjective measures. Impact analysis gives the most precise predictions in the first case, while it is obviously most needed in the third case. In this case, though, it is hard to obtain the local opinions described above which gives that the result of the analysis will be less precise than in the

7 Software Quality Management 317 first case. Therefore, it is important to consider not only mean values, but also the variance. 5.2 The Goal/Question/Metric - paradigm To obtainfigureson local impact, some kind of measurement technique must be used. It is important that this technique collects figures that represent what the models actually require as input. Goal oriented measurement provides a means to derive metrics from measurement goals. We suggest that the Goal/Question/Metric-paradigm (GQM) [Basili93a, Basili94] should be used to provide the impact analysis method with valid and relevant input data. The input requirements of the models are defined as measurement goals, which according to GQM can be used to derive metrics via questions. As examples of three measurement goals we can define: Analyse the software products for the purpose of impact analysis with respect to reliability from the point of view of the users of the products. Analyse the software development process for the purpose of impact analysis with respect to productivity from the point of view of management. Analyse the software development process for the purpose of impact analysis with respect to cycle time from the point of view of management. The models of the prediction system have been used to derive questions that must be answered to provide the prediction system with input data: 1. How many faults are introduced in the sub-processes if the improvement proposal is implemented and if it is not implemented? 2. To what extent are faults in the sub-processes discovered if the improvement proposal is implemented and if it is not implemented? 3. What is the number of person months required to complete the sub-processes if the improvement proposal is implemented and if it is not implemented? 4. What will the execution times of the sub-processes be if the improvement proposal is implemented and if it is not implemented? 5. At what stages of the sub-processes, i.e. at what time, can required products be delivered? 6. What is the accuracy of the given figures? 7. What are the required products from the other (preceding) sub-processes, if the improvement proposal is introduced and if it is not introduced? The parts of the questions referring to not implementing the proposal can in most cases be answered exactly, that is if a proper measurement programme is in place. The questions correspond to the input needed by the models of the prediction system explained in the next section. Question 1, 2 and 6 correspond to the input need of the reliability model, question 3 and 6 to the productivity

8 318 Software Quality Management Measurement goal related to impact analysis Another measurement goal Figure 3: Example of goals, questions, Qj, and metrics, M,\ model and question 4, 5, 6 and 7 correspond to the input need of the cycle time model. To increase the accuracy of the figures which serve as input to the prediction system, every question ought to be answered from more than one source. This is especially important if the questions are answered by subjective measures, such as answers in an interview. In a project or organization where GQM is applied, measures are taken on the processes and the products. These measures (or metrics, which is the notation used in GQM) include both subjective and objective measures taken both prior, during and after system development. For the impact analysis method, the most interesting measures are subjective and objective measures taken prior to the project at hand. As can be seen in the example in figure 3, some of the metrics for impact analysis and for other measurement goals may be common. This implies that the measures collected to provide the impact analysis method with input data should be collected by the same organization that collects other measures. In addition to providing the impact analysis method with input data, measurement is important to improve the impact analysis method. This type of measures include objective and subjective measures, collected during and after the execution of the processes. These measures represent the predictability of the prediction system and can be used to adjust the parameters of the models and thereby improving the prediction system. GQM can be used to provide an organization with such measures. 6 The prediction system 6.1 Introduction When measures have been collected, they must be converted to formfigures,if they were not collected as figures from the beginning. This conversion can for example be done as a transformation from subjective ratings to integers, such as from subjective ratings of improved cycle time of a sub-process to integer figures on relative improvement, such as 0%, 10%, 20%, 30%, 40% or 50% improved execution time.

9 Software Quality Management 319 "«T Figure 4: In every sub-process a number of faults is introduced and found. A number of faults remains undetected and is thus propagated to the next subprocess. The prediction system consists of a number of prediction models, which can be used to predict the overall impact of a process change proposal. In this section we give examples of three simple models: one reliability model, one productivity model and one cycle time model. 6.2 A reliability model We are interested in determining the reliability of the delivered system in terms of for example mean time between failures (MTBF), which can be estimated as a function of the number of faults in the system. Thus we can use a model as illustrated in figure 4, which is influenced by [Eick92]. A number, Z,, of faults is introduced and a number, F,, of faults is discovered in every sub-process. When a sub-process is completed, a number, X,, of faults still remains and is thus propagated to the next sub-process. Z, and F, correspond to the answers to the questions stated in the former section. If the improvement proposal is implemented then the number of faults in the delivered system can be expected to decrease by a factor r _ /i. old, old where X^ ^ denotes the estimated number of faults remaining after the last subprocess if the improvement proposal is not implemented and X^ denotes the estimated number of faults remaining after the last sub-process if the improvement proposal is implemented. If each fault is equally likely to cause a failure then the reliability of the system in terms of MTBF must also be improved by a factor /,_ This assumption does not hold in general and the subject is an area for further research. 6.3 A productivity model The productivity of an organization reflects how many person months that is needed in the development process. As an example of a very simple productivity model, we describe an effort model. The required input data to this model is

10 320 Software Quality Management Figure 5: A simple example of three sub-processes. the required number of person months for each sub-process, ej, e^^-e^. The relative improvement of the productivity, in terms of required number of person months can be expressed as: where e^^ denotes the required number of person months in sub-process / if the improvement proposal is not implemented and <?, denotes the required number of person months in sub-process / if the improvement proposal is implemented. 6.4 A cycle time model The development process is divided into a number of sub-processes, such as specification, design and test. Cycle time can be defined as the time from that the development of a system is started until it is delivered, i.e. the time required to execute all sub-processes. An improvement proposal can for example be to introduce a new method, which makes it possible to execute some of these subprocesses in parallel and thus to shorten the cycle time, or to introduce reviews in a number of sub-processes to improve the reliability of the delivered system. As input to this model, local opinions on the execution times and possibilities of parallelism are required. We have to collect local information on what products the sub-processes need from their respective preceding sub-processes and then when each sub-process can deliver the products that are needed by the succeeding sub-processes. Examples of questions that can be asked to representatives of each sub-process are stated in the previous section. When this data is collected we can estimatefigureson mean and variance of the execution times of the subprocesses and delivery times of products. A simple example of a process that consists of three sub-processes can be seen in figure 5, where the improvement proposal is to introduce a new method, which makes it possible to execute parts of the sub-processes in parallel. Two different time schedules are shown with respect to the expected values of the

11 Software Quality Management 321 execution times of the sub-processes. In the upper schedule tj ^, fj old ^d fj ^ represent the expected execution times, if the improvement proposal is not implemented and in the lower schedule f/, ^ and t$ represent the expected times if the improvement proposal is implemented, f// and ^/ represent the expected times until required products can be delivered. If the start of sub-process 3 depends on the completion of sub-process 1 and delivered products from subprocess 2 and the start of sub-process 2 depends on delivered products from sub-process 1, then the cycle time equals t=max(tjj+t2i^-t^ tj+t^, f/y+^j where 0/+'2/+0 < 0+0 corresponds to that the duration of sub-process 1 is longer than tj]+t2j and Oy+^27+0 < Oy+^2 corresponds to that sub-process 2 ends after sub-process 3. In figure 5 the cycle time of the improved process equals - The relative improvement of the cycle time can be computed as 7 = It is reasonable to suggest that the cycle time can be more precisely predicted if the old technique is used, than if the new technique is introduced. The distribution functions sketched in figure 5 indicate this. 6.5 Relationship between the models The three attributes of interest in our models are related. An improvement proposal affects more than just one of the attributes. No process change affects just one attribute and leaves the other two unaffected. Even if a process change affects the different sub-processes in such ways that the overall impact on the attribute is virtually none, this can not easily be seen in advance. As an example we can consider an improvement proposal to introduce reviews after a number of sub-processes. This process change will probably increase the execution times of thefirstsub-process and thus increase the overall cycle time, but the process change will probably increase the quality of the input to the later sub-processes and thus shorten the overall cycle time. We see that the quality attributes can be both directly affected by a process change and indirectly affected via the other quality attributes. The indirect relationship is not explicitly represented in the models presented above, but is an area for future research. 7 Data presentation The output of the prediction system, i.e. /,, 7^ and 7^, should be presented to management, to provide them with information to be able to decide whether or not to implement changes. Notice that decisions on whether or not to implement changes, are not part of the impact analysis method. Impact analysis is carried out to provide information to be able to take informed decisions. The form of the presentation is not critical, but naturally it is important that the presentation clearly states the impact of the proposed change. If the impact

12 322 Software Quality Management analysis method is applied on a number of change proposals, then the impact of each proposal should be presented. 8 Conclusions Impact analysis provides management with important information, when they decide whether to implement a change proposal or not. It is feasible to use available information on local impact to derive a prediction of the overall impact on reliability, productivity and cycle time. The presented models are so far simple and independent, but they provide a starting point for informed decisions regarding introduction of software process change proposals. Further research is however required to get a comprehensive prediction system. The reliability model must handle both fault content and reliability in terms of for example mean time between failures. The cycle time model must handle more general situations where the process can be divided into sub-processes in a number of different ways. The variance of the factors of the models and the dependencies between the models must be taken into account. References [Basili93a] V.R. Basili, Applying the Goal/Question/Metric Paradigm in the Experience Factory. Presented at the 10th Annual CSR Workshop, October [Basili93b] V.R. Basili, The Experience Factory and its Relationship to Other Improvement Paradigms. Proceedings of the 4th European Software Engineering Conference, Springer Verlag, LNCS 717, 1993, pp [Basili94] V.R. Basili, G. Caldiera, H.D. Rombach, Goal Question Metric Paradigm. In John J. Marciniak, editor, Encyclopedia of Software Engineering, John Wihley & Sons, New York, 1994, pp [Eick92] S.G. Eick, C.R. Loader, M.D. Long, L.G. Votta, S.V Wiel, Estimating Software Fault Content Before Coding. Fourteenth International Conference on Software Engineering, Melbourne, 1992, pp [Sommervile92] I. Sommerville, Software Engineering - Fourth Edition, Addison-Wesley, 1992.