Software Engineering Decision Support and Empirical Investigations A Proposed Marriage

Transcription

1

2 Software Engineering Decision Support and Empirical Investigations A Proposed Marriage Günther Ruhe University of Calgary ruhe@ucalgary.ca Abstract: Do you conduct empirical investigations without realizing how they contribute to decision-making? Decisions are the driving engines behind any kind of development. This is also true for all stages of software development and evolution. Decisions can be related to resources and software technologies (subsuming methods, tools, and techniques). Decision-Making in Software Engineering is extremely challenging because of a dynamically changing environment, conflicting (stakeholder) objectives and constraints, and a high degree of uncertainty and vagueness of the information that is available. Decisions related to the How? How good? When? Why? and Where? questions in the use of Software technologies is too often still based on simplistic rules of thumb. What is needed is a sound methodology and with links to validated models and experience. Empirical Software Engineering is relying on or derived from observations or experiments. It is oriented towards making decisions about Software Engineering technologies. The paradigm of Software Engineering Decision Support (SEDS) goes beyond the concept of just reusing models, knowledge or experience. For a more focused problem domain, emphasis is on providing a methodology for pro-active generation, evaluation, prioritization and selection of solution alternatives. However, the results of this process can only be as good as the underlying models and experience. This is exactly the main purpose of Empirical Software Engineering: to incrementally establish a body of empirically validated knowledge about existing or new phenomena. This paper describes the potential synergy between SEDS and Empirical Software Engineering. The need for decision-making defines the objects and attributes of empirical investigations. This avoids esoteric experiments of high statistical significance that have no impact on reality. Vice versa, validated models and experience as concluded from sound empirical studies are enablers for making good decisions. This synergy is studied in more detail for decision support in software release planning. Keywords: Software Engineering Decision Support, Empirical Software Engineering, Synergy, Release Planning, Evolutionary Approach.

3 1. Motivation Good decisions are the driving force behind a corporation s success [5]. Decisions can be related to technologies, resources, processes and products. Good or bad decisions have a tremendous impact on the success of any type of software development or evolution. These impacts are hardest felt when the pressure to attain a competitive market edge or to deliver a quality product is high. Software Engineering is still suffering from the inability to answer the crucial questions of How? How good? When? Why? and Where? as they relate to the use of technologies, processes and resources [7]. This topic is exactly addressed by Empirical Software Engineering (ESE), defined in [9] as a discipline concerned with the scientific measurement, both quantitative and qualitative, of Software Engineering process and product. But how useful are exploratory studies, assessment studies, manipulation experiments or observational studies (to follow a classification given by [4]) to achieve better decisions? From a decision-making perspective, software development and evolution is continuously confronted with different objectives and constraints, with a large number of variables under dynamically changing requirements, processes, actors, stakeholders, tools and techniques. Very often, this is combined with incomplete, imprecise, fuzzy or inconsistent information about all the involved artifacts. Good solutions of decision problems are characterized by the feature that they satisfy most of the inherent objectives and constraints, that they are reasonable to implement, costeffective and that, in addition, they provide a way for post-mortem impact analysis [14]. Software Engineering Decision Support aims to provide support for both strategic and tactical decision-making. Support here means to provide access to information that would otherwise be unavailable or difficult to obtain; to facilitate generation and evaluation of solution alternatives, and to prioritize alternatives by using explicit models that provides structure for particular decisions. Good decisions are more likely if one uses intelligent support based on sound methodology, empirically validated models and experience more often. This is exactly (one of) the purpose(s) of Empirical Software Engineering (ESE): to build a reliable base of knowledge and thus reduce uncertainty about which theories, methods, and tools are adequate [21]. This knowledge may be related to variables and their main impacting factors, to qualitative and quantitative characterization of dependencies, or to the appropriate formulation of relevant objectives and constraints. There are limitations for both SEDS and ESE that can be reduced by looking for a marriage between the two. This paper investigates the synergy between SEDS and Empirical Software Engineering (ESE) and is divided into four sections. The questions What does Software Engineering Decision Support mean? and Why do we need it? are discussed in Section 2. As a cased study, the problem of software release planning is studied in Section 3. This is intended to show the bi-directional dependency between SEDS and Empirical Software Engineering (ESE). Finally, in 2

4 Section 4 a framework for how to achieve the proposed synergy between SEDS and ESE is presented. 2. What Does Software Engineering Decision Support Mean? and Why Do We Need It? Experience factory and organizational learning approaches are increasingly used to improve software development practices [2],[20]. The main idea of experiencebased learning is to accumulate, structure, organize and to provide on demand any useful piece of information that may be reused in forthcoming problem situations. However, software development and evolution is typically of a large scale and complexity, with dynamically changing problem parameters. In this situation, reuse of experience alone is a useful, but non-sufficient, approach to enable proactive decision analysis. Diversity of project and problem situations in conjunction with the costs and availability of knowledge and information organized in a non-trivial experience (or case) base are reasons to look for supplementary ways to enable better decisions. The idea of offering decision support always arises when decisions have to be made in complex, uncertain and/or dynamic environments. Typically, a good decision support system is focused on a relatively narrow problem domain. There are three kinds of existing research results that contribute to Software Engineering Decision Support. Firstly, an effort explicitly devoted to provide decision support in a focused area of the software life cycle. Examples are decision support for reliability planning [19], architectural design decisions [10], or decision support for conducting inspections [13]. Secondly, research results that are implicitly contributing to decision support. Basically, most results from empirical software engineering, software measurement or software process simulation can be seen to belong to this category. Thirdly, all research devoted to enlarge the methodology of decision support (e.g., web-based decision support). Decision support has been successfully designed, developed and applied in many areas such as logistics, manufacturing, health care, forestry or agriculture. Why do we also need decision support in Software Engineering? Some of the major concerns we encountered in current real-world situations in software development and evolution are summarized below: Decision problems, with its main objectives and constraints, are often poorly understood and/or described. Decisions are done at the last moment and/or under time pressure. Decisions are not relying on empirically evaluated models and experience. Decisions are not based on a sound and appropriate methodology. 3

5 Decisions are made without considering the perspectives of all the involved stakeholders. Decisions are not explained or made transparent to those involved. The challenging question is how can we improve the maturity of decisionmaking? Before we address the role and importance of empirical investigations for that purpose, we discuss some of the above issues in more detail for the problem of (software) release planning. 3. Case Study: Decision Support for Software Release Planning 3.1 Informal Problem Statement Release planning for incremental software development includes the assignment of a given set of candidate requirements to releases. Typically, each increment is a complete system that is of value to the client. The results of its use feed back to the developers, who take that information into account when implementing subsequent phases. This feedback may introduce changes to requirements or new requirements, priorities, and constraints. Effectively solving the problem of release planning involves satisfying the needs of a diverse group of stakeholders. An understanding of who the stakeholders are and their particular needs are key elements in developing an effective solution for the software release planning problem. Typically, stakeholders will have different perspectives on the problem and different needs that must be addressed by the release plan. According to their main interest, they will focus more on quality, time, or benefit. Release planning is impacted by a large number of inherent constraints. Most of the requirements are not independent of each other. Typically, there are precedence and/or coupling constraints between them that have to be satisfied. Furthermore, effort, resource, and budget constraints have to be fulfilled for each increment. The overall goal is to find a relatively small set of most promising assignments of requirements to increments such that the degree of satisfaction of all the different stakeholders is maximized. 3.2 Problem Characteristics Release planning is a very complex problem including different stakeholder perspectives, competing objectives and different types of constraints. Among the 4

6 difficulties described in [17], we will focus on those that can be approached by empirical investigations: Requirements are not well specified and understood: There is usually no formal way to describe the requirements. Non-standard format of requirement specification often leads to incomplete descriptions and makes it harder for stakeholders to properly understand and evaluate the requirements. Uncertainty of data: Meaningful data for release planning are hard to gather and/or uncertain. Specifically, estimates of the available effort, dependencies of requirements, and definition of preferences from the perspective of involved stakeholders are difficult to gauge. Availability of data: Different type of information is necessary to conduct release planning. Some of the required data are available from other information sources within the organization. Ideally, release planning is incorporated into existing organizational information systems. Constraints: A project manager has to consider various constraints while allocating the requirements to various releases. Most frequently, these constraints are related to resources, schedule, budget or effort and hard to determine. Unclear objectives: Good release plans are hard to define at the beginning. There are competing objectives such as cost and benefit, time and quality, and it is unclear which target level should be achieved. 3.3 Solution Approach The above-mentioned (and some more, see [16]) difficulties make it impossible to apply a fixed and formal closed world type of solution to this problem. The problem is wicked [3] in that the objective is to maximize the benefit, but it is difficult to give a measurable definition of benefit. One way to improve meaningfulness of results is to systematically validate them. Furthermore, the underlying model is evolving : the more we study the problem, the more sophisticated the model becomes. Although we are approximating reality, implicit and tacit judgment and knowledge will always influence the actual decisions. Different solution approaches based on evolutionary algorithms have been proposed to solve the release planning problem [8], [18]. More recently, EVOLVE* [17] was designed as an iterative and evolutionary procedure mediating between the real world problem of software release planning, the available tools of computational intelligence for handling explicit knowledge and crisp data, and the involvement of human intelligence for tackling tacit knowledge and fuzzy data. EVOLVE* is generalizing its predecessors EVOLE [8], and EVOLVE+ [18]. At all iterations of EVOLVE*, three phases are passed: 5

7 Phase 1 - Modeling: Formal description of the (changing) real world to make it suitable for computational intelligence based solution techniques. This includes the definition of all decision variables, as well as their dependencies and constraints, and the description of what is, or contributes to, the goodness of a solution. Other data, such as stakeholder evaluation of all requirements, are part of modeling, too. Phase 2 - Exploration: Application of computationally powerful techniques to elaborate the solution space, and to generate and evaluate solution alternatives. Exploration phase is mainly based on evolutionary computing. Phase 3 - Consolidation: Human decision maker is invited to investigate current solution alternatives. This contributes to the understanding of the problem and results in modifying parts of the underlying model or in some local decisions (e.g., pre-assigning some requirements to an increment). Typically, these decisions reduce the size and complexity of the problem for the next iteration. For the problem of software release planning, the modeling activity mainly comprises the definition of key variables, their dependencies, as well as definitions of the main objectives and constraints. 3.4 Synergies between SEDS and ESE We have performed a series of observational studies with real world data [1],[18]. On the one side, empirical evidence from real-world projects with several hundred requirements and a large number of involved stakeholders (150 stakeholders in one case) has been used to considerably improve the relevance of the decision support offered by the tool system (see On the other side, in the course of evolutionary refining of the problem model with all its inherent objectives and constraints, new and relevant experimental investigations were defined. Here are some of the concrete synergies that we have observed from the interaction between SEDS and ESE: Release planning is heavily based on estimates related to the necessary effort to implement candidate features or requirements. Empirical validation of those effort estimates has resulted in the availability and improved accuracy of the underlying model data. The same arguments are applicable for estimation of risk related to the implementation of features or requirements. Both aspects have improved the meaningfulness of the proposed decision support. Empirical evaluation of the practical validity of the proposed solutions has resulted in modification of the objective function(s). As a consequence, more meaningful results were achieved (empirically validated). Empirical evaluation of the practical validity of the proposed solutions has resulted in a modification of the problem constraints. For example, release 6

8 planning can no longer be done without proper resource planning, as the nonavailability of resources impacts the planning results. Empirical evaluation of stakeholder performance has resulted in a modification of the evaluation schema of requirements (from just looking at importance to later on including urgency as well). Offering decision support (SE-DSS) has raised the question of its impact. Hypotheses to be empirically validated are [15]: o Hypothesis 1: SE-DSS enables making more effective decisions (im-proved quality). o Hypothesis 2: SE-DSS enables making more efficient solutions. o Hypothesis 3: SE-DSS allows more transparent decisions (to be better understood by involved individuals), reflecting trade-offs between conflicting criteria or stakeholder opinions. o Hypothesis 4: SE-DSS increase awareness of proper data collection and usage of process and product models. 4. The Future: Marriage between SEDS and ESE The case study on conducting software release planning has demonstrated how Software Engineering Decision Support and Empirical Software Engineering can benefit from each other. We propose and foresee a future marriage between the two. A question still to be answered is How will the two partners benefit from each other? Experiments are very expensive and/or time consuming, so the insights gained are worth its critical judgment. Self-indulging experiments that offer no new knowledge for making better decisions (maybe based on a sound theory that was verified by experiments as well) are a waste of time and effort. The decision-prone character of software development and evolution is an excellent orientation for the selection of the most essential topics and questions addressed by empirical investigations. This avoids experiments that are formally correct, but meaningless in their impact. For conducting formally correct experiments, a set of guidelines was proposed in [11]. Referring to the rime value of any technology or innovation, empirical studies should be done within a time frame actually allowing the implementation of the suggested changes. As pointed out by [12], real-world decision-making is rarely based on models that have empirically based assumptions may be because it often takes too long to specify. This can be seen as another inspiration of how ESE benefits from the demands of SEDS. 7

9 Offering decision support based on vague and uncertain models is meaningless as well. Availability of reliable qualitative or quantitative data is needed for model description. Reflecting reality in terms of the underlying objectives and constraints is not an easy task, and it needs empirical validation to do the right things. Without an appropriate description of what are the main objectives, and how the different alternatives are restricted by resources, budget or effort constraints, support will not be achievable. Finally, empirical evaluation contributes to the qualification of the support itself, as it can validate the proposed improvements in terms of more efficiency, effectiveness or transparency. Software Engineering related Problem Solving Interation 1 Increment 1 Interation 2 Increment 2 Interation 3 Increment 3 Empirical Investigations based on Human Intelligence Decision Support based on Computational Intelligence Figure 1: Synergy between Decision Support and empirical investigations resulting in improved problem solving capabilities for release planning. The synergy between SEDS and ESE is not static, but evolving. The progress achieved for the one part will accelerate the progress of the other. In an idealized form that relates to the example problem of release planning, the evolution can be illustrated as shown in Figure 1. There is a continuous cycle of modeling and formalizing the real world by offering decision support based on computational intelligence, and conducting the different kinds of evaluations triggered by human capabilities. The results achieved from those validation efforts are the input for (in tendency) improved (more valid) problem solving capabilities. As implemented in 8

10 EVOLVE*, the progress achieved for the one part will accelerate the progress of the other. Acknowledgements The author would like to thank the Alberta Informatics Circle of Research Excellence (icore) for the financial support of this research. Many thanks are due to Amandeep and Kenny Tsang for their implementation efforts on EVOLVE* and to An Ngo-The, Sebastian Maurice and Mark Standford (igraphix, Corel) for stimulating discussions. 5. References [1] A. Amandeep, G. Ruhe, M. Standford, Intelligent Support for Software Release Planning. In: preparation. [2] V. Basili, G. Caldiera, D. Rombach, Experience Factory. In: J. Marciniak: Encyclopedia of Software Engineering, Volume 1, 2001, pp [3] P. Carlshamre, Release Planning in Market-Driven Software Product Development: Provoking an Understanding. Requirements Engineering 7, pp , [4] P. Cohen, Empirical Methods for Artificial Intelligence, MIT Press, [5] G. DeGregorio. Enterprise-wide Requirements and Decision Management. Proc. Int'l Symp. Int'l Council on Sys. Eng., [6] A. Endres, D. Rombach, A Handbook of Software and Systems Engineering. Pearson [7] N. Fenton, Conducting and Presenting Empirical Software Engineering. Empirical Software Engineering 6 (2001), pp [8] D. Greer, G. Ruhe, Software Release Planning: An Evolutionary and Iterative Approach. Appears in: Information and Software Technology [9] R. Jeffrey, L. Scott. Has twenty five years of empirical software engineering made a difference. Proc. Asia Pacific Softw. Eng. Conf., 2002, pp [10] R. Kazman, J. Asundi, M. Klein. Making architecture design decisions: An economic approach. Tech. rep. CMU/SEI-2002-TR-035, Software Engineering Institute, [11] B. Kitchenham, S. Pfleeger, L. Pickard, P. Jones, D. Hoaglin, K. El-Emam, J. Rosenberg, Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Transactions on Software Engineering vol. 28, pp , [12] G. Melnik, Questioning Empirical Software Engineering. Working Paper. Department of Computer Science, University of Calgary, [13] J. Miller, F. Macdonald, J. Ferguson, ASSISTing Management Decisions in the Software Inspection Process. Information Technology and Management, vol.3 (2002), pp [14] S. Pfleeger, Making Goog Decisions: Software Development and Maintenance Projects. Tutorial at 8th IEEE Symposium on Software Metrics, [15] G. Ruhe. Software Engineering Decision Support A new paradigm for Learning Software Organizations. Proc. Wkshp. Learning Software Organizations, Springer, [16] G. Ruhe, Software Engineering Decision Support: Methodology and Applications. In: Innovations in Decision Support Systems (Ed. by Tonfoni and Jain), International Series on Advanced Intelligence Volume 3, 2003, pp [17] G. Ruhe, A. Ngo-The, Hybrid Intelligence in Software Release Planning. In preparation. 9

11 [18] G. Ruhe, D. Greer, Quantitative Studies in Software Release Planning under Risk and Resource Constraints. Proceedings of the 2003 IEEE International Symposium on Empirical Software Engineering (ISESE 2003). [19] I. Rus, J.S. Collofello, A Decision Support System for Software Reliability Engineering Strategy Selection. Proceedings of the 23rd Annual International Computer Software and Applications COMPSAC 99, Scottsdale, AZ, October 1999, pp [20] I. Rus, M. Lindvall, Knowledge Management in Software Engineering. IEEE Software May/June 2002, pp [21] W. Tichy, Should Computer Scientists Experiment More? 16 Excuses to Avoid Experimentation. IEEE Computer 31(5): pp 32-40,