DESIGNING A MEASUREMENT PROGRAMME FOR SOFTWARE DEVELOPMENT PROJECTS

Transcription

1 DESIGNING A MEASUREMENT PROGRAMME FOR SOFTWARE DEVELOPMENT PROJECTS Master s Thesis Richard Kettelerij

2 Disclaimer: Due to confidentiality reasons, the real name of the company involved in this thesis project is replaced by the fictitious name of Daniro. Other information is left intact. c 2006 Richard Kettelerij, Daniro System Integration and Development B.V. This work, excluding cover picture and images shown in chapters 1 and 2, is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. To view a copy of this license, visit or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA. Cover picture: M.C. Escher s Spher Spirals (1958) c 2006 The M.C. Escher Company B.V. - the Netherlands. All rights reserved. Used by permission. Document typeset with L A TEX in Bitstream Charter font. Bibliography formatted with BIBTEX in Alpha/AMS style. Adobe PDF file generated with MIKTEX PDFTEX on :41

3 DESIGNING A MEASUREMENT PROGRAMME FOR SOFTWARE DEVELOPMENT PROJECTS THESIS submitted in fulfillment of the requirements for the degree of MASTER OF SCIENCE in SOFTWARE ENGINEERING by ing. Richard Kettelerij born in Zutphen, the Netherlands Committee in charge: Prof. Dr. P. Klint, Chair Prof. Dr. J. van Eijck, Vice-Chair Drs. H. Dekkers, University supervisor Ing. D. Koopman, Company supervisor Dr. J. Vinju, Committee member Institution: University of Amsterdam Faculty of Science 1098 SM Amsterdam The Netherlands Company: Daniro J-Technologies Plotterweg BB Amersfoort The Netherlands (url removed)

4

5 DESIGNING A MEASUREMENT PROGRAMME FOR SOFTWARE DEVELOPMENT PROJECTS Abstract Software measurement is generally recognized as an effective means to understand, control, predict and improve software development projects. However, despite the attention given by the academic community, measurement is far from common practice in certain sectors of the software engineering industry. In the case of Daniro J-Technologies, measurement was found to be used for two purposes; monitoring source code quality and monitoring finances. A measurement programme was, however, not in place. Therefore, this research project seeks to design a process-centric measurement programme that suits the specific needs of stakeholders in development projects. In order to do so, the present author initiated a thorough literature survey. This led amongst others to the discovery of the Goal/Question/Metric method (GQM), which formed the basis of the research approach followed during the thesis project. In light of GQM, a series of interviews were held with stakeholders at different organizational levels. This led to identify five viewpoints and four measurement goals. These goals involved respectively; understanding productivity, defects and scope. In response to these goals, questions were formulated that captured the information needs of stakeholders. In turn these questions led to the definition of a balanced set of (approximately) thirty software measures. These measures, together with questions and goals, were validated during an interactive presentation session and prioritized by means of a survey. Yielding an initial set of thirteen productivity and defect-related measures (scope measurement was excluded). Furthermore, a prototype Measurement Support System (MSS) was constructed to give stakeholders a more concise and visual representation of the information they could expect from the programme. The indicators (chart and tables) of high priority measures in this MSS were validated during personal walkthroughs. Subsequently feedback from stakeholders expressed during walkthroughs, and during the presentation was incorporated in the current programme. The above process resulted in the initial definition of a strategic measurement programme, specifically tailored toward Daniro s Java division. Secondly, the project produced a prototype MSS for use in (future) pilot projects. The research also contributed to more subjective matters, such as increased awareness of stakeholders with respect to software process measurement. Based on the results obtained and the literature consulted, the present author concludes that software measurement in an outsourcing organization is in some ways different from measurement programmes discussed in literature. This is mainly due to the service oriented or customer-intimate nature of outsourcing projects. As a result it is difficult to mandate a fixed set of measures. Nevertheless by concentrating on a number of core improvement areas applicable in any project, a measurement programme can still be defined. Hereby it was found that productivity is a major driver behind measurement information needs. This involves not only productivity in the sense of size vs. effort, but also measures related to schedule adherence and rework time. Although these types of process measures mainly concern managerial stakeholders, the present author believes that engineers can also benefit from this intelligence. Provided that timely feedback sessions are organized. Whether or not the measurement programme is effective in the long run, can only be determined after or during organizational implementation. The latter is, however, recommended as future work. As a result, it is currently not possible to verify whether the proposed measures satisfy the goals and questions stated. Nevertheless, the initial design of the programme brought a number of important prerequisites in place for Daniro to archive a higher level of measurement and improvement capability. Keywords: software engineering, measurement, goal/question/metric, process improvement

6

7 I don t see how we can have software engineering in the 21st century without measurement David N. Card 1 1 Fellow of the Software Productivity Consortium and Editor-In-Chief of the Journal of Systems and Software. Quote from The Best Influences on Software Engineering by Steve McConnell [McC00].

8

9 Preface THIS thesis is the result of the master project carried out between January 9 - April 4 (part time) and April 10 - July 15 (full time) at Daniro J-Technologies. In short, the objective of this master project is to design a measurement programme for use in software development projects. More specific, attention is given to process-related measures that suit the specific needs of Daniro and that provide value to the stakeholders involved. Originally, my interest in this project was attracted by the strong focus on the concept of software development processes. As such, this project allowed me to cover a wide range of software engineering aspects that I ve learned during the past year. From a personal perspective however, this project has been a challenge since it was the first time I did an internship on a less technical subject like software process measurement. Addressing this subject on a conceptual level (pure thought stuff) while working toward a practical solution wasn t easy. Nevertheless it was exciting and it helped me to expand my horizon. This thesis would not have been possible without the help of a number of people. First I would like to thank my supervisors Dirk Koopman (Daniro) and Hans Dekkers (UvA) for their valuable feedback and guidance during the project. In addition, I would like to thank dr. ir. Rini van Solingen (LogicaCMG/Drenthe University) for his willingness to cooperate in an interview, which led to some important insights during the project. Furthermore I would like to thank all interviewees; E. Fokker, M. Willemsen, E. Dieleman, H. Jansen, R. Ligtmans, D. Koopman, M. Loggere, K. Grosskop and A. Willemse, as well as all other colleagues at Daniro for their help and support during this project. Above all I want to express my gratitude to my parents for their everlasting care and support. Finally I wish to dedicate this thesis to my younger brother Robert, whose fight for cancer over the last five years strengthened me in this accomplishment. The way you live your life is without doubt admirable. Richard Kettelerij Maurik, the Netherlands August 14, 2006 v

10

11 Contents Preface Contents List of Figures v vii ix 1 Introduction and Motivation Context Problem definition Scope Research question Outline Background and Context Software Process Software Process Improvement Software Engineering Management Software Measurement Measurement Theory Functional Size Measurement Measurement Methods Goal Oriented Measurement Practical Software and Systems Measurement Success Factors Research Method and Approach Methodology Approach Contribution Characterization and Goal Identification Project environment Progress reporting Knowledge acquisition Goals Viewpoints Measurement goals Concluding remarks vii

12 CONTENTS 5 Information needs and Constraints Terminology Strategy Information needs Productivity questions Defect questions Concluding remarks Measurement Definition Granularity Measures Productivity measures Defect measures Prioritization Concluding remarks Implementation Aspects Measurement Support System Construction Usage and Validation Measurement Specifications Formalizing measures Data Collection and Reporting Concluding remarks Conclusions and Future Work Conclusions Information; strategic measurement for understanding purposes Measurement; project measures to address multiple stakeholders Organization; measures in outsourcing projects Final conclusion Evaluation Information gathering Goal/Question/Metric method Validation and implementation Future Work Recommendation Remaining Work Bibliography 47 A Interview questions 51 B Scope Measurement 53 C Measurement Specification 57 D Measurement Support System 59 E Overview of Measurement-CMM 69 F Basic level measurement 71 viii

13 List of Figures 1.1 Daniro s software factory concept Software Process Improvement focus areas [Sol99] The Goal/Question/Metric (working both top-down and bottom-up) [Bas94] Four phases of the PSM measurement process [Jon03c] Funnel approach; overview of thesis phases and activities Organization of projects (visualized in concordance with Business Unit Manager) Process viewpoints GQM measurement goals Effort terminology Defect terminology Levels of granularity Conceptual model of the proposed introduction plan B.1 From change request to requirements C.1 Example measurement specification (M6) C.2 Example measurement specification (M15) D.1 Project level view of the Measurement Support System D.2 Iteration planning with effort, duration and size data D.3 Iteration planning with effort, duration and size data (cont.) D.4 Activity planning with effort, duration and type data D.5 Activity planning with effort, duration and type data (cont.) D.6 Periodic progress registration at the activity level D.7 Defect tracking sheet with time, impact and effort data D.8 Productivity reporting based on progress and costs (showing M19 & M13) D.9 Productivity reporting based on progress and costs (showing M5 & M6) D.10 Productivity reporting based on progress and costs (showing M4, M9 & M15) D.11 Quality reporting based on defect information (showing M17, M21 & M22) ix

14

15 Chapter 1 Introduction and Motivation DANIRO J-Technologies is a recently formed organization unit, founded through the acquisition of Solidium B.V by the former Daniro Development Centre Java. The unit is part of Daniro System Integration and Development (SI&D), a division of Daniro that focuses on the development, integration and consolidation of enterprise information systems. In general Daniro SI&D is concerned with two types of activities; outsourcing (project implementation) and consultancy. Especially outsourcing receives a lot of attention lately, since Daniro SI&D aims to become the national champion in software projects by expanding its share in this market. 1.1 Context To distinguish itself from other IT service companies on the market, Daniro SI&D is working towards the establishment of a software factory known as SMART Op Maat 1 (figure 1.1). In this context the term software factory refers to the establishment of a generic project approach in the broadest sense of the word. The idea is that everything from tooling to training to methods and procedures is organized in a uniform way, and applied consistently throughout the various software projects. One should not confuse this concept with the (recent) interpretation of Microsoft that software factories are the convergence of key ideas in software product lines, componentbased development and model-driven development [Gre03]. Although these technologies can (and will) be utilized in, especially.net related software factory projects, it is not the primary focus of SMART Op Maat. The latter is more a combination of suitable infrastructures, process features, and managerial guidelines as described in the research of [Aae97]. A central role in the factory concept of Daniro is the use of a development line. This is an integrated set of platform specific tools that support project teams in their software development process. Since Daniro J-Technologies is concerned with Java/J2EE development its development line, SMART-Java, is constructed around a set of (open source) Java tools. Currently the SMART- Java development line offers services such as issue tracking, version management, automated builds and shared storage. Another key aspect of Daniro s software factory concept is the adoption of the Rational Unified Process (RUP) as a standard project methodology. Since RUP is not meant to be used as an out-ofthe-box software process, Daniro decided to tailor the process to its specific needs. This process of RUP tailoring resulted in the publication of a book called RUP Op Maat Problem definition Although it is not presented that way, the software factory initiative of Daniro can be seen as a form of software process improvement (SPI). In essence Daniro aims to increase the maturity and quality of its software development process through the implementation of a uniform project approach. An important concept in the field of software process improvement is software measurement. However,

16 1.2 Problem definition Introduction and Motivation Figure 1.1: Daniro s software factory concept in the case of Daniro J-Technologies this concept has not yet received explicit attention. There have been attempts to introduce software measurement in the development organization, but these measures focused only on the quality of source code. The current development line for example is mainly concerned with measures of source code quality through the use of static code analyzers. Furthermore measurement is used in the bidding process (i.e. Function Point Analysis), and on more managerial sides for the purpose of financial monitoring (i.e. hour-logs). Until now, however, little effort is put in a rigorously defined set of measures that focus on the entire software process, and provide value to different stakeholders involved in software development projects. Since measurement is generally recognized [Abr04] as a key factor in the understanding, management and improvement of software related activities, it is worthwhile to devote more attention to this subject. Therefore the idea was raised to investigate the possibilities of establishing a measurement programme at Daniro J-Technologies for use in software development projects Scope Establishing a measurement programme is, like most other software process improvement initiatives, a cost and resource intensive operation. Therefore the scope of this research is limited in a number of ways. Most notably, the primary concern of this research is the definition of a measurement programme. Although some form of implementation is required to demonstrate the validity of the programme, there are important reasons to initially focus on definition activities. For example without proper definition the purpose of a measurement programme is unclear, and one could well be measuring the wrong things. Consequently the information gathered may not satisfy the needs of stakeholders, which could result in deceased buy-in. Furthermore without a defined programme, data collection is unstructured and more error prone. Also it is often not clear how to analyze and interpret measurement results. Overall, definition and planning is considered to be an important prerequisite for successful measurement. Since it provides the programme with a a clear focus and avoid unnecessary costs. Furthermore this research is limited to process-related measures, since these measures haven t been explicitly addressed yet (at Daniro). In addition the research focuses on software development projects, since these type of projects account for the largest portion of all software-related projects and are therefore critical to the business. Altogether the objective of this research is to design a measurement programme that suits the specific needs of Daniro J-Technologies. 2

17 Introduction and Motivation 1.3 Outline Research question The central research question of this thesis is formulated as follows: What process-related measures, with respect to organizational goals, can be defined to satisfy the information needs of stakeholders in the context of software development projects? In order to answer this central research question, a number of subquestions have been defined. These questions are listed below and grouped by three research topics, respectively: Information which concerns organizational goals and the information needs of stakeholders. Measurement, which deals with process-related measures that fit within the context of software development projects. Organization, which roughly concentrates on the context of the programme, and the costs/benefits associated with implementation. Information 1. What purpose(s) does the measurement programme serve? 2. What goals does the organization tries to archive? 3. What stakeholders are involved in software development projects? 4. What stakeholders have an interest in the measurement programme? 5. What information do stakeholders require from the measurement programme? Measurement 6. What kind of methods or techniques exist for defining a measurement programme? 7. What process-related measures are available in literature and at Daniro? 8. What process-related measures satisfy the information needs of stakeholders? 9. What process-related measures are feasible in the context of development projects? 10. What information about process-related measures should be specified in the programme? Organization 11. To what extend do environmental factors influence the measurement programme? 12. What are the critical success factors in the establishment of a measurement programme? 13. What guidelines are important for measurement programme implementation? 14. What benefits are to be expected from the measurement programme? 15. Does software measurement fit within the context of Daniro s development projects? 1.3 Outline Subsequent chapters elaborate various aspects of the research questions listed above. To start, the next chapter addresses the background and context of this research by a discussion of relevant literature. Thereafter chapter 3 discusses the approach and methodology of the research, as well as the scientific and practical relevance. Chapter 4 summarizes the process of environment characterization and goal identification. Chapter 5 describes stakeholder information needs and the constraints of the measurement programme. Chapter 6 discusses the process of measurement definition in response to these information needs. Chapter 7 describes the work on implementation aspects, such as data collection and measurement support tools. Finally, chapter 8 concludes and provides directions for future work. 3

18

19 Chapter 2 Background and Context BEFORE addressing the research question of this thesis a review of existing literature is required. This contributes to a basic understanding of the process measurement field and the related concepts. Moreover, the literature study performed throughout this project resulted in a subdivision of existing work in several categories. These categories are discussed in subsequent sections of this chapter. 2.1 Software Process ISO standard describes the term software process as the process or set of processes used by an organization or project to plan, manage, execute, monitor, control and improve its software related activities. This definition illustrates that the process field in software engineering includes a wide range of activities. From a measurement perspective these activities can be grouped in two areas; software process improvement and software engineering management. The paragraphs below discuss the two research areas in detail Software Process Improvement Software Process Improvement (SPI) is an extensive research area. There are many models, standards, and methods that can be used to improve the state of software engineering practice within an organization. An interesting classification of SPI related methods is given by Cannegieter in [Can03]. Cannegieter distinguishes three types of methods: system development methods, quality models and project management methods. System development methods such as extreme Programming (XP), SCRUM, Rational Unified Process (RUP) and the Dynamic Systems Development Method (DSDM) define activities, artifacts and roles that are necessary to develop software products. If the scope of a SPI initiative is system or application development, then organizations can use these methods to create or tailor their software development process accordingly. On the other end of spectrum there are methods such as; Software Process Improvement and Capability determination (SPICE/ISO 15504), TickIT, BOOTSTRAP, the Capability Maturity Model (CMM) and its successor CMMI [Can06]. These methods serve as reference models and define requirements that an organization should meet in order to reach a particular maturity level. However, these methods do not specify how to implement the software process. If an organization wants to attain a certain CMM(I) level, it should implement its software process in a way that complies with the requirements of the desired maturity level. Finally an assessment can be performed to identify (and possibly certify) the maturity of the organization, as well as to propose relevant improvements. The last type of methods, identified by Cannegieter, focus on the managerial side of process improvement. These methods, such as PRINCE2, are supplementary to SPI, and not discussed in further detail. Management in the context of software engineering, however, is not a trivial matter and therefore elaborated in Apart from the classification of Cannegieter, Solingen et al [Sol99] divide the SPI area in two streams: top-down and bottom-up. The quality models of Cannegieter; CMM, SPICE and BOOT

20 2.1 Software Process Background and Context Risk Quality Cost Time Figure 2.1: Software Process Improvement focus areas [Sol99] STRAP all classify as top-down approaches, since these methods are based on assessment and benchmarking. On the contrary, bottom-up approaches such as the Goal/Question/ Metric (GQM) and the Quality Improvement Paradigm (QIP) are based on the application of measurement as the basic guide for process improvement. The focus of this master research will be on the measurement-based (bottom-up) stream of software process improvement. However, the actual focus of improvement is unknown at this time. In [Sol99] four main areas of software process improvement focus are identified. As illustrated in figure 2.1 these areas are: quality, cost, risk and time. Quality improvement usually starts with some kind of defect detection and defect measurement, often via inspections. Also subjective measurement such as customer satisfaction or documentation quality can be defined as improvement goals. Cost improvement is mainly concerned with a more efficient development process. Measurement in this context is often related to size, such as costs per line of code or costs per function point ( 2.2.2). Improvement goals concerning risk are mainly targeted toward managing risk factors, by applying measurement to risky areas in the development process (e.g. requirements engineering). Finally, time related measurement and improvement is concerned with aspects such as productivity and time-to-market Software Engineering Management A research area closely related to SPI, and especially important from a measurement point of view, is Software Engineering Management (SEM). The Software Engineering Body of Knowledge (SWE- BOK) defines this area as the application of management activities - planning, coordinating, measuring, monitoring, controlling, and reporting - to ensure that the development and maintenance of software is systematic, disciplined and quantified [Abr04]. The notion of project management plays an important role in this area. In general, project management is an umbrella for many activities in software engineering (as well as other engineering disciplines). Typical activities include, amongst others; task planning, risk management, cost and resource estimation, process control and contract management. Together these activities evolve around five primary attributes namely: time, money, quality, information and organization [Gri00]. During a project it is the responsibility of the project manager to control (and possibly improve) the application of these attributes. One way to accomplish this is by taking decisions based upon quantitative data. Measurement is hereby of important [Abr04]. However it should be noted that measurement is a means to and end, not and end itself [Bas95]. The insight and experience of the project manager and his staff (particularly their understanding of social issues) are at least as important in successful project management. This relates, amongst others, to awareness of possible risks associated with measurement (mis)use. Such as; dysfunction (i.e. forcing people to make measures look better) and distortion (i.e measurement results distort people s behavior, causing them to provide less value to the organization) [Kan04]. 6

21 Background and Context 2.2 Software Measurement 2.2 Software Measurement Software measurement can be loosely defined as the process of defining, collecting and analyzing data on the software development process and its products in order to understand and control the process and its products, and to supply meaningful information to improve that process and its products. [Sol99]. As the definition indicates there are several reasons for conducting software measurement. The first reason is to understand the product, process or resource in question. This may lead to the establishment of a baseline for future comparisons. Once basic understanding is reached, measurement information can be used to control a particular product, process or resource. This involves performing corrective and perfective actions. Thereafter an analysis of measurement data, can help to identify opportunities and inefficiencies in products, processes and resources in order facilitate improvement actions. Finally as part of improvement, measurement information can be applied to predict the development of products, processes and resources over time [Par96]. The (widely accepted) classification in product-, process- and resource-oriented measures is suggested by Fenton and Neil [Fen00]. In general, product measures describe the (quality) characteristics of the product under development. Example measures are: size, complexity, performance and the level of test coverage. Process measures on the other hand are used to characterize software development and maintenance processes. Typical process measures include: development time, effort, cost and the number of requirement changes. Finally, resource measures describe the characteristics of the project or organization under consideration. Resource measures relate to: productivity, cost, schedule and maturity. The classification of Fenton and Neil provides valuable insight in the measures research field, and gives a handful reference to focus on a particular type of measures (process in this case). Nevertheless, the three types cannot be considered in isolation. There s a large overlap, which is illustrated by use of the Metrel (Metric Relationship) rules [Woo01]. The Metrel rules state that: For any valid product measure, its derivative with respect to time is a valid process measure. Subsequently, for any valid process measure, its derivative with respect to time is a valid measure for the organization 2. In example: The number of defects in a system is a useful and valid product measure. Then the rate of insertion of defects into code per phase (the error proneness of the methodology) is a valid process measures. The rate of removal of defects per inspection (the test efficiency) is also a valid measure of the process. And the rate of improvement in test efficiency, over a series of projects, is a valid organization measure [Woo01]. In the field of software measurement there are a number of important (research) areas that need additional explanation. An introduction to two of these areas, measurement theory and functional size measurement, is provided in the remaining sections of this paragraph Measurement Theory In general measurement is defined as the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to describe them according to clearly defined rules [Fen94]. In this context an attribute is seen as a measurable property of an entity. For example, size, complexity and testability are attributes of entity source code. Underlying these basic definitions is a principle known as the representational theory of measurement. This is an extensive subject concerned with the mapping of relations between entities and attributes. A fundamental concept in this theory is the notion of an empirical relational system. This system represents the entities in the real world and the empirical knowledge of the entities attributes. Intuitive understanding of attributes gives rise to relations between entities. For instance, if one is interested in document length the relations is longer than (document A is longer than document B) can be formulated. In order to measure an attribute represented by an empirical relation system, one needs to map the entities and relations from the empirical system to a numerical relational system (formal world), with respect to the representational condition. This condition implies that for every relation defined in the empirical system, there is a equivalent relation defined 2 In this context the term organization is similar to resource 7

22 2.2 Software Measurement Background and Context on the measures of those entities in the numerical system. The link between both worlds is expressed through measures and scales. Hereby distinction is made in two types of measures; direct and indirect. A direct measure, i.e. document length, is not dependent upon a measure of any other attribute. However, indirect measures such as productivity do involve measures of other attributes. Note that the term measure 3 is explicitly used in this thesis to differentiate from the definition of measurement. Use of the term metric is avoided since there is no general/unambiguous definition known in the context of software engineering [Gar06; Off97; Par96]. In measurement theory, scale types determine the kind of statements that can be made about measurement data. A scale type of a particular measure is determined by the admissible transformations that can be made to the scale of that measure. That is, meaningful statements about a value in one scale should also apply to values of the same measure in another scale. Scale types commonly used in software measurement are [Par96]: Nominal: Objects with the same scale value are equal on some attribute. programming languages (Java, C#, Cobol, Pascal). For example; Ordinal: Objects with a higher scale value have more of some attribute. For example; a quality rating (+, ++, +++). Interval: A certain distance along the scale means the same thing, regardless of position. For example; the difference between 5 and 7 is the same as the difference between 10 and 12. Ratio: Equal to interval scales, except that there is a true zero point. For example; a building with a height of 100 meters is twice as large as a building of 50 meters. Absolute: Used when there is only one possible way to measure an attribute. Principles from measurement theory, such as the representation of scale types, are controversial in software engineering research (see the debate between [Kit95], [Mor97] and [Kit97]). For example, some scientists argue that properties that imply or exclude measurement scales in the definition of a measure cannot be used since scales change, depending on the questions ask during data analysis [Kit95]. Others disagree, and claim that without such properties one abstracts away all relevant measurement structure and limits the ability to say anything of interest [Mor97]. However, overall most scientist agree that attributes in software engineering, e.g. correctness, are not yet sufficiently understood to be certain about a particular type of measurement scale [Fen94] Functional Size Measurement Quantifying size is one of the most basic activities in software measurement [Gra94]. A frequently used size measure is lines of code (LOC). Although LOC can be useful in many, especially coding related, situations it has a number of drawbacks [Low90]. First of all the number of LOC depends on the implementation language and coding style used. Secondly, LOC cannot be estimated a priori to implementation. Only when software construction is (partially) finished it becomes possible to faithfully count the number of lines in a software product. In response to these issues many organizations rely on functional size measurement for their products/projects. In short, functional size measurement is aimed at measuring the size of the software product from the perspective of what gets delivered to the (end)user. Function Points The idea of measuring software size in terms of functionality as opposed to physical components such as LOC, was first put forward by Albrecht in 79. Albrecht introduced the concept of function points and the accompanying Function Point Analysis (FPA) method [Low90]. FPA is a structured technique to measure software size by quantifying functionality in the form of function points, based on requirements and design. The technique breaks the system into smaller components in a way that these can be analyzed and understood. Function point counts can be applied to development projects, maintenance projects, and existing applications. There are five major components of 3 The assignment of numbers or symbols to entities, in order to characterize a specific attribute (derived from [Gar06]) 8

23 Background and Context 2.3 Measurement Methods FPA which capture the functionality of the application. These are: External Inputs (EI), External Outputs (EO), External Inquiries (EQ), Internal Logical Files (ILF) and External Interface Files (EIF). The first three are treated as user transactions, and the last two are referred to as logical data collections. At first, FPA was only used to measure productivity after the completion of development and maintenance activities. However, soon it became clear that FPA could also be used to support the software process in its inception, since the required data can be available early in the project. This aspect made FPA a widely adopted technique for software engineering tasks such as cost and resource estimation. This development puts heavy responsibility on the use of function points. Since accurate cost estimation is vitally important in software process improvement [Miz98] and especially interesting in terms of process measurement. To conclude, a typical 4 function point analysis consists out of the following steps [Low90]: Determine the type of function point count (users, purpose, dependencies) Identify and rate user transaction- and logical data collection functions, to calculate their contribution to the Unadjusted Function Point count (UFP) Determine the Value Adjustment Factor (VAF) by using general system characteristics Finally, calculate the adjusted function point count Use Case Points In addition to FPA, many modern development projects working on object oriented software apply the concept of Use Case Point (UCP) analysis [Car05] to estimate project costs. UCP is in many ways the same as FPA, since use cases consist of goals and scenarios that provide functionality to a business domain. Therefore these specifications can be used to provide insight into an applications complexity. To derive size and effort estimates, one needs to examine the actors and scenarios of the use case. In a nutshell, use case points are determined by the complexity of the actor, the number of transactions in the scenario, the estimated technical weight of the implementation and the experience of the organization. UCP was originally invented at Rational Software in 93, but over the years multiple modifications were made to the original method. Unfortunately until now no official standard on UCP analysis exists. This makes it hard to compare UCP results of different projects (and organizations) with each other. 2.3 Measurement Methods Over time various methods have been proposed to implement software measurement programmes in development organizations. Well known are; the Goal/Question/Metric (GQM), Practical Systems and Software Measurement (PSM) and Statistical Process Control (SPC). A popular instance of the latter method is Six Sigma. In general however, SPC based methods are dependent on a well-defined and repeatable software processes (say CMM-4). This is in most organizations (including Daniro) not the case. Therefore only the two most practical and promising methods ( 3.2), GQM and PSM are discussed in further detail Goal Oriented Measurement Goal Oriented Measurement can be described as the definition of a measurement programme based on explicit and precisely defined goals that state how measurement will be used [Bri96]. The most widely known method for applying goal-oriented measurement is the Goal/Question/ Metric (GQM) method [Sol99]. The principle behind GQM is that measurement should be goaloriented. Therefore organizations have to define their measurement goals based upon corporate goals. Subsequently, in order to improve their process, organizations need to transform these goals into activities that can be measured during the execution of projects. These actions take place in a top-down fashion. As illustrated in figure 2.2; goals are refined into question that in turn translate to measures. The opposite is true for the analysis and interpretation steps. By measuring 4 There are (minor) differences between the available function point guidelines. However, the two largest function point users groups, IFPUG and NESMA, are working together for almost sixteen years now to eliminate these differences. 9

24 Subjective: If they depend on both the object that is being measured and the viewpoint from which they are taken; E.g., readability of a text, level of user satisfaction. 2.3 Measurement Methods Figure 1 Background and Context Goal 1 Goal 2 Question Question Question Question Question Metric Metric Metric Metric Metric Metric Figure 2.2: The Goal/Question/Metric (working both top-down and bottom-up) [Bas94] attributes, questions can be answered that in turn led to identify whether or not goals are reached. To facilitate these actions, the GQM method contains four phases that are as listed below. These phases can be executed sequentially, however it is also possible to incorporate the GQM in the 3 six step Quality Improvement Paradigm (QIP) [Sol99; Lat98; Bri96]; resulting in an integrated software process improvement and measurement method. 1. Planning phase, in which a project for measurement is selected, defined, characterized and planned. This phase results in a project plan. 2. Definition phase, in which the measurement programme is defined (goal, questions, measures, hypotheses,... ) and documented (in GQM-, measurement- and analysis plans). 3. Data collection phase, where the actual gathering of (raw) measurement data takes place. 4. Interpretation phase, the collected data is processed into measurement results that provide answers to the defined questions. After this phase goal attainment can be evaluated. Goal/Question/Metric evolution The GQM method described above reflects to a large extend the original GQM approach, invented by Basili et al (summarized in [Bas94]). Originally, the GQM approach was developed to evaluate defects in projects of NASA Software Engineering Laboratory (SEL) [Bas95]. Over time, however, the method gained in popularity and since then numerous extensions have been made. A few notable GQM extensions, relevant in the context of this research, are highlighted below. In 96 Park et al. [Par96] studied the GQM method at the Software Engineering Institute (SEI). As part of their research, they extended the method with an extra aspect known as an indicator: a visual representation (e.g. chart or table) of data which helps to answer specific questions. Park et al. state our experience is that sketches of pictures and displays helps significantly in identifying and defining appropriate measures. Furthermore, Park et al. paid explicit attention to the role of mental models in the GQM definition process. In short the addition of, amongst others, indicators and mental models resulted in the establishment of the Goal/Question/Indicator/Metric method. This GQ(I)M method consists out of the following ten steps: 1. Identify your business goals. 2. Identify what you want to know or learn. 3. Identify your subgoals. 4. Identify the entities and attributes related to your subgoals. 5. Formalize your measurement goals. 6. Identify quantifiable questions and the related indicators that you will use to help you achieve your measurement goals. 7. Identify the data elements that you will collect to construct the indicators that help answer your questions. 8. Define the measures to be used, and make these definitions operational. 9. Identify the actions that you will take to implement the measures. 10. Prepare a plan for implementing the measures. 10

25 Background and Context 2.3 Measurement Methods In 97 a European research initiative known as PERFECT [Bir97] was started. The goal of this initiative directed by Fraunhofer IESE, was to assist in measurement-based improvement of software processes. In this light a number of techniques, methods, and tools were developed. Hereby, attention was also given to the GQM method. A notable extension resulting from this project is the distinction between project-level and strategic-level measurement. Until then GQM was mostly used for defining goals, questions and measure within a single project. The PERFECT booklet, however, summarizes the process of applying GQM to organizational goals and issues in multiple projects. In 99 Solingen and Berghout [Sol99] published the first book on GQM. This book was largely based on master projects performed at Schlumberger RPS (involving various aspects of GQM), and partially based on Solingen its Ph.D project [Sol00]. Apart from existing work on GQM, the book DoD emphases Implementation the importance Guidance of providing feedback to stakeholders with respect to measurement data collected in software projects. The primary instrument used for this purpose are feedback sessions. Pre-ACAT During these technology sessions projects measurement need to be data managed is analyzed, correctly. presented The PSM and interpreted process can by be project members. applied The to latter these helps efforts, tojust increase it the can overall to any learning other project. effect of However, measurement the range programmes. of Furthermore, information the needs book discusses may be narrower, a few other since extensions the objectives to GQM; of such these asdemonstrations process modeling are [Bri96]. limited. Moreover, the ideal technology project should not only demonstrate that 2.3.2something Practical can Software be done, but and should Systems also provide Measurement quantitative information about the likely Practical cost Software and resulting and Systems quality of Measurement a product from [Jon03b; the demonstrated Jon03c] ortechnology. PSM in short, Measurement is a measurement methodology can support initiated this requirement. and sponsored by the United States Department of Defense (DoD). PSM served as the reference model for ISO measurement standard 15939, which in turn was used as the basis for Measurement the Measurement results and from Analysis pre-acat Key Process technology Area projects (KPA) of can CMMI be useful ( 2.1.1). in Important the early detail of thestages method of is the that acquisition PSM information life cycle, as driven explained instead below. of goal-driven, like GQM. Implementing a measurement programme using PSM includes, defining organization and project information needs and then 1.2 Measurement selecting measures in the that Acquisition supply information Process relative to those needs. These measures can be selected from several categories: schedule and progress, resources and cost, product size and stability, product quality, process performance, technology effectiveness and customer satisfaction. The acquisition life cycle contains two major activities related to a contract: acquisition In contrast to GQM, PSM is specifically designed to meet management information needs. planning and acquisition management. A separate contract may be established to support Although there are other (important) differences, the PSM and GQM method are in many ways the same each when phase. viewed from a purely technical perspective. For instance PSM evolves like GQM around four comparable phases, as illustrated in figure 2.3. Furthermore the success factors for During each phase of the acquisition life cycle, a measurement process as illustrated in PSM are similar to those of GQM. Figure 1-1 can be applied to support contract requirements. Objectives and Issues Technical and Management Processes User Feedback Analysis Results Core Measurement Process Establish & Sustain Commitment Plan Measurement Measurement Plan New Issues Perform Measurement Improvement Actions Evaluate Measurement Analysis Results and Performance Measures Scope of PSM Figure 2.3: Four phases of the PSM measurement process [Jon03c] Figure 1-1. Software and Systems Measurement Activities 11 2