THE NECESSARY SOFTWARE MEASUREMENT KNOWLEDGE IN SOFTWARE ENGINEERING EDUCATION FROM THE PRACTITIONERS POINT OF VIEW

Transcription

1 THE NECESSARY SOFTWARE MEASUREMENT KNOWLEDGE IN SOFTWARE ENGINEERING EDUCATION FROM THE PRACTITIONERS POINT OF VIEW Monica Villavicencio 1,2, Alain Abran 1 1 École de technologie supérieure, Montréal, Canada 2 CIDIS-FIEC, Escuela Superior Politécnica del Litoral (ESPOL), Guayaquil, Ecuador ABSTRACT The enhancement of the teaching of software measurement in software engineering education must be based on a sound assessment of the impact of previous teaching and the needs in industry. This paper investigates through a survey of software practitioners their opinions about the software measurement knowledge that new software engineers currently bring to their employers organizations. In addition, the survey asked for their opinions on software measurement topics they see as needed in software engineering university curriculum. Survey findings indicate that basic concepts of software measurement are of great relevance for organizations. Additionally, the software measurement process, techniques and tools, measures for the requirements phase and standards are some of the preferred topics to be taught in software engineering programs. Index Terms Software measurement, software engineering, engineering education. 1. INTRODUCTION Software measurement is one of the factors that influence Software Process Improvement (SPI) [1, 2]. According to the literature, some of the key reasons identified for measuring software processes and products are [3-6]: 1) to understand the current software development process; 2) to monitor and verify the progress of projects in course; 3) to improve the quality of the products generated; 4) to improve productivity; 5) to experimentally validate best practices; 6) to observe trends; 7) to know if the process and product goals have been met; 8) to conduct benchmarking; and 9) to make better decisions. Although the number of studies related to software measurement is growing, only a few are addressing the teaching challenges of this subject [2, 7, 8]. To improve the teaching and learning of software measurement topics in software engineering programs at the undergraduate level a series of studies have been planned at École de technologie supérieure (ETS), including the two summarized in [9, 10]. This paper reports on a third survey of software practitioners to gain insights on: A) The impact in industry of the teaching of software measurement in software engineering. B) The software measurement topics that should be taught in undergraduate programs in software engineering. The remainder of this paper is organized as follows. Section 2 describes the research methodology. Section 3 presents the survey findings and Section 4 presents the conclusions. 2. RESEARCH METHODOLOGY To gather data for this study a survey was administered to software practitioners. The sample includes, by design, only professionals working on software process improvement programs as well as software measurement specialists from private or public organizations. Software measurement is a specialized subject and, software measurement-related issues in organizations should be handled by qualified practitioners, such as those in charge of software process improvement programs. The list of worldwide potential survey respondents was obtained from several sources, including: software measurement associations (i.e. GUFPI- ISMA); software measurement conferences organizers (i.e. IWSM-MENSURA, UKSMA) and digital libraries (i.e. engineering village and IEEE Xplore). From these sources, 659 people were contacted by and invited to participate. The leaders of the contacted organizations (i.e. GUFPI-ISMA, IWSM-MENSURA, etc) were asked to forward the to their colleagues and to provide to the researchers the total number of recipients. Based on the information provided by them, 292 extra people received the invitation. In addition, an announcement of the study was published in Linkedin specialized groups (i.e. Measurement & analysis forum). Sixty one practitioners answered the questionnaires from March 30 to May 30, Out of this sample, only 52 were considered valid responses, meaning that all the data was provided for a proper analysis (see Table 1) th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) /12/$ IEEE

2 Table 1: Demographic information (N=52)* Organizations characteristics Location of headquarter North America 27 South America 35 Europe 37 Asia 2 Size of the organization** Micro (< 10 employees) 17 Small (< 50 employees) 8 Medium (<250 employees) 15 Large (250 or more employees) 60 Business sector Aerospace-Automotive 6 Banking 8 Computers and software 33 Defense 2 Education 4 Electricity - water gas 4 Electronics 4 Finance and business services 6 Government 4 Insurance 2 Manufacturing 2 Professional services 6 Telecommunications 14 Other 8 Market Local 29 Both - mainly local 40 Both - mainly foreign 31 SPI programs with SPI 96 without SPI 4 Certification with certification 39 without certification 62 Measurement programs with measurement program 52 without measurement program 48 Role of the respondent Requirements or Business Analyst 8 Software architect or Designer 8 Developer-Programmer 6 QA-Tester 6 Project manager leader 27 IT-MIS manager 4 Metrics manager consultant 35 Other 8 (*) The total may be greater than 100 due to rounding (**) European commission of micro, small and medium-sized enterprises[11] hand, this is a positive bias in the sense that they are more competent to assess and answer this specialized survey within their domain of expertise; much more that the overall none specialized population of software and non software engineers. The survey instrument was pilot-tested with two distinct questionnaires for two types of respondents: practitioners and consultants. This pilot-testing was performed twice: 1) using at least 3 individuals to test the initial version of each questionnaire; and 2) using the IWSM-MENSURA 2010 attendees who volunteered to take part in the survey (see details in [10]). The two questionnaires of the pilot-test were next consolidated into one instrument since the initial versions of the questionnaires had similar questions. Based on the lessons learned from the pilot tests, the wording and structure of the questions were improved. 3. RESULTS AND ANALYSIS This section presents the findings of the questionnaire for practitioners. The questionnaire included 22 questions divided in two sections: 1) general information; and 2) software process improvement and measurement. Table 1 presents the demographics of the 52 respondents representing organizations from 18 countries. From them, 96 of the organizations have software process improvement programs and 39 are certified. Table 2 shows a detailed view of the SPI programs followed by the respondents organizations. It is important to mention that this classification refers not only to organizations following SPI programs but also to those that are in the process of getting certified. Table 2 shows that 46 of the respondents have in house SPI programs, 40 are with ISO9001 and 37 with CMMI. Table 3 shows that ISO9001 and CMMI certifications are prevalent within the organizations already certified. Table 2: SPI programs followed by organizations Type of SPI program e 4 In house 46 CMMI 37 ISO ISO ISO Six Sigma 17 Other 23 This survey sample is partially biased since people attending international software measurement conferences are therefore typically much more specialized in the main topic of such conference than the general population of software developers or software engineers. On the other

3 Table 3: Certifications obtained by organizations Certification CMMI 45 ISO ISO Other 15 Table 4 presents the percentages of employees in the organizations (column 1) that respondents categorized by educational level (columns 2 to 5): high school, bachelor, masters and PhDs. To better understand this table, what follows is a brief explanation of how the employees are categorized in high school level: 50 of the organizations do not have employees in this category; 27 of them have equal or less than 25 of their employees with high school level; 15 with more than 25 and less than 50 employees in this category; and so on. Employees in the organization Table 4: Education level of employees High School Bachelor Master PhD Equal or less than More than 25 and less than Equal or more than 50 and less than Equal or more than For a better insight on the categorization of employees in certified organizations, Table 5 shows the data divided in certified and non certified organizations. According to these percentages, certified organizations represented in this survey hire more people with master and PhD degrees than those without certification. Table 6 presents the percentages of certified organizations according to their size; while Table 7 shows the type of certifications of these organizations. It is interesting to highlight that the certified-micro organizations have an employee population with master and PhD degrees. The representativeness of this sample with respect to the overall population is not known. Table 6: Certified organizations according to their size Size of the organization Micro 15 Small 5 Medium 10 Large 70 Table 7: Certifications vs. the organizations size Type of Certification (N2=20) Size of the ISO CMM organization 9001 ISO27000 Other Micro Small Medium Large To explore the characteristics of employees, respondents were asked to rank the level of software measurement knowledge possessed by new employees coming with bachelor and master degree level (see Table 8). An ordinal scale from none to more than expected was used to capture the perception of respondents regarding the initial knowledge that their employees or team members had at the beginning of their measurement activities. It is worth mentioning that each level of the scale had an explanatory text to avoid misinterpretations and subjectivity. For example, Little and e were presented as follows: Table 5: Educational level of employees in certified and non certified organizations High School Bachelors Masters PhD Employees Cert. Cert. Cert. Cert. Cert. Cert. Cert. Cert Equal or less than More than 25 and less than Equal or more than 50 and less than Equal or more than

4 Little : New employees have heard about some measurement concepts and remember them, but they are not able to work right away in measurement activities into the organization. Some training is required to let them understand how to apply the theory in practice. e : knowledge of software measurement. A whole training program is needed to teach them the basis of measurement and their application in the real world. Table 8 shows that nearly 60 of respondents consider that new employees with a bachelor degree have little or none knowledge in software measurement. For employees with a master s degree, nearly 27 of respondents believe that these employees have good or more than expected knowledge in this subject compared to the 13 reached by the bachelors in these scale levels. Nevertheless, 50 of respondents think that masters have little or none knowledge in software measurement. To complement the collected data, respondents were asked to choose, in order of importance, three software measurement topics that should be emphasized in university courses, according to the relevance for their organizations. The questionnaire included the 12 topics of the software measurement body of knowledge [12], known as the SWEBOK guide chapter 12 [13]. They were listed in the same order of appearance of the abovementioned document. Only one of the topics was not selected by respondents as the three most relevant, that is, Measures for the maintenance process. Table 9 summarizes the percentage obtained for each topic. The first column ( Total) shows the percentage without considering the certification status, while the others ( Cert. and Cert) do consider certification for comparison purposes. The first topic of the list Basic concepts in software measurement was considered as the most important by respondents regardless of the certification status of their organizations. For positions of importance 2 and 3 there are differences between certified and non certified organizations. For the former, the topics B (The measurement process) and C (Measurement standards) are both important. On the other hand, non certified organizations are of the opinion that the topics J (Software engineering management measures) and D (Techniques and Tools) are the ones that deserve positions 2 and 3. One possible explanation for these results might be that non certified organizations have difficulties managing their processes: that may be why they need measures, techniques and tools. Another important finding is that practitioners opinions vary significantly for two of the software measurement topics, including: F (Measures for the requirement phase) and I (Measures for testing). In the case of certified organizations, it seems to be crucial to do effort estimation for project planning, assuming that the functional size of the product (measures for requirement phase) is known. It is also important for these organizations to know, for instance, the number of the specified, executed and failed test cases (measures for the testing phase) to determine the cost of software quality. Table 8: Perceived software measurement knowledge of new employees Perceived knowledge in software Educational level of employees measurement of new employees Bachelor Master More than expected 9 8 Good 4 19 Normal Little e Table 9: Software measurement topics to be taught at university courses Software Measurement Topics Total Cert. Cert. A. Basic concepts in software measurement B. The measurement process C. Measurement standards D. Techniques and tools E. Repositories for quantitative data F. Measures for requirement phase G. Measures for the design phase H. Measures for construction phase I. Measures for testing J. Software engineering management measures K. Measures related to quality CONCLUSIONS AND FUTURE WORK The present study was designed to gain insights about the opinion of software practitioners regarding a number of issues related to software measurement teaching at the university level. The focus of this paper was on 3 key issues: the educational level of the employees population working at software related organizations or software engineering departments; the level of software measurement knowledge of new employees; and the software measurement topics that practitioners consider as important to be taught at university. The findings suggest that there is a need, especially at the undergraduate level, for improving the teaching of software measurement in higher education: new software engineering hires from universities present a lack of knowledge on software measurement subject. This confirms Jones 2008 concern about the absence or inadequate training on software measurement that undergraduate and graduate students receive at universities [14]. Another important observation is that in this survey sample working

5 in software process improvement and measurement programs, is mainly composed by people holding a bachelor diploma. Furthermore, the results indicate that no clear direction is observed regarding a consensus on software measurement topics in university programs. This means that the differences in opinions of people coming from certified and non certified organizations do not allow scholars to have clear guidance for improving the teaching of software measurement. The findings, however, provide some initial thoughts for further research to determine common areas that have to be emphasized in university curriculum for software engineers. To address some of the educational issues identified in this survey, a Delphi study has been planned for It will include a survey for asking software measurement experts the same questions regarding the topics that have to be emphasized in higher education. They will also be asked about the complementary skills to be instilled in university students. There will be two experts panels: university teachers and practitioners. Both types of participants will be selected according to certain criteria to assure their level of expertise. REFERENCES in International Conferences on Software Measurement IWSM/MetriKon/Mensura, Stuttgart, Germany, 2010, pp [10] M. Villavicencio and A. Abran, "Facts and Perceptions Regarding Software Measurement in Education and in Practice: Preliminary Results," Journal of Software Engineering and Applications vol. 4, pp , [11] EuropeanCommission. (2009, 28 Abril 2010). Definition of micro, small and medium-sized enterprises. European Commission, Enterprise and Industry. Available: [12] A. Abran, et al., "Software Measurement Body of Knowledge," in Encyclopedia of Software Engineering vol. 1:1. London: Taylor & Francis, 2010, p [13] P. Bourque, et al. (2008, October 15, 2010). SWEBOK Version Available: cms/computer.org/swebok/measurementka-draft- Feb2008.pdf [14] C. Jones, Applied Software Measurement: Global Analysis of Productivity and Quality, Third ed.: McGraw-Hill Osborne Media, [1] A. Rainer and T. Hall, "A quantitative and qualitative analysis of factors affecting software processes," Journal of Systems and Software, vol. 66, pp. 7-21, [2] C. Gresse von Wangenheim, et al., "Empirical evaluation of an educational game on software measurement," Empirical Software Engineering, vol. 14, pp , [3] N. E. Fenton and S. L. Pfleeger, Software Metrics: A Rigorous and Practical Approach. London: PWS Publishing Co., [4] W. Florac and A. Carleton, Eds., Measuring the software process. Statistical process control for software process improvement (The SEI Series in Software Engineering. Westford, Massachusetts: Addison Wesley, [5] A. Gopal, et al., "Measurement programs in software development: determinants of success," IEEE Transactions on Software Engineering,, vol. 28, pp , [6] C. Gresse von Wangenheim, et al., "Software Measurement for Small and Medium Enterprises - A Brazilian-German view on extending the GQM method," in 7th International conference on Empirical Assessment in Software Engineering (EASE), Keele University, Staffordshire, UK, April 8-10th, [7] M. Bush and N. Ashley, "Metkit: toolkit for metrics education," Software, IEEE, vol. 10, pp , [8] J. Cuadrado-Gallego, et al., "Visual learning techniques for software measurement," presented at the 4th International Conference on Computer Science and Software Engineering, Montreal, Quebec, Canada, [9] M. Villavicencio and A. Abran, "Software Measurement in Software Engineering Education: A Comparative Analysis,"