A Framework for Teaching Software Development Methods

Computer Science Education Vol. 15, No. 4, December 2005, pp. 275 296 A Framework for Teaching Software Development Methods Yael Dubinsky* and Orit Hazzan This article presents a study that aims at constructing a teaching framework for software development methods in higher education. The research field is a capstone project-based course, offered by the Technion s Department of Computer Science, in which Extreme Programming is introduced. The research paradigm is an Action Research that involves cycles of data collection, examination, evaluation, and application of results. The research uses several research tools for data gathering, as well as several research methods for data interpretation. The article describes in detail the research background, the research method, and the gradual emergence process of a framework for teaching software development methods. As part of the comprehensive teaching framework, a set of measures is developed to assess, monitor, and improve the teaching and the actual process of software development projects. 1. INTRODUCTION It is a well-known fact that the process of software development is a complicated task, composed of many aspects, such as cognitive, social, and technical ones (Hamlet & Maybee, 2001; Tomayko & Hazzan, 2004). Accordingly, the academia has a significant and non-trivial role in the education of future software engineers toward this multifaceted challenge. The objective of the research described in this paper is to construct a framework for the teaching of software development methods (SDMs) not by means of a lecture, but rather by basing the teaching process on active learning (Revans, 1983/1988), in which the students actually apply an SDM. This work supports recent calls for the integration of software development skills into computer science and software engineering programs (Denning, 1992; Meyer, 2001). The main theoretical contribution of this research is by shaping a teaching framework composed of conceptual principles and teaching practices for projectbased capstone courses in higher education. In addition, since the research subject is strongly related to the software industry in which students are supposed to work, and to the community of customers and users of the software industry, the research has also a practical contribution. Specifically, the application of SDMs in general, while *Corresponding author. Computer Science Department, (adjunct lecturer), Taub Building, Technion Israel Institute of Technology, Haifa 32000, Israel. Email: yael@cs.technion.ac.il ISSN 0899-3408 (print)/issn 1744-5175 (online)/05/040275-22 Ó 2005 Taylor & Francis DOI: 10.1080/08993400500298538

276 Y. Dubinsky and O. Hazzan increasing customer collaboration in particular, is part of the process improvements that occur nowadays in the software industry. The role of academia is to train software engineers who will be able to contribute and lead such processes. Teaching and using SDMs in software engineering courses can partially answer the needs for technical, managerial, and social skills expressed in software development processes. In Hazzan and Dubinsky (2003) we described ten conceptual ideas principles that create the basis for our framework, and examined each of them from both a pedagogical and an organizational viewpoint. Based on these principles, five practices were developed. This article elaborates on this framework, describes its construction process, and presents a detailed description of the five said practices. The second section of this article describes the research background. In the third section, we elaborate on the research approach and describe the research tools and the data analysis methods. After the presentation of the practices, we summarize the article in the fifth section. 2. RESEARCH BACKGROUND 2.1. Research Field: Project-based course One of the advanced courses offered by the Department of Computer Science at the Technion, and taught by the first author for the past seven years, is the project-based capstone course Projects in Operating Systems. Since the summer semester of 2002 the course has been taught in a studio-oriented format. The studio is the basic learning method used in architecture schools. In such studios, students develop projects, closely guided by an academic coach, while performing on-going reflection, both on what is created and on the creation process itself (Kuhn, 1998; Tomayko, 1996). Hazzan (2002) analysed the suitability and implementation of the studiobased teaching approach for software engineering education. Extreme Programming (Beck, 2000; Beck with Andres, 2005) on which we elaborate later, is the development method chosen to be introduced into the studio in that course. Since the summer semester of 2002, the above course format has been applied in the Operating Systems Projects course in 31 projects. The initiative is highly supported by the Department of Computer Science, which, among other things, provides the required resources and equipment as is described in what follows. Each team of 10 to 12 students works on a specific operating system project during the entire semester, in its own studio, which is equipped with computers, tables, and white boards. Each team has an academic coach who guides the development process (but does not manage it!). Attendance of all students at all of the weekly sessions lasting between 2 and 4 hours is compulsory. In these meetings, Extreme Programming is taught and employed. In between sessions, communication is conducted via an electronic forum, which is part of the course web-based environment. Although the new format of the Operating Systems Projects course also includes the teaching of an SDM, the operating systems-related material traditionally taught in the course has remained

A Framework for Teaching Software Development Methods 277 unchanged. Furthermore, the introduction of Extreme Programming into the course has been found to enrich the development environment with respect to topics such as customer needs and process management (Dubinsky & Hazzan, 2004). 2.2. Software Engineering Methods, the Agile Approach, and Extreme Programming The field of software engineering is relatively young. One of its main aims is to provide principles and guidelines for software development processes. The main problems encountered by practitioners in the field, as well as by customers of software products, are bugged software systems and projects that exceed their planned costs, deviate from their estimated schedule and fail to meet their deadlines. In the past 20 years, professionals in the field have tried (in vain) to find a silver bullet (Brooks, 1987) that would solve these main problems for all software projects. So far, and as Brooks specifically noted in his 1987 article, such a solution is not expected to be found. However, increased awareness to these common problems has led many professionals to realize that software development processes should and can be improved. Agile SDMs have emerged over the past decade as a response to the abovementioned unique problems that characterize software development processes (Highsmith, 2002). In general, agile SDMs emphasize customer needs, communication among team members, short releases and heavy testing throughout the entire development process (See the Manifesto for Agile Software Development http:// agilemanifesto.org/). These principles are implemented quite differently by different agile SDMs. Extreme Programming (XP) is one of the agile SDMs and is accepted as the most commonly used of them all. XP emphasizes four values as the backbone principles of the method (Beck, 2000), namely communications, feedback, simplicity, and courage. 1 These values are implemented using 12 practices, which are outlined as follows: XP programmers work at a sustainable pace, on small releases that are divided into frequent small iterations. The customer is always on-site, and helps make the decisions during the planning game of each release and iteration. According to the determined scope, the development work is estimated and divided among the team members, ensuring a balanced workload for each. A metaphor is used to enhance communication between the customer and the team, and among the team members themselves. The team members perform (unit) testing, as well as periodic code refactoring, the goal of which is to obtain a simple design. Continuous integration of the code is performed by adding each small part of code to the system and running the full suite of automated tests. This provides the programmers with immediate feedback and small changes are subsequently performed in the code if needed. The pair programming practice, that is, production of code by two programmers working together, ensures that each line of code is inspected and reviewed by at least two pairs of eyes. During the development process there is a turnover of pairs according to the developers knowledge and needs. The code complies with coding standards determined by the team and is collectively owned by the team. Thus, a greater number of programmers view, simplify, and

278 Y. Dubinsky and O. Hazzan inspect the code, in addition to the original pair who wrote it. This in turn increases the teammates commitment to the entire process of software development. As can be observed, XP introduces a new approach into the software development world. More specifically, unlike SDMs that specify development stages, XP outlines specific activities that are to be conducted on a daily basis in the course of software development processes. This unique characteristic of XP, together with its acceptance as the leading agile SDM, led to the decision to introduce it into the Projects in Operating Systems course. A similar approach was applied by other researchers (Allen, Cartwright & Reis, 2003; Melnik & Maurer, 2003), though not within the studio framework. 3. RESEARCH FRAMEWORK 3.1. Research Paradigm Action Research is the research arena in which researchers study the domain within which they work, employing a systematic and criticized process. This research approach is usually adopted in cases in which researchers wish to improve their understanding of an action they perform, identify problems, find solutions to these problems and examine the application of these solutions in order to improve the process undergone in the research field. The origin of this research perspective is Kurt Lewin s work (Lewin, 1946, reproduced in Lewin, 1948), which is based on a spiral set of steps, each of which is composed of a circle of planning, action and factfinding about the result of the action (p. 206). The discussion about Action Research tends to fall into two distinctive camps (Smith, 1996). The British tradition views Action Research as the research approach oriented toward the enhancement of direct practice, and largely associates it with educational research. The second tradition, more accepted in the USA, conceives of Action Research as the systematic collection of information that is designed to bring about social change (Bogdan & Biklen 1992, p. 223). It seems that the research described in this article touches on both traditions. As can be observed, Action Research inspires an inductive research method in which a framework is developed by means of an iterative process, in which the researcher develops a theory while simultaneously testing it and gathering additional data in the course of the research itself. When a study is conducted within the paradigm of Action Research, there is a continuous interplay, which is an integral element of the ongoing research process, between data collection, data analysis, and the application of intermediate results. This research paradigm is appropriate for the research described in this paper for at least two reasons. First, as far as we know, the teaching of SDMs by using an active learning approach is not common. Therefore, an appropriate teaching framework was not available at the time when the first author wished to introduce this approach into her course. When the construction of such a teaching framework was decided on, Action Research seemed perfect to guide this process.

A Framework for Teaching Software Development Methods 279 Second, the Action Research paradigm was deemed suitable for our research because it was possible to apply it in a very natural manner. Specifically, the research field an academic environment consists of semesters of study enabled to establish an iterative process during which our proposed teaching framework was shaped in an inductive process, while the results obtained so far were constantly applied, and the suitability of the developed framework was continually examined and compared to newly gathered data and findings. 3.2. Research Method The research paradigm (Action Research) is applied in our research by the use of a measured-qualitative research method, that is, a qualitative research method accompanied by a set of measures which provides complementary quantitative data with respect to several factors evolved during the research. In what follows we describe the main characteristics of the qualitative research method, along with an explanation regarding its suitability to our research. This analysis is based on Cook and Reichardt, 1979, Strauss, 1987, and Patton, 1990. Qualitative research is interpretive. Its objective is to explain situations and processes. Indeed, in our research we aim to describe the process of teaching SDMs. A qualitative researcher feels a relative lack of control. Indeed, at the beginning of this research, we felt that the integration of XP into the course activities might reveal interesting and important findings that, eventually, would help us in the construction of a teaching framework. Although at that time we had neither an initial idea about the teaching framework nor a complete design for the study, the fact that we had adopted the Action Research paradigm for our research helped us to navigate the vague and unclear first steps of the course of research. Qualitative research is inductive. In other words, categories and theories emerge from the data. Indeed, our teaching framework has evolved based on data analysis and is, in fact, still undergoing refinement. Qualitative research is also process oriented and insider centered process. In our case, the first author works in the research field, guides a team of students each semester, works in parallel to three other academic coaches who each guides his/her own team of students. The second author observes studio activities and participates in meeting of the academic coaches, and is also involved in the investigation of human aspects of software engineering (Tomayko & Hazzan, 2004). Both authors conduct XP training sessions for the academic coaches. Our set of measures has evolved naturally in the course environment since one of the roles of the academic coach is to evaluate the work of each student as well as the software product that the team develops. The emerged measures relate to two main categories. The first category is the customer category that deals with meeting the customer s requirements and schedule. The second category is the process category that deals with management aspects of the project such as roles and team work (see later).

280 Y. Dubinsky and O. Hazzan The application of the research paradigm (Action Research) using the measuredqualitative research method led us to the establishment of an iterative process that consists of data gathering, data analysis, reflection, refinement of the framework constructed so far, and so on and so forth (data gathering, data analysis etc.). Specifically, the research began with the collection of data in the form of observations conducted in the research field, interviews with students and academic coaches, projects electronic forums and questionnaires. After the analysis of this data, initial findings started to emerge. Based on these preliminary findings, an initial rough and general version of the framework was formulated. At subsequent stages of the study, the framework was cyclically refined through the above-described iterative process. Such iterations reoccurred during our research at various intervals, and had different cycle times. Some iterations were long (a full academic year); other iterations were shorter (one semester); yet, there were even shorter cycles, such as those that were performed when we wished to improve our understanding of a specific topic. The iterations were characterized also in terms of practices. Different iterations were carried out for different practices. From the accumulative interpretations obtained during all iterations of all lengths, a framework was gradually formulated and conclusions were derived. The above process was applied during the past nine semesters (Summer of 2002 through Spring of 2004 2005) in 31 projects, developed by more than 300 students who worked in teams of 10 to 12 students each. 3.3. Research Tools This section describes the research tools that were used in this research for the purpose of data gathering. Videotapes. Each semester, all of the meetings held by one team were videotaped. The videotapes were viewed during, as well as after, the semester in which the data were gathered. Focus was placed mainly on the discourse that took place during the meetings, in order to find repeated discourse patterns. The videotapes were complemented by the second author s observations of two teams throughout the entire semester period. Student reflections. After each weekly meeting, the students reported their reflections using the web-based tool that was used for course management. These reflections addressed general themes as well as subjects dealt with at the most recent meeting. Electronic forum. In between the weekly meetings, the students complemented their communications using an electronic forum. Analysis of the forum messages revealed the story of the group in general, and the group dynamics and status of the project development in particular. Interviews. In order to improve our understanding of specific issues, whether they emerged from observations, videotapes, or student reflections, open

A Framework for Teaching Software Development Methods 281 interviews and semi-structured interviews with students and academic coaches were conducted. Questionnaires. Questionnaires enabled us to examine the responses of a relatively large number of participants. Usually, after the questionnaire results were gathered and analysed, we refined our findings by means of interviews. Questionnaires were also used in some instances in preparation for a retrospective process. Researcher s diary. The academic coach (the first author) kept a researcher s diary, which included personal observations, reflections, thoughts, short notes made on different occasions, etc. The data gathered using these research tools provided a multi-faceted picture of the process of teaching and learning SDMs. It enabled us to see the development environment, hear the voices of the students and academic coaches, and read their reflections and questionnaire answers. In addition, the data were used for the measures required for the assessment of the development process. 3.4. Data Analysis Methods The data gathered using the above research tools were analyzed using common qualitative research methods (see upper part of Table 1) as well as quantitative data analysis that fits our set of measures (see lower part of Table 1). In what follows, we describe each of the data analysis method. Constant comparison is a data analysis method in which the researcher examines the data, codes (transcribes) events and behaviours, and then begins an analysis process Table 1. Data analysis methods and their corresponding research tools Data analysis method Constant comparison Hermeneutical analysis Discourse analysis Narrative analysis Measures of quantitative data Research tools Interviews with students Interviews with academic coaches Observations Questionnaires Questionnaires Electronic text: Electronic forum and students online reflections Videotape of studio sessions Record of academic coaches training Electronic text: Electronic forum Electronic text: Students online reflections Questionnaires Electronic text: Electronic forum and students online reflections

282 Y. Dubinsky and O. Hazzan based on the comparison of different pieces of data with the objective of identifying consistencies that may reveal categories. In our research, for example, we transcribed interviews with the course s academic coaches and constantly compared findings that emerged from them in order to reveal categories that later influenced the construction of the Coaching Practice (see Dubinsky & Hazzan, 2003, and later in this article). Hermeneutical analysis is a data analysis method in which the researcher makes sense of a written text while looking for its meaning from the perspective of the people involved in the situation (Manen, 1990). The researcher uses the participants own words to tell their stories and to describe social interactions among them. In our research, for example, we used this method for the analysis of students written reflections, in which they described their perception of their individual role in the software project (see Dubinsky & Hazzan, 2004, and later in this article). Discourse analysis is a linguistic analysis of an ongoing flow of communication (Gee, 1992). The researcher refers to linguistic aspects, while considering the way people conceive a specific topic and the influence this conception has on their behaviour. In our research, for example, we used discourse analysis to study videotapes of studio activities. Results obtained fromthis analysis were used in the construction of the Course Practice. Narrative analysis is the study of the speech or narrative of the individual (Reisman, 1993). Unlike discourse analysis, in which the interaction among individuals is at the focus of the analysis, in narrative analysis the focus is placed on the story as it is told by the individual. This kind of analysis was used in our research in different ways. For example, narrative analysis was used when we examined students personal narratives presented as reflections by using the web-based tool. This kind of analysis was also used in the analysis of videotapes. Narrative analysis is indirectly illustrated later in this article. Measuring analysis provides a way to assess by quantitative means activities that occur during the development process. The measuring is performed after the activities have been conducted by using quantitative data. In our research, quantitative data is gathered from questionnaires, students communication via the course electronic forum and students reflections after each activity. 3.5. Validity and Reliability Ways exist to ensure validity and reliability in qualitative research and quantitative measures (Kirk & Miller, 1986). In what follows, we describe in brief how we ensured validity and reliability in our research. In order to ensure validity, the consistency of our results was checked using several resources (students, academic coaches, and researcher), several research tools and several data analysis methods. Furthermore, the research was repeated 31 times during the past nine semesters, with two to four studios and three to five academic coaches each semester (one of whom is the first author), who worked according to the

A Framework for Teaching Software Development Methods 283 same schedule and guidelines. Thus, the teaching framework that emerged was rechecked and re-refined each semester. Reliability is ensured by letting different people analyse the same data. Thus, for example, videotapes were examined by two (and sometimes three) researchers and written material was examined by the two researchers. In some cases, we asked different groups of researchers to analyse written material, and then compared the different analyses obtained. 4. RESEARCH FINDINGS Our research outcome is composed of a comprehensive teaching framework that emerged during the research. Based on conceptual principles, five practices were developed. The practices refer to the teaching activities and enable the academic staff to evaluate the students as well as the development process. 4.1. A Framework for Teaching a Software Development Method The framework for teaching software development methods (SDMs) is based on ten conceptual principles (Hazzan & Dubinsky, 2003). Specifically, Principle I addresses the course structure; Principless II V address the actual teaching of the SDMs; Principles VI IX address the people involved in the educational environment the students and the coaching team; and Principle X is a meta-principle that suggests action over preaching. The ten principles are presented in Table 2. In order to facilitate memorization of the principles, we have given each of them a short nickname. Table 2. Principles of the emerged framework for teaching software development methods Number Category Nickname Single sentence description I Course Project-based Course structure: Project-based team-based course II Teaching Cognition Teaching a software development method: What aspects should be emphasized? III Adjustment Adjustment of the software development method to an academic course framework IV Projection Projection of the software development method s notions into the project development environment V Connectivity The software development method in the context of the world of software development VI People Evaluation An expression of the software development method in student evaluation policy VII Listening Listening to students conceptions, objections, and expressions VIII Reflection Students reflections and progress diaries IX Coaching Coaches conceptions, hesitations, and feedback X Meta Inspiration Inspiring the actual use of the software development method rather than preaching it

284 Y. Dubinsky and O. Hazzan While the ten principles are conceptual ideas on which the framework is based, the five practices are more practical and address the actual implementation of the teaching framework. The first practice is the Course Practice. It presents a detailed course structure constructed in the course of the research. The second is the Role Practice. This practice presents a detailed role scheme that is composed of four groups of roles. One of the students roles is a coach; but it is different than the role of the academic coach. Specifically, while the student-coach is responsible for matters related to the software project development, the academic coach is responsible for the teaching and guidance part, thus assists all role holders. The third practice is the Measurement Practice. It presents measures that help to assess the development process. The fourth practice is the Coaching Practice. The fifth practice is the Evaluation Practice, described below with the details of the evolved grading policy. There are mutual relationships among the practices. Specifically, each practice contributes to the development and formulation of other practices. For example, the Evaluation Practice, which consists of the grading policy, leans on the course schedule and the development measures which are part of the Course and the Measurement Practices respectively, on the role of the student which is part of the Role Practice, and on the academic coaches feedback which is part of the Coaching Practice. In line with the Action Research paradigm, these practices were developed based on the ten principles through an iterative process, and their contribution and fitness to the overall emerging teaching framework was examined throughout the research process. As it turns out, three of the ten principles are fundamental to our research field. These are principles I, III, and X, and they can be found at the basis of each of the above practices. In addition to these fundamental principles, each practice is based on practice-focused principles, according to its context. Table 3 presents the relationship between the practices and the two kinds of principles. 4.2. The Teaching Practices In what follows we delve into the details of the five practices mentioned above. 4.2.1. The Course Practice. Since an SDM is, by nature, a complex topic, several questions must be addressed when one attempts to teach a specific SDM: What aspects of the SDM should be emphasized? Should all of the SDM ideas be taught? Perhaps only some of them? Which ones? This section addresses these questions from a cognitive perspective. Specifically, we describe a course structure that takes into consideration the cognitive aspect of learning SDMs. The research tools used for the construction of this practice were videotapes of project meetings, student reflections, forum messages, questionnaires, weekly interviews with two students from one project team, and the researcher s diary. In line with the Action Research, the suggested course structure was developed over the nine last semesters, and continues to be further developed. The teaching framework for one semester is composed of one release that is composed of three

Table 3. Practices according to the principles they are based upon Practice Fundamental principles Practice-focused principles Course Role Cognition Projection Projection Evaluation Listening Reflection Connectivity Measurement Project-based Evaluation Adjustment Coaching Inspiration Reflection Listening Connectivity Coaching Evaluation A Framework for Teaching Software Development Methods 285 Connectivity Listening Coaching Evaluation Coaching Reflection iterations. The framework is based on the following three working assumptions: a) each project is developed by a team of 12 students who are guided by a coach; b) each team has an XP studio dedicated to the project development; and c) team members meet for mandatory weekly meetings lasting between 2 and 4 hours. The first iteration deals with the topic of the project (in our case, operating systems) and with software engineering issues. It enables the students to study all of the related issues. In this iteration, the team s spirit and atmosphere are created. This aspect is extremely important, since the team members are usually not familiar with each other prior to this course. With respect to the development itself, the purpose of the first iteration is to develop a skeleton for the project so as to enable the performance of additions and improvements during the second and third iterations. By the time of the second iteration, the students are more relaxed, are more familiar with the development method and with their teammates, and most of them invest efforts in order to increase and improve the project outcomes. Most of the meetings are dedicated to development activities. Features and improvements are added to the project in the two remaining iterations. The detailed semester schedule is presented in the Appendix. The purpose of each meeting, along with its main activities, and the XP values and practices on which it is based, are presented. For reasons of space limitation, these descriptions are presented in a very condensed format. As Table 3 indicates, several of the ten conceptual principles have influenced the shape of this practice (specifically, Principles I, II, III, IV, and X). However, for

286 Y. Dubinsky and O. Hazzan reasons of space limitation, in what follows, we examine the course structure by focusing only on how it is based on and related to Principle II Cognition, which deals with the cognitive aspect of learning a SDM. The course schedule is constructed according to the following suggested mapping of an SDM s ideas. Such mapping highlights core ideas of an SDM, and for each specific idea, reflects the complexity of teaching it. Specifically, the main ideas of an SDM are mapped along two dimensions: Aspect and Cognitive Awareness. On the Aspect dimension, the SDM s ideas are divided into either a code/technical aspect or a human/social aspect. The second dimension maps the main ideas of the SDM according to the level of cognitive awareness (or cognitive complexity) required to implement each of the ideas throughout the development process. This mapping technique is illustrated using the dozen XP practices. Table 4 presents the two-dimensional mapping of the XP practices based on a cognitive analysis of the different practices. Accordingly, some of the practices are easily understood and, accordingly, they require low cognitive awareness when adopted (e.g., sustainable pace); other XP practices require high cognitive awareness and thus, in most cases, are harder for implementation (e.g., simple design); and yet others require intermediate level of cognitive complexity (e.g., coding standards). As can be seen, the emerged course schedule (see Appendix) is compatible with this mapping. The different practices of XP are introduced gradually so that, on the one hand, they do not overwhelm the students but, on the other hand, they do indeed address all aspects of the SDM. In addition, practices that require high or intermediate cognitive awareness (such as metaphor, refactoring and test driven development) are revisited several times during the semester period. In this way, students can come to an understanding of these practices in stages, based on their current and updated experience in the process of project development. This is consistent with the constructivism perspective, which ascribes the learner s experience a significant role in learning processes (Confrey, 1995; Davis, Maher & Noddings, 1990; Kilpartrick, 1987). Table 4. Mapping of the XP practices Cognitive awareness Code/Technical perspective Human/Social perspective High Refactoring Metaphor Simple design Collective ownership Intermediate Coding standards Pair programming Test driven development Planning game Low Small releases Customer on site Continuous integration Sustainable pace

A Framework for Teaching Software Development Methods 287 4.2.2. The role practice. This section deals mainly (but not only) with adjustments to be made in an SDM when it is applied to an academic university software project (see Principle III Adjustment). We illustrate this idea with a student s role scheme for software teams, together with quantitative measures. In Dubinsky and Hazzan (2004) we suggested an extended role scheme for software development teams and described the evolution of this scheme over six (out of nine) semesters. This scheme emerged as a result of the need to adjust the SDM to the academic environment. According to this scheme, all teammates are developers and, in addition, each teammate has a special personal role that addresses one aspect of software project management. Thus, the project management is covered by all teammates. Table 5 presents the suggested role scheme for an academic XP team. As can be seen, some of the roles are XP roles, like coach, tracker, tester, and customer; other roles represent XP practices that thinking our opinion should be emphasized. Various research tools were used in the construction of this practice. First, interviews with students, interviews with coaches, observations, and questionnaires were analysed by means of constant comparison. Second, questionnaires, the electronic forum, students online reflections and tape recordings of training sessions given to the coaches were analysed using hermeneutical analysis and measurement. In what follows, we briefly illustrate data related to the construction of this practice. We limit our illustration to data taken from written online reflections that were analysed using the hermeneutical analysis method. The following data is only small portion of the data that is used for the construction of this practice. In fact, the practice was developed in stages based on the analysis of data gathered using different research tools (see above), which helped us in different stages of the practice construction process. Specifically, feedbacks, such as the ones presented in what follows, when expressed at late stages of the practice formulation, naturally supported the practice formulation. At the same time, when we faced problems with intermediate stages of the practice formulation and implementation, in the spirit of Action Research we looked for ways to solve these problems. When interpreting the written reflections of students, we found that, among other subjects, students expressed their feelings and awareness with respect to their roles. Most of the students reported that they handled this additional responsibility well and benefited from it. For example, one of the Customers wrote: The role gave me a real life feeling, rather than having a predefined task and just performing it. This is very real, a customer with requirements. Satisfaction in being able to gain an overall view of the entire project in addition to the accomplishment of specific development tasks is also expressed. For example, one of the Integrators wrote: I enjoyed seeing that everything was integrated.... This data, as well as additional reflections, indicate that student accountability, as well as commitment and involvement, increased significantly when the Role Practice was used.

288 Y. Dubinsky and O. Hazzan Table 5. Roles in an academic XP team Group of roles Role Description Leading Group Coach Coordinates and solves group problems, checks the web forum and responds on a daily basis, leads some development sessions. Tracker Manages the group diary, measures group progress with respect to the estimations and test scores, Methodologist manages and updates the boards. Learns the software development method applied in the course and guides the software process when required, guides and supports the other team members with their role. Customer Group Customer Tells customer stories, makes decisions pertaining to each iteration, provides feedback, defines and develops acceptance tests. Acceptance tester Works with the customer to define and develop acceptance tests, learns the topic of test-driven development and instructs it to the other team members. Maintenance Group Presenter Plans, organizes and presents version presentations, Documenter Installer demos, and time schedule allocations. Plans, organizes and presents the project documentation: process documentation, user guide, and installation instructions. Plans and develops an automated installation kit, supports and instructs other teammates as to the appropriate way to develop software for easy and correct installation. Code Group Designer Maintains current design, works to simplify design, searches for locations in the software that need refactoring and ensures proper execution of such. Code reviewer Unit tester Integrator Establishes and refines group code standards, searches for development tools that can help the team, guides pair programming, guides and supports in the maintaining of the code standards and tools. Learns about unit testing, establishes an automated test suite, guides and supports other team members in the development of unit tests. Establishes an integration environment including source control, publishes rules pertaining to the addition of new code using the test suite, guides and supports other teammates in the integration task.

A Framework for Teaching Software Development Methods 289 4.2.3. The measurement practice. In this section we describe how quantitative data is used to derive measures. Three measures are defined and illustrated. An examination of the role activities from the academic coach perspective, defined three main measures (Dubinsky & Hazzan, in press): a) Role Time Measure (RTM): it measures the ratio between development and role performance; that is, the time invested in development tasks relative to the time invested in role activities. b) Role Communication Measure (RCM): it measures the level of communication of a team at each development stage. This measure evolves since each role holder needs to communicate with the other team members in order to perform the individual role more efficiently. c) Role Management Measure (RMM): it measures the level of the project management. Since the role scheme aimed at covering all management aspects, maximal level is obtained when all role holders provide maximum role performance. With respect to RTM, most of the students predicted in the first weeks of the semester a RTM ratio of 70% (development) 30% (role); only few predicted the rations 60% 40% or 80% 20%. However, in practice, when checking students weekly reflections, the RTM semester average turned out to be 80% 20% for all roles. The calculation of the RCM is based on the examination of the electronic forum used by the students. It illustrates how measures can be used as a pedagogical tool. For illustration, we use the electronic forum of one particular project. In this project, 714 messages were sent during the semester, out of them 698 messages were sent by the ten teammates and 16 messages were sent by the academic coach. Since coding load-balance during the actual development is ensured by the planning game, we refer to the number of messages in the electronic forum sent by each student as an indicator for the role portion of students overall contribution to the project. As it turned out, the leading group was the most communicative (49.2%) while the customer group was the least (6.5%). (The maintenance group s portion was 15.2% and the code group 29.1%). Looking into the messages details we observed that, indeed, this measure reflects the project status. Specifically, on the one hand, the students of the customer group did not perform properly as their role required; On the other hand, the coach and the tracker over-performed (the coach sent many responses during some periods, and both the coach and the tracker took control of the continuous integration in order to ensure the project success). As part of RMM, we calculate three averages for each iteration: the weekly average number of original messages, the weekly average number of responses to original messages, and the weekly average total number of messages. Figure 1 shows these averages for the three iterations of the above mentioned project. These averages, however, reflect only a partial picture of the project status. In order to improve our understanding with respect to the project dynamics, we calculate an inner RMM ratio between original messages and responses. As it turns out, iterations

290 Y. Dubinsky and O. Hazzan Figure 1. Weekly RMM for each iteration 2 and 3 have almost the same RMM ratio between original messages and responses (41.66/28 ¼ 1.48 for iteration 2 and 21.25/17 ¼ 1.25 for iteration 3), while iteration 1 has a different ratio (29.75/54.25 ¼ 0.55). We conclude, however, that iteration 2 is the most high-managed since it also has a relatively high number of messages in total. This conclusion makes sense since the first iteration launches the project as well as the role scheme, and the last iteration is largely influenced by the stress of the end of the semester. During the second iteration, the students are more experienced with the project method and perform at their best. 4.2.4. The coaching practice. In order to inspire the way a practice is developed in an Action Research manner, we present one intermediate stage of the construction process of this practice. Prior to the introduction of XP into the course, we held four XP training sessions for the coaches. These sessions were the start of a training programme that included on-going communication and guidance that continued intensively during the first year and continues till today in a less intensive fashion. The first year training programme ended with retrospective interviews with the coaches (Dubinsky & Hazzan, 2003). The objective of the interviews was to learn about the effect of the training sessions on the coaches coaching approaches, to determine the necessity of an additional series of training sessions, and to reflect on the actual process applied by each coach. Two consecutive interviews were performed with each coach at the end of the first year. In the first interview, the coaches started out by completing a questionnaire that was almost identical to a questionnaire they completed prior to the first training programme, one year earlier. Both questionnaires referred to software development projects in general, and did not address any specific SDM. In the questionnaires, the coaches were asked to describe the phases of a software development project and to specify the more important phases in their opinion. In addition, they were requested to describe their role as coaches in the process of guiding students in software development, to rank the main activities performed during software development as they perceive them, to specify the main problems of software development processes, and to suggest solutions to those problems.

A Framework for Teaching Software Development Methods 291 After completing feeling in the questionnaire in the first interview, the coaches were asked to describe the implementation of XP in each of their studios, to assess the affect of the XP training, and to mention topics that they felt should be the focus of a second series of training sessions. In the second interview, the coaches were asked to reflect on the two questionnaires they completed (at the beginning and at the end of the first year) and to describe how the use of XP influenced them personally. They were also asked to compare their teaching approach before and after this year, during which they guided students using XP. At the end of the second interview, the coaches were requested to put in writing their reflections on the two interviews. The analysis of these data revealed six categories that describe the teaching framework from the coaches perspective (see Table 6). These six categories were continuously compared and validated by the coaches based on data obtained from questionnaires and interviews. In Dubinsky and Hazzan (2003) we presented the six categories in detail. For demonstration purposes, we focus in what follows on the Customer category and illustrate the gathering of data from interviews that were analysed by using the constant comparison method. As it turns out, within the XP development framework, the awareness to the role of the customer is increased. One of the coaches said: While reflecting [... ] I thought of something new based on the experience of the past year. It is an increased awareness of the customer, and of the human aspect of the development team in general. Indeed, the coaches perceived the customer as playing a key role in their projects. One of the coaches concluded: If a customer joins the project, it changes the coaching significantly. One of the coaches compared two coaching process (with and without a customer): During the summer semester, we had a [real] customer. Moshe was the customer; he entered [the studio]. He really gave feedback. His sentences were in the background even when he was not participating. [... ] The issue of the customer is really important, and this semester it is missing. It is hard for the students to enter the customer s shoes since the scope is not clear. They are in conflict all the time. In line with Principle IX (Coaching) and Principle VII (Listening), this perspective of the coaches, together with additional feedback they provided, guides the construction of the Coaching Practice (still under development). In its future form, Table 6. The teaching framework from the coaches perspective Category Category Name Brief Description I Project Management of resources: time and organizational aspects II Method Practices and tools used in the project III Development Team Development of environment and communication among team members IV Customer Customer requirements and product acceptance V Feelings Inner feelings of people involved in the project VI Coaching Team Support given to coaching team in order to maintain continuous learning and feedback

292 Y. Dubinsky and O. Hazzan we anticipate, this practice will present a detailed coaching training plan that will address both on-going guidance, as well as condensed training sessions. Yet, in spirit of the Action Research paradigm, its specific final form is unknown at this point. 4.2.5. The evaluation practice. The evolution of the practices is interlaced with the evolution of the grading policy used for the evaluation of the students work. In the first XP training, the academic team constructed a grading policy that has been refined several times since then. The grading policy is composed of two main components. The first one is a group component (65%) whose main criterion is the meeting of the customer stories as well as the time estimations given by the students at each of the three iterations. The second ingredient of the grading policy is a personal-individual component (35%) whose main criterion is the personal performance of the student with respect to his or her development tasks as well as with respect to his or her personal role. Table 7 presents the grading policy. 5. SUMMARY This article describes a study during which a comprehensive framework that describes the teaching of software development methods was shaped. The framework is based on ten conceptual principles that address teaching and learning processes, students and academic coaches, as well as the development process itself. The framework consists of practices that provide practical guidelines of how to teach software development methods in capstone project-base course. The importance of software developing according to a specific software development method is obvious. The innovation of our research, however, is the construction of a comprehensive and detailed teaching framework that can be implemented in academic courses in higher education that use any software development method. The future Table 7. The grading policy Group component (65%) Individual component (35%) 60% - 50% - Answer the customer stories and P Weekly reflection meeting the schedule according to the team P Pair programming experience time estimations: P Test-Driven-Development exercise P (10%) for iteration 1 P Weekly presence P (25%) for iteration 2 P (25%) for iteration 3 25% - Performance of a personal role: 25% - P Actual implementation Project documentation P Further development and enhancement 15% - Group evaluation of the academic coach 25% - Personal evaluation of the academic coach

A Framework for Teaching Software Development Methods 293 implication of such a teaching framework is expressed in its potential implementation in different courses that deal with different subject matters of software development and in different university environments. NOTE 1. In the second edition, a fifth value respect is introduced (Beck with Andres, 2005). REFERENCES Allen, E., Cartwright, R., & Reis, C. (2003). Production programming in the classroom. Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education (pp. 89 93). New York: ACM Press. Beck, K. (2000). Extreme Programming Explained: Embrace change. Boston, MA: Addison-Wesley. Beck, K., & Andres, C. (2005) Extreme Programming Explained: Embrace change (2nd ed.). Boston, MA: Addison-Wesley. Bogdan, R.C., & Biklen, S.K. (1992). Qualitative Research for Education. Boston: Allyn and Bacon. Brooks, F.P. (1987). No silver bullet: Essence and accidents of software engineering. Computer, 20(4), 10 19. Confrey J. (1995). A theory of intellectual development. The Learning of Mathematics, 15(2), 36 45. Cook T.D., & Reichardt, C.S. (1979). Beyond Qualitative versus Quantitative Methods. Qualitative and Quantitative Methods in Evaluation Research. London: Sage Publications. Davis, R.B., Maher, C.A., & Noddings, N. (Eds.) (1990). Constructivist views on the teaching and learning of mathematics (The National Council of Teachers of Mathematics, Inc. Monograph No. 4). Journal for Research in Mathematics Education Monograph 4. Denning, P. J. (1992). Educating a new engineer. Communications of the ACM, 35(12), 82 97. Dubinsky, Y., & Hazzan, O. (2003). Extreme programming as a framework for student-project coaching in computer science capstone courses. IEEE International Conference on Software Science, Technology & Engineering (pp. 53 59). Piscataway, NJ: IEEE Publishing. Dubinsky, Y., & Hazzan, O. (2004). Roles in agile software development teams. In J. Eckstein & H. Baumeister (Eds), 5th International Conference on extreme Programming and Agile Processes in Software Engineering (pp. 157 165). Berlin: Springer-Verlag. Dubinsky Y., & Hazzan, O. (in press). Using a role scheme to derive software project metrics. Journal of Systems Architecture. Gee, J.P. (1992). Discourse analysis. In M. LeCompte, W.L. Millroy, & J. Preissle. The Handbook of Qualitative Research in Education. San Diego, CA: Academic Press. Hamlet, D., & Maybee, J. (2001). The Engineering of Software. Boston, MA: Addison-Wesley. Hazzan, O. (2002). The reflective practitioner perspective in software engineering education. The Journal of Systems and Software, 63(3), 161 171. Hazzan O., & Dubinsky, Y. (2003). Teaching a software development methodology: The case of Extreme Programming. In P. Knoke, A. Moreno, & M. Ryan. 16th International Conference on SE Education and Training (pp. 176 184). Piscataway, NJ: IEEE Publishing. Highsmith, J. (2002). Agile Software Developments Ecosystems. Boston, MA: Addison-Wesley. Kilpatrick, J. (1987). What constructivism might be in mathematics education. In J.C. Bergeron, N. Herscovics & C. Kieran (Eds.), 11th International Conference for the Psychology of Mathematics Education (PME11) Vol. I (pp. 3 27). Montréal: University of Montréal. Kirk, J., & Miller, M.L. (1986). Reliability and Validity in Qualitative Research. Beverly Hills, CA: Sage Publications.

294 Y. Dubinsky and O. Hazzan Kuhn, S. (1998). The software design studio: An exploration. IEEE Software, 15(2), 65 71. Lewin, K. (1948). Resolving Social Conflicts; Selected papers on group dynamics. G.W. Lewin (Ed.). New York: Harper & Row. Manen, M.V. (1990). Researching Lived Experience. New York: State University of New York Press. Melnik, G., & Maurer, F. (2003). Introducing agile methods in learning environments: Lessons learned. Proceedings of the XP/Agile Universe Conference (pp. 172 184). Berlin: Springer-Verlag. Meyer, B. (2001). Software engineering in the academy. Computer, 34(5), 28 35. Patton, M.Q. (1990). Qualitative Evaluation and Research Methods (2nd ed.). Newbury Park, CA: Sage Publications. Reisman, C.K. (1993). Narrative Analysis. Newbury Park, CA: Sage. Revans, R. (1988). The ABC of Action Learning. London: Lemos & Crane (Original work published 1983). Smith, M.K. (1996). Action research: A guide to reading. Retrieved 28 January 2005, from http:// www.infed.org/research/b-actres.htm Strauss, A.L. (1987). Qualitative Analysis for Social Scientists. New York: Cambridge University Press. Tomayko, J.E. (1996). Carnegie-Mellon s software development studio: A five-year retrospective. SEI Conference on Software Engineering Education. Daytona Beach, Florida. Tomayko J., & Hazzan, O. (2004). Human Aspects of Software Engineering. Hingham, MA: Charles River Media. APPENDIX: THE COURSE SCHEDULE Week Number Purpose of meeting Main activities (academic coach and students) XP Values and practices 1 Opening of course and Introduce the studio environment Communication getting acquainted Create a team Introduce the project subject and gather relevant materials Present student XP roles 2 Planning game of Assign student XP roles On-site customer Iteration 1: Phase I Play customer: Develop customer Planning game stories and make customer Metaphor decisions Courage 3 Planning game of Design Planning game Iteration 1: Phase II Write development tasks Simple design Small releases 4 Coding and testing Estimate time for the development tasks Balance students development work loads Teach/learn unit testing Sustainable pace Pair programming Testing (continued )